Using PROGMEM to store array of structs

Question

I am hitting the limits of my arduino's SRAM and found that SRAM usage can be reduced by storing static stuff in flash memory instead of SRAM.

My project can (optionally) be built with an included SimulatorClass. I use this to 'playback' data at home which I gathered during real-life testing of my project. My project is based on photoelectric sensors, and it's timing-critical, so all timekeeping is done using micros(), which returns unsigned long datatype.

To simulate these photoelectric sensor triggers, I need an array containing a struct of 3 variables each: unsigned int _iDataPos;

   struct SimulatorRecord
   {
      uint8_t iSensorNumber;
      unsigned long lTriggerTime;
      uint8_t iState;
   };

So one these structs is 6 bytes long in my counting (2x uint8_t = 2 bytes, unsigned long = 4 bytes). A typical set of simulator records I need to use is about 25-30 records long, so worst case scenario the simulator array is 180 bytes in size.

I initialize these arrays like so:

SimulatorRecord SimulatorQueue[28] = {
      { 1, 41708, 1 }
      , { 2, 60692, 1 }
      , { 1, 176848, 0 }
      , { 2, 197732, 0 }
      , { 1, 4675580, 1 }
      , { 2, 4692252, 1 }
      , { 2, 4830180, 0 }
      , { 1, 4849032, 0 }
      , { 2, 9058416, 1 }
      , { 1, 9074780, 1 }
      , { 1, 9215868, 0 }
      , { 2, 9234968, 0 }
      , { 2, 13497276, 1 }
      , { 2, 13500924, 0 }
      , { 2, 13502728, 1 }
      , { 1, 13514992, 1 }
      , { 1, 13668000, 0 }
      , { 1, 13669344, 1 }
      , { 1, 13681828, 0 }
      , { 2, 13701464, 0 }
      , { 2, 18028788, 1 }
      , { 2, 18034252, 0 }
      , { 2, 18036540, 1 }
      , { 1, 18046892, 1 }
      , { 2, 18160312, 0 }
      , { 1, 18178224, 0 }
      , { 1, 20454580, 1 }
      , { 1, 20456368, 0 }
   };

In my simulator code I then loop through these arrays like so:

   unsigned int _iDataPos = 0;
   SimulatorRecord PendingRecord = SimulatorQueue[_iDataPos];

   //main simulator loop
   if (PendingRecord.lTriggerTime <= lRaceElapsedTime)
   {
      //simulate signal
      digitalWrite(PendingRecord.iSensorNumber, PendingRecord.iState);

      //And increase pending record
      _iDataPos++;
      PendingRecord = SimulatorQueue[_iDataPos];
   }

Now I've been reading various PROGMEM tutorials, and storing the SimulatorQueue array in flash seems to be piece of cake, just change

SimulatorRecord SimulatorQueue[28]

to

const PROGMEM SimulatorRecord SimulatorQueue[28]

But for the life of me I can't find out how I should change the copying of each array item to the PendingRecord variable, as my iDataPos counter increases.

Edit: I should have mentioned that this simulator class and all it's member functions & variables reside in a seperate *.cpp and *.h file. I alreadt found out that since i'm initializing the SimulatorQueue array in the class itself, and not in the main *.ino file, I should do it like so:

constexpr static SimulatorRecord SimulatorQueue[28] PROGMEM = {<structs go in here>};

Edit2: Sample code which reproduces the problem when the simulator array is available here: http://1drv.ms/1TRwiIB

Answer 1

An easy way of accessing any type of data in PROGMEM is to use this small library:

PROGMEM_readAnything.h

#include <Arduino.h>  // for type definitions

template <typename T> void PROGMEM_readAnything (const T * sce, T& dest)
  {
  memcpy_P (&dest, sce, sizeof (T));
  }

template <typename T> T PROGMEM_getAnything (const T * sce)
  {
  static T temp;
  memcpy_P (&temp, sce, sizeof (T));
  return temp;
  }

You can put that file into a new tab in your IDE, or make a library by putting it inside a folder called PROGMEM_readAnything and put that folder inside the libraries folder, which is inside your sketchbook folder.

That lets you copy from the memory in PROGMEM (using memcpy_P) into RAM. The template is used to work out how many bytes to copy.

---

Example, using your structure:

#include <PROGMEM_readAnything.h>

struct SimulatorRecord
{
  uint8_t iSensorNumber;
  unsigned long lTriggerTime;
  uint8_t iState;
};

const SimulatorRecord SimulatorQueue[28] PROGMEM = {
      { 1, 41708, 1 }
      , { 2, 60692, 1 }
      , { 1, 176848, 0 }
 //    ...   omitted for brevity
      , { 1, 18178224, 0 }
      , { 1, 20454580, 1 }
      , { 1, 20456368, 0 }
   };

// number of items in an array
template< typename T, size_t N > size_t ArraySize (T (&) [N]){ return N; }

void setup ()
{
  Serial.begin (115200);
  Serial.println ();

  for (int i = 0; i < ArraySize (SimulatorQueue); i++)
    {
    SimulatorRecord thisItem;
    PROGMEM_readAnything (&SimulatorQueue [i], thisItem);
    Serial.print (i);
    Serial.print (F(" = "));
    Serial.print (thisItem.iSensorNumber);
    Serial.print (F(" / "));
    Serial.print (thisItem.lTriggerTime);
    Serial.print (F(" / "));
    Serial.print (thisItem.iState);
    Serial.println ();
    }  // end of for loop
}  // end of setup

void loop ()
{
}  // end of loop

I found that it decreased RAM usage by 166 bytes, which is almost 28 * 6 (a couple of bytes must have been used by something).

---

Reference

Putting constant data into program memory (PROGMEM)

---

Putting PROGMEM into a class

In response to some comments below, this explains how to put the constants into a class.

Alex originally had:

class SimulatorClass
{

...

   typedef struct SimulatorRecord
   {
      uint8_t iSensorNumber;
      unsigned long lTriggerTime;
      uint8_t iState;
   } SimulatorRecord;

   constexpr static SimulatorRecord SimulatorQueue[28] PROGMEM = {
      { 1, 41708, 1 }
      , { 2, 60692, 1 }
   //    ...   omitted for brevity
      , { 1, 20454580, 1 }
      , { 1, 20456368, 0 }
   };

   SimulatorRecord PendingRecord;
};

Putting aside whatever "constexpr" is, you can't put constants inside a class like that. You need to make it static, and have the actual definition elsewhere. So, change the constants to:

class SimulatorClass
{

...

   typedef struct SimulatorRecord
   {
      uint8_t iSensorNumber;
      unsigned long lTriggerTime;
      uint8_t iState;
   } SimulatorRecord;

  static const SimulatorRecord SimulatorQueue[28] PROGMEM ;

  SimulatorRecord PendingRecord;
};

Now in your main class file (Simulator.cpp in your case):

 const SimulatorClass::SimulatorRecord SimulatorClass::SimulatorQueue[28] PROGMEM = {
    { 1, 41708, 1 }
    , { 2, 60692, 1 }
   //    ...   omitted for brevity
    , { 1, 20454580, 1 }
    , { 1, 20456368, 0 }
   };

That now compiles OK.

Answer 2

Using it with heterogeneous types is trickier, since it has to be copied by hand instead of using one of the pgm_read_*() macros.

SimulatorRecord PendingRecord;
memcpy_P(&PendingRecord, &SimulatorQueue[_iDataPos], sizeof(PendingRecord));