Reading a 10-byte buffer out of a PROGMEM string stored in a string table

Question

I'm storing a large snippet of text in three separate PROGMEM char arrays:

const char script1[] PROGMEM = "....";
const char script2[] PROGMEM = "....";
const char script3[] PROGMEM = "....";

The first two are 32766 characters, and the last one is 23867 characters (the whole string is 89400 characters in total). I store these in PROGMEM on an Arduino Mega 2560 (which then uses about 90K of 256K of available program memory).

I stored these three strings in a table, together with their lengths:

const int TABLE_SIZE = 3;
const char * const scriptTable[] PROGMEM = 
{   
  script1,
  script2,
  script3
};

const int scriptLengths[] {32767, 32767, 23867};

I have some issues trying to chunk up the string into 10-byte buffers, and processng it that way. I want to iterate through the table, process the first part in 10-byte chunks first, then move on to the second part, and finally the last part.

For a single string (code), the following works fine:

const char code[] PROGMEM = "....";

// Chunk up large code string in PROGMEM.
const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
char workingSpace[CHUNKSIZE + 1];
int baseIndex = 0;

while (baseIndex < sizeof(code)) {
    int chunk;
    if (baseIndex + CHUNKSIZE > sizeof(code)) {
        chunk = sizeof(code) - baseIndex;
        }
        else {
          chunk = CHUNKSIZE;
        }
        memcpy_PF(workingSpace, ((uint_farptr_t)code) + baseIndex, chunk);
        workingSpace[chunk] = '\0';

        // for debugging
        Serial.print("#");
        Serial.print(baseIndex);
        Serial.print("– Got a chunk [");
        Serial.print(workingSpace);
        Serial.println("]");        

        baseIndex += CHUNKSIZE;

However, I'm having some difficulties doing the memcpy_PF call using the table.

int tableIndex = 0;
// for each of the three script strings
for (tableIndex = 0; tableIndex < TABLE_SIZE; tableIndex++) {         
    // Chunk up large script strings in PROGMEM.
    const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
    char workingSpace[CHUNKSIZE + 1];
    int baseIndex = 0;

    // we can only use sizeof if it's a static var.
    // strlen should only be calculated once, expensive operation (loop until we encounter '\0').
    // now stored statically in scriptLengths table.
    Serial.println("tableIndex: ");
    Serial.println(tableIndex);     
    int strLength = scriptLengths[tableIndex];
    Serial.print("length script string: ");
    Serial.println(strLength);

    while (baseIndex < strLength) {
        int chunk;
        if (baseIndex + CHUNKSIZE > strLength) {
          chunk = strLength - baseIndex;
        }
        else {
          chunk = CHUNKSIZE;
        }
        // do memcpy_PF call here

I have tried several options here, mostly having difficulties with the second parameter (how to address the string stored in the table, and increase its indexing parameter). I read that the way to get a string out of a string table is to use pgm_read_word. However, with both of these below, I either get gibberish in the buffer string, or it seems to start in an incorrect location in the string.

memcpy_PF(workingSpace, ((uint_farptr_t) &scriptTable[tableIndex][baseIndex]), chunk);
memcpy_PF(workingSpace, ((uint_farptr_t) (char*)pgm_read_word(&(scriptTable[tableIndex])) + baseIndex), chunk);       

Any help would be greatly appreciated!

Thanks!

Answer 1

The first thing you need to fix is this:

/home/nick/Arduino/Large_Progmem_test/Large_Progmem_test.ino: In function 'foo()':
/home/nick/Arduino/Large_Progmem_test/Large_Progmem_test.ino:42:27: warning: iteration 3276 invokes undefined behavior [-Waggressive-loop-optimizations]
     baseIndex += CHUNKSIZE;

You are adding 10 to 32760 hoping to be >= sizeof(code) however now baseIndex will become negative as an int goes from -32768 to +32767.

---

There were some interesting subtleties to getting this working. :)

One of your major problems is this:

const char * const scriptTable[] PROGMEM = 

That will give you 2-byte pointers (const char *) which will not be large enough to index into 89400 characters.

So you need to have an array of 4-byte pointers like this:

uint_farptr_t  scriptTable [TABLE_SIZE] =

However you can't just initialize those like you tried to. That has to be done procedurally with a macro that pulls in the full address. My working code is below:

#include "code.cpp"

const int TABLE_SIZE = 3;
uint_farptr_t  scriptTable [TABLE_SIZE];  // values calculated at runtime
const int scriptLengths[TABLE_SIZE] 
  {
  sizeof (script1), 
  sizeof (script2), 
  sizeof (script3)
  };

void foo ()
{
  // for each of the three script strings
  for (byte tableIndex = 0; tableIndex < TABLE_SIZE; tableIndex++) 
    {
    // Chunk up large script strings in PROGMEM.
    const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
    char workingSpace[CHUNKSIZE + 1];
    long baseIndex = 0;

    Serial.println ("----------------");
    Serial.print("tableIndex: ");
    Serial.println(tableIndex);
    long strLength = scriptLengths[tableIndex];
    Serial.print("length script string: ");
    Serial.println(strLength);

    while (baseIndex < strLength) {
      int chunk;
      if (baseIndex + CHUNKSIZE > strLength) {
        chunk = strLength - baseIndex;
      }
      else {
        chunk = CHUNKSIZE;
      }
      memcpy_PF(workingSpace, scriptTable [tableIndex] + baseIndex, chunk);
      workingSpace[chunk] = '\0';

      // for debugging
      Serial.print("#");
      Serial.print(baseIndex);
      Serial.print("- Got a chunk [");
      Serial.print(workingSpace);
      Serial.println("]");
      delay (50);
      baseIndex += CHUNKSIZE;
    } // end of while
  } // end of for
} // end of foo


void setup() 
{
  // get proper pointers to the various pieces of progmem
  scriptTable [0] = pgm_get_far_address (script1);
  scriptTable [1] = pgm_get_far_address (script2);
  scriptTable [2] = pgm_get_far_address (script3);

  Serial.begin (115200);
  foo ();
}

void loop() 
{
}

The file "code.cpp" just has the actual data in it (generated at http://www.lipsum.com).

As in:

#include <Arduino.h>

const char script1[] PROGMEM =  { 

"Lorem ipsum dolor sit amet, consectetur adipiscing elit.  ... " };

const char script2[] PROGMEM =  { 

"Aliquam erat volutpat. Praesent aliquet accumsan arcu,  ... " };

const char script3[] PROGMEM =  { 

"Ut neque dolor, condimentum ut odio iaculis,  ... " };

Output:

----------------
tableIndex: 0
length script string: 32767
#0- Got a chunk [Lorem ipsu]
#10- Got a chunk [m dolor si]
#20- Got a chunk [t amet, co]
#30- Got a chunk [nsectetur ]
#40- Got a chunk [adipiscing]
#50- Got a chunk [ elit. Pro]
#60- Got a chunk [in sit ame]
#70- Got a chunk [t purus cu]
#80- Got a chunk [rsus, dict]
#90- Got a chunk [um magna q]
#100- Got a chunk [uis, place]
#110- Got a chunk [rat dui. I]
#120- Got a chunk [nteger ali]

...

#32700- Got a chunk [, eget tem]
#32710- Got a chunk [por dui te]
#32720- Got a chunk [mpor a. Do]
#32730- Got a chunk [nec malesu]
#32740- Got a chunk [ada mi et ]
#32750- Got a chunk [dolor cras]
#32760- Got a chunk [ amet.]
----------------
tableIndex: 1
length script string: 32767
#0- Got a chunk [Aliquam er]
#10- Got a chunk [at volutpa]
#20- Got a chunk [t. Praesen]
#30- Got a chunk [t aliquet ]

It successfully transitioned from the first array to the second one.