0

I'm storing a large snippet of text in three separate PROGMEM char arrays:

const char script1[] PROGMEM = "....";
const char script2[] PROGMEM = "....";
const char script3[] PROGMEM = "....";

The first two are 32766 characters, and the last one is 23867 characters (the whole string is 89400 characters in total). I store these in PROGMEM on an Arduino Mega 2560 (which then uses about 90K of 256K of available program memory).

I stored these three strings in a table, together with their lengths:

const int TABLE_SIZE = 3;
const char * const scriptTable[] PROGMEM = 
{   
  script1,
  script2,
  script3
};

const int scriptLengths[] {32767, 32767, 23867};

I have some issues trying to chunk up the string into 10-byte buffers, and processng it that way. I want to iterate through the table, process the first part in 10-byte chunks first, then move on to the second part, and finally the last part.

For a single string (code), the following works fine:

const char code[] PROGMEM = "....";

// Chunk up large code string in PROGMEM.
const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
char workingSpace[CHUNKSIZE + 1];
int baseIndex = 0;

while (baseIndex < sizeof(code)) {
    int chunk;
    if (baseIndex + CHUNKSIZE > sizeof(code)) {
        chunk = sizeof(code) - baseIndex;
        }
        else {
          chunk = CHUNKSIZE;
        }
        memcpy_PF(workingSpace, ((uint_farptr_t)code) + baseIndex, chunk);
        workingSpace[chunk] = '\0';

        // for debugging
        Serial.print("#");
        Serial.print(baseIndex);
        Serial.print("– Got a chunk [");
        Serial.print(workingSpace);
        Serial.println("]");        

        baseIndex += CHUNKSIZE;

However, I'm having some difficulties doing the memcpy_PF call using the table.

int tableIndex = 0;
// for each of the three script strings
for (tableIndex = 0; tableIndex < TABLE_SIZE; tableIndex++) {         
    // Chunk up large script strings in PROGMEM.
    const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
    char workingSpace[CHUNKSIZE + 1];
    int baseIndex = 0;

    // we can only use sizeof if it's a static var.
    // strlen should only be calculated once, expensive operation (loop until we encounter '\0').
    // now stored statically in scriptLengths table.
    Serial.println("tableIndex: ");
    Serial.println(tableIndex);     
    int strLength = scriptLengths[tableIndex];
    Serial.print("length script string: ");
    Serial.println(strLength);

    while (baseIndex < strLength) {
        int chunk;
        if (baseIndex + CHUNKSIZE > strLength) {
          chunk = strLength - baseIndex;
        }
        else {
          chunk = CHUNKSIZE;
        }
        // do memcpy_PF call here

I have tried several options here, mostly having difficulties with the second parameter (how to address the string stored in the table, and increase its indexing parameter). I read that the way to get a string out of a string table is to use pgm_read_word. However, with both of these below, I either get gibberish in the buffer string, or it seems to start in an incorrect location in the string.

memcpy_PF(workingSpace, ((uint_farptr_t) &scriptTable[tableIndex][baseIndex]), chunk);
memcpy_PF(workingSpace, ((uint_farptr_t) (char*)pgm_read_word(&(scriptTable[tableIndex])) + baseIndex), chunk);       

Any help would be greatly appreciated!

Thanks!

1

The first thing you need to fix is this:

/home/nick/Arduino/Large_Progmem_test/Large_Progmem_test.ino: In function 'foo()':
/home/nick/Arduino/Large_Progmem_test/Large_Progmem_test.ino:42:27: warning: iteration 3276 invokes undefined behavior [-Waggressive-loop-optimizations]
     baseIndex += CHUNKSIZE;

You are adding 10 to 32760 hoping to be >= sizeof(code) however now baseIndex will become negative as an int goes from -32768 to +32767.


There were some interesting subtleties to getting this working. :)

One of your major problems is this:

const char * const scriptTable[] PROGMEM = 

That will give you 2-byte pointers (const char *) which will not be large enough to index into 89400 characters.

So you need to have an array of 4-byte pointers like this:

uint_farptr_t  scriptTable [TABLE_SIZE] =

However you can't just initialize those like you tried to. That has to be done procedurally with a macro that pulls in the full address. My working code is below:

#include "code.cpp"

const int TABLE_SIZE = 3;
uint_farptr_t  scriptTable [TABLE_SIZE];  // values calculated at runtime
const int scriptLengths[TABLE_SIZE] 
  {
  sizeof (script1), 
  sizeof (script2), 
  sizeof (script3)
  };

void foo ()
{
  // for each of the three script strings
  for (byte tableIndex = 0; tableIndex < TABLE_SIZE; tableIndex++) 
    {
    // Chunk up large script strings in PROGMEM.
    const int CHUNKSIZE = 10; // CHUNKSIZE *must* be an even number!!
    char workingSpace[CHUNKSIZE + 1];
    long baseIndex = 0;

    Serial.println ("----------------");
    Serial.print("tableIndex: ");
    Serial.println(tableIndex);
    long strLength = scriptLengths[tableIndex];
    Serial.print("length script string: ");
    Serial.println(strLength);

    while (baseIndex < strLength) {
      int chunk;
      if (baseIndex + CHUNKSIZE > strLength) {
        chunk = strLength - baseIndex;
      }
      else {
        chunk = CHUNKSIZE;
      }
      memcpy_PF(workingSpace, scriptTable [tableIndex] + baseIndex, chunk);
      workingSpace[chunk] = '\0';

      // for debugging
      Serial.print("#");
      Serial.print(baseIndex);
      Serial.print("- Got a chunk [");
      Serial.print(workingSpace);
      Serial.println("]");
      delay (50);
      baseIndex += CHUNKSIZE;
    } // end of while
  } // end of for
} // end of foo


void setup() 
{
  // get proper pointers to the various pieces of progmem
  scriptTable [0] = pgm_get_far_address (script1);
  scriptTable [1] = pgm_get_far_address (script2);
  scriptTable [2] = pgm_get_far_address (script3);

  Serial.begin (115200);
  foo ();
}

void loop() 
{
}

The file "code.cpp" just has the actual data in it (generated at http://www.lipsum.com).

As in:

#include <Arduino.h>

const char script1[] PROGMEM =  { 

"Lorem ipsum dolor sit amet, consectetur adipiscing elit.  ... " };

const char script2[] PROGMEM =  { 

"Aliquam erat volutpat. Praesent aliquet accumsan arcu,  ... " };

const char script3[] PROGMEM =  { 

"Ut neque dolor, condimentum ut odio iaculis,  ... " };

Output:

----------------
tableIndex: 0
length script string: 32767
#0- Got a chunk [Lorem ipsu]
#10- Got a chunk [m dolor si]
#20- Got a chunk [t amet, co]
#30- Got a chunk [nsectetur ]
#40- Got a chunk [adipiscing]
#50- Got a chunk [ elit. Pro]
#60- Got a chunk [in sit ame]
#70- Got a chunk [t purus cu]
#80- Got a chunk [rsus, dict]
#90- Got a chunk [um magna q]
#100- Got a chunk [uis, place]
#110- Got a chunk [rat dui. I]
#120- Got a chunk [nteger ali]

...

#32700- Got a chunk [, eget tem]
#32710- Got a chunk [por dui te]
#32720- Got a chunk [mpor a. Do]
#32730- Got a chunk [nec malesu]
#32740- Got a chunk [ada mi et ]
#32750- Got a chunk [dolor cras]
#32760- Got a chunk [ amet.]
----------------
tableIndex: 1
length script string: 32767
#0- Got a chunk [Aliquam er]
#10- Got a chunk [at volutpa]
#20- Got a chunk [t. Praesen]
#30- Got a chunk [t aliquet ]

It successfully transitioned from the first array to the second one.

  • Good point, I missed that, thanks! Probably best to limit the size of script1,script2,script3, etc. to avoid overflows (e.g., to 30.000)? – jozilla Feb 13 '17 at 23:16
  • Or use a long as an index. – Nick Gammon Feb 13 '17 at 23:58
  • Right, thanks! Any thoughts on the memcpy? – jozilla Feb 14 '17 at 0:04
  • Maybe the avr-gcc compiler can not handle a PROGMEM section beyond 64kbytes. With a test sketch I am able to declare a lot of data (150kbytes), but the pointers in the Table[] stay 16-bit. They should be forced to 32-bit, but I don't know how. Even then, the memcpy_PF might not read beyond the first 64kbytes of the PROGMEM segment. So far I can't even read script3 directly with a function (when it is at the end of 150kbytes data). – Jot Feb 14 '17 at 3:26
  • See my amended answer. – Nick Gammon Feb 14 '17 at 3:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.