Reading from SPIFFS - What's the best way to deal with a string (or char) array with undefined length?

Question

I've read that it's best to avoid using String in Arduino so I'm ready to use a char array if it's better. I'm reading the file names from an SD Card and my goal is to store them in an array and then be able to read that array. What I'm currently doing is:

String lib_arr[100];
int lib_arr_size;

and then

File dir = SD.open("/");
int c = 0;
while (true) {
  File entry = dir.openNextFile();
  if (!entry) {
    // If no more files
    lib_arr_size = c - 1;
    break;
  }
  if(strcmp(entry.name(), "SYSTEM~1") == 0) {
    continue;
  }
  lib_arr[c] = entry.name();
  entry.close();
  c++;
}

and then

for (int i = 0; i < lib_arr_size; i++) {
  if(i == pos) {
    tft.print("> ");
    tft.println(lib_arr[i]);
  } else {
    tft.println(lib_arr[i]);
  }
}

This is not a solution because, for my purposes, the String lib_arr[100] can be 1000 or 10000 or 100000... and I found out that by increasing that value the global variables use skyrockets and Arduino says that it's out of memory. My goal is memory optimization. I don't know how to go about this problem.

Answer 1

Well, you cannot use more memory than you have. If the length of file names is (probably/possibly) too much, you just cannot store it in memory. Some alternatives:

• Use some compression... This will save you a bit, but at the end if you don't know how much the maximum bytes you need to store is, it will not help much
• Use external SRAM memory... You need to write your own memory functions. Also a slight performance hit.
• Use SD card memory... As SRAM, but much slower. Also not sure if you can use the same SD card to write on.
• The best solution, but could be a big performance hit, is not to store at all the file names, but go through the file names and process what you need along the way. This is not always possible or feasible though. In your solution you can print the file names directly without storing (but I guess you want to do some additional processing).

Answer 2

Please look into the SPIFFS/ LittleFS library. You will see that the length of a filename (in those file systems it includes the path and the root indicator "/" as well as the extension .XXXXX as in .shtml ). The maximum length defined in the libs is 32 as max length this means 31 chars to use + 1 terminator. To all the wise guys claiming you can change that (Yes you can but you should not). Do not do it - there are reasons for that. Read more.
The maximum files is defined by blocksize and howlargeyou defined your SPIFFS partition.
There is no difference between writing to SPIFFs or a SD card both have around 100.000 r/w cycles per individual physical "cell" Sometimes it is cheaper to get a new ESP than a SDcard.

The question of the OP:
As long as you read its no problem to do it often if you do mass renames you might have a problem after some time. What you do is actually partly copying what the libs do - so utterly obsolete. Speed is the same because its the same flash memory anyway (+ your processing overhead).

You do not really have directories its a flat table system, so searching for path beginning with or containing (combined with a clever read strategy) can be done on the fly without problems.

If you need data elsewhere (browser) filter & stream it - do not make in between storage (=duplication) with the small memory


As a final thought from the tech specs:

The main complication writing spiffs was that it cannot be assumed the target has a heap. Spiffs must go along only with the work ram buffer given to it. This forces extra implementation on many areas of spiffs. ....

SPI flash devices are physically divided in blocks. On some SPI flash devices, blocks are further divided into sectors. Datasheets sometimes name blocks as sectors and vice versa.

Common memory capacaties for SPI flashes are 512kB up to 8MB of data, where blocks may be 64kB. Sectors can be e.g. 4kB, if supported. Many SPI flashes have uniform block sizes, whereas others have non-uniform - the latter meaning that e.g. the first 16 blocks are 4kB big, and the rest are 64kB.

The entire memory is linear and can be read and written in random access. Erasing can only be done block- or sectorwise; or by mass erase.