0

I've read that it's best to avoid using String in Arduino so I'm ready to use a char array if it's better. I'm reading the file names from an SD Card and my goal is to store them in an array and then be able to read that array. What I'm currently doing is:

String lib_arr[100];
int lib_arr_size;

and then

File dir = SD.open("/");
int c = 0;
while (true) {
  File entry = dir.openNextFile();
  if (!entry) {
    // If no more files
    lib_arr_size = c - 1;
    break;
  }
  if(strcmp(entry.name(), "SYSTEM~1") == 0) {
    continue;
  }
  lib_arr[c] = entry.name();
  entry.close();
  c++;
}

and then

for (int i = 0; i < lib_arr_size; i++) {
  if(i == pos) {
    tft.print("> ");
    tft.println(lib_arr[i]);
  } else {
    tft.println(lib_arr[i]);
  }
}

This is not a solution because, for my purposes, the String lib_arr[100] can be 1000 or 10000 or 100000... and I found out that by increasing that value the global variables use skyrockets and Arduino says that it's out of memory. My goal is memory optimization. I don't know how to go about this problem.

7
  • 1
    read and print/process in one go. don't store the list in memory. here I have a SD browser github.com/jandrassy/lab/blob/master/SDWebServer/… – Juraj Apr 1 '20 at 8:37
  • @Juraj That's a great idea! But when I'm browsing, whenever I go up or down to select different files, I would have to each time go to the SD card and get the files. Wouldn't that be bad for the memory? Is it a good practice? Is it better than storing in array? – JingleBells Apr 1 '20 at 9:26
  • The SD library uses these old "DOS" file names in 8.3 format, with automatic renaming. This a) limits the required size and b) makes it useless to deal with 10,000 files at once, as they will have unrecognizable names... – DataFiddler Apr 1 '20 at 10:37
  • you don't have the RAM to cache the names. but you have time to load them. on a MCU with more RAM you could cache some last seen folder, but this would be premature optimization and that is the root of all evil in coding. en.wikipedia.org/wiki/Program_optimization#When_to_optimize – Juraj Apr 1 '20 at 12:00
  • @Juraj So I should get and read the file names on every file select? – JingleBells Apr 1 '20 at 12:19
0

Well, you cannot use more memory than you have. If the length of file names is (probably/possibly) too much, you just cannot store it in memory. Some alternatives:

  • Use some compression... This will save you a bit, but at the end if you don't know how much the maximum bytes you need to store is, it will not help much
  • Use external SRAM memory... You need to write your own memory functions. Also a slight performance hit.
  • Use SD card memory... As SRAM, but much slower. Also not sure if you can use the same SD card to write on.
  • The best solution, but could be a big performance hit, is not to store at all the file names, but go through the file names and process what you need along the way. This is not always possible or feasible though. In your solution you can print the file names directly without storing (but I guess you want to do some additional processing).
2
  • The user can create 10000 files with a single character in each for example. – JingleBells Apr 1 '20 at 7:40
  • I can imagine, but you still cannot put all those file names in memory. You need either external memory or process them as you go through each file (without storing inbetween data/results). – Michel Keijzers Apr 1 '20 at 7:46
0

Please look into the SPIFFS/ LittleFS library. You will see that the length of a filename (in those file systems it includes the path and the root indicator "/" as well as the extension .XXXXX as in .shtml ). The maximum length defined in the libs is 32 as max length this means 31 chars to use + 1 terminator. To all the wise guys claiming you can change that (Yes you can but you should not). Do not do it - there are reasons for that. Read more.
The maximum files is defined by blocksize and howlargeyou defined your SPIFFS partition.
There is no difference between writing to SPIFFs or a SD card both have around 100.000 r/w cycles per individual physical "cell" Sometimes it is cheaper to get a new ESP than a SDcard.

The question of the OP:
As long as you read its no problem to do it often if you do mass renames you might have a problem after some time. What you do is actually partly copying what the libs do - so utterly obsolete. Speed is the same because its the same flash memory anyway (+ your processing overhead).

You do not really have directories its a flat table system, so searching for path beginning with or containing (combined with a clever read strategy) can be done on the fly without problems.

If you need data elsewhere (browser) filter & stream it - do not make in between storage (=duplication) with the small memory


As a final thought from the tech specs:

The main complication writing spiffs was that it cannot be assumed the target has a heap. Spiffs must go along only with the work ram buffer given to it. This forces extra implementation on many areas of spiffs. ....

SPI flash devices are physically divided in blocks. On some SPI flash devices, blocks are further divided into sectors. Datasheets sometimes name blocks as sectors and vice versa.

Common memory capacaties for SPI flashes are 512kB up to 8MB of data, where blocks may be 64kB. Sectors can be e.g. 4kB, if supported. Many SPI flashes have uniform block sizes, whereas others have non-uniform - the latter meaning that e.g. the first 16 blocks are 4kB big, and the rest are 64kB.

The entire memory is linear and can be read and written in random access. Erasing can only be done block- or sectorwise; or by mass erase.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.