Optimizing Arduino File Selection from SD card

Question

the following code will display the files on an SD card in an Arduino and allow the user to open a file of their choosing. Here's the code with a walkthrough of what it does.

In the Init state, I call the libraries needed to run the code

// Example use of lfnOpenNext and open by index.
#include<SPI.h>
#include "SdFat.h"
#include "FreeStack.h"

// SD card chip select pin.
const uint8_t SD_CS_PIN = 10;

SdFat sd;
SdFile dirFile;
SdFile file;

// Number of files found.
uint16_t numberOfFiles = 0;

This is the area of the real problem, calling SdFile parIndex[50]; Takes a large amount of dynamic memory. For number of files on an SD card, this is a pretty small number, so I need to get at something better

// Position of file's directory entry.
uint16_t dirIndex[50];
SdFile parIndex[50];
//------------------------------------------------------------------------------

In the Setup phase, we make sure the SD card is plugged in and, if it is, we call the function to give the files on the SD card.

void setup() 
{
  Serial.begin(38400);
  while (!Serial) {}

  // Initialize at the highest speed supported by the board that is
  // not over 50 MHz. Try a lower speed if SPI errors occur.
  if (!sd.begin(SD_CS_PIN, SD_SCK_MHZ(50))) 
  {
    sd.initErrorHalt();
  }
  if (dirFile.open("/", O_READ)) 
  {
  printDirectory(dirFile, 0);
  }
}

The printDirectory function (This is the part most open to edits, if there's any way to improve performance here, please let me know as well).

void printDirectory (SdFile CFile, int numTabs)
{
  SdFile file;
  while (file.openNext(&CFile, O_READ))
  {
  if (file.isHidden()||false)
    {
      //file hidden, skip
    }
    else
    {
      for (uint8_t i = 0; i < numTabs; i++) 
      {
        //create tabs for spacing
        Serial.print('\t');
      }
      if (file.isSubDir())
      {
        SdFile SubDirFile;
        printDirectory(SubDirFile, numTabs+1);
      }
      else 
      {
        // Save dirIndex of file in directory.
        dirIndex[numberOfFiles] = file.dirIndex();
        parIndex[numberOfFiles] = CFile;
        // Print the file number and name.
        Serial.print(numberOfFiles);
        Serial.write(' ');
        file.printName(&Serial);
        Serial.println();
        numberOfFiles++;
      }
    }
  file.close();
  }
}

---

This is the user interface portion, it was largely slapped together as a proof of concept for the above code. As it's written, the code only accepts index values 0-9, but my actual program could call any index value.

void loop() {
  int c;

  // Read any existing Serial data.
  do {
    delay(10);
  } while (Serial.available() && Serial.read() >= 0);
  Serial.print(F("\r\nEnter File Number: "));

  while (!Serial.available()) {
    SysCall::yield();
  }
  c = Serial.read();
  uint8_t i = c - '0';
  if (!isdigit(c) || i >= numberOfFiles) {
    Serial.println(F("Invald number"));
    return;
  }

After making sure that the input value exists, it calls up the arrays that contain the parent's directory SdFile object and the index of the desired file.

  Serial.println(i);
  if (!file.open(&parIndex[i], dirIndex[i], O_READ)) {
    sd.errorHalt(F("open"));
  }
  Serial.println();

  char last = 0;

  // Copy up to 500 characters to Serial.
  for (int k = 0; k < 500 && (c = file.read()) > 0; k++)  {
    Serial.write(last = (char)c);
  }
  // Add new line if missing from last line.
  if (last != '\n') {
    Serial.println();
  }
  file.close();
  Serial.flush();
  delay(100);
}

The main issue I have is that SdFile parIndex[50]; uses 22% of my dynamic memory vs the rest of the program using 12%. How can I rewrite my code to not require keeping the SDFile objects in dynamic memory?

For anyone unfamiliar, documentation for the SdFat library can be downloaded from that link.

---

Edit: In terms of use: The SD files are .csv files that store speed and time parameters for running a motor test. Each test lasts for a few minutes to hours.

Answer 1

You have several possibilities:

• Do not store the names of all files, but use the index and iterate through all files again. Disadvantage: this takes a lot of time (every time you need the file name)
• Add SRAM to the Arduino (e.g. 23LC1024, 23K256), depending on your memory needs. Disadvantage: you have to 'manage' this memory yourself.
• If you know something about the file names, you might generate the name. Probably it's not the case, but just to add another option. E.g. if you know that file nr 10 will always be called 'File10.txt', you can 'generate' the name.
• I don't know exactly how you want to use the file names, but you might store only the latest x files used, to prevent having to iterate again reading the SD card. Of course, this only helps if there is a high chance the last x files are used more than the non recent files.

Update: If you need to read the files while leaving the Arduino unblocked, you have to incorporate it in your loop, but don't read all the files in a loop. Something in pseudo code like:

bool readingSd = false; // True between reading first and last file uint16_t lastTimeRead = 0; // Last time read

void loop()
{
   // Check if file read needed (initial or one hour past, I don't take
   // overflow into account
   if (lastTimeRead == 0 || (lastTimeRead + 1000 * 60 < millis()) 
   {
      checkReadFile();
   }

   // Perform your normal code... this will run between processing/reading
   // one file/dir.
   ...     
}

void ProcessSingleFile()
{
   readingSd = true; // Within reading first/last file
   "read/process ONE file or dir (without containing files)";
   if ("last file read")
   {
       readingSd = false;
       lastTimeRead = millis(); // Move to top to prevent reading time
         // between first/last to take into account
   }
}