SdFat: Why does the first write to the file take so long?

Question

I have tested this problem on 6 micro-SDHC cards of 3 different types. 3 are 32-GB Sandisk, 2 are 16-GB Kingston, and 1 is 16-GB Samsung. These cards are used in data loggers and typically contain hundreds or thousands of 5-MB text files whose names are formatted like FILE1234.TXT.

The problem is that when a logger containing such a card (containing data) is started, the first write takes a long time--from 7 to 30 seconds. Opening the file (the smallest available filename) doesn't take too long. Subsequent writes don't take long. Just the first write.

Now, the plot thickens: this delay can be eliminated by first opening FILE0000.TXT and immediately closing it. Further, once this is done, that disk is never affected by this delay again. This has happened with each of the disks.

Here is some minimal code demonstrating the issue:

#include <SdFat.h>
SdFat sd;
SdFile file;

void setup(){
  Serial.begin(57600);
  sd.begin();
  int m = millis(); // just to time everything

  // critical line that prevents delays
  file.open("FILE0000.TXT", O_WRITE | O_TRUNC | O_CREAT); file.close();

  // file we actually want to write to
  file.open("FILE6005.TXT", O_WRITE | O_TRUNC | O_CREAT);
  Serial.println(millis()-m); // print how long it took to open

  file.println(F("test"));  
  Serial.println(millis()-m); // print how long it took to write
  file.close();
}
void loop(){
}

When the "critical line" in the code is commented out, I get a big delay in the first write to the file. When it's uncommented, there's no delay, and I never get a delay on that disk again even with the line commented. I saw this on all six disks.

One further bit of mystery: at first, this did not work on one of the cards. I checked and found that this card alone had a hidden ".Trash-1000" directory. I removed that directory and tried again, and this fix did work that time.

Does anyone have any idea what's going on here?

Answer 1

Most file systems exhibit poor performance if you store many files in one directory. 'Normal' desktop computers are perhaps so fast, that you don't notice the performance impact, but accessing the file system from a relatively slow micro-controller, which also have limited means of caching the file system data structures in RAM, you may have to consider for this and either write a smaller number of files or build a tree of directories, so that you only have a limited number of files in each directory.

Answer 2

I have observed this as well, but with bare-metal programming rather than arduino. I use an STM32F7 chip to write to a class 10 SD card using block writes from the SD card standard. I use a custom file system rather than FAT. My first write to the SD card takes roughly 8-10 times longer than all of my other writes. This is each time I remove power and resupply power to the device. This doesn't happen if I reset the processor without removing the power supply. If I only reset the processor (which means I reconfigure and initialize the SD card) this write delay doesn't happen.

The only thing that doesn't happen to the SD card before the first write is the configuration of the data path state machine (DPSM). Could the DPSM being in a state that isn't conducive to writing on start-up be the culprit?