Efficient way to save accelerometer data on sd card

Question

I have analog accelerometer that transfer acceleration on 3 axis with frequency 5 kHz. I want to store that data in SD card for in txt file. My sketch is the following:

void loop()
{
  
  // Get raw accelerometer data for each axis
  int rawX = analogRead(A0);
  int rawY = analogRead(A1);
  int rawZ = analogRead(A2);

  
  // Scale accelerometer ADC readings into common units
  // Scale map depends on if using a 5V or 3.3V microcontroller
  float scaledX, scaledY, scaledZ; // Scaled values for each axis
  if (micro_is_5V) // Microcontroller runs off 5V
  {

    scaledX = mapf(rawX, 0, 1023, -scale, scale); // 3.3/5 * 1023 =~ 675
    scaledY = mapf(rawY, 0, 1023, -scale, scale);
    scaledZ = mapf(rawZ, 0, 1023, -scale, scale);

  }
  else // Microcontroller runs off 3.3V
  {
    scaledX = mapf(rawX, 0, 675, -scale, scale);
    scaledY = mapf(rawY, 0, 675, -scale, scale);
    scaledZ = mapf(rawZ, 0, 675, -scale, scale);
  }
 
   String dataString = "";
   // Concatenate string
   dataString += String(scaledX);
   dataString += "\t\t";
   dataString += String(scaledY);
   dataString += "\t\t";
   dataString += String(scaledZ);
   dataString += "\t\t";
   saveSD();  
}

// Same functionality as Arduino's standard map function, except using floats
float mapf(float x, float in_min, float in_max, float out_min, float out_max)
{
  return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}

void saveSD()
{
  // create file for writing
  File dataFile = SD.open("datalog.txt", FILE_WRITE);
  // if file is available
  if (dataFile) {
    // save the data
    dataFile.println(dataString);
    // close file
    dataFile.close();
  } else {
    // if file is unavailable
    Serial.println("Error opening datalog.txt");
  }
}

So in loop I open file, write to it and after that close it. When my file will be big enough there will be bottle neck.It will take a lot of time to read and write file. How to deal with saving data to files on SD card?

UPD: I measured the time of executing code like that:

  ...
  start_open_t=micros(); // time in milli seconds
  File dataFile = SD.open("datalog.txt", FILE_WRITE);
  delta_open_t=micros() - start_open_t ;
  // if file is available for writing
  if (dataFile) {
    // save the data
    start_write_t=micros(); // time in milli seconds
    dataFile.println(dataString);
    delta_write_t=micros() - start_write_t ;
    // close the file
    start_close_t=micros(); // time in milli seconds
    dataFile.close();
    delta_close_t=micros() - start_close_t ;
    ...

And there is the result:

Open time:
7656
Write time:
2176
Close time:
8076

The open time is increasing there is no surprise (with increasing the file size).
Write to file and close time is almost constant. I thought to open file in setup() and initialize timer. Write to file in loop() and after some time interval to close the file, after that open new file and so on. But as can be seen the print operation dataFile.println(dataString); takes approximately 2 ms that is 500 Hz, and it is too slow. I need to register with 5 kHz. If I print in serial the data is printed much faster but I can not connect to usb because arduino is located to far (approx 30 meters). May be there is the way to connect to serial with wifi or bluetooth or may be there are another methods to store the data fast?

Answer 1

I can see it is safe to open/close the file every time, especially closing, as if the Arduino loses power, the entire file can be corrupted possibly.

However, I can find of some 'solutions':

1. Backup battery

Open the file in setup(), and only close the file when the backup battery becomes active (instead of closing the file after every write). This will however need an external battery, and some implementation to check if it is active. Also, during the external battery is active, no data should be written.

2. Using multiple files

More easier is to use multiple files, and write each time period (less than the size of the file that cause problems) in a separate file. You can either close the file after each write, or when starting a new period, meaning when the Arduino loses power, you only lose the last time period.

3. Using binary format

This can be used in combination with solution 1 and 2, and is to not write the files as text files but in binary format, reducing the size considerably.