1

I have analog accelerometer that transfer acceleration on 3 axis with frequency 5 kHz. I want to store that data in SD card for in txt file. My sketch is the following:

void loop()
{
  
  // Get raw accelerometer data for each axis
  int rawX = analogRead(A0);
  int rawY = analogRead(A1);
  int rawZ = analogRead(A2);

  
  // Scale accelerometer ADC readings into common units
  // Scale map depends on if using a 5V or 3.3V microcontroller
  float scaledX, scaledY, scaledZ; // Scaled values for each axis
  if (micro_is_5V) // Microcontroller runs off 5V
  {

    scaledX = mapf(rawX, 0, 1023, -scale, scale); // 3.3/5 * 1023 =~ 675
    scaledY = mapf(rawY, 0, 1023, -scale, scale);
    scaledZ = mapf(rawZ, 0, 1023, -scale, scale);

  }
  else // Microcontroller runs off 3.3V
  {
    scaledX = mapf(rawX, 0, 675, -scale, scale);
    scaledY = mapf(rawY, 0, 675, -scale, scale);
    scaledZ = mapf(rawZ, 0, 675, -scale, scale);
  }
 
   String dataString = "";
   // Concatenate string
   dataString += String(scaledX);
   dataString += "\t\t";
   dataString += String(scaledY);
   dataString += "\t\t";
   dataString += String(scaledZ);
   dataString += "\t\t";
   saveSD();  
}

// Same functionality as Arduino's standard map function, except using floats
float mapf(float x, float in_min, float in_max, float out_min, float out_max)
{
  return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}

void saveSD()
{
  // create file for writing
  File dataFile = SD.open("datalog.txt", FILE_WRITE);
  // if file is available
  if (dataFile) {
    // save the data
    dataFile.println(dataString);
    // close file
    dataFile.close();
  } else {
    // if file is unavailable
    Serial.println("Error opening datalog.txt");
  }
}

So in loop I open file, write to it and after that close it. When my file will be big enough there will be bottle neck.It will take a lot of time to read and write file. How to deal with saving data to files on SD card?

UPD: I measured the time of executing code like that:

  ...
  start_open_t=micros(); // time in milli seconds
  File dataFile = SD.open("datalog.txt", FILE_WRITE);
  delta_open_t=micros() - start_open_t ;
  // if file is available for writing
  if (dataFile) {
    // save the data
    start_write_t=micros(); // time in milli seconds
    dataFile.println(dataString);
    delta_write_t=micros() - start_write_t ;
    // close the file
    start_close_t=micros(); // time in milli seconds
    dataFile.close();
    delta_close_t=micros() - start_close_t ;
    ...

And there is the result:

Open time:
7656
Write time:
2176
Close time:
8076

The open time is increasing there is no surprise (with increasing the file size).
Write to file and close time is almost constant. I thought to open file in setup() and initialize timer. Write to file in loop() and after some time interval to close the file, after that open new file and so on. But as can be seen the print operation dataFile.println(dataString); takes approximately 2 ms that is 500 Hz, and it is too slow. I need to register with 5 kHz. If I print in serial the data is printed much faster but I can not connect to usb because arduino is located to far (approx 30 meters). May be there is the way to connect to serial with wifi or bluetooth or may be there are another methods to store the data fast?

3
  • 1
    I thought about offering this as a potential duplicate, but I didn't feel that strongly about it, and you got some good advice (not mentioned there) below that I'm going to vote up. Still, you may find it an interesting read.
    – timemage
    Jun 24, 2022 at 18:01
  • Thanks for advice, I'll read
    – Kracozebr
    Jun 26, 2022 at 11:57
  • You'd edited to include the line: "May be there is the way ... serial with wifi or bluetooth or ... another methods to store the data fast?" It's a valid question (or at least a kernel) but it is a different question that's been shoehorned in there. And that's been done after the original question had already been given a thoughtful answer. IMO, if you want to ask something like that, remove it from this question and start a new one. You'd need to be careful that you're asking something that can be factually answered and not something that is only a matter of opinion.
    – timemage
    Jul 2, 2022 at 16:49

1 Answer 1

1

I can see it is safe to open/close the file every time, especially closing, as if the Arduino loses power, the entire file can be corrupted possibly.

However, I can find of some 'solutions':

  1. Backup battery

Open the file in setup(), and only close the file when the backup battery becomes active (instead of closing the file after every write). This will however need an external battery, and some implementation to check if it is active. Also, during the external battery is active, no data should be written.

  1. Using multiple files

More easier is to use multiple files, and write each time period (less than the size of the file that cause problems) in a separate file. You can either close the file after each write, or when starting a new period, meaning when the Arduino loses power, you only lose the last time period.

  1. Using binary format

This can be used in combination with solution 1 and 2, and is to not write the files as text files but in binary format, reducing the size considerably.

4
  • 1
    Thanks for replying. About battery, for now I'm gonna to use crone battery as main power for arduino because is located away from electrical power. About second your suggestion I thought about it, so I need to measure time of loop and time of opening file with different file sizes and find the max size that it can open not to be too slow to catch data from acceleromerter. And your third suggestion is great I should try it!
    – Kracozebr
    Jun 26, 2022 at 12:21
  • If your file doesn't need to be humanly readable, that is the easiest solution. I think it can save a factor 4 , and maybe more if you use interpunction and spaces. Jun 26, 2022 at 23:05
  • Thanks, I can decode the file after that. I have an update, I measured the time of open, write and close operation and it seems that it not the best way to store logs on sd. See updates
    – Kracozebr
    Jun 27, 2022 at 12:38
  • Use binary format, that will reduce the write time. Jun 27, 2022 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.