Best buffering practice for continuous SD card writing

Question

In a simplified example, I have a microcontroller functioning as a data acquisition device - sensor data is received over UART, and then saved to an SD card.

This received data is in the form of a 12 byte struct, received at 1kHz. The microcontroller receiving the data is an esp32, with an SD card connected over spi. Not an uncommon issue, but if a typical SD card write is triggered everytime new data is received, it will take longer than the gap between samples to write the data.

What is typically considered the best practice for this type of system, presuming the data will always be received faster than it can be written. I have seen examples of circular buffers, but presumably these will eventually fill completely and samples will be missed.

In my particular example, I have the bonus second cpu core in the esp32. Therefore, my thinking is to use 2 alternating buffers. Data will be received on core 1, and placed into the active buffer. When the buffer is full, incoming data will be written to the second buffer, and the first will be written to the SD card by core 2.

Is this a sensible solution to the problem? I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

Answer 1

Is this a sensible solution to the problem?

No, unfortunately not. The size of the buffer is limited by the available RAM on your chip. Jumping between multiple buffers or cores will not change that, since it is a hardware limitation.

I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

When you are continuously receiving faster than you can write, you only have 2 options:

• Improve write speed
• Reduce the needed data size

There is no other way, and that's a logical conclusion, not so much a programmical or hardware conclusion. It that situation the amount of "buffered" data will increase indefinitely, so you would need an infinite amount of memory. And that doesn't exist in real world. The same principle would apply to a high end PC with lots of memory, it will just be full after longer time, than with an ESP.

• Improving write speed:

  • On various sites I have read, that you can use the SD card with different SPI frequencies. A higher frequency would mean a higher transaction speed. You can investigate further at this point.
  • Writing a full 512 byte page at once and only closing and opening the file again, if really needed, will improve the overall speed.

• Reduce the needed data size

  • Maybe you can compromise with your data resolution and then pack those values as dense as possible (like packing two 4-bit values into a single byte of transmission).
  • You can research for compression code, so that you can compress your data lossless before transmitting it. Though the result will always depend on the kind of your data. Also mind, that in this case you are trading transmission size for computation time and RAM.

About buffers:

I would say, a normal circular buffer does not bring you much advantages here, since with the SD card you don't want to do bytewise data transmission. The 2 buffer approach, that you mentioned, seems good to me. Though doing handling with different cores does not make much sense to me. The UART receiving itself is done by hardware and some ISRs. You can do all the SD card transmissions in your main code, just making sure, that you regularly copy the received data from the Serial buffer into your buffer.

Answer 2

Buffers will help you if at some point the input stream stops, so the "slower" device (e.g. the SD card) can catchup. If the data will always be received faster than it can be written (note always) then no buffer will help you.

But I assume you meant that there times where you get bursts of input data at 1Khz, then it will stop. So, as long as the average input data rate is lower than the ability for the SD to persist the data, you will be fine.

The SD card library already has a buffer (as @Majenko pointed out), so most likely you won't need to do anything. Seem like something you can test relatively simply.

The 2 core approach sounds somewhat overkill (not to mention some synchronization issues you will have to deal with)