In a simplified example, I have a microcontroller functioning as a data acquisition device - sensor data is received over UART, and then saved to an SD card.
This received data is in the form of a 12 byte struct, received at 1kHz. The microcontroller receiving the data is an esp32, with an SD card connected over spi. Not an uncommon issue, but if a typical SD card write is triggered everytime new data is received, it will take longer than the gap between samples to write the data.
What is typically considered the best practice for this type of system, presuming the data will always be received faster than it can be written. I have seen examples of circular buffers, but presumably these will eventually fill completely and samples will be missed.
In my particular example, I have the bonus second cpu core in the esp32. Therefore, my thinking is to use 2 alternating buffers. Data will be received on core 1, and placed into the active buffer. When the buffer is full, incoming data will be written to the second buffer, and the first will be written to the SD card by core 2.
Is this a sensible solution to the problem? I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.
data will always be received faster than it can be written
... that means that you will lose data ... data compression may help .... please provide few examples of the 12 bytes of data – jsotola Nov 29 '20 at 15:32if a typical SD card write is triggered everytime new data is received,
-- It isn't. An internal 512 byte buffer is maintained by the SD library to store an entire data block. When that is filled it will flush it to the SD card in one go and get (or allocate) the next. – Majenko♦ Nov 29 '20 at 17:15