In a simplified example, I have a microcontroller functioning as a data acquisition device - sensor data is received over UART, and then saved to an SD card.

This received data is in the form of a 12 byte struct, received at 1kHz. The microcontroller receiving the data is an esp32, with an SD card connected over spi. Not an uncommon issue, but if a typical SD card write is triggered everytime new data is received, it will take longer than the gap between samples to write the data.

What is typically considered the best practice for this type of system, presuming the data will always be received faster than it can be written. I have seen examples of circular buffers, but presumably these will eventually fill completely and samples will be missed.

In my particular example, I have the bonus second cpu core in the esp32. Therefore, my thinking is to use 2 alternating buffers. Data will be received on core 1, and placed into the active buffer. When the buffer is full, incoming data will be written to the second buffer, and the first will be written to the SD card by core 2.

Is this a sensible solution to the problem? I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

  • data will always be received faster than it can be written ... that means that you will lose data ... data compression may help .... please provide few examples of the 12 bytes of data
    – jsotola
    Nov 29, 2020 at 15:32
  • SDCards can be written over a 1 bit bus or 4 bit bus. The latter is faster and, as I recall, requires a license to implement. Also, the speed at which SDCards work varies greatly. If your driver can take advantage of this, try a faster SDCard. Finally, all SDCards contain a propriety controller. So picking the right OEM for your needs may help. (I'd go w/@jsotola recommendation 1st, try losses compression if you can. Simple is best.)
    – st2000
    Nov 29, 2020 at 15:43
  • 1
    if a typical SD card write is triggered everytime new data is received, -- It isn't. An internal 512 byte buffer is maintained by the SD library to store an entire data block. When that is filled it will flush it to the SD card in one go and get (or allocate) the next.
    – Majenko
    Nov 29, 2020 at 17:15
  • I suppose the question is not so much how the SD writing process itself can be sped up, but how incoming data can be preserved while the write is taking place
    – nuggetbram
    Nov 30, 2020 at 2:33

4 Answers 4


Is this a sensible solution to the problem?

No, unfortunately not. The size of the buffer is limited by the available RAM on your chip. Jumping between multiple buffers or cores will not change that, since it is a hardware limitation.

I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

When you are continuously receiving faster than you can write, you only have 2 options:

  • Improve write speed
  • Reduce the needed data size

There is no other way, and that's a logical conclusion, not so much a programmical or hardware conclusion. It that situation the amount of "buffered" data will increase indefinitely, so you would need an infinite amount of memory. And that doesn't exist in real world. The same principle would apply to a high end PC with lots of memory, it will just be full after longer time, than with an ESP.

  • Improving write speed:

    • On various sites I have read, that you can use the SD card with different SPI frequencies. A higher frequency would mean a higher transaction speed. You can investigate further at this point.
    • Writing a full 512 byte page at once and only closing and opening the file again, if really needed, will improve the overall speed.
  • Reduce the needed data size

    • Maybe you can compromise with your data resolution and then pack those values as dense as possible (like packing two 4-bit values into a single byte of transmission).
    • You can research for compression code, so that you can compress your data lossless before transmitting it. Though the result will always depend on the kind of your data. Also mind, that in this case you are trading transmission size for computation time and RAM.

About buffers:

I would say, a normal circular buffer does not bring you much advantages here, since with the SD card you don't want to do bytewise data transmission. The 2 buffer approach, that you mentioned, seems good to me. Though doing handling with different cores does not make much sense to me. The UART receiving itself is done by hardware and some ISRs. You can do all the SD card transmissions in your main code, just making sure, that you regularly copy the received data from the Serial buffer into your buffer.


Buffers will help you if at some point the input stream stops, so the "slower" device (e.g. the SD card) can catchup. If the data will always be received faster than it can be written (note always) then no buffer will help you.

But I assume you meant that there times where you get bursts of input data at 1Khz, then it will stop. So, as long as the average input data rate is lower than the ability for the SD to persist the data, you will be fine.

The SD card library already has a buffer (as @Majenko pointed out), so most likely you won't need to do anything. Seem like something you can test relatively simply.

The 2 core approach sounds somewhat overkill (not to mention some synchronization issues you will have to deal with)

  • Unfortunately in this case the input data is at a constant rate, start to finish with no interruptions. There's also up to a maximum of 30 minutes of data, meaning it's a bit outside the bounds of just writing to onboard memory and dumping to the SD card at the end, hence the 2 core idea. If a full SD 512 byte write only takes 3000us, data can be written at an average speed higher than its coming in - but what to do with the data received while the SD write is occurring?
    – nuggetbram
    Nov 30, 2020 at 0:53

Forgive my non-technical analogy, but this is how my brain works. Hope you can get past the conclusion that "this isn't a technical answer", until you at least see it from this different perspective.

Buffers, input streams, and sampling rates, storage rates all relate strongly to filling up a bath. Taps, water, baths and plug holes? WTF?

Try slightly blocking the plug hole in your bath, and turn both taps on full.

The water that's drained away is the rate of which data is stored to the SD card. That's the storage-rate.

The remaining water which doesn't drain away fast enough causes the bath the fill up. The water is the sampling rate, the bath is the buffer.

If the taps remain on, and water is leaving the taps faster than the water can escape from the partially blocked hole will cause the bath to fill up with water.

So, if the blockage isn't cleared, the bath will eventually overflow.

Either you turn the taps down, use a bigger bath, or increase the drainage rate.


One approach would be to use Circular Buffering. It involves using a fixed-size buffer that wraps around itself, allowing data to be continuously written and overwritten. This approach eliminates the need for costly memory reallocation or shifting operations. As new data is written, older data is overwritten in a circular fashion. But, you will need to choose the appropriate buffer size. A larger buffer size can help handle bursts of data However, larger buffers also consume more memory.

If one buffer isn't enough, then you could use Two buffers instead. While one buffer is being written to the SD card, the other buffer can be used to receive new incoming data. Once the write operation is completed, the roles of the buffers are swapped. This technique can help improve efficiency by minimizing data transfer and write latency, which is the goal you aim at. I've used this type of buffer for audio data acquisition, and it worked well, got rid of the buffer overflow issues.

The other method would be to use DMA, if your microcontroller or system supports it, which I think it does, using DMA for SD card writing can offload the data transfer task from the CPU. DMA allows direct memory access without CPU intervention, enabling faster and more efficient data transfers. DMA can significantly reduce the overhead associated with continuous SD card writing.

For DMA, you might want to check this: https://github.com/RobTillaart/DAC8552/issues/8 and this: https://github.com/espressif/esp-idf/blob/master/examples/peripherals/spi_master/lcd/main/spi_master_example_main.c

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.