2

In a simplified example, I have a microcontroller functioning as a data acquisition device - sensor data is received over UART, and then saved to an SD card.

This received data is in the form of a 12 byte struct, received at 1kHz. The microcontroller receiving the data is an esp32, with an SD card connected over spi. Not an uncommon issue, but if a typical SD card write is triggered everytime new data is received, it will take longer than the gap between samples to write the data.

What is typically considered the best practice for this type of system, presuming the data will always be received faster than it can be written. I have seen examples of circular buffers, but presumably these will eventually fill completely and samples will be missed.

In my particular example, I have the bonus second cpu core in the esp32. Therefore, my thinking is to use 2 alternating buffers. Data will be received on core 1, and placed into the active buffer. When the buffer is full, incoming data will be written to the second buffer, and the first will be written to the SD card by core 2.

Is this a sensible solution to the problem? I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

Improve this question
4
  • data will always be received faster than it can be written ... that means that you will lose data ... data compression may help .... please provide few examples of the 12 bytes of data – jsotola Nov 29 '20 at 15:32
  • SDCards can be written over a 1 bit bus or 4 bit bus. The latter is faster and, as I recall, requires a license to implement. Also, the speed at which SDCards work varies greatly. If your driver can take advantage of this, try a faster SDCard. Finally, all SDCards contain a propriety controller. So picking the right OEM for your needs may help. (I'd go w/@jsotola recommendation 1st, try losses compression if you can. Simple is best.) – st2000 Nov 29 '20 at 15:43
  • if a typical SD card write is triggered everytime new data is received, -- It isn't. An internal 512 byte buffer is maintained by the SD library to store an entire data block. When that is filled it will flush it to the SD card in one go and get (or allocate) the next. – Majenko♦ Nov 29 '20 at 17:15
  • I suppose the question is not so much how the SD writing process itself can be sped up, but how incoming data can be preserved while the write is taking place – nuggetbram Nov 30 '20 at 2:33
3

Is this a sensible solution to the problem?

No, unfortunately not. The size of the buffer is limited by the available RAM on your chip. Jumping between multiple buffers or cores will not change that, since it is a hardware limitation.

I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

When you are continuously receiving faster than you can write, you only have 2 options:

  • Improve write speed
  • Reduce the needed data size

There is no other way, and that's a logical conclusion, not so much a programmical or hardware conclusion. It that situation the amount of "buffered" data will increase indefinitely, so you would need an infinite amount of memory. And that doesn't exist in real world. The same principle would apply to a high end PC with lots of memory, it will just be full after longer time, than with an ESP.

  • Improving write speed:

    • On various sites I have read, that you can use the SD card with different SPI frequencies. A higher frequency would mean a higher transaction speed. You can investigate further at this point.
    • Writing a full 512 byte page at once and only closing and opening the file again, if really needed, will improve the overall speed.

  • Reduce the needed data size

    • Maybe you can compromise with your data resolution and then pack those values as dense as possible (like packing two 4-bit values into a single byte of transmission).
    • You can research for compression code, so that you can compress your data lossless before transmitting it. Though the result will always depend on the kind of your data. Also mind, that in this case you are trading transmission size for computation time and RAM.

About buffers:

I would say, a normal circular buffer does not bring you much advantages here, since with the SD card you don't want to do bytewise data transmission. The 2 buffer approach, that you mentioned, seems good to me. Though doing handling with different cores does not make much sense to me. The UART receiving itself is done by hardware and some ISRs. You can do all the SD card transmissions in your main code, just making sure, that you regularly copy the received data from the Serial buffer into your buffer.

Improve this answer
1

Buffers will help you if at some point the input stream stops, so the "slower" device (e.g. the SD card) can catchup. If the data will always be received faster than it can be written (note always) then no buffer will help you.

But I assume you meant that there times where you get bursts of input data at 1Khz, then it will stop. So, as long as the average input data rate is lower than the ability for the SD to persist the data, you will be fine.

The SD card library already has a buffer (as @Majenko pointed out), so most likely you won't need to do anything. Seem like something you can test relatively simply.

The 2 core approach sounds somewhat overkill (not to mention some synchronization issues you will have to deal with)

Improve this answer
1
  • Unfortunately in this case the input data is at a constant rate, start to finish with no interruptions. There's also up to a maximum of 30 minutes of data, meaning it's a bit outside the bounds of just writing to onboard memory and dumping to the SD card at the end, hence the 2 core idea. If a full SD 512 byte write only takes 3000us, data can be written at an average speed higher than its coming in - but what to do with the data received while the SD write is occurring? – nuggetbram Nov 30 '20 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.