I have an Arduino project that is converting floats to characters and then transferring those characters via the serial to a Python program. This is a portion of the Arduino code:

#define MEM_LEN (16 * 4 * 4) + 1
uint8_t dataBuf[MEM_LEN]; 

typedef union {
  float f;
  uint8_t b[4];
} cvt_t;

cvt_t cvt1;

// set up sensor stuff
MLX90621 sensor; 

void loop() {
    sensor.measure(); //get new readings from the sensor
    count = 0;
    for(uint8_t y=0;y<4;y++){ //go through all the rows
      for(uint8_t x=0;x<16;x++){ //go through all the columns
        cvt1.f = sensor.getTemperature(y+x*4); // data at position x/y
        dataBuf[count + 0] = cvt1.b[0];
        dataBuf[count + 1] = cvt1.b[1];
        dataBuf[count + 2] = cvt1.b[2]; 
        dataBuf[count + 3] = cvt1.b[3];
        count += 4;
    dataBuf[MEM_LEN - 1] = '\n'; // not a good delimeter
    Serial.write(dataBuf, MEM_LEN);

The problem is that float conversion loads any and all characters into dataBuf, so I don't have a good delimiter or "end-of-line". When the Python program ingests the incoming string, there's nothing to use to know it's received a complete frame. I don't want the Python to just count characters to know when it's received a frame. No good delimiter - what should I do?

  • You have to use an "escape" character to signal that the next character must be read as a simple char, not as a control char. Use as escape any char you doesn't use to much in your data, said "%". Then you write "%/n" to send a normal char with value "\n". For line ending, you send just "\n". – user31481 Jul 5 '17 at 11:02

Framing binary messages is not as simple as choosing a delimiter. Counting the characters could be an option, but you need some way to check that your reception has not gotten out of sync. You could for example send messages that looks like:

struct Message {
    uint32_t magic;
    uint8_t data[MEM_LEN];
    uint16_t crc;

Where magic in a magic number, i.e. a random but fixed value that is very unlikely to happen by accident, and you end the message with a CRC. On the Python side, you check that both the magic number and the CRC are OK. If they are not, you have a communication error, but you can re-synchronize by looking for the next occurrence of the magic number.

Another option is to choose both a delimiter (say, ASCII 10 = LF = '\n') and an escape character (say, ASCII 92 = backslash = '\\'). On the sending side, you prefix by the escape character every accidental occurrence of either the escape character or the delimiter:

void send_byte(uint8_t c)
    if (c == '\\' || c == '\n')
        Serial.write('\\');  // prefix by escape character

and on the Python side you do the reverse operation. The only issue with this approach is that your messages are not constant size anymore.


Generally the method to ensure binary data is correctly decoded is to have some form of header and then a checksum. While the header may appear in the data the chances of the correct checksum also being randomly in the data at the correct point is fairly slim. The longer the header and checksum the lower the probability of invalid data somehow getting accepted.
The header could be as simple as a 1 or 2 byte fixed pattern or could be a fixed pattern followed by a message ID and a length.
The checksum is normally 16 bits, that means that if the system got out of sync somehow and then header happened to appear in the data before the next real header the invalid data has a 1 in 65,000 chance of being accepted.
Exact probabilities of invalid data being accepted depend upon the packet lengths but assuming random data then for a 16 bit header with a 16 bit checksum and 32 bits of data it's in the region of a 1 in 4 billion chance of allowing incorrect data in. A 32 bit header + 16 bit checksum increases that something like 1 in 100 trillion. At that point you've got more chance of winning every lottery in the world while being simultaneously struck by lightning and a meteor.
Data is rarely truly random, if you can use a header that is less likely to appear in the data then you can improve things further.

Since you only have 4 bytes of data I'll assuming you want to keep things simple and don't want to add a dozen bytes of overhead to each value. A header of 0xff is a good bet, if that was the first byte of a float the value will be in the -10E38 region. If you use a two byte header the 0xffc0 is a good choice, that would be NaN, an invalid floating point value. Since this means the first two bytes of your data aren't going to match the header this greatly reduces the chance of the header appearing in the data. Your checksum could then be a simple xor of the next for bytes. So your data packet ends up being:
0x ff c0 <d1> <d2> <d3> <d4> <d1^d2^d3^d4>

One other alternative approach is through the use of escape codes/characters. This is a special value that never appears on it's own in the data. For example let's define our escape character as being 0xFF and the header value as being 0xC0 C0.
In that situation 0xC0 C0 is always the start of a data packet.
If 0xC0 happens to appear in the data we insert the escape character in front of it. Similarly if the escape character happens to appear in the data we insert an extra escape character in front of it.
e.g. for a data value of 0x11 0xC0 22 FF we would send 0xC0 C0 11 FF C0 22 FF FF or <header> <d1> <escape> <d2> <d3> <escape> <d4>
For hopefully obvious reasons whenever possible the header and escape characters should be values that rarely occur in the data.

This approach has the advantage that if designed correctly there is no risk no matter how small of invalid data getting through and (on average) it has a lower overhead in terms of bytes sent, most of the time all you need to add is the 2 byte header. The down side is that inserting and removing the escape characters can get complex and messages are no longer a fixed size, the length depends on the data content. This indeterminacy in message length can potentially cause problems depending upon the application.

To be honest for a single floating point number text may be easier. Binary is more efficient, it's less CPU load and less data to send, but for something as short as a single number the savings aren't normally worth the extra complexity.

  • Good answer. You could also mention the possibility of escaping bytes in the data stream - maybe an 0xFF or 0xC0 in the data stream must be preceded by an escape character (and escape characters must be doubled up) so there is zero chance you see 0xFF 0xC0 in the data (you'd get, say 0x1B 0xFF 0x1B 0xC0, and every 0x1B by itself becomes 0x1B 0x1B). May make encoding/decoding a little harder, but the data integrity would be more reliable. – Majenko Jun 21 '17 at 9:30
  • @Majenko Good point. I'll add that. – Andrew Jun 21 '17 at 10:14
  • most of the time all you need to add is the 1 byte header. - actually having escape characters relies on always having at least a two byte header - that way the header is the only place where 0xC0 is guaranteed not to be preceded by 0xFF. If you have only one byte header and the final byte of the previous packet is 0xFF it looks like an escaped mid-data byte. You need a two-byte sequence which is 100% guaranteed to be the header because the escape character breaks that sequence up in the data portion. – Majenko Jun 21 '17 at 11:59
  • @Majenko Fixed. Although you could still get away with a one byte header if you made the escape sequence for FF to be something like FF FE. That way FF C0 would only ever be an escape sequence and never data followed by header since no valid data symbol would end in FF. However that means modifying data values rather than just padding them which is getting even messier to implement. – Andrew Jun 21 '17 at 12:13
  • Great answer, and by the way I'm just just sending a single floating point - notice it's in a loop that fills the dataBuf[] array with 256 characters and that gets transmitted. – Owen White Jun 21 '17 at 14:46

Using a version of Pascal strings could work: the first byte of the buffer is the number of characters, the rest of the sequence the bytes.

Or convert the numbers to ASCII strings. Inefficient but you'll have your '\0' delimiter.

  • Doesn't solve the problem of how to identify the start and end of data in a datastream. In fact it just makes it worse. – Majenko Jun 21 '17 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.