If any kind of character gets loaded into a string, what do I use for an end-of-line character?

Question

I have an Arduino project that is converting floats to characters and then transferring those characters via the serial to a Python program. This is a portion of the Arduino code:

#define MEM_LEN (16 * 4 * 4) + 1
uint8_t dataBuf[MEM_LEN]; 

typedef union {
  float f;
  uint8_t b[4];
} cvt_t;

cvt_t cvt1;

// set up sensor stuff
MLX90621 sensor; 
 .
 .
 .

void loop() {
    sensor.measure(); //get new readings from the sensor
    count = 0;
    for(uint8_t y=0;y<4;y++){ //go through all the rows
      for(uint8_t x=0;x<16;x++){ //go through all the columns
        cvt1.f = sensor.getTemperature(y+x*4); // data at position x/y
        dataBuf[count + 0] = cvt1.b[0];
        dataBuf[count + 1] = cvt1.b[1];
        dataBuf[count + 2] = cvt1.b[2]; 
        dataBuf[count + 3] = cvt1.b[3];
        count += 4;
      }
    }
    dataBuf[MEM_LEN - 1] = '\n'; // not a good delimeter
    Serial.write(dataBuf, MEM_LEN);
}

The problem is that float conversion loads any and all characters into dataBuf, so I don't have a good delimiter or "end-of-line". When the Python program ingests the incoming string, there's nothing to use to know it's received a complete frame. I don't want the Python to just count characters to know when it's received a frame. No good delimiter - what should I do?

Answer 1

Framing binary messages is not as simple as choosing a delimiter. Counting the characters could be an option, but you need some way to check that your reception has not gotten out of sync. You could for example send messages that looks like:

struct Message {
    uint32_t magic;
    uint8_t data[MEM_LEN];
    uint16_t crc;
};

Where magic in a magic number, i.e. a random but fixed value that is very unlikely to happen by accident, and you end the message with a CRC. On the Python side, you check that both the magic number and the CRC are OK. If they are not, you have a communication error, but you can re-synchronize by looking for the next occurrence of the magic number.

Another option is to choose both a delimiter (say, ASCII 10 = LF = '\n') and an escape character (say, ASCII 92 = backslash = '\\'). On the sending side, you prefix by the escape character every accidental occurrence of either the escape character or the delimiter:

void send_byte(uint8_t c)
{
    if (c == '\\' || c == '\n')
        Serial.write('\\');  // prefix by escape character
    Serial.write(c);
}

and on the Python side you do the reverse operation. The only issue with this approach is that your messages are not constant size anymore.

Answer 2

Generally the method to ensure binary data is correctly decoded is to have some form of header and then a checksum. While the header may appear in the data the chances of the correct checksum also being randomly in the data at the correct point is fairly slim. The longer the header and checksum the lower the probability of invalid data somehow getting accepted.
The header could be as simple as a 1 or 2 byte fixed pattern or could be a fixed pattern followed by a message ID and a length.
The checksum is normally 16 bits, that means that if the system got out of sync somehow and then header happened to appear in the data before the next real header the invalid data has a 1 in 65,000 chance of being accepted.
Exact probabilities of invalid data being accepted depend upon the packet lengths but assuming random data then for a 16 bit header with a 16 bit checksum and 32 bits of data it's in the region of a 1 in 4 billion chance of allowing incorrect data in. A 32 bit header + 16 bit checksum increases that something like 1 in 100 trillion. At that point you've got more chance of winning every lottery in the world while being simultaneously struck by lightning and a meteor.
Data is rarely truly random, if you can use a header that is less likely to appear in the data then you can improve things further.

Since you only have 4 bytes of data I'll assuming you want to keep things simple and don't want to add a dozen bytes of overhead to each value. A header of 0xff is a good bet, if that was the first byte of a float the value will be in the -10E38 region. If you use a two byte header the 0xffc0 is a good choice, that would be NaN, an invalid floating point value. Since this means the first two bytes of your data aren't going to match the header this greatly reduces the chance of the header appearing in the data. Your checksum could then be a simple xor of the next for bytes. So your data packet ends up being:
0x ff c0 <d1> <d2> <d3> <d4> <d1^d2^d3^d4>

One other alternative approach is through the use of escape codes/characters. This is a special value that never appears on it's own in the data. For example let's define our escape character as being 0xFF and the header value as being 0xC0 C0.
In that situation 0xC0 C0 is always the start of a data packet.
If 0xC0 happens to appear in the data we insert the escape character in front of it. Similarly if the escape character happens to appear in the data we insert an extra escape character in front of it.
e.g. for a data value of 0x11 0xC0 22 FF we would send 0xC0 C0 11 FF C0 22 FF FF or <header> <d1> <escape> <d2> <d3> <escape> <d4>
For hopefully obvious reasons whenever possible the header and escape characters should be values that rarely occur in the data.

This approach has the advantage that if designed correctly there is no risk no matter how small of invalid data getting through and (on average) it has a lower overhead in terms of bytes sent, most of the time all you need to add is the 2 byte header. The down side is that inserting and removing the escape characters can get complex and messages are no longer a fixed size, the length depends on the data content. This indeterminacy in message length can potentially cause problems depending upon the application.

To be honest for a single floating point number text may be easier. Binary is more efficient, it's less CPU load and less data to send, but for something as short as a single number the savings aren't normally worth the extra complexity.

Answer 3

Using a version of Pascal strings could work: the first byte of the buffer is the number of characters, the rest of the sequence the bytes.

Or convert the numbers to ASCII strings. Inefficient but you'll have your '\0' delimiter.