19

Every time I design a serial protocol to be used between two arduinos, I feel a bit like I'm reinventing a wheel. I wonder if there are any best practices or patterns people follow. This question is less about the actual code, but more about the format of the messages.

For example, if I wanted to tell an arduino to flash it's first LED 3 times I might send:

^L1,F3\n
  • '^': starts a new command
  • 'L': Defines the command, (L: target this command to an LED)
  • '1': Target the first LED
  • ',': Command line separator, new value in this message to follow
  • 'F': Flash sub-command
  • '3': 3 times (Flash the LED three times)
  • '\n': End the command

Thoughts? How do you usually approach writing a new serial protocol? What about if I wanted to send a query from arduino 1 to arduino 2 and then receive a response?

13

There are a lot of ways to write a serial protocol depending on what functionality you might want and how much error checking you need.

Some of the common things you see in point to point protocols are:

End of message

The simplest ASCII protocols just have an end of message character sequence, often \r or \n as this is what gets printed when the enter key is hit. Binary protocols might use 0x03 or some other common byte.

Start of message

The problem with just having end of message is that you don't know what other bytes have already been received when you send your message. These bytes would then be prefixed to the message and cause it to be interpreted wrongly. For example, if the Arduino just woke from sleep there might be some garbage in the serial buffer. To get around this you have a start of message sequence. In your example, ^, in binary protocols often 0x02

Error Checking

If the message can be corrupted we need some error checking. This could be a checksum or a CRC error or something else.

Escape Characters

It could be that the checksum adds to a control character, such as the 'start of message' or 'end of message' byte, or the message contains a value equal to a control character. The solution is to introduce an escape character. The escape character is placed before a modified control character so that the actual control character is not present. E.g. if a start character is 0x02, using the escape character 0x10 we can send the value 0x02 in the message as the byte pair 0x10 0x12 (byte XOR control character)

Packet number

If a message is corrupted we could request a resend with a nack or retry message, but if multiple messages have been sent then only the latest message can be resent. Instead the packet can be given a number that rolls over after a certain number of messages. For example, if this number is 16, the transmitting device can store the last 16 messages send and if any were corrupted the receiving device can request a resend using the packet number.

Length

Often in binary protocols you see a length byte which tells the receiving device how many characters are in the message. This adds another level of error checking as if the correct number of bytes were not received then there was an error.

Arduino specific

When coming up with a protocol for Arduino the first consideration is how reliable is the communications channel. If you are sending over most wireless mediums, XBee, WiFi, etc, there is already built in error checking and retries and thus no point in putting these in your protocol. If you are sending over RS422 for a couple of kilometres then it will be necessary. The things I would include are the start of message and end of message characters, as you have. My typical implementation looks something like:

>messageType,data1,data2,…,dataN\n

Delimiting the data parts with a comma allows for easy parsing, and the message is sent using ASCII. ASCII protocols are great because you can type messages into the serial monitor.

If you want a binary protocol, maybe to shorten the message sizes, you will have to implement escaping if a data byte can be the same as a control byte. Binary control characters are better for systems where the full spectrum of error checking and retries is desired. The payload can still be ASCII if desired.

  • Isn't it possible that the garbage ahead of the real start of message code could contain a start of message control code? How would you deal with this? – CMCDragonkai Oct 6 '16 at 14:53
  • @CMCDragonkai Yes this is a possibility, especially for single byte control codes. However if you encounter a start control code halfway through parsing a message, the message is discarded and parsing restarts. – geometrikal Oct 7 '16 at 13:29
9

I do not have any formal expertise on serial protocols, but I've used them quite a few times, and more or less settled on this scheme:

(Packet header)(ID byte)(data)(fletcher16 checksum)(Packet Footer)

I usually make the header 2 bytes and the Footer 1 byte. My parser will dump everything when it sees a new packet header, and attempt to parse the message if it sees a footer. If the checksum fails, it will not ditch the message, but continue adding until the footer character is found and a checksum succeeds. That way the footer only needs to be one byte since collisions don't disrupt the message.

The ID is arbitrary, sometimes with the length of the data section being the bottom nibble (4 bits). A second length bit could be used but I normally don't bother since the length does not need to be known to parse correctly, so seeing the right length come through for a given ID is just more confirmation that the message was correct.

The fletcher16 checksum is a 2 byte checksum with nearly the same quality as CRC but is much easier to implement. some details here. The code can be as simple as this:

for(int i=0; i < bufSize; i++ ){
   sum1 = (sum1 + buffer[i]) % 255;
   sum2 = (sum2 + sum1) % 255;
}
uint16_t checksum = (((uint16_t)sum1)<<8) | sum2;

I've also used a call and respone system for critical messages, Where the PC will send a message every 500ms or so until it gets an OK message with a checksum of the whole original message as data (including the original checksum).

This scheme is, of course, not well suited for being typed into a terminal like your example would be. Your protocol seems pretty good for being limited to ASCII and I'm sure is easier for a quick project that you want to be able to directly read and send messages. For larger projects it is nice to have the density of a binary protocol and the security of a checksum.

  • Since "[your] parser will dump everything when it sees a new packet header" I wonder if this does not create problems if by chance the header is encountered inside of data? – humanityANDpeace Aug 13 '15 at 13:43
  • @humanityANDpeace The reason for dropping it is that when a packet gets cut off it will never parse correctly, so when do you decide its garbage and move on? The easiest, and in my experience good enough, solution is to drop an bad packet as soon as the next header comes in. I've been using a 16 bit header without issue, but you could make it longer if certainty is more important that bandwidth. – BrettAM Aug 13 '15 at 17:17
  • So what you refere to as a header is somewhat of a Magic 16bit combination. i.e. 010101001 10101010, right? I agree it is only 1/256*256 change to hit, but it also disables ever using this 16bit inside your data, else it gets misinterpreted as a header and you discard the message, right? – humanityANDpeace Aug 14 '15 at 12:31
  • @humanityANDpeace I know it's a year later, but you need to introduce an escape sequence. Before sending, the server checks the payload for any special bytes, then escapes them with another special byte. Client side, you have to put the original payload back together. This does mean that you cannot send fixed length packets and does complicate the implementation. There are many serial protocol standards to choose from that all address this. Here is a very good read on the topic. – RubberDuck Aug 13 '16 at 14:23
1

If you're into standards, then you may take a look at ASN.1 / BER TLV encoding. ASN.1 is a language used to describe data structures, made specifically for communication. BER is a TLV method of encoding the data structured using ASN.1. Problem is that ASN.1 encoding may be tricky at best. Creating a full fledged ASN.1 compiler is a project in itself (and a particularly tricky one at that, think months).


It's probably better to keep just the TLV structure. TLV basically consists of three elements: a tag, a length and a value field. The tag defines the type of the data (text string, octet string, integer etc.) and the length the length of the value.

In BER the T also denotes if the value is a set of TLV structures itself (a constructed node) or directly a value (a primitive node). That way you can create a tree in binary, much like XML (but without the XML overhead).

Example:

TT LL VV
02 01 FF

is an integer (tag 02) with a length of the value of 1 (length 01) and value -1 (value FF). In ASN.1 / BER the integers are sign big endian numbers, but you can of course use your own format.

TT LL (TT LL VV, TT LL VV VV)
30 07  02 01 FF  02 02 00 FF

is a sequence (a list) with length 7 containing two integers, one with value -1 and one with value 255. The two integer encodings together make up the value of the sequence.

You can simply toss this into an online decoder too, aint that nice?


You can also use indefinite length in BER which will allow you to stream data. In that case you do need to parse your tree correctly though. I'd consider it an advanced topic, you need to know about breadth first and depth first parsing, for one.


Using a TLV scheme basically allows you to think of any kind of data structure and encode it. ASN.1 goes a lot further than that, giving you unique identifiers (OID's), choices (a lot like C-unions), includes of other ASN.1 structures etc. etc. but that may be overkill for your project. Probably the best well known ASN.1-defined structures today are the certificates used by your browser.

0

If not, you've got the basics covered. Your commands can be created and read by both humans and machines which is a big plus. You might add a checksum to detect a mal-formed command or one damaged in transit, especially if your channel includes long cable or a radio link.

If you need industrial strength (your device mustn't cause or allow someone to get hurt or die; you need high data rates, fault recovery, missing packet detection etc.) then look to some of the standard protocols and design practices.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.