XBee: How do I create a packet?

Question

I am currently using a simple protocol I created to send three bytes (unit information, data, and XOR) individually over an XBee radio (3 bytes are considered a single transmission). However, this is not always reliable if a byte is dropped. For this reason I'd like to encapsulate each individual byte as a packet in a way that the Arduino could identify which byte is which and if a byte is lost then just wait for the next transmission. What I am not sure about is creating a packet. I haven't been able to find any tutorials or example code.

Answer 1

The biggest mistake people always make with serial communications of any form is to just send raw data over the connection. While you might just about get away with that on a 100% reliable direct serial connection over a short distance, as soon as you introduce anything with any form of uncertainty to it you are, as you have seen, doomed to failure.

So you have to come up with a reliable method of transferring data in such a way that the receiver knows that it has received valid data, and how to find valid data.

So you need to create a packet. That's not something where you just wave a magic wand and a packet is created. A packet starts it's life in your own head.

In order to reliably transfer data there are a few things you need to consider from the point of view of the receiver:

• Where is the start of my data?
• How much data is there?
• Is this data correct?

The receiver has to have some way of knowing where the start of a packet is amid the torrent of bytes it is being sent. It needs to look out for something unique, something it cannot mistake for anything else. This is called synchronisation. It is often done with a specific sequence of bytes (known as a preamble) that can, through various means, never appear anywhere else.

And at that point things start to get a little more complex, for now you need some way of wrestling that kind of thing into a system that can only send bytes of 0 to 255 with no way of distinguishing raw data from control information like this synchronisation sequence. And that is where the concept of escaping comes in. This is where you nominate certain byte values as control bytes and decide that these bytes will never appear in your data without some kind of marker to say they are data bytes. A common escape marker is 255 (used in the telnet protocol) or 27 (the ANSI ESC character) although any character can be chosen.

For instance if you choose to have a synchronisation sequence of 0xAA, 0x55, you can then decide that neither of those bytes will ever appear in your data without first being preceeded by character 0xFF. So if you happened to have that sequence in your data you would expand it out to 0xFF, 0xAA, 0xFF, 0x55. But what if you want to send the value 0xFF, in your data? How do you handle that? Well, you also escape 0xFF as well. 0xFF, being the escape character, has no actual value in the data. It only says that the next character must be interpreted as data. So to send 0xFF you would actually send 0xFF, 0xFF. Your receiver sees the first 0xFF and thinks "ah, the next byte that arrives, whatever it is, just be data." It then gets a second 0xFF and knows it is data. (Actually this is the opposite of escaping which says the next character is a control character, but the underlying concept is the same).

If it ever sees 0xAA, 0x55 then it knows, without a shadow of a doubt, that this is the start of a packet, or random noise. And that is where the next two points above come in: Validating your packet.

If your packets are always the same size then all you need to know is how many bytes to expect, which can be hard coded into your software. If you have different sizes of data to send then you also need to tell the receiver how big the data is. This forms part of the packet header. A small block of meta data that preceeds the actual data that holds information about the packet. Depending on your topology that could be things like the size of the data, where the packet came from, where it is going, etc, and maybe a checksum, though some people like to put the checksum after the data since it simplifies sending.

And then there is the checksum. Possibly the most important thing next to the synchronisation. And so easy to get wrong.

Having a checksum that is too simple is not so good. It can be fooled. That is, if values get corrupted in the data with a simple checksum it is far more likely that the checksum still says it's correct than with a more complex checksum. You should read up on existing checksum algoritms, such as Cyclic Redundancy Check (CRC) and things like the checksums used in IP packets. All are good lessons on how to deal with checksums properly.

So far we have just tackled the serial equivalent of UDP - throw a packet at a receiver and hope it gets there. Doing anything more reliable becomes much more complex. If you want to know if your packet actually got there you will need your receiver to acknowledge reception, possibly including some information about the packet it received, such as a sequence number. If the transmitter never receives that acknowledgement it needs to send the packet again, which means it needs to remember what the packet is that it sent. So you need a packet buffer, or even better, a packet queue. And timeouts. And sequence numbers.

That latter is most important. A sequence number is a unique (or unique over a reasonable period of time) identifier for a packet. When a transmitter doesn't receive an acknowledgement is it because the receiver never got the packet, or that the receiver did get the packet but the acknowledgement never made it back to the transmitter? If the latter then the receiver is liable to get the same packet multiple times through retransmission. By including a sequence number the receiver can go "hang on a moment, I have seen this packet already" and it can throw it away sending a new acknowledgement in the process.

So as you can see, reliable communication isn't as simple as just sending data, or even sending data in the form of a struct. There is a whole lot more to learn and understand.

I suggest you read some of the IP networking RFCs to learn more about reliable packet transmission and reception.

Oh, and I haven't even touched on sliding window reception and such like for how to best program your receiver software...

Answer 2

• The x-ctu xbee config tool has a few options worth exploring such as error checking. This means the xbee would do the work.

OR using the Arduino(s)

• Adding a couple more bytes to you your current system with a checksum of the first 3 bytes would be a simple way to check for a valid packet.

• You could also use a struct of values (along with a checksum). Bill Porter has written an Arduino Library called Arduino Easy Transfer. The library is designed to send and receive a struct along with checksum validation. This is an overview page for the library.

• Sending and receiving structs over Serial is also covered here