Unpacking Arduino binary data on a python sketch

Question

As a follow up to my earlier post: Simultaneously reading data from multiple serial ports, I am stuck trying to develop a python sketch that can read the incoming serial data from the com port on my Arduino and unpack them from the buffer into two separate cvs files.

One of the csv files would contain the data from the first sensor, while the other would contain the data from the second sensor. I have been able to develop the sketch to a point where on receiving the first header byte the first sensor data is unpacked. I am finding it hard to adjust the sketch to allow for the second byte header to be called to unpack the second sensor data from the data packet. Below is my abbreviated sketch thus far. Thanks all.

def run(self):
    #Creating a csv file to read data from the first sensor
    self.click = open("ball_Data.csv", "wb")
    #Creating a csv file to read data from the second sensor
    self.flick = open("rugby_Data.csv", "wb")
    #creating the serial port object
    self.push = serial.Serial('COM6', 9600)
    while True:
        #If the incoming byte is equivalent to '$'
        if node == b'$':
            # the byte object is equivalent to the available bytes
            node = self.push.read()
            #an array to hold the sensor data
            sensor_array = []
            i = 0
            #This while loop initiates a count from 0 to 9
            #through the sensor array
            while i < 10:
                #The available byte present
                malcolm = self.push.read()
                #The next byte
                turnbull = self.push.read()
                #The incoming uint_16 data is unpacked into the variable lingo
                lingo= (struct.unpack('b', turnbull)[0] << 8) | struct.unpack('B',malcolm)[0]
                #The variable lingo is then appended to the sensor array
                sensor_array.append(lingo)
                #If the length of the sensor array is equivalent to 9
                #write the data through each count into the sensor array
                if len(sensor_array) == 9:
                    self.click.write(str(sensor_array[0] ..+ str(sensor_array[9]))
                    #My idea is that this else statement was supposed to trigger
                    #the next header for the second sensor which would be an '#'
                    #sign before unpacking the data
                    else:
                        oldde = self.push.read()
                        sensor_array = []
                        i = 10
                        while i < 20:
                            #The available byte present
                            malcolm = self.push.read()
                            #The next byte
                            turnbull = self.push.read()
                            lingo= (struct.unpack('b', turnbull)[10]<<8)|struct.unpack('B',malcolm)[10]
                            sensor_array.append(lingo)
                            if len(sensor_array) > 19:
                                self.flick.write(str(sensor_array[10]..+ str(sensor_array[19]))

                                The Packet data structure for the first and second sensor is:
                                    Packet1 from sensor 1:
                                        data_array[0] = headerbyte('#')
                                        data_array[1] = acceleration_xAxis
                                        data_array[2] = acceleration_yAxis
                                        data_array[3] = acceleration_zAxis
                                        data_array[4] = temperature
                                        data_array[5] = gyro_xAxis
                                        data_array[6] = gyro_yAxis
                                        data_array[7] = gyroz_xAxis
                                        data_array[8] = checksum
                                        Packet2 from sensor 2:
                                            data_array[9] = headerbyte('$')
                                            .
                                            .
                                            .
                                            data_array[17] = checksum

Answer 1

I'm somewhat confused by the flow of your program, so instead I will tell you how I would handle the data.

You have a single-byte header, then 7 bytes of data, followed by a single byte checksum.

PySerial's read is (I believe) blocking by default, so it's simple to read the next serial character - unlike an Arduino. It also has a "multi-byte" read version where it can read a pre-set number of bytes from the stream for you.

So my methodology would be:

1. Read a byte. Is it a '#' or a '$'?
2. If it is, remember which stream you are dealing with, then:
3. Read 7 bytes with this.push.read(7) into a byte array (or a string in python < 2.6)
4. Read one byte for the checksum. If the checksum is correct (whatever calculation you use for that) then unpack the data in the array.
5. Write the data to the correct file depending on the stream you remembered in step 2.

That way you're not duplicating code for each sensor stream, you are just saying "this packet is stream 2" and using that remembered fact once you have received the packet and verified that it's correct.

A better protocol arrangement, though, which is more reliable, involves both a start byte and an end byte, and your whole packet is wrapped in that pair of bytes. You then have a "sliding window" arrangement whereby the incoming data bytes scroll past a window of a fixed width, and when the first and last characters in that window match your start and end bytes you then extract the middle section and do what you need to do with it. That way if you have a $ or a # that is midway through a stream of data it won't trigger reception since the end byte doesn't match. It makes the whole synchronisation of the data more reliable.

For instance, if you have the current data packet that looks like $ho9*$fup and you miss the first bit of the data for whatever reason (arduino started transmitting before you were ready to receive, for instance) it might trigger reception on the second $ which would give you incorrect values. So instead if you have the start character as $ and the end character as # and include a stream number, so your packet looks like:

$1ho9*$fup#

Then when you scroll that past the window it would look like:

 $         #
 v         v
[..........$]
[.........$1]
[........$1h]
[.......$1ho]
[......$1ho9]
[.....$1ho9*]
[....$1ho9*$]
[...$1ho9*$f]
[..$1ho9*$fu]
[.$1ho9*$fup]
[$1ho9*$fup#]

And the first and last characters match, so you receive the content.

If you were to miss the first $ the second $ wouldn't match - say the extended stream looks like: $1ho9*$fup#$2hdiw83bf# and you missed $1, the window would look like:

 $         #
 v         v
[..........o]
[.........o9]
[........o9*]
[.......o9*$]
[......o9*$f]
[.....o9*$fu]
[....o9*$fup]
[...o9*$fup#]
[..o9*$fup#$]
[.o9*$fup#$2]
[o9*$fup#$2h]
[9*$fup#$2hd]
[*$fup#$2hdi]
[$fup#$2hdiw] <= Doesn't match even though it starts with $
[fup#$2hdiw8]
[up#$2hdiw83]
[p#$2hdiw83b]
[#$2hdiw83bf]
[$2hdiw83bf#] <= Matches - starts with $ and ends with #

As you can see the synchronising of your packet is then much more reliable. It would normally have incorrectly triggered on the first $ it saw.

There is always a chance that there may be a $ and a # in just the right positions within successive packets to fool the system, but the chances of that are somewhat slim. You can add extra protection against that by "escaping" the start and stop bytes if they appear within your stream. There's many possible ways of handling that if you really need it.