Sending Large Amounts of Serial Data

Question

So in the fields of robotics sometimes you need multiple boards and or computers linked together to share information or save statistical data. Currently I need to send a few different variables over a serial connection and was wondering what was the best way to go about doing such?

So far I've determined that sending structures would probably be the easier way to send data. Does anyone know of any other way that might be more efficient?

Please keep in mind that I will essentially be having to send the data for 4 motors, compressor, different temperatures, random things, and the 3 sections of the arm.

Answer 1

With my personal thoughts on Structs being the most efficient way to send many different variables I've constructed a library to help make it easier to send structs and variables over serial. Source Code

In this library it makes sending through serial easily. I've used with with hardware and software serial. Usually this is used in conjunct with xbee's so I can wirelessly send the data to and from the robot.

When sending data it make it simple as it allows you to either send a variable or a struct (it doesn't care).

Here is an example of sending a simple char over the serial:

// Send the variable charVariable over the serial.
// To send the variable you need to pass an instance of the Serial to use,
// a reference to the variable to send, and the size of the variable being sent.
// If you would like you can specify 2 extra arguments at the end which change the
// default prefix and suffix character used when attempting to reconstruct the variable
// on the receiving end. If prefix and suffix character are specified they'll need to 
// match on the receiving end otherwise data won't properly be sent across

char charVariable = 'c'; // Define the variable to be sent over the serial
StreamSend::sendObject(Serial, &charVariable, sizeof(charVariable));

// Specify a prefix and suffix character
StreamSend::sendObject(Serial, &charVariable, sizeof(charVariable), 'a', 'z');

Example of sending a simple int over the serial:

int intVariable = 13496; // Define the int to be sent over the serial
StreamSend::sendObject(xbeeSerial, &intVariable, sizeof(intVariable));

// Specify a prefix and suffix character
StreamSend::sendObject(xbeeSerial, &intVariable, sizeof(intVariable), 'j', 'p');

Example of sending a struct over serial:

// Define the struct to be sent over the serial
struct SIMPLE_STRUCT {
  char charVariable;
  int intVariable[7];
  boolean boolVariable;
};
SIMPLE_STRUCT simpleStruct;
simpleStruct.charVariable = 'z'; // Set the charVariable in the struct to z

// Fill the intVariable array in the struct with numbers 0 through 6
for(int i=0; i<7; i++) {
  simpleStruct.intVariable[i] = i;
}

// Send the struct to the object xbeeSerial which is a software serial that was
// defined. Instead of using xbeeSerial you can use Serial which will imply the
// hardware serial, and on a Mega you can specify Serial, Serial1, Serial2, Serial3.
StreamSend::sendObject(xbeeSerial, &simpleStruct, sizeof(simpleStruct));

// Send the same as above with a different prefix and suffix from the default values
// defined in StreamSend. When specifying when prefix and suffix character to send
// you need to make sure that on the receiving end they match otherwise the data
// won't be able to be read on the other end.
StreamSend::sendObject(xbeeSerial, &simpleStruct, sizeof(simpleStruct), '3', 'u');

Receiving Examples:

Receiving a char that was send via Streamsend:

char charVariable; // Define the variable on where the data will be put

// Read the data from the Serial object an save it into charVariable once
// the data has been received
byte packetResults = StreamSend::receiveObject(Serial, &charVariable, sizeof(charVariable));

// Reconstruct the char coming from the Serial into charVariable that has a custom
// suffix of a and a prefix of z
byte packetResults = StreamSend::receiveObject(Serial, &charVariable, sizeof(charVariable), 'a', 'z');

Receiving an int that was sent via StreamSend:

int intVariable; // Define the variable on where the data will be put

// Reconstruct the int from xbeeSerial into the variable intVariable
byte packetResults = StreamSend::receiveObject(xbeeSerial, &intVariable, sizeof(intVariable));

// Reconstruct the data into intVariable that was send with a custom prefix
// of j and a suffix of p
byte packetResults = StreamSend::receiveObject(xbeeSerial, &intVariable, sizeof(intVariable), 'j', 'p');

Receiving a Struct that was sent via StreamSend:

// Define the struct that the data will be put
struct SIMPLE_STRUCT {
  char charVariable;
  int intVariable[7];
  boolean boolVariable;
};
SIMPLE_STRUCT simpleStruct; // Create a struct to store the data in

// Reconstruct the data from xbeeSerial into the object simpleStruct
byte packetResults = StreamSend::receiveObject(xbeeSerial, &simpleStruct, sizeof(simpleStruct));

// Reconstruct the data from xbeeSerial into the object simplestruct that has
// a prefix of 3 and a suffix of p
byte packetResults = StreamSend::receiveObject(xbeeSerial, &simpleStruct, sizeof(simpleStruct), '3', 'p');

Once you read the data using StreamSend::receiveObject() you need to know if the data was GOOD, Not Found, or BAD.

Good = Successful

Not Found = No prefix character was found in the specified ostream

Bad = Somehow there was a prefix character found, but the data isn't intact. Usually it means there was no suffix character found or the data wasn't the correct size.

Testing Validity of Data:

// Once you call StreamSend::receiveObject() it returns a byte of the status of
// how things went. If you run that though some of the testing functions it'll
// let you know how the transaction went
if(StreamSend::isPacketGood(packetResults)) {
  //The Packet was Good
} else {
  //The Packet was Bad
}

if(StreamSend::isPacketCorrupt(packetResults)) {
  //The Packet was Corrupt
} else {
  //The Packet wasn't found or it was Good
}

if(StreamSend::isPacketNotFound(packetResults)) {
  //The Packet was not found after Max # of Tries
} else {
  //The Packet was Found, but can be corrupt
}

SteamSend Class:

#include "Arduino.h"

#ifndef STREAMSEND_H
#define STREAMSEND_H


#define PACKET_NOT_FOUND 0
#define BAD_PACKET 1
#define GOOD_PACKET 2

// Set the Max size of the Serial Buffer or the amount of data you want to send+2
// You need to add 2 to allow the prefix and suffix character space to send.
#define MAX_SIZE 64


class StreamSend {
  private:
    static int getWrapperSize() { return sizeof(char)*2; }
    static byte receiveObject(Stream &ostream, void* ptr, unsigned int objSize, unsigned int loopSize);
    static byte receiveObject(Stream &ostream, void* ptr, unsigned int objSize, unsigned int loopSize, char prefixChar, char suffixChar);
    static char _prefixChar; // Default value is s
    static char _suffixChar; // Default value is e
    static int _maxLoopsToWait;

  public:
    static void sendObject(Stream &ostream, void* ptr, unsigned int objSize);
    static void sendObject(Stream &ostream, void* ptr, unsigned int objSize, char prefixChar, char suffixChar);
    static byte receiveObject(Stream &ostream, void* ptr, unsigned int objSize);
    static byte receiveObject(Stream &ostream, void* ptr, unsigned int objSize, char prefixChar, char suffixChar);
    static boolean isPacketNotFound(const byte packetStatus);
    static boolean isPacketCorrupt(const byte packetStatus);
    static boolean isPacketGood(const byte packetStatus);

    static void setPrefixChar(const char value) { _prefixChar = value; }
    static void setSuffixChar(const char value) { _suffixChar = value; }
    static void setMaxLoopsToWait(const int value) { _maxLoopsToWait = value; }
    static const char getPrefixChar() { return _prefixChar; }
    static const char getSuffixChar() { return _suffixChar; }
    static const int getMaxLoopsToWait() { return _maxLoopsToWait; }

};

//Preset Some Default Variables
//Can be modified when seen fit
char StreamSend::_prefixChar = 's';   // Starting Character before sending any data across the Serial
char StreamSend::_suffixChar = 'e';   // Ending character after all the data is sent
int StreamSend::_maxLoopsToWait = -1; //Set to -1 for size of current Object and wrapper



/**
  * sendObject
  *
  * Converts the Object to bytes and sends it to the stream
  *
  * @param Stream to send data to
  * @param ptr to struct to fill
  * @param size of struct
  * @param character to send before the data stream (optional)
  * @param character to send after the data stream (optional)
  */
void StreamSend::sendObject(Stream &ostream, void* ptr, unsigned int objSize) {
  sendObject(ostream, ptr, objSize, _prefixChar, _suffixChar);
}

void StreamSend::sendObject(Stream &ostream, void* ptr, unsigned int objSize, char prefixChar, char suffixChar) {
  if(MAX_SIZE >= objSize+getWrapperSize()) { //make sure the object isn't too large
    byte * b = (byte *) ptr; // Create a ptr array of the bytes to send
    ostream.write((byte)prefixChar); // Write the suffix character to signify the start of a stream

    // Loop through all the bytes being send and write them to the stream
    for(unsigned int i = 0; i<objSize; i++) {
      ostream.write(b[i]); // Write each byte to the stream
    }
    ostream.write((byte)suffixChar); // Write the prefix character to signify the end of a stream
  }
}

/**
  * receiveObject
  *
  * Gets the data from the stream and stores to supplied object
  *
  * @param Stream to read data from
  * @param ptr to struct to fill
  * @param size of struct
  * @param character to send before the data stream (optional)
  * @param character to send after the data stream (optional)
  */
byte StreamSend::receiveObject(Stream &ostream, void* ptr, unsigned int objSize) {
    return receiveObject(ostream, ptr, objSize, _prefixChar, _suffixChar);
}
byte StreamSend::receiveObject(Stream &ostream, void* ptr, unsigned int objSize, char prefixChar, char suffixChar) {
  return receiveObject(ostream, ptr, objSize, 0, prefixChar, suffixChar);
}

byte StreamSend::receiveObject(Stream &ostream, void* ptr, unsigned int objSize, unsigned int loopSize, char prefixChar, char suffixChar) {
  int maxLoops = (_maxLoopsToWait == -1) ? (objSize+getWrapperSize()) : _maxLoopsToWait;
  if(loopSize >= maxLoops) {
      return PACKET_NOT_FOUND;
  }
  if(ostream.available() >= (objSize+getWrapperSize())) { // Packet meets minimum size requirement
    if(ostream.read() != (byte)prefixChar) {
      // Prefix character is not found
      // Loop through the code again reading the next char
      return receiveObject(ostream, ptr, objSize, loopSize+1, prefixChar, suffixChar);
    }

    char data[objSize]; //Create a tmp char array of the data from Stream
    ostream.readBytes(data, objSize); //Read the # of bytes
    memcpy(ptr, data, objSize); //Copy the bytes into the struct

    if(ostream.read() != (byte)suffixChar) {
      //Suffix character is not found
      return BAD_PACKET;
    }
      return GOOD_PACKET;
  }
  return PACKET_NOT_FOUND; //Prefix character wasn't found so no packet detected
}


boolean StreamSend::isPacketNotFound(const byte packetStatus) {
    return (packetStatus == PACKET_NOT_FOUND);
}

boolean StreamSend::isPacketCorrupt(const byte packetStatus) {
    return (packetStatus == BAD_PACKET);
}

boolean StreamSend::isPacketGood(const byte packetStatus) {
    return (packetStatus == GOOD_PACKET);
}

#endif

Answer 2

If you really wanted to send it fast, i recommend Full Duplex Serial (FDX). It's the same protocol that USB and ethernet use, and it's a lot faster than UART. The downside is that it usually requires external hardware to facilitate the high data rates. I've heard that the new softwareSreial supports FDX, but this may be slower even than hardware UART. For more on communication protocols, see How to connect two Arduino without shields?

Answer 3

Sending a structure is fairly simple.

You can declare the structure as you normally would, and then use memcpy(@myStruct,@myArray) to copy the data to a new location, and then use something similar to the below code to write the data as a datastream.

unsigned char myArraySender[##];   //make ## large enough to fit struct
memcpy(&myStruct,&myArraySender);  //copy raw data from struct to the temp array
digitalWrite(frameStartPin,High);  //indicate to receiver that data is coming
serial.write(sizeof myStruct);     //tell receiver how many bytes to rx
Serial.write(&myArraySender,sizeof myStruct);   //write bytes
digitalWrite)frameStartPin,Low);   //done indicating transmission 

Then you can attach an interrupt routine to the pin on the other device that does the following:

volatile unsigned char len, tempBuff[##];   
//volatile because the interrupt will not happen at predictable intervals.

attachInterrupt(0,readSerial,Rising);  

//tell the mcu to call fxn when pinhigh. This will happen during virtually any moment. if that is not desired, remove the interrupt and simply watch for new characters in your main executive loop (aka, UART polling).

void readSerial(unsigned char *myArrayReceiver){
    unsigned char tempbuff[sizeof myArrayReceiver];
    while (i<(sizeof myArrayReceiver)) tempBuff[i]=Serial.read();
    memcpy(&tempbuff,&myArrayReceiver);
    Serial.flush();
}

Syntax and use of pointers will need some review. I pulled an all-nighter so I'm sure the above code won't even compile, but the idea is there. Fill your structure, copy it, use out-of-band signalling to avoid framing errors, write the data. On the other end, receive the data, copy it to a struct, and then the data becomes accessible via normal member access methods.

The use of bitfields will also work, just be aware that the nibbles will appear to be backwards. For instance, attempting to write 0011 1101, may result in 1101 0011 appearing at the other end if the machines differ in byte order.

If data integrity is important, you can also add a checksum to make sure you aren't copying misaligned garbage data. This is quick and effective check that I recommend.

Answer 4

If you can tolerate the data volume, debugging communicatons is so much easier when sending strings than when sending binary; sprintf()/sscanf() and their variants are your friends here. Enclose the communication in dedicated functions in their own module (.cpp file); if you need to optimize the channel later - after you have a working system - you can replace the string-based module with one coded for smaller messages.

You'll make your life lots easier if you hold tight to protocol specs on transmission and interpret them more loosely on reception, regard to field widths, delimiters, line endings, insignificant zeros, presence of + signs, etc.

Answer 5

I have no official credentials here, but in my experience things have gone pretty efficiently when I pick a certain character position(s) to contain the state of a variable, so that you might designate the first three characters as the temperature, and the next three as the angle of a servo, and so on. On the sending end I would save the variables individually and then combine them in a string to send serially. On the receiving end I would have the string picked apart, getting the first three characters and turning them into whatever variable type i need, then doing so again to get the next variable value. This system works best when you know for certain the amount of characters each variable will take up, and you always look for the same variables (which I hope is a given) each time the serial data loops through.

You can pick one variable to put last of indeterminate length and then get that variable from it's first character to the end of the string. Granted, the serial data string could get really long depending on the variable types and the sheer amount of them, but this is the system I use and so far the only setback I've hit is the serial length, so that's the only disadvantage I know of.

Answer 6

Send struct data across serial

Nothing fancy. Sends a struct. It uses an escape character '^' to delimit the data.

Arduino code

typedef struct {
 float ax1;
 float ay1;
 float az1;
 float gx1;
 float gy1;
 float gz1;
 float ax2;
 float ay2;
 float az2;
 float gx2;
 float gy2;
 float gz2;

} __attribute__((__packed__))data_packet_t;

data_packet_t dp;

template <typename T> void sendData(T data)
{
 unsigned long uBufSize = sizeof(data);
 char pBuffer[uBufSize];

 memcpy(pBuffer, &dp, uBufSize);
 Serial.write('^');
 for(int i = 0; i<uBufSize;i++) {
   if(pBuffer[i] == '^')
   {
    Serial.write('^');
    }
   Serial.write(pBuffer[i]);
 }
}
void setup() {
  Serial.begin(57600);
}
void loop(){
dp.ax1 = 0.03; // Note that I didn't fill in the others. Too much work. ;p
sendData<data_packet_t>(dp);
}

Python Code:

import serial
from  copy import copy
from struct import *


ser = serial.Serial(
#   port='/dev/cu.usbmodem1412',
  port='/dev/ttyUSB0',
#     port='/dev/cu.usbserial-AL034MCJ',
    baudrate=57600
)



def get_next_data_block(next_f):
    if not hasattr(get_next_data_block, "data_block"):
        get_next_data_block.data_block = []
    while (1):
        try:
            current_item = next_f()
            if current_item == '^':
                next_item = next_f()
                if next_item == '^':
                    get_next_data_block.data_block.append(next_item)
                else:
                    out = copy(get_next_data_block.data_block)
                    get_next_data_block.data_block = []
                    get_next_data_block.data_block.append(next_item)
                    return out
            else:
                get_next_data_block.data_block.append(current_item)
        except :
            break


for i in range(1000): # just so that the program ends - could be in a while loop
    data_ =  get_next_data_block(ser.read)
    try:
        print unpack('=ffffffffffff', ''.join(data_))
    except:
        continue