Advice for checking integrity of serial char strings?

Question

I'm trying to create a flexible but reliable system for arduinos to interact using serial ascii strings. I'm using a message structure comprised of Command[10] + (optional) Params[20] + (optional) Header[30] + '\0' ( terminator).

Each arduino node looks for incoming serial Commands (plus any appropriate optional Parameters) that it must respond to.

During development and debugging it's sufficient to send Commands and Parameters using the IDE serial monitor without optional Header block or error-checking.

But for interactive node-to-node communications, an optional 'Header' (technically a Footer) block is automatically added to the end of the message.

If an incoming serial message includes a header block, it means that the message was sent automatically from another node (rather than manually from the IDE serial monitor), and consequently the message integrity should be checked against any relevent accompanying header contents, then an acknowledgement returned to the sender if the message was received ok.

Header contents can optionally contain any of the following, and in any order:- s(source)=nnn, d(destination)=nnn, i(id)=nnn, l(msg length)=nnn, and non-critical mixtures of commas and/or spaces can be used as delimters.

To test for message length, the transmitted l=nnn value (text representation) is padded to 3 digits with leading zeros, so the length stays the same whatever the value (eg:.anything from 000 to 999 still requires 3 digits).

So far so good. But although a length test is better than nothing, it doesn't trap for corrupt characters, and the plan is for nodes to communicate via UDP broadcasts, so some form of error-checking and acknowledgement is essential.

So the plan was to (also, or instead) use an optional c(checksum)=nnn in the header, whose value would be the mod sum of everything EXCEPT itself in the header entry. This is easily enough achieved for transmission by taking the mod sum of everything else when assembling the message, then adding the c=modsum on the end (or anywhere in the Header for that matter).

In theory, the receiving end just needs to remove the c=nnn entry (plus any preceding delimter) from wherever it is in the Header, then compare the remaining message's calculated mod sum value against the transmitted c=nnn value.

In practice, I don't know how to exclude the c=nnn entry from the header in order to calculate the remaining modsum, without being forced to accept and pander to unwelcome restrictive assumptions, such as the c=nnn value having to be the final entry, and having to use very strict delimiting.

I'm trying to keep things as flexible and informal as possible, so that other x=nnn type instruction options may be added anywhere to the header if needed without risk of breaking existing rigidity.

Perhaps I should add that this time a year ago I hadn't touched an arduino or C, and that this project has long ago changed from my original automation/security needs to become a goal for providing a simple-to-use interactive arduino distributed functionality cluster system suitable for anyone to use.

The end finally seems to be in sight, which is thanks to the help and advice from you guys (a quick look through my questions shows how far you've brought me), so hopefully your greater experience and wisdom may also be able to offer a more satisfactory integrity-test solution if it is possible.

I would also welcome any other comments that might help me improve on what I'm trying to do.

ADDITIONAL INFO

New incoming serial messages are received in char Buffer[64]. This gets split by delimiters into Command[10], Params[20] and Header[30], but these sizes are only arbitrary first guestimates which may change.

Commands and Parameters can be whatever the node has been programmed to respond to. A trivial 'Blink' example might be a couple of nodes, one looking for incoming "PING" Commands and responding by returning "PONG" Commands, and the other node vice- versa.

A more useful example could be an IR receiver node issuing instructions to a remote IR blaster node, so the "Command+Param+Header" string might be:-

"IRSEND 'NEC 0xFFEEAB99, 32', s=1, d=2, l=044"

The Command/Params parser accepts comma or space as delimiter except if they are contained within quotes. Quotes may be sent inside of quotes of the other type (single or double). Mr T's invaluable Header parser is slightly modified to also accept comma's or spaces as delimiters in the Header, therefore " , s=nnn,d=nnn, i=nnn l=nnn , " all extracts ok.

All of the above is working ok in a very messy and disjointed proof-of-concept form.

    static char buffer[64];
    int msgpos = 0;
    int msglength;
    int quotes=0;
    int parseMode=0;
    bool echo=false;
    bool ack=false;
    byte chksum=0;
    int modchk = 0;                 

    char command[10];
    char param[20];
    char header[30];
    int snum=0, dnum=0, inum=0, lnum=0, cnum=0;

  void setup()
  {   
    Serial.begin(9600);
    Serial.println("Ready");
  }

  int readline(int readch, char *buffer, int len) { //  If a CR terminated ascii string is received, returns string length, else returns -1
    static int pos = 0;
    int rpos;
    if (readch >0) {                               // if a valid ascii character
      msgpos++; 
      if (msgpos == 1) resetBuffer();              // First chr, so reset everything to make a new start
      if (!(pos < len-1)) Serial.println("Error: Msg too big for buffer");
      else if ((readch !=10) && (readch != 13)) {
        buffer[pos++] = readch;
        modchk += readch; 
        modchk %= 100;                  
    //  Serial.print("modsum%='"); Serial.print(modchk); Serial.println("'"); 
        buffer[pos] = 0;
      }  // end if (pos < len-1) 
      switch(readch) {      
      case 10: break;                                                                             // Ignore new-lines
      case 39: if (quotes==0) quotes=1; else if (quotes==1) quotes=0; else addChr(readch); break; // Turn on or off single-quotes
      case 34: if (quotes==0) quotes=2; else if (quotes==2) quotes=0; else addChr(readch); break; // Turn on or off double-quotes
      case 33: if (quotes==0) parseMode=-1; else addChr(readch); break;                           // ! Header info
      case 32: if ((quotes>0) || (parseMode==-1)) addChr(readch); else parseModeBump(); break;    // space delimiter
      case 44: if ((quotes>0) || (parseMode==-1)) addChr(readch); else parseModeBump(); break;    // comma delim 
      case 13:                                                                                    // CR stop char
        if (parseMode==0) {Serial.println("Error: blank or corrupt message"); pos=0;  return 0; }
        else if(quotes>0) {Serial.println("Error: open quotes not closed"); pos=0;  return 0; }
        else {// new msg recvd ready for processing. TODO: check if msglength and/or checksum sent, if so and if correct, return ack to sender
          rpos = pos;  
          pos = 0;    // Reset position index ready for next time
          return rpos;
        }
        break;     
      default:
        addChr(readch); 
        break;     
      }  // end case
    } // end if (readch >0);
    return -1;        // No end of line CR yet, so return -1.
  }

  void resetBuffer() {
    modchk=0; quotes=0; parseMode=0; command[0]='\0'; param[0]='\0'; header[0]='\0';
    snum=0; dnum=0; inum=0; lnum=0; cnum=0;
  }

  bool parseModeBump() {
    switch (parseMode) {
    case  0: parseMode=1; break;
    case  1: parseMode=2; break;
    case  2: parseMode=-1; break;
    case -1: Serial.println("Error: too many delimited parameters"); return false;
    }
    return true;   
  }

  bool addChr(char readch)  {                  //
    char addch[2];
    bool ret=true;
    addch[0]=readch; addch[1]='\0'; 
    if (parseMode==0) parseMode=1;
    switch (parseMode) {
    case  1: strcat(command,addch); break;
    case  2: strcat(param,addch); break;
    case -1: strcat(header,addch); break;
    default: 
      Serial.println("addChr Error: too many delimited parameters"); ret = false;
    }
    return ret;  
  }    

  void show() {
    Serial.print("SHOW: Command='"); Serial.print(command); Serial.print("'"); 
    Serial.print(" Parameter='"); Serial.print(param); Serial.print("' "); 
    Serial.print(" Header='"); Serial.print(header); Serial.println("'"); Serial.println();
    Serial.print("Msg Length='"); Serial.print(msglength); Serial.println("'");
    if (snum) { Serial.print("snum="); Serial.println(snum); }
    if (dnum) { Serial.print("dnum="); Serial.println(dnum); }
    if (inum) { Serial.print("inum="); Serial.println(inum); }
    if (lnum) { Serial.print("lnum="); Serial.println(lnum); }
    if (cnum) { Serial.print("cnum="); Serial.println(cnum); }
  }

  /*
  bool padzeros() {  // Flawed - redo from scratch using 2 for loop passes
    char lengthstr[4]="7";  
    Serial.print("lengthstr='"); Serial.print(lengthstr); Serial.println("'");
    int maxsize=sizeof(lengthstr);  // size of lengthstr array
    int steps=strlen(leng)+1;       // number of chars to be moved
    int hop=maxsize-steps;          // number of steps to be jumped over each move
    if((steps>1)&&(maxsize>steps)) {  
      for (int pos=steps-1; pos >= 0; pos--) {
        lengthstr[pos+hop] = lengthstr[pos];
        lengthstr[pos]='0';
      }
    }
    Serial.print("lengthstr='"); Serial.print(lengthstr); Serial.println("'");
  }
  */

  unsigned int calcmodsum(char* s){
    unsigned int i = 0;
    int m=0;
    byte b;
    do { b = s[i]; m += s[i]; i++; } 
    while (b != 0); //keep going until null terminator found at end of string
    m %= 100;                   //
    return m;    
  }

  int getheader() {      // extract values from header
  //  char header[] = "s=123 d=789 c=463";
    ack=false; echo=false;
    char * ptr = header; // header string pointer
    char * eq = NULL;    // search char (=) 
    int * num = NULL;    // empty at start
    int c = 0;
    while (1){
      eq = strchr(ptr, '=');
      ptr = eq;          // update the pointer
      if (ptr == NULL)  break;  // found no '=' chars 
      switch (*(ptr - 1)){ 
        case 'd': num = &dnum; break;
        case 's': num = &snum; break;
        case 'i': num = &inum; break;
        case 'l': num = &lnum; break;
        case 'c': num = &cnum; c=3; break;
        default:  num = NULL;
      }
      ptr++;
      if (num == NULL) //unrecognized var
        continue;      // locate next = char
      *num = 0;
      while (*ptr && (*ptr != ' ') && (*ptr != ',') && isdigit(*ptr)) {  // while valid digit and end of string not yet reached
        *num *= 10;    // extract each int
        *num += *ptr - '0';
        c++;
        ptr++;
      }
    }                  // Now Process any extracted header values      
    if (lnum) {
      Serial.print("Msg length="); Serial.print(strlen(buffer)); Serial.println();
      Serial.print("lnum="); Serial.print(lnum); Serial.println();
      if (lnum==strlen(buffer)) Serial.println("* Msg length OK"); else Serial.println("* Incorrect msg length");  
    }
    if (cnum) {
      Serial.print("calcmodsum='"); Serial.println(calcmodsum(buffer));
      Serial.print("cnum='"); Serial.print(cnum); Serial.println("'");
      Serial.print("count='"); Serial.print(c); Serial.println("'");
      if (modchk==cnum) Serial.println("* Checksum OK"); else Serial.println("* Incorrect checksum");  
    }
  }

  void loop() { 
    //  Check for new received serial string
    msglength = readline(Serial.read(), buffer, 64);  // Returns length of new ascii string, or -1 while receiving, or 0 if received msg is corrupt
    if(msglength==0  ) { Serial.println("ERROR: received serial message was invalid and discarded");show(); resetBuffer();} 
    else if(msglength > 0) {  // New msg to deal with
      if (header) getheader();
      show();

      msgpos=0;  // reset ready for new msg after finished with last msg
    }   
  } 

Once I can finalise a satisfactory c=nnn validity test, I shall tidy up this working 'receiver' part and move on to doing the 'sender' part, which I'm hoping should be quite trival in comparison.

I suppose it would be 'proper' to assemble the outgoing message into something like a char outmsg[64] string, but I'm trying to keep resources minimal to maximise the resources available for the arduino's 'real' functionality, so perhaps it may be better just to send the outgoing msg as consecutive serial chunks, ie:-

Serial.print(Command); Serial.print(Params); Serial.println(Header);

The header will be built up as required. If s=nnn (source) was received it will be assumed that an acknowledgement response is required by the source to prevent it repeating failed transmissions, so the received s=nnn would be used as the outgoing d=nnn destination to respond to - outgoing msgs need only include a source s=nnn if they requires a response back. The i=nnn (msg id) isn't yet used, but could eventually be for re-assembling individual chunks of larger msgs, or keeping track of msg order if multiple msgs are expected.

The l=nnn is a zero-padded count of all transmitted characters (excluding the terminating ), so can be anywhere within the Header string.

It would be preferable if the zero-padded c=nnn checksum could also be anywhere in the Header, but this will require a checksum() calculator function to be able to apply its calculation to all Header characters except the nnn data characters of c=nnn.

As things are, I can jump straight to "c=" wearing blinkers, but that offers no relative position info, so leaves me oblivious of everthing before it. And I'd prefer not having to split Header[] into seperate Left[] and Right[] chunks which would unnecessarily wastes at least [30+30] chars of memory.

So I think I must sequentially step through all Header characters looking for "=" while applying the mod sum calc to each, except for the 3 following chars when the preceeding char is "c", but I can't quite get my head around how to do that yet.

If the C string functions could return relative string positions instead of absolute * pointers it could all be so much more straightforward.

Does anyone know of any alternative string-relative functions that might be available?

• NEW INFO 2 *

Most "Hello world" examples (eg: IRlib) demonstrate functionality using the IDE serial monitor for user interaction. Whatever the functionality, they can all be considered as just a black box capable of recognising a few incoming serial plain-text commands and actioning as appropriate, possibly by also issuing serial text command responses of their own.

The aim of my project is for a user to easily turn that existing and already-proven serial user interaction into an automatic arduino to arduino serial interaction if wished. The headers are just an optional bolt-on for arduino to arduino reliability, but during dev and debugging the various optional header contents will still need to be read and written by users with the IDE serial monitor, which only allows ascii text interaction.

Although this sort of system would not be the preferred solution for experts, it offers much advantage for non-experts. Un-skilled hackers could create effective working solutions without needing to aquire any rocket science first, and without needing to worry too much about resource limitations, or the problems and complexity of trying to consolidate incompatable or conflicting functionality onto any single device.

If they have an arduino that can serially interact, they can use it as an interactive node. If two can interact over serial, then more can serially interact using RS485, or using serial Ethernet UDP slaves (ESP8266's are small and cheap). Only nodes that 'recognise' an incoming command would respond to it.

A Triggers node could send ascii Alerts such as 'Mailbox Opened', 'Gate1 Visitor', Shed Door Opened' etc. A Responder node could parse such incoming Alerts to match against entries in a config file, then parse the corresponding config entry for appropriate Response commands to be issued to other cluster nodes, such as 'Mailbox: Announce Mailbox Delivery', or 'Zone2: Announce Zone 2 Activity, CCTV: Select Zone2'.

This would allow new node functionality to be added easily just by updating the Responders appropriate config file entry to include any additional Response commands that become available from any new nodes. So adding an X10 node for instance might offer a new Response command such as 'X10, A12 ON', which only the newly added X10 node would be looking for and respond to.

A cluster system could be evolved into whatever sophisticated functionality was required, with minimal software skills and knowledge. Commands and parameters could be any plain-text names that the user prefers, which when matched during input parsing would branch to any required local function. Despite any resulting complexity, all individual nodes could still just be considered as relatively simple black boxes capable of recognising a few incoming serial plain-text trigger commands and replying with plain-text responses.

Cluster nodes aren't even limited to being arduinos, because they could include ANY devices capable of receiving and responding to simple ascii command strings. This offers easy gradual progression to other platforms if wished without needing to abandon all existing efforts and start over again from scratch.

I knew I was biting off more than I could chew, but each time I play devils advocate trying to avoid wasting my time going up a dead-end, the more benefits become apparent to make the effort seem worthwhile. The forums are littered with questions and libraries aiming in a similar direction, but I haven't yet found anything which offers that simple black-box plain-text interaction capability, hence my continued struggle. I'm not a programmer so I'll never be able to offer an efficient library solution, but I must try to keep going until I've got something that is usable, and if others find it useful then maybe eventually it might offer incentive for someone to wish to optimise and improve on it.

Hopefully the following should do what I want as far as ignoring "c=nnn" from the mod checksum wherever it finds it.

unsigned int calcmodsum(char* passedstr){     // calculates the mod sum of passedstr excluding the 3 "c=nnn" value digits
  int pos = 0;                         
  int modsum=0;
  byte b;
  do { 
    b = passedstr[pos]; 
    if ((passedstr[pos]=='c')&&(passedstr[pos+1]=='=')&&(pos>2)) pos=pos+4; else modsum += passedstr[pos]; // skip next 3 digits after finding c=
    pos++; 
  } 
  while (b != 0); // keep going until null terminator found at end of string
  modsum %= 100;         
  return modsum;    
}

Answer 1

Hmm... the length of your question makes it difficult to focus an answer. I'll try by stating my interpretation, then I'll provide a longer answer ;-)

Is there a way to wrap a text message between two Arduinos
  (to increase reliability), but would still
    allow me to do testing without the wrapper?

If I assume that's what you're asking, then the above comments have touched on various aspects of an answer: "Yes."

Let's progress through my interpretation:

• "Text" means ASCII values between 32 and 126 (SP and '~', the printable characters).
• "Wrapping a message" means sending some bytes before and possibly after the text.
• "Increasing reliability" means detecting and/or correcting characters that get corrupted.
• "Testing without the wrapper" means that you would like to use the Serial Monitor to send example commands while you're in a noise-free environment, testing the logic of the command execution, not the mechanism of communicating reliably.

Wrapping

You're on the right path by prefixing the text with a binary header, and Nick's library shows how to use a "special" start and end byte to delimit the messages. (This is also called message "framing".) If the START byte can appear inside the message elsewhere, you will need to either (1) "escape" the START byte when it appears inside the message, (2) provide a second method of verifying that the message is framed correctly (e.g., an END byte and/or a length), or (3) decrease the odds of a false start by using a multiple-byte START sequence.

You appear to have chosen (2). You are free to chose the bytes that follow the START byte. Addressing, sequence number and message type are common. Of course, the message length is in there somewhere.

Reliability.

If you're happy to simply detect an error at the receiving end, a checksum is sufficient. I'd like to suggest the Fletcher checksum, as it is simple, quick and fairly robust. CRCs are more robust, but a little more expensive to compute.

But because they are calculated over the rest of the message (the "payload"), they almost always appear at the end, before any END sequence. This also makes it easier for the receiver, because it can accumulate the checksum as the bytes are received, and reject the message if the received checksum does not match the calculated checksum.

You can also use error detection to help with acknowledgements, if you decide to implement them. An alternative to ACK/NAK messages is to request the current state (a poll command), expecting that it should reflect the previous (unacknowledged) command. If the state doesn't match, send the command again.

If you also want to perform error correction, there are numerous ways to add that. All of them will increase the size of the message, as well as the CPU time. (Sorry, Nick, but sending complemented nybbles is not a very robust detection scheme, and it doubles the message size without adding correction capability.)

Testing without the wrapper

You could use CR/LF as your message framing and assume that anything after CR/LF is the start of an unwrapped message. If it starts unwrapped, set a bool flag that skips the checksum part and END byte.

Final notes

Your current program is structured to hold the complete message in RAM for processing (sent or received). This is not always feasible in the Arduino environment. You don't put all the print chars in a buffer, then do one big Serial.print( buffer ), do you?

Just like breaking a Serial.print into several smaller prints, reception can be performed the same way. You can watch for keywords, or accumulate parameter values without buffering the entire message. While it saves a lot of RAM, it also saves CPU time because each received character is handled once. With a buffered approach, each character is handled multiple times: read() into a buffer, then compared during a search, and maybe parsed.

One of my recent posts has some general structure information, snipped a little for here:

Reading data on the Arduino is different from most PC applications. Calling Serial.read() does not always return data. Well, it returns -1 (or 0xFF) if no characters are available. In a PC app, calling read will usually wait until a data has been received: the PC app is "blocked" until a character comes in. That's not very noticeable because the PC has a nice multitasking OS so your pointer keeps moving and other windows respond, even though the one app is waiting for a character.

On the Arduino, you can make it block the same way, but there's no OS that will let you do something else while you're waiting for data (yes, there are multitasking solutions, but let's avoid that for now). So blocking on the Arduino can keep you from doing other things you may need to do, like blinking an LED or catching a button press.

There are a few common ways to write an Arduino sketch that can cope with this speed disparity:

1) Force the Arduino to wait for each character (i.e., block), and write the sketch in a very linear, top-to-bottom fashion:

while (Serial1.available() == 0)
  ; // wait
  inByte1 = Serial1.read();

  if (inByte1 == 0x3E){ // START byte
    buffer[0] = inByte1;

    while (Serial1.available() == 0)
      ; // wait
    inByte1 = Serial1.read();

    if (inByte1 == 0x03){

2) Save each character in a buffer as it arrives until you have received an entire packet. Then parse the packet all at once with similar linear code:

static uint8_t count = 0; // keeps its value across calls to loop

if (Serial1.available()) {
  inByte1 = Serial1.read();

  if (inByte1 == 0x3E) { // START character

    // Next packet starting, process the current one we've been saving,
    //   if we have saved enough bytes (skips small/bad packets).

    if ((count > 6) && (buffer[0] == 0x3E)) {

      // SNIP: handle previous packet

    }

    // Reset the count to start a new packet with the 0x3E we just received.
    count = 0;
  }

  // Save the current byte
  if (count < sizeof(buffer))
    buffer[ count ] = inByte1;
  count++;
}

3) Parse each character as it arrives, picking up where you left off after doing other things.

void loop() { // statics keep values across calls to loop static uint8_t count = 0; static bool skipping = true; static uint8_t messageLength = 0;

  if (Serial1.available()) { // or 'while'

      uint8_t inByte = Serial1.read();

      if (inByte == 0x3E) { // START  byte

          // Starting a new packet!

          buffer[0] = inByte;
          count    = 1;
          skipping = false;

      } else if (!skipping) {

          // Saving a packet, store the current byte

          if (count < sizeof(buffer))
            buffer[ count ] = inByte;

          // Use the current byte, depending on 'count'
          switch (count) {
            case 1:
              if (inByte != 0x03)
                skipping = true; // ignore other packet selector values
              break;

            case 2:
              messageLength = inByte;
              // Error check?
              if (messageLength != 40)
                skipping = true;
              break;

            case 3:
              // packetID = inByte; ?
              buffer[3] = inByte;
              break;

            case SNIPPED: other header information fields
            case SNIPPED: accumulate/parse payload
            case SNIPPED: verify checksum (ignore packet or execute packet)
         }

         count++;
     }
 }

 // Do other things, too?

}

Choice 1 is very linear. The disadvantage is that it blocks and does not recover from data errors very well.

Choice 2 is pretty easy to understand and it doesn't block; other tasks can do things while waiting for a complete packet. The disadvantage is that all the packet processing happens when the last byte is received (actually, the first byte of the next packet in that code).

Choice 3 is more difficult to understand, because the entire section is executed for each character, either starting a new packet or using the switch statement to handle each char. Different cases do a little work, depending on the count. The advantage is that the processing is spread out over each character, and you can decide to skip the rest of a packet at any point.

This is actually a Finite-State machine approach, and Nick Gammon has a great description, too.