Arduino Mega Serial2 seems to stop functioning mid-sketch

Question

thanks in advance for the advice.

I've got an Arduino Mega talking to a Cavro XL-3000 pump. After a lot of time I got the arduino successfully controlling the Cavro. However, the big problem I have now is after my Mega sends about a dozen commands, communication breaks down between the two. The Arduino sends commands and gets no response back from the Cavro. I've written code that, when the Arduino gets no response, it tries twice more before terminating the program.

For a while I thought the problem to be pump related. After all, my test code (the vast majority of my code is in a library I created to have the Mega talk to the Cavro pump) simply tells it to move the Cavro between two different positions interminably. But when I reset my arduino, it starts working again. Not when I reset the pump.

Finally, I hooked my logic analyzer up. It turns out that, when I get no response from the pump, it's because no command is reaching the Cavro. I am definitely successfully creating a command packet and sending it using Serial2.write(packet,packet_length). And though the Arduino is calling these lines of code and thinks data is going out, nothing is actually going out on the TX line. It's like Serial2 gets tired after about a dozen calls and stops shifting out data even though I'm calling Serial2.write.

I'm pretty stumped. I'm clearing the receive buffer after each response, so no overflow there. Each time I send a command I wait until it is fully shifted out before moving on, so no overflow on the write buffer either. If I get no response back from the pump I try two more times. On the last try, I shut down Serial2, begin it again, and wait until it is connected (while(!=Serial2)). But even after I do all this, Serial2.write isn't actually sending out data. My logic analyzer isn't picking anything up.

A simple reset of the Arduino and commands are actually going out again. Logic analyzer picks them up immediately. After about a dozen or so commands (it seems fairly random but never lasts very long), the same thing happens. I swapped out Arduino megas and the same thing occured.

So.

1. It seems to be a code issue (repeatable on multiple arduino megas)
2. It does not seem to concern the pump or the rs232 converter
3. A push of the reset button fixes it temporarily
4. My logic analyzer snoops in between the arduino and rs232 converter.
5. Serial2.write(packet, packet-length) is definitely being called.
6. I have no idea what's happening.

Here is a schematic. Any advice would be appreciated!

Here is my test code:

#include <Hydro.h>


Hydro pump;
void setup() {
  Serial.begin(9600);
  Serial2.begin(9600);
  pump.InitComm(0,true,"right");
  Serial.println("Communication initialized");
  delay(5000);
}

void loop() {
//  Serial2.begin(9600);
  //long mil = millis() - 10000;
//while(true){
  //if(millis() > mil + 10000){
    //Serial.println("Sending command...");

    //byte datastuff[10]={0x02, 0x31, 0x31, 0x41, 0x35, 0x30, 0x30, 0x52, 0x03, 0x27};
    //Serial2.write(datastuff,10);
    //Serial.println("Sending command...ONE");
    pump.AbsoluteMove(0,4000);
    delay(10000);
    //Serial.println("Sending command...2");
    pump.AbsoluteMove(0,9000);
    delay(10000);
    //Serial2.write(new byte[10] {0x02, 0x31, 0x31, 0x41, 0x35, 0x30, 0x30, 0x52, 0x03, 0x27}, HEX);
    //mil = millis();
}

Here is my library.

/*
Hydro.c - Library created for OEM communication with Cavro XL-3000 Pump.

11/23/2016
Written for Mega using Serial2, can we tweaked.
*/

#include "Arduino.h"
#include "Hydro.h"

Hydro::Hydro(void){}

const byte LineSynch = 0xFF;
const byte StartByte = 0x02;
byte seq = 0x31; //confirm this
const byte ETX = 0x03;
const byte EX = 0x52;
int length_total = 0;


//Initializes communication and then sets it to Fine (12000 steps) Control.
void Hydro::InitComm(int addr, boolean fc, String initdir){
    Serial2.begin(9600);
    Initialize(addr,initdir);
    delay(2000);
    FineControl(addr,fc);
}

//Top Level Send Command. First Queries the pump to make sure it's ready for a command.
//If Pump is ready, passes along actual command to MidCommand.
void Hydro::SendCommand(int addr, String command){
    byte status = Query(0);
    while(status==false){
        Serial.println("Query: Stat Error: Wait till Pump No Longer Busy.");
        delay(10000);
        status = Query(0);
    }
    Serial.println("Query: Pump Not Busy.");
    MidCommand(addr, command);
    ReadData(false);

}

//Executes the actual command by calling command.
//If no response received (command != true), tries again twice.
void Hydro::MidCommand(int addr, String command){
    bool response = Command(addr, command);
    if (response == true){}
    else if (response == false){
        //resend
        Serial.println("Command Failed. Resending.");
        bool response2 = Command(addr, command);
        if (response2 == false){
            //shut off
            Serial.println("Second Command Failed.");
            Serial2.end();
            delay(2000);
            Serial2.begin(9600);
            while (!Serial2){
                Serial.println("Waiting for S2 to Connect.");
                delay(1000);
            }
            Serial.println("Connected.");

            bool response3 = Command(addr, command);
            if (response3 == false){
                Serial.println("Final Command Failed.");
                while(true){}
            }   
        }
    }
}

//Send a Command.
//Returns True if a response is received from the pump.
bool Hydro::Command(int addr, String command){
    command.concat("R");
    byte address = 0x31+ addr;
    int length = command.length()+1;
    byte dat[length];

    Serial.print("Command: The string command is: ");
    Serial.println(command);
    command.getBytes(dat,length);

    //Serial.println("Data----");
    //for (int i = 0; i < length; i++){
        //Serial.print("i equals: ");
        //Serial.println(i);
        //Serial.println(dat[i], HEX);
    //}
    //Serial.println("----");
    //Serial.print("Dat Length is: ");
    //Serial.println(length);

    //line sync, stx, add, seq, etc, checksum. 6. 6 + (length.data) == length.
    //Decided to ignore Line Sync. Pump seems to have no problem without it. So 5+length.data.
    length_total=5+(length-1);
    byte packet[length_total];
    int p = 0;

    //packet[p++] = LineSynch;
    packet[p++] = StartByte;
    packet[p++] = address;
    packet[p++] = seq;

    for (int i = 0; i<(length-1); i++){
        //Serial.print("i equals: ");
        //Serial.print(i);
        packet[p++] = dat[i];
        //Serial.print(".  Inserting Data: ");
        //Serial.println(dat[i], HEX);  
    }

    packet[p++] = ETX;
    byte checksum = StartByte;

    for (int i = 1; i<length_total-1; i++){
        checksum = checksum^packet[i];
    }
    packet[p++] = checksum;
    int x = Serial2.write(packet,length_total);
    Serial.print("bytes written: ");
    Serial.println(x);
    seq++;

    //print out packet to verify.
    Serial.println("Packet----");
    for (int i = 0; i < length_total; i++){
        Serial.println(packet[i], HEX);
    }
    Serial.println("----");

    //Waits for packet to be fully shifted out.
    //Serial.flush could be used instead.
    while(Serial2.availableForWrite() != 63){
        Serial.println("Command: Waiting to finish sending command...");
        delay(1000);    
    }
    Serial.println("Command: Finished Sending.");

    unsigned long time = millis();

    //looks for a response from the pump.
    //returns true if received.
    while(Serial2.available()==0){
        Serial.println("Waiting for Response...");
        delay(1000);
        if (millis() > (time+5000)){
            Serial.println("Resend Command.");
            return false;
            }
    }
    Serial.println("Command: Response Received.");
  return true;
}

//Both initialize at half force.
//Init with volve output to Right or Left.
void Hydro::Initialize(int addr, String dir){
    if (dir == "right"){SendCommand(addr,"Z1");}
    else if (dir == "left"){SendCommand(addr,"Y1");}
}

//If Fine Control is True (active), Resolution is 0-12000.
//Else If Fine Control is False (deactivate), 0-3000.
void Hydro::FineControl(int addr, boolean control){
    if (control==true){SendCommand(addr,"N1");}
    else if (control==false){SendCommand(addr,"N0");}
}

//See Fine Control for Integer Values
void Hydro::AbsoluteMove(int addr, int move){
    String mc = "A";
    mc.concat(move);
    SendCommand(addr,mc);
}

void Hydro::Execute(int addr){
    SendCommand(addr,"R");
}

//Sends a Query for Status.
//Returns true if Status byte is 60, ready for command.
bool Hydro::Query(int addr){
    MidCommand(addr,"Q");
    if ((ReadData(true)) != 0x60)return false;
    else return true;   



}

//True to print out data, false to not.
//Grabs and returns the Status Byte.
byte Hydro::ReadData(bool print){
    byte status;
    int i=0;
    if (print == false){
        while(Serial2.available()){
            if (i==3){
                status = Serial2.read();
            }
            else Serial2.read();
            i++;
        }
    }

    else if (print == true){
        while(Serial2.available()){
            if (i==3){
                status = Serial2.read();
                Serial.print("i is: ");
                Serial.print(i);
                Serial.print(". Reading (s): ");
                Serial.println(status, HEX);
            }
            else {
                Serial.print("i is: ");
                Serial.print(i);
                Serial.print(". Reading: ");
                Serial.println(Serial2.read(), HEX);
            }

            i++;

        }

    }
    return status;  
}

Answer 1

I'm not a C++ guy, but I have looked at the code and fail to find where some of the index's are set to the proper values (whatever that is.) Some of it may occur during instantiation, but... Most of these I've seen are index related (out of bounds), so.. As an example, the code sequence of

for (int i = 0; i < (length - 1); i++)
    packet[p++] = dat[i];

Seems to me, if the bytes (length) of 'dat' is 6 assuming 6 bytes, then the loop will run from 0 to 4. That would seem to lose 2 bytes? Some of the array indexes are difficult to track for their value, not knowing what you were thinking.

My suggestion is

1.) ensure indexes stay within bounds
2.) ensure every function call (related to problem) is checked for it's expected return status
3.) attempt to ascertain the number of operations that occur before failure. This may give you a clue about indexes or lead to discover of other problems.

I don't think you have a hardware problem. Intermittent hardware doesn't do this either. This is a software bug, and it's sounds repeatable. Step three may be useful.

Best of luck :)

Answer 2

Difficult to say, but if your last trace is "Final Command Failed." , you are probably stuck in the following infinite loop.

if (response3 == false){
    Serial.println("Final Command Failed.");
    while(true){}
}

as @Jack says, you might as well have problems with your bound arrays. You have a lot of "length+1, length-1" scattered everywhere, this is really hard to follow.