What is the best (fastest and most robust) way to send messages back and forth between Python on a PC and an Arduino, over serial?

Question

I am trying to communicate between a PC running Python using PySerial and an Arduino. The Arduino itself has a CAN shield, and is responsible for interfacing with a motor. My goal is for the PC to construct the desired CAN frame (8 bytes), send it over Serial to the Arduino, which then itself uses a CAN library to speak to the motor. The motor then send a return frame of the same size, which the Arduino copies to an array, and also prints it to serial.

This works in some cases, although I'm struggling to make it robust. As an example, I am driving the motor in torque control mode (Command 37 in this datasheet). For an arbitrary forward torque of '50', the sending frame should be:

[161, 0, 0, 0, 50, 0, 0, 0]

And for a command of '-50', the frame should be:

[161, 0, 0, 0, 206, 255, 255, 255]

I didn't appreciate this would be an issue, but my guess is that because this actually requires more characters, it takes longer to send. Positive torque commands are sent with no issues, while the negative ones cause the motor to skip or judder. I can see that the Arduino isn't actually receiving the correct command at times, see below (left is sent frame from Python, and right is what the Arduino receives printed straight back).

I have tried:

1. Flushing serial buffers before and after sending messages, on both devices
2. Adding delays to try and slow the communication down
3. Changing baudrates
4. Trying to use Serial.write() instead of Serial.print() as I assumed it might be faster (?)

I have attached my code below, but in general, what is the best way to communicate over serial between Python and Arduino, robustly and quickly? Even if my current application is ignored, what might ideal scripts for each look like?

Python:

    def serial_begin(self, baud, com):
        self.ser = serial.Serial(com, baud)
        self.ser.flushInput()
        self.ser.flushOutput()
        print("Connected to Serial Port " + com)
        t.sleep(1)

    def send_cam_frame(self, frame):

        self.send_frame = frame
        string_to_send = "<" + str(int(frame[0])) + "," + \
                         str(int(frame[1])) + "," + \
                         str(int(frame[2])) + "," + \
                         str(int(frame[3])) + "," + \
                         str(int(frame[4])) + "," + \
                         str(int(frame[5])) + "," + \
                         str(int(frame[6])) + "," + \
                         str(int(frame[7])) + ">"

        self.ser.write(string_to_send.encode('UTF-8'))
        self.receive_can_frame()

    def receive_can_frame(self):
        get_data = self.ser.readline().decode('UTF-8', errors='ignore')[0:][:-2]
        self.receive_frame = np.fromstring(get_data, dtype='int', count=8, sep=' ')

    def set_torque(self, torque):

        self.command = self.command_list["SET_TORQUE"]

        torque = int(self.constrain(torque * 2000 / 32, -2000, 2000))
        frame = [self.command, 0, 0, 0, torque & 0xFF, (torque >> 8) & 0xFF, (torque >> 16) & 0xFF,
                 (torque >> 24) & 0xFF]
        self.send_cam_frame(frame)

def main():

    arduino_port = "COM5"  # Default COM port
    baud_rate = 57600  # Default Baud Rate

    rmd = Motor()
    rmd.serial_begin(baud=baud_rate, com=arduino_port)

    set_torque = 0

    while True:
        try:

            if keyboard.is_pressed('w'):
                set_torque += 0.001
            if keyboard.is_pressed('s'):
                set_torque -= 0.001

            rmd.set_torque(set_torque)

        except KeyboardInterrupt:
            rmd.disable_motor()
            print("Keyboard Interrupt")

Arduino:

byte recvFrame[8] = {0, 0, 0, 0, 0, 0, 0, 0};
byte sendFrame[8] = {0, 0, 0, 0, 0, 0, 0, 0};

// Serial Read Parameters
const byte numChars = 32;  // Length of received char array from Serial
char receivedChars[numChars];  // Received char array from Serial
char tempChars[numChars];        // Temporary array for use when parsing
bool newData = false;  // Flag to check if newData has been received


void setup() {
  Serial.begin(57600);
  delay(1000);
  
}

void loop() {

  CANMessage frame;

  frame.id = 0x140 + 1;
  frame.len = 8;

  // Do not touch this section
  recvWithStartEndMarkers();  // Check Serial and receive data if there is newData
  if (newData == true) {
      strcpy(tempChars, receivedChars);  // Copy variables to prevent them being altered
      parseData();  // Parse data (split where there are commas)
      
      for (int i = 0; i  < 8; i++){
        frame.data[i] = sendFrame[i];
      }

      can.tryToSend(frame);

      newData = false;  // Set to false
  }
  
  if (can.available()){
    can.receive(frame);
  }
  
  for (int i = 0; i < 8; i++){
    recvFrame[i] = frame.data[i];
  }

  printFrame();

}


// Do not touch this function
void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

// Only touch this function if more data is being sent from Python code
void parseData() {      // split the data into its parts

    char * strtokIndx; // this is used by strtok() as an index

    strtokIndx = strtok(tempChars,",");
    sendFrame[0] = atoi(strtokIndx);

    strtokIndx = strtok(NULL,",");
    sendFrame[1] = atoi(strtokIndx);

    strtokIndx = strtok(NULL,",");
    sendFrame[2] = atoi(strtokIndx);

    strtokIndx = strtok(NULL,",");
    sendFrame[3] = atoi(strtokIndx);

    strtokIndx = strtok(NULL,",");
    sendFrame[4] = atoi(strtokIndx);

    strtokIndx = strtok(NULL,",");
    sendFrame[5] = atoi(strtokIndx);

    strtokIndx = strtok(NULL,",");
    sendFrame[6] = atoi(strtokIndx);
 
    strtokIndx = strtok(NULL, ",");
    sendFrame[7] = atoi(strtokIndx);


    // How to add a new variable
    // Currently, data is sent as <0, 0, 0, 0>
    // If a fourth parameter was to be sent (<0, 0, 0, 0, 1>), the following lines need to be added

    // strtokIndx = strtok(NULL, ", "); This reads the string, from where it was previously cut, up until the next comma
    // newVariableName = atoi(strtokIndx); atoi is 'to integer'. If newVariable is a float, atof is needed etc


}


void printFrame(){

  for (int i = 0; i < 8; i++){
    Serial.print(sendFrame[i]);
    Serial.print(" ");
  }
  Serial.println();
  //Serial.write(sendFrame, 8);
  //delay(3);
  
}

N.B: I have removed some of the unnecessary functions from both scripts (e.g: the rest of the Motor class for the Python, and the CAN Shield setup from the Arduino) for clarity.

Answer 1

There is no best way, but there is a better way (and more efficient way).

For an arbitrary forward torque of '50', the sending frame should be:

[161, 0, 0, 0, 50, 0, 0, 0]

And for a command of '-50', the frame should be:

[161, 0, 0, 0, 206, 255, 255, 255]

Most of the programmers who are new to embedded program, prepare their data to be send via communication as string instead of sending raw binary data, the problem is that

1. to send a value of 215, you are sending 3 bytes of ASCII('2', '1', '5') while the data can actually sent by one single byte.
2. both integer 256 (0x0100) and 32766 (0x7FFE) only took up two bytes, but when you encode it as ASCII, you are dealing with various length from 3 chars to 5 chars. This not only take more bytes to send the data but also make parsing of the data more difficult unless you had some delimiter to separate the data, e.g. ("256,32766" separated by comma).

Looking at your data set and datasheet, it is clear that the each data point is represented by a 4-byte signed data with small endian (i.e. the lower byte get send first), so decimal 50 is 0x00000032, it is store in memory as 0x32, 0x00, 0x00, 0x00 with small endian.

A negative value is represented in two's compliment of the positive value, so -50 in decimal is 0xFFFFFFCE and stored as 0xCE, 0xFF, 0xFF, 0xFF.

In python, the struct library allows you to converts between Python values and C structs represented as Python bytes objects.

form struct import *

torque = -50

data = pack('<i', torque) # pack torque as a 4-byte integer with little endian
ser.write(torque)         # send over serial as bytes

On the Arduino side, read the data from Serial as byte(i.e. uint8_t) and store it in an array. You can then convert it back to a int32.

// assuming you had read these 4 bytes of data from Serial and add them into an array
uint8_t data[4]{0xCE, 0xFF, 0xFF, 0xFF};

// Convert received data bytes to int32_t
int32_t torque = static_cast<int32_t> (data[3]<< 24 | data[2] << 16 | data[1] << 8 | data[0]);

// This will return result of -50
Serial.println(torque);

This should reduce the code and improve the efficiency of the sending the data significantly.

The code along however does not improve the reliability, one advantage of sending data as string is that you could determine the end of data stream by detecting the \0 terminator at end of the string. When sending raw binary, ideally you need to have the mechanism to signify the number of bytes the data of bytes that you are going to send over.