Serial Comm. timing issue between Arduino and Pyserial

Question

This is my first post to Stack Exchange so please forgive any formatting errors.

I am conducting an experiment that has 7 total sensors: 5 thermocouples, 2 humidity sensors and 1 pressure sensor. I am reading their signals via Arduino Mega analog input ports. I chose to use Python to save and compile the data in a nice csv file, therefore I am using Pyserial.

My problem is that I cannot go above 1 sample/second. As you can see in my code, I have a 1 second delay in Arduino and Python. I have ran successful calibration tests with my thermocouples, so I know at that rate Python and Arduino are communicating well. Anything faster than 1 sample/second (i.e. I change my delays to .5 or .25 seconds), a timing error occurs and my data is "cut up" and sent in odd chunks instead of the standard layout I programmed.

Additionally, I tried to increase the baud rate from 9600 and anything above that will send "garbage" ASCII characters (e.g. little blank blocks and odd characters) to the Serial Comm. I changed the baud rate in Arduino, Python and the COM port in my devices manager settings to no avail.

I would really like to increase my sampling rate to increase my signal-to-noise ratio and I believe it's good practice as a metrologist to have a higher sampling rate than 1 samp/sec. I am sending ~1072 bits each time I write to the Serial Comm (if I'm counting right??) so I should have enough room to increase the amount of bits to send per second (around 3 times what I am currently sending).

In summary, my two main issues are: if I increase the baud rate, I get garbage in my Serial Comm and if I try to increase my "sampling rate" by decreasing my delays, timing issues appear and my data is split up erroneously. Should I be using a different function, like Serial.flush()?

I greatly appreciate any feedback, thank you for your time!

// Data acquisition
// updated  March 30, 2017


//Note: This version is for TEMPERATURE, HUMIDITY and PRESSURE data         
acquisition ALONE. No motor/conveyor control.

//***GLOBAL VARIABLES***//

float lfactor = 125.0; 
float afactor = 1.15;

/////////////////////////

//Standard setup with analogReference to 5Volts. 

void setup() {

Serial.begin(9600);  
}

void loop() {


//******THERMOCOUPLES*******//

//get voltage reading
// [code]; TC # - Location

float tcv1 = analogRead(A4) * ( 5.0 / 1024.0 ); //TC 1 
float tcv2 = analogRead(A7) * ( 5.0 / 1024.0 ); //TC 2 
float tcv3 = analogRead(A8) * ( 5.0 / 1024.0 ); //TC 3 
float tcv4 = analogRead(A9) * ( 5.0 / 1024.0 ); //TC 4 
float tcv5 = analogRead(A10) * ( 5.0 / 1024.0 ); //TC 5

//convert to temperature using custom equation based on 5V reference
float tc1 = ( 188.7755 * tcv1 ) - 245.3959;  
float tc2 = ( 188.7755 * tcv2 ) - 245.3959;  
float tc3 = ( 188.7755 * tcv3 ) - 245.3959;  
float tc4 = ( 188.7755 * tcv4 ) - 245.3959;  
float tc5 = ( 188.7755 * tcv5 ) - 245.3959;    

//******HUMIDITY SENSORS********//

float hs1 = ((( analogRead(A13) * ( 5.0 / 1024.0) ) / 5.0 ) * 100.0 ); //HS 
1
float hs2 = ((( analogRead(A15) * ( 5.0 / 1024.0) ) / 5.0 ) * 100.0 ); //HS 
2

//******PRESSURE SENSORS********//

//obtain voltage signal from Sensirion sensor
float pvolt = analogRead(A0) * (4.995 / 1024.0); 

//convert to Pascals using equation from datasheet, including altitude 
compensation
//P = lfactor * (voltage[V] - .250 ) / 3.750

float pressure =  lfactor * ( pvolt - .250 ) / 3.750;  
float pascals = pressure * afactor; 

Serial.print("TC1");
Serial.print(":");
Serial.print(tc1);
Serial.print(";");

Serial.print("TC2");
Serial.print(":");
Serial.print(tc2);
Serial.print(";");

Serial.print("TC3");
Serial.print(":");
Serial.print(tc3);
Serial.print(";");

Serial.print("TC4");
Serial.print(":");
Serial.print(tc4);
Serial.print(";");

Serial.print("TC5");
Serial.print(":");
Serial.print(tc5);
Serial.print(";");    

Serial.print("RH 1");
Serial.print(":");
Serial.print(hs1);
Serial.print(";");  

Serial.print("RH 2");
Serial.print(":");
Serial.print(hs2);
Serial.print(";");

Serial.print("Pressure");
Serial.print(":");
Serial.print(pascals);
Serial.print(";");

Serial.println("");
delay(1000); 

  }

Python Code:

from threading import Thread
import time
import serial
import os 
global datalist
global fileName

global motorspeed

motorspeed = 0 

def serInitialization():
    #activate the serial port, if possible
    try:
        ser = serial.Serial('COM9', 9600) #initialize the serial port
        print "Serial connection successful." 
        return ser
    except:
        print "Error: serial port cannot be initialized"
        while(1):
            voidholder = 1

def getDateTime():
  """Function grabs current time and date, then returns values in a 2-
 element list. """
   timeNow = time.strftime("%H:%M:%S")
   dateToday = time.strftime("%m/%d/%y")
  return [dateToday, timeNow]

def writetocsv(data):

    """ function writes datalist values to a csv file. If daily csv file 
exists already, 
    list values are simply appended to end of file. If it does not, function 
creates the file, 
    then appends values. 
    """

    global csv_success

    header = ["date", "time", " ", "TC1", " ", "TC2", " ", 
"TC3", " ", "TC4", " ", "TC5", " ", "HS1", " ", "HS2"," ", 
"PRESSURE", " ", "PRESSURE(alt)", "\n"]

    fileName = str(time.strftime("%m_%d_%y_")+ "log.csv")

    if os.path.exists(fileName):
        f = open(fileName, "a")

    else:
        f = open(fileName, "a+")

        for element in header:
            f.write(element + ",")
        f.write("\n")

    for element in data:
        if type(element)==str:
            f.write(element + ",")
        if type(element) == list:
            for i in element:
                f.write(i + ",")

    f.write("\n")
    f.close()
    csv_success = True

def mainprogram():

    data = []


    if ser.inWaiting():
        datetimeData = getDateTime()

        for i in datetimeData:
            data.append(i)

        val = ser.readline().strip('\n\r').split(';')
        print "Current readings: "
        for i in range(0,len(val)):
            sensorData = val[i].split(':')
            data.append(sensorData)

            print sensorData

        writetocsv(data)

        time.sleep(1) 


        print "\n" * 50

#Initializes the serial port with the arduino so now we can read what 
Arduino is sending us!
print "*******************************"
print "DATA ACQUISITION PROGRAM       "
print "*******************************"
print "Data files are saved under "
print "C:\\Users\\lpaw\\Downloads\\python workspace"
print "*******************************"


ser = serInitialization()

while True:

   mainprogram()

Screenshots of errors:

Answer 1

Your main problem is in erroneously having a delay in both the sender and the receiver. While this sometimes works by lucky accident, it essentially never does what is hoped for by those who try it.

In a system such as this, you should have a delay in the sender only, in order to produce quasi-regular sample intervals, but no delay in the receiver. Rather, the receiver should accept data whenever the transmitter produces it, or more specifically in your case, every time it completes a terminated line of data.

But you also seem to want to write the data in an organized way as sets of measurements. To do that, you should reject data until you see the first "TC1", then collect lines to fill in a data block until and end of data condition is recognized (in your case a blank line), and write it out to disk in a timestamped file. You may also want to include code that abandons a corrupted block unwritten if any of the intermediate reading headers aren't as expected, and goes back to looking for the next hopefully clean TC1, or marks it as erroneous, or whatever.

To use a different baud rate, you must apply a matching setting in both the Arduino and the python ends of the link.

Also note that while the Arduino serial monitor is indeed useful for debugging, you must not have it open while trying to collect data with python - this will either cause an error in that one program is disallowed, or the two programs to fight over the data and each get incomplete results.

Another technique which can be simple and useful, especially during development, is to write every received line of serial data out to a text file with a numeric timestamp at the start of each line. You can then have a second process which reads this file, grabs complete multi-line records, and writes them out using either the timestamp of the first or last element of the record. In particular this lets you get an intermediate view of the data to understand any oddities. With various Unix style command line tricks you can even run the first program both saving the intermediate format and feeding the second record extraction program in real time.

Answer 2

You certainly should use a single Serial.flush at the end of your loop()

Serial.println("");
Serial.flush()
delay(100); 

Remember you have limited SRAM to store characters. I'd suggest you should use the inbuilt serial monitor in the Arduino IDE with the above alteration, and increase your baud rate. There should be no problems sending data at 115k baud.

You should also consider sending a signal back from Python that you have received a valid reading string (ACK/NAK) that way you can remove delays altogether (you still need the flush).

Answer 3

Does your Python code have any way of determining the start & end of a data set?
Does it 'assume' that when it reads a chunk of data from the serial port that the first byte it gets will always be the start of a "Cond.temp" and the last byte will be the ; at the end of Press(alt)?
After reading a chunk of data, do you have any check to see if it is complete or if you should wait a few 10's of milliseconds and go back to read some more?

I'm assuming the answers to these questions are No, Yes & No.

Your PC and your Arduino are running completely asynchronously - neither one really knows what the other is doing.
When your python code reads a chunk of data from the serial port (I'm making another assumption here - that you're not reading it byte-by-byte), you have no way of knowing that the beginning of that chunk is really the beginning of your data set.
Similarly, you also have no way of knowing that the end of that chunk is the end of your data set - maybe it's only a small piece - or maybe you're getting the beginning of the next set glued to the end of this on.

If you're using a USB-Serial adapter then your data is even more likely to arrive in bits & pieces which you'll need to reassemble - just by the nature of how these beasties often work - they could be buffering some number of serial bytes before packaging them together in a single USB packet.

You need some way of flagging the start and/or end of your data sets. Even something as simple as a \n newline character at the end of each one might work for you.
Then, you need to read in and buffer the data you're receiving until you know you've reached the end of a data set.
So, for the \n example:
- read some stuff
- add it to a buffer
- do you have a \n anywhere in the buffer?
-- yes - pull that section from the start of the buffer up to the \n & process it
-- no - go back to the beginning and wait for more stuff
- after you've pulled some stuff out of the buffer to process,
don't just forget the rest - it could be the start of your next data set!
- move it to the start of your buffer
- rinse & repeat

For your baud-rate issue I can't say much for sure. The garbage you're seeing is typical for a mismatched baud-rate situation.
Check that your particular Arduino supports the baud rates you're trying to use at whatever clock speed you're running it at.
Check that your USB-Serial adapter (if you're using one) works at those speeds (either loop pins 2&3 and type to yourself in a terminal or get another, preferably different one, and hook them together with a 3-wire link pin 2-3, pin 3-2, pin 5-5).