Script on pc only gets 640 characters via serial

Question

I experience that if the Arduino (Teensy 3.2) writes a rather long stream of characters to serial, the python script on the other end only receives the first 640 characters. I have found this result both by printing chars, printing lines and by using ArduinoJson's prettyPrintTo(Serial).

Note, that it does not matter what chunks of characters I read in the Python scripts, whether it is read(5) or read(1000).

With the Arduino Serial Monitor, I get all the data, ie more than the 640 characters.

From what I can understand, it is not possible for a sketch to write "too fast" or "too much" to the serial, so there must be something on the pc side receiving the data. However, the Python script for some reason does not get all the data.

What is the problem here? Where should I look for the problem/solution? Perhaps this is more a Linux or Python question, but I will try here.

Sketch - printLines()

void printLines() {
  for (int i=0; i<100; i++) {
    // length below is 40 incl end of line character
    Serial.printf("abcdefthijklmnopqrstuvxyz123456789--%03d\n", i);
  }
}

void setup() {
  Serial.begin(9600);
  delay(500);
  printLines();
  //printChars();
  //printJson();
}

void loop() {}

Sketch - printChars()

void printLines() {
  for (int i=0; i<1000; i++) {
    Serial.write('x');
  }
}

Sketch - printJson()

void printJson() {
  StaticJsonBuffer<10000> jsonBuffer;
  JsonArray &root = jsonBuffer.createArray();
  for (int i=0; i<700; i++) {
    root.add(i);
  }
  root.prettyPrintTo(Serial);
}

Python script

import serial

ser = serial.Serial('/dev/ttyACM0')
ser.timeout = 0.5

# PARAMETERS:
# (As long as num_iterations*num_chars >= 640, EXACTLY 640 characters are read)
num_chars = 100
num_iterations = 10

chars_read = 0
for i in range(num_iterations):
    s = ser.read(num_chars)
    chars_read += len(s)
    print("### i: %d, reading %d chars, char %d to %d, len(s): %d, total chars: %d" 
        % (i,num_chars, i * num_chars, i * num_chars + num_chars -1, len(s), chars_read ))

Terminal output

jan@jan:~/Desktop/batlogger/localserver$ sudo python localserver.py 
### i: 0, reading 100 chars, char 0 to 99, len(s): 100, total chars: 100
### i: 1, reading 100 chars, char 100 to 199, len(s): 100, total chars: 200
### i: 2, reading 100 chars, char 200 to 299, len(s): 100, total chars: 300
### i: 3, reading 100 chars, char 300 to 399, len(s): 100, total chars: 400
### i: 4, reading 100 chars, char 400 to 499, len(s): 100, total chars: 500
### i: 5, reading 100 chars, char 500 to 599, len(s): 100, total chars: 600
### i: 6, reading 100 chars, char 600 to 699, len(s): 40, total chars: 640
### i: 7, reading 100 chars, char 700 to 799, len(s): 0, total chars: 640
### i: 8, reading 100 chars, char 800 to 899, len(s): 0, total chars: 640
### i: 9, reading 100 chars, char 900 to 999, len(s): 0, total chars: 640

Answer 1

You are not using a serial connection, you are using a USB CDC/ACM connection, and that is a very very different thing.

With a serial connection when you send data it gets sent regardless of whether there is anything listening for it - the TX IO pin toggles high and low regardless.

However, with a CDC/ACM connection it's only possible to send data when there is something actively listening at the remote end - that is, when the port /dev/ttyACM0 has been opened by an application.

Until that port has been opened any data sent will remain in internal buffers. These buffers are 64 bytes each in size (that's the size of a CDC/ACM packet), and there are (I would guess) 10 of them on the Teensy3. After these buffers are full, because there is nothing receiving the data, there can never be any more room made for more data, so instead of blocking like a serial connection does it just discards the data.

So you have filled the buffers with 640 bytes of data and thrown away the rest. That data just sits there.

Then eventually you open the /dev/ttyACM0 port and the Teensy3 is now able to send CDC/ACM packets through the USB. And it has 10 of those packets sat there waiting to go out, so it sends them. So you receive the 640 bytes of data that you had buffered up.

Your mistake is basically not waiting for the port to be opened before sending data to the remote end. You can test the state of the port (kind of, but that's a whole other subject) by examining the Serial object:

if (Serial) {
    // ... the port is open
}

In general you can usually just wait at the start of your program before starting anything else with a simple while:

while (!Serial); // Wait for the port to be opened

Answer 2

I wouldn't be running as root. Why do that, just to execute a Python script?

Anyway, your problem is that you are not waiting for input. Your Python script is executing faster than the Arduino can pump out the bytes. At 9600 baud you send one byte every 1/960 of a second, so to send 4000 bytes (100 x 40) would take 4000 * 1/960 = 4.166 seconds. However your Python script powers along, getting the first 600 or so which may have been buffered already, and then finishes. You need to rewrite so you wait for a line to arrive. For example:

import serial

ser = serial.Serial('/dev/ttyACM0')

while True:
  s = ser.readline()
  print (s)

When I ran that I saw all 100 lines.

Of course, you need to modify that to break out of the loop. Maybe a special character would do that. Typically a newline is used.

---

PySerial's read function is blocking by default. - or it has a long timeout anyway

Not when I tested it. Using the code in the question, and without the Arduino even plugged in, I get:

### i: 0, reading 100 chars, char 0 to 99, len(s): 0, total chars: 0
### i: 1, reading 100 chars, char 100 to 199, len(s): 0, total chars: 0
### i: 2, reading 100 chars, char 200 to 299, len(s): 0, total chars: 0
### i: 3, reading 100 chars, char 300 to 399, len(s): 0, total chars: 0
### i: 4, reading 100 chars, char 400 to 499, len(s): 0, total chars: 0
### i: 5, reading 100 chars, char 500 to 599, len(s): 0, total chars: 0
### i: 6, reading 100 chars, char 600 to 699, len(s): 0, total chars: 0
### i: 7, reading 100 chars, char 700 to 799, len(s): 0, total chars: 0
### i: 8, reading 100 chars, char 800 to 899, len(s): 0, total chars: 0
### i: 9, reading 100 chars, char 900 to 999, len(s): 0, total chars: 0

Execution time: 5.5 s.

Of course, the code sets the timeout to 0.5s. But that isn't good enough. You can't assume that 400 bytes will arrive within 0.5s. They should if both sides are perfectly synchronized (at 9600 baud it would take 0.416s).

But serial doesn't work that way. You can't just read for 0.5s and hope everything you want arrives in that time. Heck, it may not have even started sending.

---

OK, I admit I tested with my Uno and not a Teensy, but this amended Python script shows similar results:

import serial

ser = serial.Serial('/dev/ttyACM2')
ser.timeout = 0.5

# PARAMETERS:
# (As long as num_iterations*num_chars >= 640, EXACTLY 640 characters are read)
num_chars = 100
num_iterations = 10

chars_read = 0
result = ""
for i in range(num_iterations):
    s = ser.read(num_chars)
    result = result + str(s)
    chars_read += len(s)
    print("### i: %d, reading %d chars, char %d to %d, len(s): %d, total chars: %d"
        % (i,num_chars, i * num_chars, i * num_chars + num_chars -1, len(s), chars_read ))
    print ("String = " + str (s))
    print ()

print ("-" * 70)
print ("Total output =")
print (result)
print ("Total length =", len (result))

Output:

### i: 0, reading 100 chars, char 0 to 99, len(s): 100, total chars: 100
String = b'thijz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmnopqrstuvxyz123456789yz123456'

### i: 1, reading 100 chars, char 100 to 199, len(s): 100, total chars: 200
String = b'789--68\nabcdefthijklmnopqrstuvxyz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmn'

### i: 2, reading 100 chars, char 200 to 299, len(s): 21, total chars: 221
String = b'opqrstuvxyz123456789\xff'

### i: 3, reading 100 chars, char 300 to 399, len(s): 0, total chars: 221
String = b''

### i: 4, reading 100 chars, char 400 to 499, len(s): 0, total chars: 221
String = b''

### i: 5, reading 100 chars, char 500 to 599, len(s): 0, total chars: 221
String = b''

### i: 6, reading 100 chars, char 600 to 699, len(s): 100, total chars: 321
String = b'abcdefthijklmnopqrstuvxyz123456789--0\nabcdefthijklmnopqrstuvxyz123456789--1\nabcdefthijklmnopqrstuvxy'

### i: 7, reading 100 chars, char 700 to 799, len(s): 100, total chars: 421
String = b'z123456789--2\nabcdefthijklmnopqrstuvxyz123456789--3\nabcdefthijklmnopqrstuvxyz123456789--4\nabcdefthij'

### i: 8, reading 100 chars, char 800 to 899, len(s): 100, total chars: 521
String = b'klmnopqrstuvxyz123456789--5\nabcdefthijklmnopqrstuvxyz123456789--6\nabcdefthijklmnopqrstuvxyz123456789'

### i: 9, reading 100 chars, char 900 to 999, len(s): 100, total chars: 621
String = b'--7\nabcdefthijklmnopqrstuvxyz123456789--8\nabcdefthijklmnopqrstuvxyz123456789--9\nabcdefthijklmnopqrst'

----------------------------------------------------------------------
Total output =
b'thijz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmnopqrstuvxyz123456789yz123456'b'789--68\nabcdefthijklmnopqrstuvxyz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmn'b'opqrstuvxyz123456789\xff'b''b''b''b'abcdefthijklmnopqrstuvxyz123456789--0\nabcdefthijklmnopqrstuvxyz123456789--1\nabcdefthijklmnopqrstuvxy'b'z123456789--2\nabcdefthijklmnopqrstuvxyz123456789--3\nabcdefthijklmnopqrstuvxyz123456789--4\nabcdefthij'b'klmnopqrstuvxyz123456789--5\nabcdefthijklmnopqrstuvxyz123456789--6\nabcdefthijklmnopqrstuvxyz123456789'b'--7\nabcdefthijklmnopqrstuvxyz123456789--8\nabcdefthijklmnopqrstuvxyz123456789--9\nabcdefthijklmnopqrst'
Total length = 669

---

Since the output of the Uno would be deterministic (it would do the same thing every time) the problem is in the way you are processing the data at the Python end. Look at the line numbers in the above list. They don't even start at 0.