0

I experience that if the Arduino (Teensy 3.2) writes a rather long stream of characters to serial, the python script on the other end only receives the first 640 characters. I have found this result both by printing chars, printing lines and by using ArduinoJson's prettyPrintTo(Serial).

Note, that it does not matter what chunks of characters I read in the Python scripts, whether it is read(5) or read(1000).

With the Arduino Serial Monitor, I get all the data, ie more than the 640 characters.

From what I can understand, it is not possible for a sketch to write "too fast" or "too much" to the serial, so there must be something on the pc side receiving the data. However, the Python script for some reason does not get all the data.

What is the problem here? Where should I look for the problem/solution? Perhaps this is more a Linux or Python question, but I will try here.

Sketch - printLines()

void printLines() {
  for (int i=0; i<100; i++) {
    // length below is 40 incl end of line character
    Serial.printf("abcdefthijklmnopqrstuvxyz123456789--%03d\n", i);
  }
}

void setup() {
  Serial.begin(9600);
  delay(500);
  printLines();
  //printChars();
  //printJson();
}

void loop() {}

Sketch - printChars()

void printLines() {
  for (int i=0; i<1000; i++) {
    Serial.write('x');
  }
}

Sketch - printJson()

void printJson() {
  StaticJsonBuffer<10000> jsonBuffer;
  JsonArray &root = jsonBuffer.createArray();
  for (int i=0; i<700; i++) {
    root.add(i);
  }
  root.prettyPrintTo(Serial);
}

Python script

import serial

ser = serial.Serial('/dev/ttyACM0')
ser.timeout = 0.5

# PARAMETERS:
# (As long as num_iterations*num_chars >= 640, EXACTLY 640 characters are read)
num_chars = 100
num_iterations = 10

chars_read = 0
for i in range(num_iterations):
    s = ser.read(num_chars)
    chars_read += len(s)
    print("### i: %d, reading %d chars, char %d to %d, len(s): %d, total chars: %d" 
        % (i,num_chars, i * num_chars, i * num_chars + num_chars -1, len(s), chars_read ))

Terminal output

jan@jan:~/Desktop/batlogger/localserver$ sudo python localserver.py 
### i: 0, reading 100 chars, char 0 to 99, len(s): 100, total chars: 100
### i: 1, reading 100 chars, char 100 to 199, len(s): 100, total chars: 200
### i: 2, reading 100 chars, char 200 to 299, len(s): 100, total chars: 300
### i: 3, reading 100 chars, char 300 to 399, len(s): 100, total chars: 400
### i: 4, reading 100 chars, char 400 to 499, len(s): 100, total chars: 500
### i: 5, reading 100 chars, char 500 to 599, len(s): 100, total chars: 600
### i: 6, reading 100 chars, char 600 to 699, len(s): 40, total chars: 640
### i: 7, reading 100 chars, char 700 to 799, len(s): 0, total chars: 640
### i: 8, reading 100 chars, char 800 to 899, len(s): 0, total chars: 640
### i: 9, reading 100 chars, char 900 to 999, len(s): 0, total chars: 640
  • You're not actually using a serial connection there - you're using a USB CDC/ACM connection, which is a whole different ball game. Are you sure you're not just missing the first chunk of the data while you're sending without the python program running? – Majenko Mar 7 '16 at 22:59
  • Typically CDC/ACM will discard characters if the port is not open at the remote end - or it will discard characters after the buffer is full. This is different to UART since a UART can transmit regardless of what is wired or not wired to it - CDC/ACM can only actively transmit if the port is open at the remote end. – Majenko Mar 7 '16 at 23:01
  • I am sure I don't miss the first chunk, because when I ask it to print, it shows that the first 640 characters is the start of the data. – Mads Skjern Mar 7 '16 at 23:11
  • It could well be that the Teensy is able to buffer 640 bytes in the USB buffer and discards everything else. When you open the port on the PC it then receives the 640 bytes that have been buffered. Make the Teensy wait for the port to be opened before doing any sending (while (!Serial);) – Majenko Mar 7 '16 at 23:14
  • CDC/ACM packets are 64 bytes in size. It makes sense that there may be 10 packet buffers allocated, which makes 640 bytes. – Majenko Mar 7 '16 at 23:19
3

You are not using a serial connection, you are using a USB CDC/ACM connection, and that is a very very different thing.

With a serial connection when you send data it gets sent regardless of whether there is anything listening for it - the TX IO pin toggles high and low regardless.

However, with a CDC/ACM connection it's only possible to send data when there is something actively listening at the remote end - that is, when the port /dev/ttyACM0 has been opened by an application.

Until that port has been opened any data sent will remain in internal buffers. These buffers are 64 bytes each in size (that's the size of a CDC/ACM packet), and there are (I would guess) 10 of them on the Teensy3. After these buffers are full, because there is nothing receiving the data, there can never be any more room made for more data, so instead of blocking like a serial connection does it just discards the data.

So you have filled the buffers with 640 bytes of data and thrown away the rest. That data just sits there.

Then eventually you open the /dev/ttyACM0 port and the Teensy3 is now able to send CDC/ACM packets through the USB. And it has 10 of those packets sat there waiting to go out, so it sends them. So you receive the 640 bytes of data that you had buffered up.

Your mistake is basically not waiting for the port to be opened before sending data to the remote end. You can test the state of the port (kind of, but that's a whole other subject) by examining the Serial object:

if (Serial) {
    // ... the port is open
}

In general you can usually just wait at the start of your program before starting anything else with a simple while:

while (!Serial); // Wait for the port to be opened
  • Great, while(!Serial) fixed it. I knew of that "trick", I know people write it. But I thought the serial connection was in place as long as the pc had "registered" the /dev/ttyACM0. Obviously there is no connection before some proces on the pc opens a connection, fx serial monitor or a python script. – Mads Skjern Mar 8 '16 at 9:34
0

I wouldn't be running as root. Why do that, just to execute a Python script?

Anyway, your problem is that you are not waiting for input. Your Python script is executing faster than the Arduino can pump out the bytes. At 9600 baud you send one byte every 1/960 of a second, so to send 4000 bytes (100 x 40) would take 4000 * 1/960 = 4.166 seconds. However your Python script powers along, getting the first 600 or so which may have been buffered already, and then finishes. You need to rewrite so you wait for a line to arrive. For example:

import serial

ser = serial.Serial('/dev/ttyACM0')

while True:
  s = ser.readline()
  print (s)

When I ran that I saw all 100 lines.

Of course, you need to modify that to break out of the loop. Maybe a special character would do that. Typically a newline is used.


PySerial's read function is blocking by default. - or it has a long timeout anyway

Not when I tested it. Using the code in the question, and without the Arduino even plugged in, I get:

### i: 0, reading 100 chars, char 0 to 99, len(s): 0, total chars: 0
### i: 1, reading 100 chars, char 100 to 199, len(s): 0, total chars: 0
### i: 2, reading 100 chars, char 200 to 299, len(s): 0, total chars: 0
### i: 3, reading 100 chars, char 300 to 399, len(s): 0, total chars: 0
### i: 4, reading 100 chars, char 400 to 499, len(s): 0, total chars: 0
### i: 5, reading 100 chars, char 500 to 599, len(s): 0, total chars: 0
### i: 6, reading 100 chars, char 600 to 699, len(s): 0, total chars: 0
### i: 7, reading 100 chars, char 700 to 799, len(s): 0, total chars: 0
### i: 8, reading 100 chars, char 800 to 899, len(s): 0, total chars: 0
### i: 9, reading 100 chars, char 900 to 999, len(s): 0, total chars: 0

Execution time: 5.5 s.

Of course, the code sets the timeout to 0.5s. But that isn't good enough. You can't assume that 400 bytes will arrive within 0.5s. They should if both sides are perfectly synchronized (at 9600 baud it would take 0.416s).

But serial doesn't work that way. You can't just read for 0.5s and hope everything you want arrives in that time. Heck, it may not have even started sending.


OK, I admit I tested with my Uno and not a Teensy, but this amended Python script shows similar results:

import serial

ser = serial.Serial('/dev/ttyACM2')
ser.timeout = 0.5

# PARAMETERS:
# (As long as num_iterations*num_chars >= 640, EXACTLY 640 characters are read)
num_chars = 100
num_iterations = 10

chars_read = 0
result = ""
for i in range(num_iterations):
    s = ser.read(num_chars)
    result = result + str(s)
    chars_read += len(s)
    print("### i: %d, reading %d chars, char %d to %d, len(s): %d, total chars: %d"
        % (i,num_chars, i * num_chars, i * num_chars + num_chars -1, len(s), chars_read ))
    print ("String = " + str (s))
    print ()

print ("-" * 70)
print ("Total output =")
print (result)
print ("Total length =", len (result))

Output:

### i: 0, reading 100 chars, char 0 to 99, len(s): 100, total chars: 100
String = b'thijz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmnopqrstuvxyz123456789yz123456'

### i: 1, reading 100 chars, char 100 to 199, len(s): 100, total chars: 200
String = b'789--68\nabcdefthijklmnopqrstuvxyz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmn'

### i: 2, reading 100 chars, char 200 to 299, len(s): 21, total chars: 221
String = b'opqrstuvxyz123456789\xff'

### i: 3, reading 100 chars, char 300 to 399, len(s): 0, total chars: 221
String = b''

### i: 4, reading 100 chars, char 400 to 499, len(s): 0, total chars: 221
String = b''

### i: 5, reading 100 chars, char 500 to 599, len(s): 0, total chars: 221
String = b''

### i: 6, reading 100 chars, char 600 to 699, len(s): 100, total chars: 321
String = b'abcdefthijklmnopqrstuvxyz123456789--0\nabcdefthijklmnopqrstuvxyz123456789--1\nabcdefthijklmnopqrstuvxy'

### i: 7, reading 100 chars, char 700 to 799, len(s): 100, total chars: 421
String = b'z123456789--2\nabcdefthijklmnopqrstuvxyz123456789--3\nabcdefthijklmnopqrstuvxyz123456789--4\nabcdefthij'

### i: 8, reading 100 chars, char 800 to 899, len(s): 100, total chars: 521
String = b'klmnopqrstuvxyz123456789--5\nabcdefthijklmnopqrstuvxyz123456789--6\nabcdefthijklmnopqrstuvxyz123456789'

### i: 9, reading 100 chars, char 900 to 999, len(s): 100, total chars: 621
String = b'--7\nabcdefthijklmnopqrstuvxyz123456789--8\nabcdefthijklmnopqrstuvxyz123456789--9\nabcdefthijklmnopqrst'

----------------------------------------------------------------------
Total output =
b'thijz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmnopqrstuvxyz123456789yz123456'b'789--68\nabcdefthijklmnopqrstuvxyz123456789--69\nabcdefthijklmnopqrstuvxyz123456789--70\nabcdefthijklmn'b'opqrstuvxyz123456789\xff'b''b''b''b'abcdefthijklmnopqrstuvxyz123456789--0\nabcdefthijklmnopqrstuvxyz123456789--1\nabcdefthijklmnopqrstuvxy'b'z123456789--2\nabcdefthijklmnopqrstuvxyz123456789--3\nabcdefthijklmnopqrstuvxyz123456789--4\nabcdefthij'b'klmnopqrstuvxyz123456789--5\nabcdefthijklmnopqrstuvxyz123456789--6\nabcdefthijklmnopqrstuvxyz123456789'b'--7\nabcdefthijklmnopqrstuvxyz123456789--8\nabcdefthijklmnopqrstuvxyz123456789--9\nabcdefthijklmnopqrst'
Total length = 669

Since the output of the Uno would be deterministic (it would do the same thing every time) the problem is in the way you are processing the data at the Python end. Look at the line numbers in the above list. They don't even start at 0.

  • PySerial's read function is blocking by default. - or it has a long timeout anyway – Majenko Mar 7 '16 at 23:34
  • See amended answer. – Nick Gammon Mar 8 '16 at 1:57
  • The exact round value (640) every time suggests something more exact than a few characters getting lost every time though. – TisteAndii Mar 8 '16 at 2:52
  • See even more amended answer. – Nick Gammon Mar 8 '16 at 4:39
  • My user does not have privileges to communicated with /dev/ttyACM0. Running with sudo is just temporary until I figure another way out. – Mads Skjern Mar 8 '16 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.