just a quick question on an oddity I noticed while playing with serial on my Arduino.

I'm using the serial monitor to test an interface I plan to adapt to Bluetooth later. Except I'm noticing if I only read fixed sizes with readBytes I can only ever input data of that exact byte length or my serial connection gets broken.

Example pseudocode:

if(Serial.available() >= NUM_BYTES){
/* Do stuff */

If I input something of length NUM_BYTES + 1, it reads the first NUM_BYTES. But then I can never send more data over the serial monitor. Why is this? Is it part of the serial implementation? (The write would always be 1 afterwards even if I try to send more.)

A fix I had was just to clear the buffer by calling Serial.read() until nothing else was available. As a matter of curiosity too... If anyone cares to answer or give me a link to some reading - is there a way to check before I write to serial to make sure there's buffer space for my message on an Arduino? I read that they have a 64 byte buffer for data but I'm unsure how the protocol caters to sending larger sets of data if the buffer ever overflowed. Perhaps the serial library is using some form of ack behind the scenes?

  • 2
    Can you show us your real code? And how do you know that you "cannot send more data" over serial? What behavior are you seeing? – chrisl Sep 18 '18 at 18:54

The purpose of the serial input buffer is to prevent data-loss between your program's Serial.read() calls. If you really want to allow long messages to accumulate in it, here is one of many articles that describes how you can increase the size of the buffer in the Arduino core-code. But this is a system-wide solution to one program's design problem - it imposes the memory overhead of that larger buffer on all programs whether they need it or not, or you have to define a new board-type in the boards file (the technique in the above article) and remember to use it for programs that need the larger buffer.

A better approach is to do your own buffering, by defining within your program, a buffer large enough for your largest message (plus the zero terminator), collect bytes from the serial input buffer into your own buffer often enough to not let the serial buffer overflow; then detect when your own buffer has a complete message before you "/* Do stuff */ ".

You appear to be asking about a situation in which you will know in advance what the length of the next message will be; in that case, you can count bytes until you've collected the whole message.

In other situations, the message may have a terminator such as the NewLine character, or the message may contain the length within the first few bytes, in which case you'd wait for those bytes, convert them to an integer, and use that number to detect the end of message.


But it doesn't explain to me really why I can't send it more bytes afterwards until I've read the entire input buffer.

I can't offer much help there; I've no experience using .readbytes(); I just read bytes as they arrive until I've got what I wanted. But my notes on it say it "Reads chars into a buffer. Terminates when len bytes have been read, or it times out" (1 sec, I think; there may be a call to change that, too). I'd also consider the possibility of the 64 byte serial buffer having over-flowed; doesn't sound like it but strange things can cause strange things....

"but I rarely work this close to the hardware" That's the fun and the challenge of embedded systems!

| improve this answer | |
  • Thanks, useful information. And yeah I was trying to have fixed sized messages except being lazy I was using the serial monitor to send them to the Arduino. I just still don't understand why I can't send another message after the first one if I exceed the "NUM_BYTES". Sure I'd expect one byte to remain after reading NUM_BYTES. But it doesn't explain to me really why I can't send it more bytes afterwards until I've read the entire input buffer. – Raishin Sep 18 '18 at 18:13
  • Also just as a side note this is just for a fun LED project. I haven't got the Android IDE working on my desktop yet but I have a Bluetooth LE that I plan to use with a phone app to control a bunch of WS2812B LEDs. But to do that I need an ICD. (And I'll probably be extra fancy and add some checksums and whatnot.) I'm a software developer for a living but I rarely work this close to the hardware/communications layer. I usually just have TCP/UDP communications that are already handled by home brewed libraries so all I end up having to do is define the structure/enumerations and parse the data. – Raishin Sep 18 '18 at 18:16

I believe you may have misunderstood the way some of the serial port functions work. Serial.available is not guaranteed to match the number of bytes that can be read, because data may be in several different parts of the pipeline and only one is counted. Furthermore, the amount available may change after you check it. Furthermore, it may not change again if more data arrives after it has been set. And the behavior may be different depending on how the port is configured.

In your case it appears that you often have one extra byte left over, so your test if(Serial.available() >= NUM_BYTES) remains set to 1.

One way to accomplish what you want is to set serial read timeout to zero, then attempt to read the maximum number of bytes that the serial buffer can hold, then check to see how many bytes you actually got. If you got a partial read, either loop around and keep checking until more arrives, or buffer the partial read somewhere and go around the main loop again, while periodically checking for more data. This is the approach I generally use for receiving large amounts of data.

Another way is just to read one byte (char) at a time as long as serial.available is true (non-zero). After each byte is read you check whether your condition has been met. If it has, you return the received bytes, and if it has not then you keep reading until serial.available() is false (zero bytes available), at which point you leave the received bytes in a buffer and return an empty string (or whatever).

With techniques like these you can accumulate incoming bytes until any condition you like has been met. No data will be lost unless you fail to read for a significant period of time and handshaking is not implemented). Data that you don't read after your condition is met will still be held in the serial buffer and will be read on the next pass.

You can apply any conditions you wish for stopping the accumulation and returning a result. If you are receiving binary COBS-encoded data, you keep going until you see a 0x00 byte (in COBS encoding this indicates the end of one block and the start of another). If you are receiving lines of text, you keep accumulating until you see a newline. If you are receiving blocks of fixed size, you accumulate until you have received enough bytes. Of course if you're receiving binary then you don't use a string to accumulate it :)

Below is the code I use to receive newline-terminated messages using the second approach I mentioned. I call this routine on every pass through the loop() and if it returns anything but an empty string, my main loop acts upon it. I use this to receive newline-delimited ascii text commands from a host (while streaming binary data from the device to the host (at 4 Mbps) over the same serial port). I pulled this code from a release that has been running for several years, and no character losses have ever been detected.

// All serial reads are non-blocking because the timeout is set to zero.
// This must be done before the serial.begin() is called
// This routine is for receiving occasional newline-terminated commands.
// Read port until a newline is received or we run out of characters in the buffer
// Returns a single newline-terminated message, or else returns "".
String inputString_Host = "";
String s = "";
String ReadFromHostSerialPort() 
    s = "";
    while (HOST_SERIAL_PORT.available())
        char inChar = (char)HOST_SERIAL_PORT.read();
        inputString_Host += inChar;
        if (inChar == '\n')
            s = inputString_Host;
            inputString_Host = "";
            return s;
    return "";

Below is another example showing how to use readBytes to accomplish a similar thing using multi-byte block reads. This code is used to read binary data that is streaming from an Oximeter. I simply take everything that's available (just to get it away from the serial port) and put it into a circular buffer of my own, from which I later decode and parse the data. Depending on the interface, this approach can easily receive floating point data samples at 100 KHz without data loss, with plenty of time left for processing the samples.

// All serial reads are non-blocking because the timeout is set to zero.
// This is essential & must be done before the serial.begin() is called. 
// Request 1500 bytes and look at numBytes to see what we really got.
// Note that spO2Buffer is a circular buffer implemented and serviced elsewhere

// OXIMETRY_SERIAL_PORT is connected to the Nonin SPO2 sensor.
// The Tx pin is not currently connected at all.
// Read port into a 3-second circular buffer

void OximetryClass::ReadOximeter()
    if (OXIMETRY_SERIAL_PORT.available())
        uint8_t inOximeterBytes[1500] = { 0 };  
        int numBytes = OXIMETRY_SERIAL_PORT.readBytes(inOximeterBytes, 1500);            
        spO2Buffer.WriteBytes(inOximeterBytes, numBytes);

I hope this helps!

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.