SoftwareSerial read until available is empty only works with delay

Question

I originally was only going to make my program work on an ESP until I needed it on an Arduino UNO (because I suspect my ESP is broken). I had to add SoftwareSerial support but because I didn't originally intend to, the strings I send from my app have no terminators on them. I thought this would be fine because I could do something like:

void loop()
{
    while (ble.available() > 0)
    {
        data += (char)ble.read();
    }

    if (ble.available() <= 0 && data.length() > 0)
    {
        ble.println(data);
        //do stuff
        data = "";
    }
}

All that is doing is creating a string out of the recieved characters. It then checks if there is nothing left to read, and stuff has already been read (ie data has a length), then print the data. What I expect the output to be is (if the input string in 03:255,255,255):

03:255,255,255

What I actually get is:

0
3
:
2
5
5
,
//and so on

That is becuase, for some reason, even though ble.available() should return a number not 0 at the if statement, it returns 0 and therefore the statement will always become true. The only way I've fixed this is by adding in a little delay at the end of the loop. Whether that be:

delay(10)

or even this, in the while loop:

Serial.println(ble.available()) //This introduces enough delay somehow

Without having to modify my app, can I pull this off without using a delay? With delays it seems hacky especially because for all I know, for older or newer devices this delay could be too high or too low.

Answer 1

Serial does not transmit data as one bunch, but byte by byte. There can easily be some little delays introduced, especially, when transmitted via a packaged protocol like bluetooth, long enough, for the Arduino to not see any new data in its buffer. Also the transmission needs some time, depending on the baudrate. When you read faster from the buffer, than you receive, you will encounter an empty buffer after every byte.

Now with that information look at your code:

void loop()
{
    while (ble.available() > 0)
    {
        data += (char)ble.read();
    }

    if (ble.available() <= 0 && data.length() > 0)
    {
        ble.println(data);
        //do stuff
        data = "";
    }
}

Let's assume, the Arduino receives one byte, then a little pause happens, then the next data is received. The code will go into the while loop and read the received byte from the serial buffer. To stay in the while loop, the next byte has to be received in the execution time of the code inside of the while loop. That is incredible short. So after one byte there is nothing more inside the buffer, thus the while loop exits, because ble.available() returns zero. Then the if statement is executed, also rather fast. Most likely, ble.available() will still return zero. But you already put a character into the buffer, so the if statement is executed.

The delay solves your problem because then you read way slower than you receive. The data is mainly received while the delay runs. Then the rest of your code can read, what was already received and put into the buffer.

**What to do instead of delay(): You need to make sure, that you always read a full message. There are different ways, depending on your requirements. If your messages always have the same length, you could wait, until the specified number of characters was received, and only then read them. But that will fail, if you for some reason (for example a reset of the Arduino) receive a partial message, which would cause a misalignment. You could clear the receive buffer, when you read a full message, that might help, if your data transmissions don't happen to often.

You could try to parse the incoming data according to the message format (which needs to stay constant for this purpose), like:

Read a number.
If it has a colon following it, it is a new message.
Then read number
    read comma
    read number
    read comma
    read number
Repeat.

For this purpose it would be fitting to read every byte, that gets received, and write then into your own circular buffer. You can then continuously add newly received characters at the end, while removing the already parsed messages from the start of the buffer.

---

You see, these ways have caveats and might be convoluted. The easiest way would be to add a delimiter/terminator to your messages, though from your question it sounds, as if this is not possible for you currently.

Note:

• None of the above is specific to SoftwareSerial. It's the same for hardware Serial.
• I suspect the code worked your ESP, because you had some other code there, which introduced a big enough delay (just like your delay() call)