SoftwareSerial will not read all of the printed string when calling readString()

Question

I have an Arduino UNO and an ESP32 that need to communicate to each other using SoftwareSerial. The problem I am coming across is that when I call readString and print it out using Serial, it will not give me everything that has been printed out from my ESP32.

I have my Arduino RX Pin hooked up to 4 and TX hooked up to 5 I have my ESP32 RX2 Pin hooked up to 16 and TX2 pin hooked up to 17, with the GND hooked up to directly to the Arduino GND I am not using any resistors, straight up 5v of pure power

Arduino UNO Code:

// C-standard library
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

#include <SoftwareSerial.h>

#define RX_PIN 4
#define TX_PIN 5

SoftwareSerial ardSerial(RX_PIN, TX_PIN);

void setup(void) {
  // Get access to network
  pinMode(RX_PIN, INPUT);
  pinMode(TX_PIN, OUTPUT);
  Serial.begin(9600);
  while(!Serial) {}
  ardSerial.begin(9600);
  while(!ardSerial) {}
}

void loop(void) {
  while (ardSerial.available() > 0) {
    String payload = ardSerial.readString();
    Serial.println(payload);
  }
  delay(1000);

}

ESP32 Code

#include <stdlib.h>
#include <string.h>

#include <SoftwareSerial.h>

SoftwareSerial ardSerial;

void setup(void) {
  // Begin connections
  Serial.begin(115200);
  while (!Serial) {}

  ardSerial.begin(9600, SWSERIAL_8N1, RX2_PIN, TX2_PIN);
  while (!ardSerial) {}
  return;
}

void loop(void) {
  {
    String spotData = "Huge JSON File that needs transferred";
    ardSerial.print(spotData);
  }

  delay(5000);
}

Expected result:

• Able to retrieve every little bit of the JSON file that was printed out

Actual result:

• Only receive like half of the JSON file

Answer 1

I see 2 problems with your code:

• You wrote in the comments, that the JSON file is huge. The Uno has only 2kB of RAM and you are trying to lead the complete file into RAM at once. Such a JSON file from Spotify will easily exceed the 2kB (in fact you have even less, because the Arduino framework and SoftwareSerial also use some of the RAM). So you just cannot handle the complete file at once with an Uno.

• You use readString(). This function is sure easy, but also has some pitfalls. It uses the String class, which utilizes dynamic memory allocation. It allocates a bigger character array in RAM, if the current array is not big enough. That means, that you would need up to 2 versions of your String fitting into RAM while receiving, leaving only about 1kB. Also it will read from the SoftwareSerial interface, until a timeout is hit (1s by default I think). For building a robust communication between the two devices you should read until you reach a special character, for example via readStringUntil(). That special character is often the newline character '\n'.

Possible solution: First we should talk about the limited RAM problem. As stated above you cannot handle the full file at once. You need to handle it in chunks. Whatever you want to do with the data in the end, the Uno will need some time handling the current chunk of data. So you need to pause the transmission of data at the ESP, until the Uno is ready to receive more. Here you can go two ways:

• Send the data in chunks of predefined size (small enough for the Uno to hold at once in its RAM) and wait a predefined time between each chunk, that is long enough for the Uno to handle every data it might have gotten in that chunk. This is the easiest to program, but will necessarily take more execution time.
• Let the Uno signal to the ESP, that its buffer is full, for example by sending a special character to the ESP. The ESP should then stop sending, until it gets the OK from the Uno to proceed the transmission. To send without loosing data the Uno should probably also send the number of bytes received in that chunk. That way the ESP knows exactly, where in the data String the Uno is currently. This is a bit more difficult, though still easy enough and robust.

To handle the sending in chunks I would use the write(const uint8_t *buffer, size_t size) method of SoftwareSerial (actually this function comes from the Print class, from which SoftwareSerial inherits). The buffer is the pointer to the first element of the current chunk in the char array with the data. You can get the char array via spotData.c_str() and you can get the pointer to the n-th element of it via &(spotData.c_str()[n]). The second parameter is the size of the chunk in bytes. This needs to be small enough, that the data will fit into the RAM of the UNO, together with all the RAM needed for processing the data.

On the Uno I would then do the reading bytewise. Something like this:

char data[BUFFER_SIZE]="";
int pos=0;

void loop(){
    if(ardSerial.available()){
        data[pos] = ardSerial.read();
        if(pos == BUFFER_SIZE - 1){
            // Handle chunk of data
            pos = 0; // reset position to reuse buffer
        } else {
            pos++;
        }
    }
}

I have left out some parts to only demonstrate the principle. Also you need to define BUFFER_SIZE yourself, depending on the available RAM.

Obviously this does not implement the second option with the pause signal, which is more complex.

---

Apart from sending and receiving you will need to think about how to process the chunks of data. This depends largely on what exactly you want to do with that data. Just sending them via Serial is easy. Interpreting the data in chunks is more difficult.

Answer 2

There are two ways to implement a parser for a language like JSON or XML:

1. Have the parser swallow the whole document at once, and produce an in-memory representation of the same data.

2. Have the parser return control to the caller for each “event” it detects, where an “event” is typically a primitive data item, or the start or the end of a container.

Parsers of the first kind are much easier to use, but this can be expensive in RAM usage if you want to parse a alrge document. The second kind of parsers are harder to use, but they are able to parse huge documents using a minimal amount of memory.

On an Arduino, you would typically use ArduinoJson, which is a parser of the first kind, not ideal for parsing huge documents on little memory. A quick search for “event driven C++ JSON parser” got me JsonReader. I did not try it, and do not know whether that would work on Arduino, but it shows that at least this kind of parsers do exist for JSON in C++. You may want to search further along this way.

Before you dig into this, however, I suggest you give ArduinoJson a second chance. ArduinoJson has a couple of tricks that can help dealing with large documents:

• You can ask it to filter the data as it reads it, in order to only keep in memory the items you are actually interested in.

• You can deserialize in chunks, and use the data extracted from one chunk at a time.

Both techniques are explained in the tutorial How to deserialize a very large document?.

Also, let's not forget the obvious: you should not attempt to store the raw JSON document in memory. Instead, you should give ArduinoJson a reference to your stream (namely ardSerial), and let it directly pull the bytes as it parses.