How do I make a proper long string scanner to detect specific tags in a stream of characters?

Question

Here's my code:

#include <SoftwareSerial.h>
SoftwareSerial BTserial(2, 3); // RX, TX

void setup() {
    Serial.begin(9600);
    BTserial.begin(9600);
    Serial.println("Communication with HC-05 successfully started.");
}


bool is_uploading_book = false;
char c = ' ';
String recv_scanner = "";
void loop(){
    if (BTserial.available()) {
        c = BTserial.read();
        recv_scanner += c;
        if(recv_scanner.length() >= 50) {
          recv_scanner.remove(0, 1);
        }
        // I believe there's something wrong with the recv_scanner.
        if(is_uploading_book) {
          Serial.write(c);
        } else {
          if(recv_scanner.indexOf("(__BOOK_UPLOAD_START__)") >= 0) {
            is_uploading_book = true;
          }
        }
        Serial.println("\n\n" + recv_scanner + "\n\n");
    }
}

I'm sending a book (long text) in the form of characters one by one from my phone to my HC-05 module (and the Arduino).
My goal is to spot the tag "(_BOOK_UPLOAD_START_)" in the incoming characters.
Arduino allows me to receive the characters of the book one by one on each loop.
I'm storing those characters in a string called recv_scanner where I allow it's maximum size to be 50 characters and once it reaches those 50 characters, I start to remove the first character and add a new one on each loop (so it can only see 50 characters at a time because maximum String length is 200 and for my purposes, 50 is enough).
The thing is, it doesn't work properly.
When I Serial.println the recv_scanner on every loop (so I see how it progresses) it starts to look like this:

⸮qsnieq⸮cn .laslvanm.⸮⸮⸮d⸮alvuv⸮⸮urosqiam tr⸮⸮⸮

(It starts normally and then slowly progresses into this)
I have no idea what that is.
As far as my predictions go, it should show 50 characters and they should be parts of the books.
Is there something fundamental about Strings that I don't know of?
I'm doing this so I'm able to detect a tag (_BOOK_UPLOAD_END_) at the end of the book and finish uploading.
Feel free to ask my questions if something is unclear.

Answer 1

Another option is to not save the characters in buffer but instead check each one as they arrive to see if the character is the next in the tag sequence.

This has the advantage that you don't need a buffer to match the tag and you don't have to repeatedly search for the string in the buffer.

const char* TAG_STR="(__BOOK_UPLOAD_START__)";
int tag_pos = 0;
char c;
bool is_uploading_book = false;


void loop() {
  if (BTserial.available()) {
    c = BTserial.read();
    if (is_uploading_book) {
      //do book stuff
    } else {
      if (c == TAG_STR[tag_pos]) {
        // c matched the next character in the string
        tag_pos++;
      } else if (c == TAG_STR[0]) {
        // matched first char in the middle, restart
        tag_pos = 1;
      } else {
        // Did not match the next character so we should start over
        tag_pos = 0;
      }
      // The next character to match is the string terminator
      // so we must have matched the whole string.
      if (TAG_STR[tag_pos] == '\0') {
        is_uploading_book = true;
      }
    }
  }
}

Answer 2

I think your problem lies in the usage of the String class. Every time, that you concat a String variable, a new buffer for the result is allocated in the background via dynamic memory allocation. That can result in heap fragmentation, which eats up your memory pretty fast (especially on the smaller ones, like the Uno).

Instead you should use a char array as ring buffer. Let's define the buffer first and initialize it to zero:

#define BUFFER_SIZE    50
char buffer[BUFFER_SIZE] = {0};

Now we define position variables, where the valid data starts and ends in the buffer:

uint8_t buffer_start = 0;
uint8_t buffer_end = 0;

When buffer_start == buffer_end, the buffer is empty. When buffer_end is one position before buffer_start, the buffer is full. To put one character into the buffer we do the following (note, that c is the same variable, as in your code):

if( ((buffer_end + 1) % BUFFER_SIZE) == buffer_start)
{
    buffer_start = (buffer_start + 1) % BUFFER_SIZE;
}
buffer[buffer_end] = c;
buffer_end = (buffer_end+1) % BUFFER_SIZE;

What are we doing here? First we need to check, if our buffer is full. We are doing this by comparing buffer_start to the next position of your buffer_end. The calculation of the new position is where the magic happens. First we increment buffer_end by one and then we are taking the modulus with the buffer size. The modulus is the rest of an integer division. When the position is at the end of the buffer array, it will be (49 + 1) % 50, which is zero. That wraps the array end to the array end for us.

If our buffer is full, then we increment the start counter of the buffer. That means, that we are ditching one character at the start of our buffer to add one at the end (which is what you want to do). (Note, that normally with ring buffers you don't want to throw away data, so you refuse to put more data into the buffer, when it is full).

Then we store the received character into our buffer and increment buffer_end like described above.

The above principle is also called circular FIFO buffer (First In First Out), though there are missing some function to a real FIFO buffer, that you don't need here.

---

Now, that we have a ring buffer, which gets filled with the data, we also need to implement a function to find a string in the buffer, meaning an equivalent to the String.indexOf() method. It might look something like this:

int find_string(char str[], const char search[]){
    uint8_t start = buffer_start;
    uint8_t search_pos = 0;
    uint8_t result_pos = -1;
    while(start != buffer_end){
        if(str[start] == search[search_pos]){
            if(search_pos == 0) result_pos = start;
            search_pos++;
            if(search[search_pos] == '\0') return result_pos;   
        } else {
            result_pos = -1;
            search_pos = 0;
        }
        start = (start + 1) % BUFFER_SIZE;
    }
    return -1;
}

Note, that this implementation is only roughly tested, not thoroughly.