Parse a large json response with ESP8266

Question

I'm doing a project where my Arduino uses an external ESP8266 wifi module (with AT firmware) to send http requests and receive server response.

I'm trying to get an NBA scoreboard, and here's what my Arduino sends to ESP:

> AT+CIPSTART=0,"TCP","data.nba.net",80
0,CONNECT

OK

> AT+CIPSEND=0,103
SEND OK

> GET /data/10s/prod/v1/20190214/scoreboard.json HTTP/1.1
Host: data.nba.net
Connection: keep-alive

+IPD,0,1460:HTTP/1.1 200 OK..
.............................
...here comes the response...
.............................

Now the code works fine, and I'm able to get the json file needed. The problem is that the response is sometimes too long and the file gets truncated, thus the response is incomplete. The file itself is sometimes pretty large, but I only need a few numbers from it (current teams + current score). So my questions are:

1. I use readString() method (of SoftwareSerial object instance) to read the response. Is there a way to parse the response chunk by chunk without loading everything in my ESP memory?
2. Is there a way to limit the response?
3. Are there any alternatives to the method I'm using? Maybe parsing the website (nba.com or espn.com/nba) directly? If so, how can I do that?
4. Maybe there's a way to write a separate (free) online API to parse json, constraint the response and send the small version upon request?

PS. I tried doing the same without AT firmware, flashing the module with a code that used HTTPClient directly (with ESP8266WiFi.h and ESP8266HTTPClient.h libraries):

HTTPClient http;
http.begin(request);
if (http.GET() > 0) {
    response = http.getString();
}
http.end();

That didn't help much. I'm still running out of memory since I'm directly loading the response to memory with getString().

PPS. Just in case it matters, I'm using ATmega2560 and a small ESP-01 module (similar to this).

Answer 1

Ok, so I was able to do this. Here's the workaround.

Let's say the GET response is a flat string containing large .json file with all the commas and curly brackets. The idea is to read it byte-by-byte and instead of storing it just find the key you need and extract its value.

We want to find the field containing, say, "97.5 FM" from http://data.nba.net/data/10s/prod/v1/20190223/scoreboard.json (that's at the very end), skip 3 bytes (which is just " | ") and then read and Serial.print 4 bytes (which is "1280")*. We would do something like this (of course make sure to connect to wifi and include <ESP8266HTTPClient.h>:

HTTPClient http;
http.begin("http://data.nba.net/data/10s/prod/v1/20190223/scoreboard.json");
int httpCode = http.GET();
if (httpCode == HTTP_CODE_OK)
  Serial.println(getValue(http, "97.5 FM", 3, 4));

where the getValue() function is defined as follows:

String getValue(HTTPClient &http, String key, int skip, int get) {
  bool found = false, look = false;
  int ind = 0;
  String ret_str = "";

  int len = http.getSize();
  char char_buff[1];
  WiFiClient * stream = http.getStreamPtr();
  while (http.connected() && (len > 0 || len == -1)) {
    size_t size = stream->available();
    if (size) {
      int c = stream->readBytes(char_buff, ((size > sizeof(char_buff)) ? sizeof(char_buff) : size));
      if (len > 0)
        len -= c;
      if (found) {
        if (skip == 0) {
          ret_str += char_buff[0];
          get --;
        } else
          skip --;
        if (get <= 0)
          break;
      }
      else if ((!look) && (char_buff[0] == key[0])) {
        look = true;
        ind = 1;
      } else if (look && (char_buff[0] == key[ind])) {
        ind ++;
        if (ind == key.length()) found = true;
      } else if (look && (char_buff[0] != key[ind])) {
        ind = 0;
        look = false;
      }
    }
  }
  return ret_str;
}

You can call this getValue() whatever many times you like, but the strings you're looking for need to be only after each other, because the response to GET is read byte-by-byte just once.

There is probably (i.e., most likely) a better way of doing this, but this approach works and it's pretty fast, so...

---

* This, of course, is just an example, and you can overload this getValue() function definition to, say, read until it gets a curly bracket or whatever.

Answer 2

Check out the Streaming JSON Parser. It's designed to parse large JSON objects on processors with little memory. You'll just read one character at a time from the HTTP connection rather than the whole response; pass the character to the library and it'll call code you provide every time it sees an array, object, key or value. Then you can extract just the pieces you want without the overhead of loading it all in memory at once.

The library has good examples that'll show you how to use it.