Send multiple sensor values over serial

Question

I want to send a JSON string over serial from arduiono device to my computer. The JSON string contains a sensor value array. To fill this array I use String.concat("value") function. But If I have more than 100 values I get the JSON string with an empty array and an error message:

{"deviceID":"ZB_ROUTER_1","sensorID":"MMA8451","sensorType":"accelerator","aX":[],"landscape": PUF}
VMDPE_1|3147_VMDPE

Is there any other way to send more than 100 values in one message? The arduino device is running with a battery and the communication is wireless. So I want to save energy and reduce the send interval.

EDIT:

After reading the answers I updated my sketch:

Serial.print("{\"deviceID\":\"ZB_ROUTER_1\",");
    Serial.print("\"sensorID\":\"MMA8451\",");
    Serial.print("\"sensorType\":\"accelerator\",");    

    Serial.print("\"aX\":[");
    for (int i = 0; i < samples; i++)
    {
        if(i != 0)
        {
            Serial.print(",");
        }

        Serial.print(xArray[i]);
    }   
    Serial.print("],");

    Serial.print("\"aY\":[");
    for (int i = 0; i < samples; i++)
    {
        if(i != 0)
        {
            Serial.print(",");
        }

        Serial.print(yArray[i]);
    }
    Serial.print("],");

    Serial.print("\"aZ\":[");
    for (int i = 0; i < samples; i++)
    {
        if(i != 0)
        {
            Serial.print(",");
        }

        Serial.print(zArray[i]);
    }
    Serial.print("],");

    Serial.print("\"landscape\":\"");
    Serial.print(orientation);
    Serial.println("\"}");  

In the xArray, yArray and zArray, there are 50 double values stored. One double value needs 4B. So that shouldn't be too much. But there are problems when I receive the JSON string on my computer:

{"deviceID":"ZB_ROUTER_1","sensorID":"MMA8451","sensorType":"accelerator","aX":[-1.17,-1.31,-1.26,-1.25,-1.16,-1.09,-1.40,-0.91,-0.94,-0.97,-1.39,-1.06,-0.91,-0.75,-0.78,-0.83,-0.72,-0.62,-0.74,-0.53,-0.82,-0.86,-0.84,-0.79,-0.79,-0.76,-0.83,-0.82,-0.97,-1.07,-0.92,-0.87,-0.97,-1.20,-1.51,-1.53,-1.82,-0.99,-1.04,-0.41,-0.54,-0.27,-0.24,-0.08,-0.02,-0.63,-0.{"deviceID":"ZB_ROUTER_1","sensorID":"MMA8451","sensorType":"accelerator","aX":[0.16,-0.04,-0.18,-0.03,0.01,-0.17,-0.10,-0.30,-0.31,-0.21,-0.16,-0.16,-0.15,-0.19,-0.17,-0.17,-0.17,-0.18,-0.16,-0.17,-0.17,-0.20,-0.18,-0.16,-0.17,-0.16,-0.15,-0.13,-0.20,-0.19,0.37,-0.88,-0.20,-0.05,-0.12,-0.21,-0.20,-0.17,-0.17,-0.17,-0.17,-0.17,-0.19,-0.17,-0.19,-0.16,-0.18,-0.17,-0.18,-0.17],"aY":[9.80,8.63,8.37,9.05,9.19,8.83,9.01,9.69,8.42,7.94,7.72,7.83,8.43,8.82,9.24,9.21,8.79,8.60,8.39,8.44,8.61,8.77,8.87,8.76,8.66,8.59,8.54,8.73,8.879,8.73,8.66,8.60,8.61,8.68,8.71,8.76,8.71,8.66,8.60,8.63,8.69,8.71],"aZ":[-4.81,-4..28,-4.02,-4.02,-4.29,-4.49,-4.67,-4.63,-4.46,-4.32,-4.28,-4.38,-4.42,-4.52,-4.58,-4-4.52,-4.52,-4.42,-4.39,-4.34,-4.40,-4.47,-4.51,-4.47,-4.44,-4.38],"landscape":" PDB{"deviceID":"ZB_ROUTER_1","sensorID":"MMA8451","sensorType":"accelerator","aX":[-0.18,-0.17,-0.19,-0.20,-0.18,-0.17,-0.18,-0.19,-0.19,-0.20,-0.17,-0.18,-0.17,-0.18,-0.16,-0.18,-0.18,-0.18,-0.17,-0.16,-0.19,-0.18,-0.17,-0.17,-0.16,-0.18,-0.17,-0.18,-0.18,-0.18,-0.16,-0.19,-0.17,-0.16,-0.17,-0.18,-0.17,-0.17,-0.17,-0.16,-0.17,-0.19,-0.18,-0.18,-0.17,-0.18,-0.17,-0.18,-0.17,-0.18],"aY":[8.67,8.68,8.69,8.68,8.68,8.68,8.66,8.68,8.69,8.69,8.69,8.67,8.69,8.70,8.68,8.67,8.68,8.69,8.69,8.68,8.67,8.69,8.69,8.70,8.68,8.66,8.67,8.67,66,8.76,8.80,8.98,8.99,8.76,8.46,8.37,8.55,8.71,8.80,8.85,8.74,8.59,8.58,8.64,8.70,8.44,-4.43,-4.43,-4.44,-4.42,-4.43,-4.43,-4.43,-4.42,-4.44,-4.43,-4.44,-4.43,-4.41,-44.13,-4.20,-4.44,-4.58,-4.68,-4.60,-4.41,-4.31,-4.29,-4.40,-4.50,-4.52,-4.48,-4.41,-{"deviceID":"ZB_ROUTER_1","sensorID":"MMA8451","sensorType":"accelerator","aX":[-0.18,-0.19,-0.22,-0.15,-0.15,-0.19,-0.23,-0.13,-0.16,-0.21,-0.20,-0.17,-0.15,-0.10,-0.16,-0.18,-0.24,-0.17,-0.12,-0.20,-0.28,-0.20,-0.13,-0.19,-0.21,-0.17,-0.18,-0.20,-0.19,-0.17,-0.17,-0.17,-0.17,-0.15,-0.18,-0.16,-0.18,-0.19,-0.18,-0.16,-0.17,-0.18,-0.19,-0.17,-0.19,-0.18,-0.17,-0.16,-0.17,-0.17],"aY":[8.67,8.55,8.61,8.73,8.82,8.73,8.66,8.54,8.54,8.60,8.67,8.60,8.77,8.88,8.79,8.57,8.60,8.51,8.79,8.72,8.83,8.72,8.66,8.68,8.79,8.76,8.67,8.58,75,8.78,8.63,8.63,8.70,8.79,8.77,8.72,8.71,8.67,8.67,8.66,8.70,8.70,8.70,8.69,8.66,87,-4.47,-4.45,-4.32,-4.36,-4.31,-4.41,-4.30,-4.37,-4.46,-4.42,-4.20,-4.39,-4.49,-4.4.40,-4.35,-4.36,-4.39,-4.41,-4.43,-4.40,-4.41,-4.39,-4.40,-4.40,-4.41,-4.42,-4.44,-4{"deviceID":"ZB_ROUTER_1","sensorID":"MMA8451","sensorType":"accelerator","aX":[-0.19,-0.18,-0.18,-0.17,-0.17,-0.18,-0.18,-0.17,-0.19,-0.18,-0.18,-0.17,-0.17,-0.17,-0.18,-0.18,-0.18,-0.18,-0.20,-0.18,-0.16,-0.17,-0.17,-0.18,-0.18,-0.17,-0.18,-0.19,-0.17,-0.19,-0.19,-0.19,-0.17,-0.18,-0.16,-0.18,-0.18,-0.18,-0.19,-0.20,-0.18,-0.17,-0.17,-0.17,-0.18,-0.22,-0.18,-0.17,-0.17,-0.17],"aY":[8.69,8.68,8.69,8.68,8.69,8.68,8.69,8.70,8.69,8.69,8.70,8.68,8.69,8.69,8.70,8.70,8.69,8.67,8.66,8.62,8.73,8.80,8.79,8.75,8.68,8.57,8.60,8.66,8,8.67,8.66,8.69,8.62,8.67,8.76,8.75,8.71,8.71,8.70,8.66,8.67,8.66],"aZ":[-4.43,-4.4-4.41,-4.40,-4.41,-4.42,-4.43,-4.44,-4.47,-4.36,-4.30,-4.30,-4.39,-4.45,-4.51,-4.52,,-4.49,-4.42,-4.35,-4.32,-4.37,-4.38,-4.41,-4.42,-4.46,-4.43],"landscape":" PDB"}

Sometimes there is "aZ" missing, sometimes I get wired array values like 8.83.2 (values with two "."). Do I have to many Serial.print operations? The sending interval is 500ms.

Answer 1

I have had a very similar problem and the Arduino core team has accepted a change from me to facilitate this functionality. It is a complex story so you may have to reread a couple of times.

1) Serial.print returns before the data is send over serial.

The first thing to realize is that when you do serial.print() the data is not actually send over serial. When your code looks like

serial.print("I'm dead");
resetArduino(); //This method does not exists in the arduino core

The message "I'm dead" will not be send over serial. That is because Serial print does not wait until the data has been delivered.

2) There is a software buffer used for serial.

The second thing to realize is that when you do serial.print() the data is not actually send but is put into a software buffer. A interrupted routine sends the data from the buffer over the "transport layer" (like in the osi model) here the serial connection. This buffer uses the same memory that your string is using. In other words you are storing the string 2 times. This is why I agree with majenko when he states

Don't use String. Don't use concat(). Just print each bit as a separate Serial.print() function call.

3) If the software buffer is full serial.print will wait

If you do serial.print and the software buffer is full the function will not return until all data is in the buffer. In other words: If a sketch tries to send more data than is transmittable over the serial connection the behaviour of the sketch will change over time.

4) Serial is slow 115200 baut may seem lightning fast but it is slow compared to what your arduino can do. In other words: if you send lots of data you will need to calculate wether the serial connection can handle that amount of data.

So how can you improve on the situation.

1. As majenko implied: Only use the buffer of serial.
2. Reduce the amount of data you send. Jason is not a good way to send lots of similar data over a serial line because you have to add the variable name each time. I have split my variable names from the data to do reduce the amount of data to send.
3. If you send bursts of data bigger than the software buffer; increase the buffer. This is the change Arduino core team accepted from me.you can set the define SERIAL_TX_BUFFER_SIZE for changing the send buffer size and SERIAL_RX_BUFFER_SIZE for changing the receive buffer size. The default is 64 which means 64 characters.
4. Only send changed data.
5. Preprocess your data so you can send less data

Answer 2

Is there any other way to send more than 100 values in one message?

Yes. Don't use String. Don't use concat(). Just print each bit as a separate Serial.print() function call.

Answer 3

Maybe avoid doing the dynamic string manipulations?

Serial.print("{\"deviceID\":\"ZB_ROUTER_1\",\"sensorID\":\"MMA8451\",\"sensorType\":\"accelerator\",\"aX\":[");
for(i=0; i <10000; i++){ 
   if (! i==0) Serial.print(',');
   Serial.print(i);
}
Serial.print("],\"landscape\": PUF} VMDPE_1|3147_VMDPE");