Arduino Sending faster/more data to a web server

Question

I'm working on a project that reads vibration on a building using an accelerometer and displaying it's data on a web server. The plan is to read 10 samples/second from the accelerometer and send that data every 5 seconds to a web server using arduino yun.

So there would be 10(samples/sec) x 5(seconds) x 3(3 axis - x,y,z) = 150 data points that will be sent to the webserver, I tried this with the client.get() command but it cannot send that much data to a web server, I also tried POST request but i've had trouble with the sending it multiple times since it needs to have a delay.

What are the other options that I can do in order to achieve this? Any suggestion would help and please if you think my question is bad or not stated properly don't report me, I'm still new here in stack exchange. Any help/suggestion is much appreciated

Arduino Code for POST request

#include <Bridge.h>
#include <HttpClient.h>
#include <YunClient.h>

//IPAddress server(192,168,1,200);
byte php_server[] = {192,168,1,9};

YunClient client;
String data = "", datax="", datay="", dataz="";
unsigned long ts_0 = 0, tr_0 = 0, t_read = 100;
int counter=0;

void setup() {
  // put your setup code here, to run once:
  Bridge.begin();
  Serial.begin(115200);

}

void loop() {
  // put your main code here, to run repeatedly:

  if((millis()-tr_0>=t_read)) {

    if(counter<49) {
      counter++;
    }
    else if(counter == 49) {
      //Serial.print("sent");
      String dat = "0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12,0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12, 0.24,1.24,4.56,8.97,3.15,7.23,9.34,0.12,6.21,8.12";

      send_to_ws(dat);
      Serial.println(dat);

      counter = 0;
    }
    tr_0=millis();
  }
}

void send_to_ws(String data) {
  if(client.connect(php_server, 80)) {
    Serial.println("Connected");        
    client.println("POST /arduino/add.php HTTP/1.1");
    client.print("Content-Length:");
    client.println(data.length());
    Serial.println(data.length());
    Serial.println(data);
    client.println("Connection: Close");
    client.println("Host:192.168.1.9");
    client.println("Content-Type: application/x-www-form-urlencoded");
    client.println();
    client.println(data);
  } else {
    Serial.println("Connection error");
  }
}

Serial monitor output when running this sketch, It prints it in the serial monitor but only blank data is being pushed in the database of the server

Answer 1

Option 1. Send a single message from the client to the server. This message should contain all 150 points of data.

Option 2.
Send the data in a format other than as a string. Base64 encoding of the raw numeric values would be ideal, but its going to add extra load to the CPU. Depending on the range of the data (0-359) and the resolution (use 2 DP) you might be able to use just 4 bytes for each data point. Because you are using a fixed length value to represent each point you no longer need separators in you data so a single reading goes from 21 bytes to 12 bytes.

Option 3. Hand off the data to a second processor that is dedicated to sending the data. The Yun would write the data into a shard circular buffer and the other processor (possible another Yun or an ESP8266) would read the data from the buffer, convert it and write it to the server.

I can't say which is best, because I don't know enough about you system, but if you could expand on your details we should be able to narrow it down.

---

After reading you answer above. If you ranges are 0-10 with two DPs then I would look at option 2 first. If each reading can range from -9.99 to +9.99 you take the absolute value and multiply by 100 and write that as a 2 digit hex number to your message body.

uint8 number = ((reading < 0) ? 0x80 : 0x00);  // Set the high bit for negative.
number += (abs (reading) * 100);
messageBodyString += String (number, HEX);

That will give you 2 bytes per point, and a 300 byte (+overhead) message should be transferable within 5 seconds.

---

Here is some quick code to explain the encoding and decoding

unsigned int Encode(const float& value)
{
    unsigned int result = (value < 0) ? 0x80000000 : 0x00000000;// Work out and set the sign bit
    result |= (unsigned int)abs(value * 100.0);             // Get the absolute value (ignore sign) and multiply
    return result;
}

float Decode(const unsigned int& value)
{
    const bool isNegative = ((value & 0x80000000)  != 0);       // Get the sign bit
    float result = (float)(value & 0x7FFFFFFF);                 // To remove the sign bit
    result /= 100.0;                                            // Divide to return to the original number of dps.
    if (isNegative)
        result = 0 - result;
    return result;
}

Answer 2

Personally for this kind of frequency of data I wouldn't even consider sending it via HTTP. The extra bulk of the HTTP protocol makes the transfer way too slow.

Instead I would set up a UDP-based gateway server which accepts UDP messages from your sensor system and stores the data directly into the database (or wherever you are storing the data) for the website to display it at its leisure.

Yes, it is more complex to create since you need to write the UDP server gateway software. However you then completely remove the massive overheads of a text-based bulky protocol like HTTP.

Answer 3

You can use 2d Array for this and send all data at once in 5 Sec. It Read all 3 Axis data each 100Ms and make a matrix of 50 X 3 = 150 Values and then Print it in Serial Port. You can use this in your code to send data to server in desirable format.

int Xaxis=A0; 
int Yaxis=A1;
int Zaxis=A2;
int axisMatrix[50][3]; 
void setup(){
    Serial.begin(9600);
    pinMode(Xaxis,INPUT);
    pinMode(Yaxis,INPUT);
    pinMode(Zaxis,INPUT);}

void loop(){
        for (int i = 0; i <= 50; i++) {
            axisMatrix[i][1]=analogRead(Xaxis);
            axisMatrix[i][2]=analogRead(Yaxis);
            axisMatrix[i][3]=analogRead(Zaxis);
            delay(100);}
        for (int i = 0; i <= 50; i++) {
            Serial.print(axisMatrix[i][1]);
            Serial.print(",");
            Serial.print(axisMatrix[i][2]);
            Serial.print(",");
            Serial.print(axisMatrix[i][3]);
            Serial.print(",");
            Serial.print("\n");
        }}