How to send Json to a webservice from pc/RPI with live data from Arduino(serial)

Question

I'm making a project where my plan is to take sensor input from the Arduino, and then pass that info (by serial/usb connection) through either a laptop or a Raspberry Pi and convert it to Json and send it to a webservice in asp.net c#, where it gets stored in a database, and then the data can be shown on a website, so the visitor can see live data from the Arduino sensor on the site and also be able to see what the sensor measured yesterday, last month, or the average for a extended period of time, or a graph.

It might be easier to illustrate what I'm talking about:

The PC / RPI part is optional, it can be using either one.

There are a few options, as I see it, as to which methods could work. But I am unsure about how to implement them and what programs to use for it.

First one is to log data to a file on the PC/RPI and then have some shell script either listen for changes and then combine a Json string with the newly added info to the file and send it of to the webservice.

Second one is to somehow take the Serial data from the Arduino and have some shell script or program listen for any data coming from the Arduino and converting it to Json, and sending it of to the webservice. The pro's by this method would be omitting the extra "file-step", but it would make the system more vulnerable to any interruptions in the connection.

Does anyone have experience with doing something like this? Or maybe have an idea how I can do what I'm trying to do ? Or just a nudge in the right direction, as to which approach you think is best, or what programs I could/should use. Any help is very much appreciated!

Best regards Daniel

Answer 1

I'm kind of tackling the same thing at the moment.

I have been using MQTT for a while for things like this. Including on an Netduino, an Arduino, and today a LEGO NXT.

But it is TCP based. I have been wanting to use serial, not TCP (ethernet) for a while, and when the need came for the NXT, it was time to write my own bridge to existing code.

I runs on the same host as the MQTT broker, but listens to a serial port, and mirrors the data to a TCP MQTT broker.

The code is turning out to be much simpler than I expected, I should have done this months ago.

Nxt Side

while (true)
{
    float x, y, z;
    DIMUreadGyroAxes(IMU, x, y, z);

    char json[128];
    sprintf(json, "PUBnxt/Gyro,{\"x\":%f,\"y\":%f,\"z\":%f}\n", x, y, z);
    nxtWriteRawHS(json, strlen(json), 0); // wireless serial over xbee
    wait1Msec(50);
}

PC side

 void Serial_DataReceived(object sender, SerialDataReceivedEventArgs e)
    {
        SerialPort p = sender as SerialPort;
        while (p.BytesToRead > 0)
        {
            string line;
            try
            {
                line = p.ReadLine();
            }
            catch (IOException)
            {
                throw;  // +++timeout
            }
            // parse it, it should be CMDtopic[,payload]
            if (line.Substring(0, 3).Equals("PUB"))
            {
                int commaIdx = line.IndexOf(',', 3);
                string topic = line.Substring(3, commaIdx - 3);
                string payload = line.Substring(commaIdx + 1);
                Console.WriteLine("{3}: {0}, publish topic({1}) payload({2})", p.PortName, topic, payload, DateTime.Now.ToLongTimeString());
                Mqtt.Publish(topic, System.Text.Encoding.UTF8.GetBytes(payload));

            }
            else if (line.Substring(0, 3).Equals("SUB"))
            {
                string topic = line.Substring(3);
                Console.WriteLine("{0}, subscribe topic({1})", p.PortName, topic);
                throw new NotImplementedException("MQTT Subscribe not implemented yet");
            }
            else
            {
                //Debugger.Break();   // ignore it, framing error
                Console.WriteLine("{0}, Data dropped", p.PortName);
            }
        }
    }

You might also want to look at data.sparkfun.com. A similar, complete concept, already existing.