MQTT bridging with Arduino and ESP8266 - What software should I use?

Question

Sorry for the vagueness in my question, I am just looking for ideas to point me in the correct direction please...

I wish to connect many (read: 20+) different sensors to a local wifi network and have them all datalogged and controlled by a single server on the network. Pretty standard stuff for the home automation people etc. Some sensors are connected to esp8266 boards directly, others communicate via serial and will employ an 8266 to provide wifi.

I am having trouble deciding which direction to go or which software / firmware to combine since its changing rapidly and a fair number of tutorials are around but all have different approaches and end goals.

This must be entirely contained offline - I have a raspberry pi, a wireless router, and the various sensors (mostly arduino based).

Now from research it would seem I need to setup a serial to MQTT 'bridge' - and have the esps 'talk' to the pi by MQTT. And that the pi should be running 'mosquitto' if I wish the MQTT to be bi-directional (i.e. I can receive serial data AND send serial commands. I have never dealt with MQTT so I am trying to learn as I go.

I have been working for a time with 'ESP-Link' software and had success using it as a simple serial to tcp-ip bridge and using virtual serial ports to communicate with the attached device. However using virtual serial ports on the server is clunky, unreliable and not what I wish to do - does anyone know if it can be setup so I can send 'raw' serial messages over TCP to the server for use? And from the server to control the connected devices? Or if there is an easier way? I cannot modify all of the serial devices I wish to connect, so just want to get the data out/in and have it manipulated by the server where necessary.

In essence I wish to end up with several sensors all connected to the router, and logged / controlled through a web interface hosted on the PI, all 'offline' and contained. Eventually I would like the server to be able to recognize what devices it can 'see' automatically, however at this moment I simply wish to gain insight into how I can most easily achieve this? Or even if I am completely mistaken in my approach?

Thanks

Answer 1

MQTT is perfectly suited to this application. Here is an overview of a possible configuration. You should be able to find tutorials about each piece of the system.

On the Raspberry Pi:

• Setup a Mosquitto server that can receive messages published by the ESP devices.
• Write and run a MQTT client on the RPi that subscribes to the messages published by the ESP devices. This client would then store the data in some sort of permanent data store (database, flat files, whatever).
• Write and run on the RPi a web app that reads and presents the data from the data store.

On the ESP devices:

• Connect to the same wireless network as the RPi (or set up the RPi as an access point and have the devices connect to it)
• Read data from the sensors and publish it to the MQTT server on the RPi.

If you want to push data to the ESP devices -- have them subscribe to certain messages (topics) on the MQTT server and then publish messages in those topics on the server. Your web app could do this.

All of this can be done on an isolated network.

Answer 2

As you also asking about hardware I would recommend the following setup:

For sensors that only need to send data I would go with a ATtiny85. It has very low power consumption, is very small and easy to program. If you use 10-N of them then a programming board and only the micro controller will be cheaper and even smaller.

For sensors that have some logic on it, maybe multiple attached sensors I would go with a Wemos D1 Mini. It has build in WIFI, you can debug it, it is still very small and there are a ton of extension (shield) boards for it to speed up your development.

For the communication part I would go away from WIFI since 20+ devices can generate quiet some traffic. Also WIFI costs you a lot in terms of power consumption since it is not optimized for IoT (small data packages). I would add a 433mhz Mini Low Power RF Receiver/Transmitter to my IoT hardware and communicate through that. It has a long range and has a very small power footprint.

Additionally you could think about using a Wemos as MQTT gateway between your 433mhz devices and the WIFI instead of your proposed Raspberry Pi solution. The nice benefit is that you will have a dedicated device managing an MQTT gateway that is far less error-prone as a Linux server (Raspberry Pi) and you don't need to update as often as with the Pi to keep it secure. There is a Wemos D1 Mini Pro that has a better WIFI antenna and a little bit more power that could be perfect for this use case.

The MQTT protocol is perfect for a sensor network since it is optimized for low bandwidth, high latency, and fragile connections. It supports Quality of Service (QoS) and you can configure what shall happen if a device disconnects/is not reachable anymore. There are a lot of library's already available for MQTT so that the setup is very quick. HiveMq has a nice how to get started with MQTT guide for further reading.