Sending textual and nontextual data simultaneously across a serial port

Question

I am currently using an Arduino Uno and creating an LCD simulator that works by sending LCD data through the hardware UART (serial interface). This LCD data is very short in length; in this case, it's 2 bytes. The LCD simulator has already been made, and it is capable of detecting these LCD packets on the serial port and using them to control the simulated LCD.

However, I don't want to disturb the other traffic passing through the serial port; in this case, it's just text. The problem is that most serial monitors are able to pick up the LCD packets, which aren't supposed to be printed out because they appear as gibberish.

For now, I'm prefixing these short LCD packets with two magic numbers for it to be detectable by the LCD simulator alongside non-LCD traffic, and perhaps removable by a modified serial monitor.

I thought about using ASCII control characters to inform regular terminals that following data must not be printed, but there does not seem to be a specific control character to "hide" text/data; the closest thing there is is the BS character. All other control characters seem to give an inconsistent result among terminals - i.e. ESC doesn't actually hide text with the Windows console host, but it does with mintty.

My goal is to not have to integrate my own serial monitor and force others to use it if there is traffic present that can only be read by the LCD simulator. I want to allow others to use their own terminals and serial monitors to exchange textual data with the Arduino while seamlessly having LCD packets being sent invisibly and "under the hood."

Is this possible? Can I mix different types of data across the same serial port? If not, what are my options, or am I essentially forced to "reinvent the wheel" by making my own protocol that can mix the two types of traffic?

Answer 1

The idea that sundry terminal programs can cleanly deal with binary data intended for your LCD simulator doesn't seem realistic. Instead, look for a method to suppress the binary data when the Arduino isn't talking to an LCD simulator.

For example, during startup the Arduino could send some fixed text and a number to the host, and if the simulator is running, it would reply to the Arduino with some other fixed text and the 9's complement of the number. If the Arduino doesn't get such a reply within a couple of seconds, it turns off LCD-simulation data transmission.

Answer 2

I would simply send the data as ASCII, in the form of hex bytes. It wil make your two bytes become four, plus the two "leader" characters to uniquely identify them becomes six. This minor overhead can probably be absorbed.

If you have much more binary (non-ASCII) data, you could look at using a different encoding format such as Base64. This only has a 2 character overhead for every 6 bytes of binary data.

You don't mention what data rate you use, either in baud rate or "packets per second", but if you're reading it on a serial monitor it can't be very much.

Answer 3

If I were to do something like this on Linux, I would probably have the LCD simulator sit between the Arduino and the serial monitor, like this:

                    +---------------------------------------------------+
                    |                    host computer                  |
+-------------+     | +-------------------+         +-----------------+ |
|             |     | |                   |         |                 | |
| Arduino  UART-----tty  LCD simulator    ptmx----pts  Serial monitor | |
|             |     | |                   |         |                 | |
+-------------+     | +-------------------+         +-----------------+ |
                    +---------------------------------------------------+

For this, you have to:

1. open the serial port connected to the Arduino (“tty” on the figure)
2. create a pair of pseudo-terminals (ptmx/pts)
3. open the master side of the pair (ptmx)
4. ask the user to open the slave side (pts) with his serial monitor
5. forward all traffic but the LCD control messages between the Arduino and the pseudo-terminal.

I don't know how this works on MacOS, but since the whole pseudo-terminal concept is a standard Unix feature, I would expect it to work in a similar way on MacOS and Linux. But then I have no idea how hard it would be to port this to Windows.