Extract binary data from text string. Like Perl pack, not just regexp

Question

If I have data coming into an Arduino in an ASCII serial format from some external device (or from a PC, or whatever) and I want to extract binary values from that data and log it or put it on a little display or react to it or whatever there are a few ways I can do it. But it seems like there should be a reversed version of sprintf, mashed up with regexp, mashed up with Perls pack/unpack that would make it easy and configurable.

For example, let's say I'm getting hex numbers from a data logger. something like c,03,1234,5678,9ABC where c is just a code telling me this line is channel data, there are 3 channels installed, and then there are the 3 sets of data. I can use split or something like that and then extract the values, convert the hex to binary, apply Y=mX+B and display the channel readings. If a channel goes flat, I can sound an alarm, etc... No worries.

But then I have a different data logger plugged in, and this one uses a different format. Like decimal raw ADC counts. It spits out 2342 1313 234236 Now I gotta recode.

But wait, what if I just had a regexp in eeprom and I used that to extract the data? Something like /^c,\X+,(\X+),(\X+),(\X+)$/ for the first logger and /^(\d+) (\d+) (\d+)$/ for the second. That would be really cool because I could pretty much extract data from any serial device for local display. No need to reprogram, just update the regexp. And there is a nice regexp library: https://github.com/nickgammon/Regexp although it uses the LUA regexp syntax.

Oh, but that won't work... because although my program in the Arduino gets back 3 values from the regexp, it doesn't know that the first set are hex strings, and the second set are decimal.

Now, if I was doing this in Perl, I'd just apply a pack string to the binary data. But then pack can't reject a bad match, it always returns a result, even if the input wasn't valid.

Doesn't it seem like the regexp should be able to return /binary/ values instead of just the string? It knows those values are hex. And shouldn't pack be able to do matching against the other characters in the string and reject the value if it's not a match?

Seems to me like there is a hole in our standard libraries. Something that does both simple matches, and converts data to binary return values on the fly.

Or is that too much to ask from an engine? Perhaps few people would use it; better to just hardcode for the rare times you need something like this?

Answer 1

Something that does both simple matches, and converts data to binary return values on the fly.

You are describing sscanf(). It is way less powerful than regexps when it comes to pattern matching, but it seems to closely match your requirements.

Dummy example:

const int FORMAT_COUNT = 2;
const char * formats[FORMAT_COUNT] = {
    "c,%*x,%lx,%lx,%lx",  // first data logger
    "%lu %lu %lu"         // second data logger
};

// Here goes the data.
unsigned long channels[3];

// Try to parse using all the known formats.
void parse(const char *message)
{
    for (int i = 0; i < FORMAT_COUNT; i++) {
        int ret = sscanf(message, formats[i],
                &channels[0], &channels[1], &channels[2]);
        if (ret == 3) {
            Serial.print("Message \"");
            Serial.print(message);
            Serial.print("\" matched with format \"");
            Serial.print(formats[i]);
            Serial.println("\"");
            return;
        }
    }
    Serial.println("Could not parse message.");
}

Sample output:

Message "c,03,1234,5678,9ABC" matched with format "c,%*x,%lx,%lx,%lx"
Message "2342 1313 234236" matched with format "%lu %lu %lu"

Answer 2

Yes, but it doesn't really know something is hex. What about the patterns:

[\dA-F]
[0-9A-Fa-f]

And so on. The regexp library is simply a way of extracting out substrings of a target string. It doesn't make a heap of sense for it to try to auto-detect the type of the returned string. For example, a GUID might be a lot of hex digits. Do you want something like 182676d07497c511f850 to be somehow converted into binary? And then what? Compressed into an int?

In fact it doesn't return (decimal) numbers anyway, it returns a string which may or may not be convertible into a number. Chris Stratton's suggestion of using atoi, sttol and so on is the logical way to go. Just have a flag in your EEPROM that tells the decoding routine what format the input data is. Also, for simple string sequences, don't forget strtok.