Can't convert string to UTF-16LE for MD5 calculation on an Arduino

Question

TLDR

I need to convert text from a website into UTF-16LE format so that I can get the proper MD5 checksum but can't figure out how to go about doing that. This is all happening on an Arduino to log in to a router.

Background

I want to read values from a smart home device connected to a Fritz!Box router using an Arduino with an ethernet connection. The router has an open API and I'm using HTTP (EN|DE) to get the values. C++ programming is not my strong point.

I have been basing my attempts on BASH and javascript examples. For MD5 calculation I'm using tzikis' ArduinoMD5 library.

Steps

Based on the manual, you need to do the following:

1. Contact the router to get a challenge string (router sends XML)
2. Generate a response by calculating an MD5 checksum using the challenge string and password: <challenge>-<password>
3. Send the response, <challenge>-<response>, via POST
4. Receive the XLM answer from the router containing the SID
5. Use SID to request data or control a smart home device

I'm stuck on step 2. I have the challenge but cannot calculate the correct checksum.

The manual states:

The MD5 hash is generated from the byte sequence of the UTF-16LE coding of this string (without BOM and without terminating 0 bytes).

Attempts

So far I can request the challenge: 70067288 for example. The reponse appears to always be alphanumeric. My password is also alphanumeric.

unsigned char* hash2 = MD5::make_hash("1234567z-äbc");
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

This was my attempt at confirmation. The manual gave 1234567z as a challenge example with äbc (the a in the example does indeed have umlauts) as a password. The response shown for this example was 1234567z-9e224a41eeefa284df7bb0f26c2913e2 but my checksum from the above code was 935fe44e659beb5a3bb7a4564fba0513.

I tried "manually" generating UTF-16LE and calculating the MD5 but that didn't work either.

byte rawTest[] = {0x3100, 0x3200, 0x3300, 0x3400, 0x3500, 0x3600, 0x3700, 0x7a00, 0x2d00, 0xe400, 0x6200, 0x6300};
char buffer[12] = {};
unsigned char* hash2 = MD5::make_hash(&buffer[0]);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

It gave me d41d8cd98f00b204e9800998ecf8427e.

I am not entirely sure the example given was correctly calculated. Using their example in the API documentation with BASH code from the site mentioned above, I get a different answer: 1234567z-bb6e5b7c7d4d485590f4e084ad3da989.

#!/bin/bash
# -----------
# definitions
# -----------
FBF="http://192.168.178.1/"
USER="root"
PASS="äbc"
AIN="AINOFYOURFRITZDECTDEVICE"
# ---------------
# fetch challenge
# ---------------
CHALLENGE="1234567z"
# -----
# login
# -----
MD5=$(echo -n ${CHALLENGE}"-"${PASS} | iconv -f ISO8859-1 -t UTF-16LE | md5sum -b | awk '{print substr($0,1,32)}')
RESPONSE="${CHALLENGE}-${MD5}"
echo $RESPONSE

Any help getting it working is greatly appreciatet.

---

Working Example (with caveats)

Requirements:

• Password using only characters found in ASCII
• Password at least 17 characters long

The password must use only ASCII characters due to the simple way it's being converted to UTF-16LE.

I can't explain why the password must be at least 17 characters long though. My code checks the length of the actual password programmatically.

With a password at least 17 characters long I get something like this:

Challenge: 6c607ee5

Challenge Pass (26): 6c607ee5-01234567890123456

Challenge Pass Preallocation Length: 26

MD5 (UTF-16LE): 3b26241ae4aab8eaf71d5c1599932178

Any password short and I get something like this:

Challenge: 621611e1

Challenge Pass (26): 621611e1-0123456789012345�

Challenge Pass Preallocation Length: 25

MD5 (UTF-16LE): 30a163ab8a267adfbb524b2b7412e81b

Please excuse any poor coding, C++ is not my strongpoint.

void fritzboxSID() {
  EthernetClient fritzBox;
  const char* pass = "01234567890123456";  // 17 ASCII character minimum
  const char* user = "fritz3456";  // Account on router, randomly generated or user created
  const size_t passLength = strlen(pass);  // Password length
  char c;
  uint8_t xml = 0;
  char challenge[9] = {0};  // Challenge
  char challengePass[8 + 1 + passLength];  // challenge-password
  uint8_t challengeCount = 0;
  
  if (fritzBox.connect("192.168.200.1", 80)) {
    // Send login request
    fritzBox.println("GET /login_sid.lua HTTP/1.1");
    fritzBox.println("Host: 192.168.200.1");
    fritzBox.println("Connection: close");
    fritzBox.println();
    
    while (fritzBox.connected()) {
      while (fritzBox.available()) {
        c = fritzBox.read();  // Read a single character from the router response
        if (c == '>') {
          xml++;
        }
        if (xml == 5) {
            // Challenge starts after the fifth >
            if (challengeCount > 0 & challengeCount < 9) {
                // Save the challenge, character by character
                challenge[challengeCount-1] = c;
                challengePass[challengeCount-1] = c;
            }
            challengeCount++;
        }
        Serial.print(c);  // Print HTTP response
      }
    }
    challengePass[8] = '-';
    for (size_t i = 0;i < passLength; i++){
      // Copy password into the combined challenge-password string
      challengePass[i+9] = pass[i];
    }
    
    // Show challenge and pass, checking lengths
    //      If the strlen(challengePass) doesn't equal the preallocation length then the MD5 will be incorrect
    Serial.println("");
    Serial.print("Challenge: ");
    Serial.println(challenge);
    Serial.print("Challenge Pass (");
    Serial.print(strlen(challengePass));
    Serial.print("): ");
    Serial.println(challengePass);
    Serial.print("Challenge Pass Preallocation Length: ");
    Serial.println(8 + 1 + passLength);

    // Convert to UTF-16LE (only works for standard ASCII characters)
    const size_t length = strlen(challengePass);
    char buffer[2*length];
    for (size_t i = 0; i < length; i++) {
      buffer[2*i] = challengePass[i];
      buffer[2*i+1] = 0;
    }

    // Generate MD5
    unsigned char* hash = MD5::make_hash(buffer, 2*length);
    char *md5str = MD5::make_digest(hash, 16);
    free(hash);
    Serial.print("MD5 (UTF-16LE): ");
    Serial.println(md5str);
  } else {
    Serial.println("Cannot connect to 192.168.200.1");
  }
}

If it helps, the response from the router for the initial login request is this:

HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/xml
Date: Fri, 16 Sep 2022 08:21:27 GMT
Expires: -1
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; connect-src 'self'; font-src 'self'; frame-src https://service.avm.de https://help.avm.de https://www.avm.de https://avm.de https://assets.avm.de https://clickonce.avm.de http://clickonce.avm.de http://download.avm.de https://download.avm.de 'self'; img-src 'self' https://tv.avm.de https://help.avm.de/images/ http://help.avm.de/images/ data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; frame-ancestors 'self'; media-src 'self'

<?xml version="1.0" encoding="utf-8"?><SessionInfo><SID>0000000000000000</SID><Challenge>621611e1</Challenge><BlockTime>0</BlockTime><Rights></Rights><Users><User last="1">fritz3456</User><User>UserName</User></Users></SessionInfo>

Answer 1

You have to convert the string "<challenge>-<password>" to UTF-16LE. If the whole string is plain ASCII, the conversion is trivial: you just have to add a zero byte after each ASCII byte.

// ASCII string to be hashed.
const char* text = "1234567z-abc";  // umlaut removed

// Convert to UTF-16LE.
const size_t length = strlen(text);
char buffer[2*length];
for (size_t i = 0; i < length; i++) {
    buffer[2*i] = text[i];
    buffer[2*i+1] = 0;
}

// Compute and print the hash.
unsigned char* hash2 = MD5::make_hash(buffer, 2*length);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);

Note that I use the two-parameter version of MD5::make_hash() in order to pass the length of the buffer. The one parameter version of this method would stop at the first zero byte.

---

Edit: Regarding your extended question (the new section “Working Example”), this is a problem completely unrelated to the previous one. It is about string termination.

In C and C++, strings are arrays of characters terminated with a NUL ASCII character (numeric value zero). The terminating NUL is not part of the string per se, but it should be stored in the array holding the string. String literals are interpreted by the compiler as NUL-terminated arrays of characters. Every function that expects a string as an input expects it to be NUL-terminated.

In this particular instance, you failed to NUL-terminate challengePass. When this array is given to strelen(), this function scans the memory until it finds a zero byte. If that zero byte happens to be right after the end of the array, your code will work as expected. Otherwise, any non-zero byte coming after challengePass in memory will be interpreted as being part of the string. The program behavior is unpredictable, because you do not know what will be stored in memory right after challengePass.

The solution is to NUL-terminate the string. Start by allocating one more byte:

char challengePass[8 + 1 + passLength + 1];  // challenge-password

Then, right after copying the password, add a zero byte to terminate the string:

challengePass[8 + 1 + passLength] = '\0';  // terminate the string

Note that '\0' means “the character with numeric value zero”.

Alternatively, you can terminate the string right after adding the dash, then use strcat() to concatenate the password. strcat() will take care of terminating the string:

challengePass[8] = '-';
challengePass[9] = '\0';  // temporarily terminate the string
strcat(challengePass, pass);  // append password