TLDR
I need to convert text from a website into UTF-16LE format so that I can get the proper MD5 checksum but can't figure out how to go about doing that. This is all happening on an Arduino to log in to a router.
Background
I want to read values from a smart home device connected to a Fritz!Box router using an Arduino with an ethernet connection. The router has an open API and I'm using HTTP (EN|DE) to get the values. C++ programming is not my strong point.
I have been basing my attempts on BASH and javascript examples. For MD5 calculation I'm using tzikis' ArduinoMD5 library.
Steps
Based on the manual, you need to do the following:
- Contact the router to get a challenge string (router sends XML)
- Generate a response by calculating an MD5 checksum using the challenge string and password:
<challenge>-<password>
- Send the response,
<challenge>-<response>
, via POST - Receive the XLM answer from the router containing the SID
- Use SID to request data or control a smart home device
I'm stuck on step 2. I have the challenge but cannot calculate the correct checksum.
The manual states:
The MD5 hash is generated from the byte sequence of the UTF-16LE coding of this string (without BOM and without terminating 0 bytes).
Attempts
So far I can request the challenge: 70067288 for example. The reponse appears to always be alphanumeric. My password is also alphanumeric.
unsigned char* hash2 = MD5::make_hash("1234567z-äbc");
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);
This was my attempt at confirmation. The manual gave 1234567z
as a challenge example with äbc
(the a in the example does indeed have umlauts) as a password. The response shown for this example was 1234567z-9e224a41eeefa284df7bb0f26c2913e2
but my checksum from the above code was 935fe44e659beb5a3bb7a4564fba0513
.
I tried "manually" generating UTF-16LE and calculating the MD5 but that didn't work either.
byte rawTest[] = {0x3100, 0x3200, 0x3300, 0x3400, 0x3500, 0x3600, 0x3700, 0x7a00, 0x2d00, 0xe400, 0x6200, 0x6300};
char buffer[12] = {};
unsigned char* hash2 = MD5::make_hash(&buffer[0]);
char *md5str2 = MD5::make_digest(hash2, 16);
free(hash2);
Serial.print("MD5: ");
Serial.println(md5str2);
It gave me d41d8cd98f00b204e9800998ecf8427e
.
I am not entirely sure the example given was correctly calculated. Using their example in the API documentation with BASH code from the site mentioned above, I get a different answer: 1234567z-bb6e5b7c7d4d485590f4e084ad3da989
.
#!/bin/bash
# -----------
# definitions
# -----------
FBF="http://192.168.178.1/"
USER="root"
PASS="äbc"
AIN="AINOFYOURFRITZDECTDEVICE"
# ---------------
# fetch challenge
# ---------------
CHALLENGE="1234567z"
# -----
# login
# -----
MD5=$(echo -n ${CHALLENGE}"-"${PASS} | iconv -f ISO8859-1 -t UTF-16LE | md5sum -b | awk '{print substr($0,1,32)}')
RESPONSE="${CHALLENGE}-${MD5}"
echo $RESPONSE
Any help getting it working is greatly appreciatet.
Working Example (with caveats)
Requirements:
- Password using only characters found in ASCII
- Password at least 17 characters long
The password must use only ASCII characters due to the simple way it's being converted to UTF-16LE.
I can't explain why the password must be at least 17 characters long though. My code checks the length of the actual password programmatically.
With a password at least 17 characters long I get something like this:
Challenge: 6c607ee5
Challenge Pass (26): 6c607ee5-01234567890123456
Challenge Pass Preallocation Length: 26
MD5 (UTF-16LE): 3b26241ae4aab8eaf71d5c1599932178
Any password short and I get something like this:
Challenge: 621611e1
Challenge Pass (26): 621611e1-0123456789012345�
Challenge Pass Preallocation Length: 25
MD5 (UTF-16LE): 30a163ab8a267adfbb524b2b7412e81b
Please excuse any poor coding, C++ is not my strongpoint.
void fritzboxSID() {
EthernetClient fritzBox;
const char* pass = "01234567890123456"; // 17 ASCII character minimum
const char* user = "fritz3456"; // Account on router, randomly generated or user created
const size_t passLength = strlen(pass); // Password length
char c;
uint8_t xml = 0;
char challenge[9] = {0}; // Challenge
char challengePass[8 + 1 + passLength]; // challenge-password
uint8_t challengeCount = 0;
if (fritzBox.connect("192.168.200.1", 80)) {
// Send login request
fritzBox.println("GET /login_sid.lua HTTP/1.1");
fritzBox.println("Host: 192.168.200.1");
fritzBox.println("Connection: close");
fritzBox.println();
while (fritzBox.connected()) {
while (fritzBox.available()) {
c = fritzBox.read(); // Read a single character from the router response
if (c == '>') {
xml++;
}
if (xml == 5) {
// Challenge starts after the fifth >
if (challengeCount > 0 & challengeCount < 9) {
// Save the challenge, character by character
challenge[challengeCount-1] = c;
challengePass[challengeCount-1] = c;
}
challengeCount++;
}
Serial.print(c); // Print HTTP response
}
}
challengePass[8] = '-';
for (size_t i = 0;i < passLength; i++){
// Copy password into the combined challenge-password string
challengePass[i+9] = pass[i];
}
// Show challenge and pass, checking lengths
// If the strlen(challengePass) doesn't equal the preallocation length then the MD5 will be incorrect
Serial.println("");
Serial.print("Challenge: ");
Serial.println(challenge);
Serial.print("Challenge Pass (");
Serial.print(strlen(challengePass));
Serial.print("): ");
Serial.println(challengePass);
Serial.print("Challenge Pass Preallocation Length: ");
Serial.println(8 + 1 + passLength);
// Convert to UTF-16LE (only works for standard ASCII characters)
const size_t length = strlen(challengePass);
char buffer[2*length];
for (size_t i = 0; i < length; i++) {
buffer[2*i] = challengePass[i];
buffer[2*i+1] = 0;
}
// Generate MD5
unsigned char* hash = MD5::make_hash(buffer, 2*length);
char *md5str = MD5::make_digest(hash, 16);
free(hash);
Serial.print("MD5 (UTF-16LE): ");
Serial.println(md5str);
} else {
Serial.println("Cannot connect to 192.168.200.1");
}
}
If it helps, the response from the router for the initial login request is this:
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/xml
Date: Fri, 16 Sep 2022 08:21:27 GMT
Expires: -1
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; connect-src 'self'; font-src 'self'; frame-src https://service.avm.de https://help.avm.de https://www.avm.de https://avm.de https://assets.avm.de https://clickonce.avm.de http://clickonce.avm.de http://download.avm.de https://download.avm.de 'self'; img-src 'self' https://tv.avm.de https://help.avm.de/images/ http://help.avm.de/images/ data:; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; frame-ancestors 'self'; media-src 'self'
<?xml version="1.0" encoding="utf-8"?><SessionInfo><SID>0000000000000000</SID><Challenge>621611e1</Challenge><BlockTime>0</BlockTime><Rights></Rights><Users><User last="1">fritz3456</User><User>UserName</User></Users></SessionInfo>
-f ISO8859-1
looks suspicious. Are you sure you are using this old, legacy character set? Your bash script gives me the expected output if I remove that option. 2. Can you manage to have an ASCII-only password? If so, the translation to UTF16LE would be practically trivial.byte rawTest[] = {0x3100, 0x3200,.....
this is not doing what you think it does. I'm assuming you weren't looking for a creative way to fill your array with 0x00 values. Probably you meant0x31, 0x00, 0x32, 0x00,.....