How to program Arduino manually via STK500?

Question

I am trying to program an Arduino Nano (with an ATmega328P) from an ESP32 that is attached to its UART and the reset pin. I have to do this via the ESP, because the Arduino is supposed to run on its own, detached from a PC (where I could use an existing programmer for it). But its program should be changable, hence, it is attached to the ESP, which is accessible from a web server via Wifi, so that the Arduino can be reprogrammed on request. I almost got the whole setup figured out: The server is running, files can be sent to the ESP, and the ESP can talk with the Arduino Bootloader. The only problem is that I cannot get the Arduino to execute new programs.

The whole procedure goes like this: The server communicates with the ESP, sends the HEX file of the Arduino sketch (which is successfully verified by a checksum, so this part is working properly). Then, the ESP pulls the reset pin of the Arduino and waits for a while before it attempts to talk to its bootloader. Then, it basically does the same sequence of STK500 commands as avrdude.

I really just copied the commands from the avrdude log after executing

avrdude -c arduino -vvvvv -p m328p -b 57600 -P /dev/ttyUSB0 -U flash:w:sketch.hex:i

and skipping query commands like which firmware version it uses etc. So, basically: enter programming mode, erase chip, re-enter programming mode, write the hex file to flash, starting from address 0, moving in steps of 128 bytes (using 1's to pad the last message to the full size of 128 bytes). Then afterwards reading back the file from the Arduino and veryfing that all bytes match (which they do!) and leaving the programming mode (which is the very last thing that happens in the avrdude log). Afterwards, I trigger the reset. I am using exactly the same type of file which I gave to avrdude, and all the responses of the Arduino bootloader exactly match the values from the avrdude log.

But whereas the Arduino just starts executing the sketch after this procedure when using avrdude, in my own implementation it just blinks its LED in a regular interval (when the test sketch should make it blink short, long, short, long etc.). I don't get what's going wrong here. I thought that the blinking might represent an error code, but I couldn't find any documentation about that. I feel like I've searched the whole Internet (including arduino stackexchange), and I even looked at the avrdude source code to check if there's something else done after the last STK500 command, that is not listed in its log. I feel (hope) that I'm very close to having it work and this whole thing just needs like a one-line fix, but I don't know what else to try anymore and would greatly appreciate a second opinion from someone more experienced.

Answer 1

Okay, by comparing the data packets my implementation sends with those from the avrdude log for the same sketch, I finally found the cause of trouble. I was sending the hex file byte-by-byte exactly as it is read from storage. But of course, the metadata needs to be skipped (i.e. the addresses at the beginning of each line, the magic bytes at the end of each line, and the newline). I added this simple fix, and now it works fine.

EDIT: Here is the preprocessing code from my Python3 script:

import codecs

program = b''
for line in open('sketch.hex', 'rb'):
    program += codecs.decode(line[9:-4], 'hex')