How do I properly trigger the ATMega328 reset pin via software so that the bootloader runs and the firmware can be updated via RX/TX pins?

Question

I am programming my Atmega328P chip via a USB to TTL converter (https://www.amazon.com/gp/product/B075N82CDL/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1)

I am able to upload hex files via the command line and the Arduino IDE when I have the RTS pin of the adapter and a capacitor attached to the !RESET pin of the Atmega. (http://www.electronics-lab.com/wp-content/uploads/2018/09/programming-Uno-on-breadboard.png)

I am also able to upload via the command line when I have a push button attached to the !RESET pin and manually pull it low then begin uploading the firmware within a few moments. (https://cdn.instructables.com/F0L/2F6U/J7AQSHD1/F0L2F6UJ7AQSHD1.LARGE.jpg)

My problem is that for my application I will only have the TX/RX/Vcc/GND pins available (no RTS or push button) and need to reset the micro controller so the bootloader runs and listens to be programmed with updated firmware.

So far I have written a batch file that sends a command to restart the Arduino, and then calls avrdude to upload a hex file. I have tried two methods of restarting the micro controller. I have told it to go to memory location zero, and I have attached an IO pin to the reset and dropped it low (although it stays low until reset, so the reset doesn't really work until I manually remove the wire between the IO pin and the !RESET pin)

My sequence is as follows:

1. Have Arduino running code:

    int recvByte;
    int delayS = 400;
    int resetPin = 9; //bttom right pin of uC

    void setup() {
      // put your setup code here, to run once:
    pinMode(txPin, OUTPUT);
    pinMode(rxPin, INPUT);
    pinMode(resetPin, OUTPUT);
    digitalWrite(resetPin, HIGH);
    Serial.begin(57600);
    Serial.write("Booted/reset");
    Serial.write("This setup was uploaded with batch");    
    }

    void(* resetFunc) (void) = 0;

    void loop() {

    if (Serial.available() )
    {    // get incoming byte:
        recvByte = Serial.read();
        if (recvByte==int('r'))
        {
        Serial.write("resetting");
        delay(10); //wait for last message to send
        //digitalWrite(resetPin, LOW); //commented out because this method stays low until wire is removed
        resetFunc();
        Serial.write("should not see if reset properly");
        }

    }

    delay(100);
    }

2.I then run my batch file.

echo Running upload bat, sending reset command then calling avrdude
cd "C:\Program Files (x86)\Arduino\hardware\tools\avr\bin\"
python "path\sendR.py"
.\avrdude.exe -C"C:\Program Files (x86)\Arduino\hardware\tools\avr/etc/avrdude.conf" -v -patmega328p -carduino -PCOM6 -b57600 -D -Uflash:w:"path\Atmega328code.hex":i 

The python sendR.py is to send the "r" character, which the Arduino views as the reset command.

import serial
ser = serial.Serial("COM6", baudrate=57600)  # open serial port
bytes = "r".encode()
ser.write(bytes)
ser.close()
print("End python script")

When I send "r" via Tera Term or python the setup code runs again, but I have not been able to upload with the .\avrdude.exe -C"C:\Program Files (x86)\Arduino\hardware\tools\avr/etc/avrdude.conf" -v -patmega328p -carduino -PCOM6 -b57600 -D -Uflash:w:"path\Atmega328code.hex":i command inside the batch file. If I use the pushbutton to do a hardware reset, this command works without issue.

How can I upload new firmware via the RX and TX pins using software to reset the microcontroller and listen for new code as the bootloader runs?

New ideas tried: Use a uniform baud rate of 57600 in python and atmega serial setup. I chose this value because the Arduino IDE output this baudrate in avrdude.exe -C"C:\Program Files (x86)\Arduino\hardware\tools\avr/etc/avrdude.conf" -v -patmega328p -carduino -PCOM6 -b57600 -D -Uflash:w:"path\Atmega328code.hex":i

Answer 1

You wrote:

void(* resetFunc) (void) = 0;

This seems to me like an approach worth trying, except that the function address is wrong. At address zero of the flash you have the start of the user's code. The bootloader, if you do use one, lives at the end of the flash. The start address of the bootloader depends on its size, and is configured by setting some fuses on the chip. If you are using the popular optiboot (the bootloader of the Uno), you have to jump to exactly 512 bytes before the end of the flash, i.e.:

void (*resetFunc)(void) = 0x7e00;

Oh, wait... the flash is organized as 16-bit words, so you may have to use the wordwise address instead:

void (*resetFunc)(void) = 0x3f00;

But do note that, unlike a real hardware reset, this approach does not reset the peripherals to their normal initial state. You may want to do it before jumping to the bootloader.

---

Edit: I have disassembled a test program that does this kind of call-by-pointer, then double checked the documentation of the icall instruction. It appears that the function address you provide is indeed interpreted as a word-wise address, so you should write half the byte-wise address. If you write anything larger than the size of the flash (0x4000 = 16 kilowords), the most significant bits get lost, and the address gets reduced to the correct range modulo the flash size.

That is, if 0x7800 worked for you, then presumably the correct address is 0x3800, i.e. 2 kilowords (4 KB) before the end of the flash. This is quite a large bootloader, and the largest size supported by the ATmega328P. Are you sure it is righ? Aren't you sliding through some uninitialized flash before hitting the bootloader? Here is a table that gives the bootloader's start address as a function of the amount of memory reserved for it:

reserved size  start address
    512 B         0x3f00
     1 KB         0x3e00
     2 KB         0x3c00
     4 KB         0x3800

Note that the Arduino Uno uses optiboot, which fits into the smallest size that can be allocated to a bootloader (512 bytes).