On microcontrollers with bootloader (not just AVRs), the bootloader is a separate executable in flash, and it's typically the program that runs on system startup. The application will have a special trigger, such as a button sequence, getting a specific byte sequence over I2C, or really anything that will tell the bootloader to enter into its "update" mode. From there, the bootloader will have some communication (I2C, USB, WiFi, serial, etc) to download and the flash the new application image.
This link has more information about the Arduino bootloader
On the PC side, to trigger a reset on the RESET pin, the PC (avrdude.exe or GNU/Linux equivalent) opens the serial port to Arduino when your press upload and the code is ready to upload. This causes the Data Transmit Ready (DTR) line of the USB/TTL chip to go LOW. The Arduino board has a capacitor charging circuit that uses this LOW (charging the capacitor) signal to momentarily pull down the RESET line of the ATMEGA328P chip before returning it to HIGH (capacitor charging completes). So Arduino resets each time its serial port is opened.
Upon reset, Arduino enters the bootloader.
The bootloader looks at the source that caused the reset. There are several sources that can cause a reset. If the reset was caused by the RESET pin, then it waits for one second for the PC to send in commands. When it receives valid commands, it will start accepting new Arduino code in HEX format and erase the existing code to load new one. If it doesn’t receive valid commands, it times out after one second and triggers a Watch Dog Timer (WDT) reset.
For Arduinos with built-in USB, such as the Micro, the Arduino library (firmware) triggers the system reset. There's a little chunk of code that looks for the virtual COM port to open at 1200 baud and then close. When that happens, the code triggers a system reset (by writing a register, typically) which launches the bootloader.
Rather than requiring a physical press of the reset button before an upload, the Micro board is designed in a way that allows it to be reset by software running on a connected computer. The reset is triggered when the Micro's virtual (CDC) serial / COM port is opened at 1200 baud and then closed. When this happens, the processor will reset, breaking the USB connection to the computer (meaning that the virtual serial / COM port will disappear). After the processor resets, the bootloader starts, remaining active for about 8 seconds. The bootloader can also be initiated by pressing the reset button on the Micro. Note that when the board first powers up, it will jump straight to the user sketch, if present, rather than initiating the bootloader.
Because of the way the Micro handles reset it's best to let the Arduino Software (IDE) try to initiate the reset before uploading, especially if you are in the habit of pressing the reset button before uploading on other boards. If the software can't reset the board, you can always start the bootloader by pressing the reset button on the board.
Personal story: I wrote an I2C bootloader for the PIC32. The only communication with the device was through the I2C bus. The main application had a special 3 byte trigger that would set a non-initialized variable in RAM, and then trigger a system reset. The bootloader, which runs on system startup, would check that variable and if it was set, would wait for commands over I2C to download and flash the new application image. That variable in RAM had to be specially marked as non-initialized, otherwise the C startup code would clear it and the bootloader would never update.
The bootloader also calculated a CRC32 over the application image to validate it before attempting to boot. If the calculated CRC32 didn't match the stored CRC32 (which was part of the application image), then the bootloader would enter its firmware update mode. This was a guard against trying to run a corrupted firmware image, which could happen if power was pulled during the update process.