Most computers begin execution at (or jump indirect through) location zero on startup. There may be variations, but this is the general idea. Whatever is put at location zero when the bootloader is installed causes the bootloader to gain control on a power-up or a reset. After that, it is up to the bootloader what happens next.
What usually happens is that the bootloader watches the serial port for the start of a "handshake" sequence; when (if) it sees what it is looking for, it replies in a manner that lets the loader program, e.g., avrdude, know that this is the bootloader replying and not some random text that was being sent just at that moment. (If the bootloader doesn't see the the handshake sequence begin within a short period, or it sees something other than the handshake, it starts the existing user program).
From there, the loader program and the bootloader cooperate to transfer the user program to the Arduino and load it into Flash memory at specified locations. Also transferred along with the user program is its starting address. When the upload is finished and verified, the bootloader transfers to the user program's starting address.
At this point, the user program owns the machine, including the serial port(s), and will continue to do so until another power-cycle or a reset. There is no need for the user program and the bootloader to negotiate use of the port; the bootloader owns it from reset to end-of-upload, and the user program owns it thereafter.
However at that point (while the user program is running) I can start
an upload (from the IDE or avrdude).
This is possible only because avrdude can send a hardware signal (not contained in the data stream) that the Arduino board uses to cause the chip to reset. It is still the reset process that transfers control from the running program to the bootloader. As an example, on a number of my hand-built boards I did not include that auto-reset circuit. On those boards, avrdude cannot begin a new upload autonomously; I have to manually reset them when I start avrdude.