Cannot upload to otherwise working bootloaded ATmega328P via RS232: “programmer is not responding”

Question

I have an ATmega328P on a breadboard with the following circuitry around it: The ISP is a Tiny AVR Programmer. The RS232 to USB converter is this thing.
Following is the test program I use:

void setup() {
    Serial.begin(28800);
    DDRC  = 0b00000001;
    PORTC = 0b00000000;
}

void loop() {
    PORTC = 0b00000001;
    delay(300);
    PORTC = 0b00000000;
    delay(300);

    int Data;
    char Text[2] = {0};
    while (true)
    {
        Data = Serial.read();
        if (Data == -1) break;
        Text[0] = 0xFF - Data;
        Serial.write(Text);
    }
}

Almost everything is working fine:

• I can upload the program via ISP by holding the shift-key while pressing the upload-button in Arduino IDE.
• The LED on Pin 23 (PC0) blinks.
• When I press the reset button the LED stops blinking and when I release the button it starts blinking again, so resetting the chip works.
• When I open a serial monitor (at 28800 baud and not the one integrated in Arduino IDE) I can send 00, 01, 02, 03, ... and I receive FF, FE, FD, FC, ... back, so the serial communication works.
• I can burn the bootloader from Arduino IDE. Every time I reset the chip the LED on pin 19 blinks three times in rapid succession and then stays off.

But I can't upload the program through the serial connection once the bootloader is burned. Arduino IDE shows the following output:

C:\Portable Program Files\arduino-1.8.5\hardware\tools\avr/bin/avrdude -CC:\Portable Program Files\arduino-1.8.5\hardware\tools\avr/etc/avrdude.conf -v -patmega328p -carduino -PCOM6 -b115200 -D -Uflash:w:C:\Users\Niko\AppData\Local\Temp\arduino_build_706815/BlinkTest.ino.hex:i 

avrdude: Version 6.3, compiled on Jan 17 2017 at 12:00:53
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "C:\Portable Program Files\arduino-1.8.5\hardware\tools\avr/etc/avrdude.conf"

         Using Port                    : COM6
         Using Programmer              : arduino
         Overriding Baud Rate          : 115200
avrdude: stk500_recv(): programmer is not responding
avrdude: stk500_getsync() attempt 1 of 10: not in sync: resp=0xe0

(And of course the 9 other attempts follow.)
I am not sure what to do here. Maybe the baudrate is incorrect? The Arduino website says that Arduino bootloaders use 19200 baud. But executing the same command in a console but changing the -b115200 to -b19200 yields the same result. I also tried 9600.
I make sure to to reset the chip before trying to upload, though I can't really say whether I get the timing right.

Something I noticed is that The Arduino website says that "Burning the bootloader may take 15 seconds or more", but burning the bootloader takes less than a second for me. Seems unusually fast but I can't see any errors in the output.

I also find it very weird that the command executed when trying to upload via serial connection contains -carduino, which would imply that it's trying to use a programmer to upload the program. The bootloader should make that unnecessary.

Any ideas?