It is possible, with some really clever code, to exploit the bootloader to overwrite itself.
The solution is actually conceptually fairly simple. A bootloader, by
its very nature is designed to download new firmware to the device.
Therefore it will contain at least one spm instruction. Because the
spm configuration register must be written no more than 4 cycles
before the spm instruction it means there are very few sequences that
practically occur: just sts, spm or out, spm sequences. So, all you
need to is find the sequence in the bootloader section; set up the
right registers and call it.
However, it turned out there was a major problem with that too. The
V-USB self-programming bootloader's spm instructions aren't a neat
little routine, but are inlined into the main code; so calling it
would just cause the AVR to crash as it tried to execute the rest of
the V-USB bootloader.
Nasty, but again there's a solution. By using a timer clocked at the
CPU frequency (which is easy on an AVR), you can create a routine in
assembler which sets up the registers for the Bootloader's out, spm
sequence; calls it and just at the moment when it's executed the first
cycle of the spm itself, the timer interrupt goes off and the AVR
should jump to your interrupt routine (in Application space). The
interrupt routine pops the bootloader address and then returns to the
previous code - which is the routine that sets up the out, spm
sequence. This should work, because when you apply spm instructions to
the bootloader section the CPU is halted until it's complete.
So basically, you can do some fancy programming, and get the bootloader to hose itself, rendering the MCU useless without a dedicated programmer.