The Arduino can be programmed using SPI without a boot loader. Without a boot loader, the Arduino starts running its sketch right away. So, how does the Arduino know that it is being programmed via SPI vs regular SPI traffic already present on the SPI bus? Was is the pin-level, electrical signal that let's the Arduino know the difference?
In-system programmer is activated with RESET line. During normal operation reset line is not active and ISP module is disabled.
You can find detailed description of AVR ISP in this document: