According to my program running on the chip, it acts as master for SPI peripherals. If using ICSP, how does the chip know that it is supposed to act as SPI slave and receive a program?
A special signal is sent using the RESET pin. From the datasheet (p361):
- Power-up sequence: Apply power between VCC and GND while RESET and SCK are set to “0”. In some systems, the programmer can not guarantee that SCK is held low during power-up. In this case, RESET must be given a positive pulse of at least two CPU clock cycles duration after SCK has been set to “0”.
- Wait for at least 20ms and enable serial programming by sending the Programming Enable serial instruction to pin MOSI.
- The serial programming instructions will not work if the communication is out of synchronization. When in sync. the second byte (0x53), will echo back when issuing the third byte of the Programming Enable instruction. Whether the echo is correct or not, all four bytes of the instruction must be transmitted. If the 0x53 did not echo back, give RESET a positive pulse and issue a new Programming Enable command.
Is there anything I must take into account with regards to what is connected to the SPI pins? i.e. does it harm in any way if I have other SPI slaves on the board connected to these same pins?
Yes. It is possible that a chip may respond to the ICSP signals. You should have pullup resistors on all your CS (or SS if you prefer to call it that) pins on all your slave chips so that while the ATMega328P is in programming mode (and all the GPIO pins are in INPUT mode by default) the CS pins of your slaves aren't floating.
Same for serial programming: how does the chip know that it's supposed to receive a program on the serial interface, when my application uses this port for a different purpose?
Simple: it listens for 2 seconds after power-on or reset for a special sequence of bytes arriving on the serial port. If they arrive within that 2-second window (8 seconds for ATMega32U4 based boards, to allow time for USB enumeration) the bootloader keeps running and looking for programming instructions. If it doesn't see that start sequence within the timeout period it exits the bootloader and runs your sketch.
The STK500 protocol (which the Arduino bootloader is based around) is detailed here. The trigger sequence is the "CMD_SIGN_ON" command detailed on page 4.