While not physically impossible it would be a really silly thing to attempt.
However, if I were to attempt it I'd probably use just 3 switches (or rather 2 switches and one button though you could use a switch instead of the button).
I'd wire it up so that the RESET, MISO and SCK pins were all connected to the switches in such a way that I could hold the whole board in RESET for extended periods (i.e., switch between RESET and GND), set a HIGH or a LOW as needed on the MISO pin, and toggle the SCK pin HIGH and LOW (I'm not sure off hand if you need to idle LOW or HIGH with the clock - the datasheet will tell you).
Then I'd become extremely intimate with the ICSP protocol of the ATMega328P.
That would then allow me, extremely slowly, to act like a hardware programmer and instruct the ATMega328P to program itself with data I sent.
You might also like to attach a shift register to the MOSI and SCK pins with 8 LEDs on it so you can monitor the responses from the Arduino while programming.
It would take forever to upload even the simplest of programs, and the error rate would be far to high, but it could be done if you were really desperate and had nothing else to do (including eat or sleep) for a number of days.