You can use something like the ATtiny25/45/85 then have a button cell battery to power it, make it a sleeper that listens to the button, and when clicked it activates the Arduino.
The Arduino is then turned on keeping a capacitor alive holding a transistor active, then add another button that drains the capacitor, thus closing the transistor and turning the Arduino off. Or you can program the ATtiny to drain the capacitor if the button is held down.
A button cell on an ATtiny in sleepmode should last around a year or so.
Just some information to get you going further:
http://playground.arduino.cc/Learning/arduinoSleepCode : The sleeper code, it is power saving using interrupts.
http://www.instructables.com/id/Program-an-ATtiny-with-Arduino/ : How to burn an ATtiny using Arduino. Fairly simple, did it myself a few times.