The first and most smart solution is to use a switch between the batteries and everything else. This way, you slide the switch in the off position and the whole circuit is completely detached from the batteries: power consumption: 0.
If you can't (because, for instance, you want it to automatically power off after a while) you can use a transistor (PMOS, for instance) instead of the switch. you have then to put a complementary transistor (NMOS) on its gate, then a button in parallel to it. Just another passve component is required: a resistor (value e.g. can be 100k). When you press the button, the circuits becomes momentarily powered on. You will have to pull the nmos gate to a logic 1 with the atmega to keep the circuit powered. When you want to turn everything off, pull the pin to a logic 0 and your circuit will immediately shut down.
If, third solution, you want to give the atmega complete control over the powering, it needs to be always on. You can then omit the complementary transistor and use a reverse logic on the pin (so 0 = on, 1 = off) and then you can do whatever you want with the power (turn on and off whenever you want).
edit: as a comment suggests, it can be better to put a resistor to set the appropriate voltage on the gate at power up. Even if the commenter told to put a pull-down, I suggest you to put a pull-up (pull-down = normally on, pull-up = normally off). edit 2 as Paul suggested, there could be cases in which you need to be able to turn on and off automatically the circuit, and then store it for longer times without consuming battery. In this case it could be a good idea to put something (a mechanical switch, a jumper, a detachable wire) to completely detach the battery even from the arduino (even if, in this case, you can just remove the batteries from the housing).
Here is a small paint-made sketch to show the solutions. If you have any questions just ask ;)
EDIT: If you want to let the atmega control the power, you'll have to give him a way to do so. Which means.. You should use a pin. If you want you can use safely the RX and TX pins, as long as you don't enable the serial interface. If you need more pins, you can try to optimize your pins usage (for instance using a 4:1 multiplexer for the buttons you will have 2 control pins and 1 input, for a total of 3 pins, instead of 4), even if I think that the best solution is to use a port expander. for instance in the past I used the MCP23017, which has a I2C interface, or the MCP23S17, with a SPI interface. You will have to use the serial interface you choose (I2C: 2 pins - SCL and SDA, SPI: 4 pins - MISO, MOSI, SCK, SS, but they are the same pins used for programming), and then you will have 16 more pins for your uC
EDIT BY foivaras:
The same result can be achieved utilizing an NMOS instead of a PMOS.This would have an advantage over PMOS solution if power line was >5v and arduino was powered by a dc to dc converter. On this scenario, PMOS solution would need one more transistor to work.
Notice that the NMOS has to be placed on the low side of the power line and it will cutoff the power to joystick and lcd with 0v and enable the power with 5v on its gate.(See 4th image)
COMMENT BY fra87
This solution is not a valid alternative in my opinion. Usually interrupting ground is a bad idea, because usually the rest position of the I/O pins is ground: if you interrupt the ground for the LCD, for instance, you will have +5V on the VCC pin, and a floating ground, but you will have 0V on all the I/O pins, possibly damaging the internal circuits. Moreover having a floating ground is bad in terms of noise protection.
The only advantage COULD be in the particular case explained by the editor, which is a DC/DC converter powering the arduino but not the LCD and the Joystick. This is a strange setup, though, and the proposed solution is better at a first glance (requires one less transistor), but is wrong. For instance, let's suppose the joystick is powered at 12V. If you detach the ground, the joystick will "shut down", which means that all its pins will float up to 12V. This means that on the arduino pin you will see 12V: you can damage it.
So, in the end, I discourage this solution. The only case when you can need it is when you are in a hurry and do not have any feasable PMOS: even in this case, however, I suggest you to replace the NMOS with a PMOS as soon as possible.
Just one more note: edits to the posts, in my opinion, should be done to improve the quality of the answer, by better formatting the content, removing orthographic errors and so on. If someone needs to add some comments or other solutions, it's better to use the comments or add another answer to the thread
