Why do you think it should show 0 when not connected to anything?
Does the antenna of your radio give a zero signal just because the end isn't connected to anything? An Arduino's input pin (either analog or digital) is just the same. If it isn't connected to anything then it just picks up noise - be that radio transmissions, induction from nearby mains wiring, neutrinos from Alpha Centauri... Basically if it's not connected to anything then there is no way you can possibly know what the value should be since there is no set value.
In order to read 0 on the ADC the ADC has to be connected 0V. That means connect it to ground.