The behaviour you are seeing is what happens when you have high-impedance inputs. Basically, if you don't drive them, you get random values.
You can solve the problem by making sure the value is always driven, even when the sensor is disconnected. Add a resistor between the op-amp's + input and ground.
The value of the resistor should be large enough that it doesn't affect the output from the pH sensor much when it is connected, so find the output impedance of the pH sensor and pick a resistor, say, 100 times larger.
The resistor should also be much smaller than the input impedance of the op-amp; they have very high input impedance (mega-ohms). Let's hope there exists a suitable value to meet both criteria.
You might not reliably get down to 0, but you should get pretty low.