The behaviour you are seeing is what happens when you have high-impedance inputs. Basically, if you don't drive them, you get random values.

You can solve the problem by making sure the value is always driven, even when the sensor is disconnected. Add a resistor between the op-amp's + input and ground.

The value of the resistor should be large enough that it doesn't affect the output from the pH sensor much when it is connected, so find the output impedance of the pH sensor and pick a resistor, say, 100 times larger.

The resistor should also be much smaller than the input impedance of the op-amp; they have very high input impedance (mega-ohms). Let's hope there exists a suitable value to meet both criteria.

You might not reliably get down to 0, but you should get pretty low.

The behaviour you are seeing is what happens when you have high-impedance inputs. Basically, if you don't drive them, you get random values.

Unfortunately in your case, where the sensor you are using has no output current to speak of, I don't think there's anything you can do about this electronically.

You'll have to solve the problem in some other way if you really need to be able to disconnect the sensor. A mechanical solution would be a connector which shorts when disconnected. Switched headphone connectors do this, for example - when you pull out the jack, they make a connection so that the speakers can be switched on.