You need to isolate the slave device from the I2C bus since it is causing interference with the signals.
Adafruit have published a good application note about level shifting in I2C busses using MOSFETs, and it also contains a bit about bus isolation:
2.3.4 Isolation of the powered-down “Lower voltage” section.
An additional feature of the level shifter circuit in figure 2 is the isolation of the “Lower voltage” section when
VDD1 is switched off. In that case VDD1 is about 0 Volt and the MOS-FET’s [sic] are switched off because VGS is
below the threshold voltage. The “Higher voltage” section is not hindered and stays operational. To assure a
noise margin, the MOS-FET’s should have a minimum threshold voltage VGS(th) of e.g. 0.4V and VDD1 must
stay below this value. The isolation feature can also be applied if no level shifting is required, VDD1 and VDD2
may have the same value, e.g. both 3.3V or both 5V.