No, it's not going to "fry" the ESP32, and here's why:
Yes, there are pullups to 5V. Those are 4.7K in your example. The IO pins are connected to 3.3V through ESD diodes. That means that any voltage over and above 3.3V plus the forward voltage of those diodes will be "diverted" away from the IO pin through the diode. IO pins have these ESD diodes built in, but this board has extra external ones added to make it more rugged.
These ESD diodes can handle a finite amount of current flowing through them - the internal ones only a small amount, but the external ones much more.
The 5V will only be a low current because it is connected with a resistor. We can even do some maths here.
Assuming the ESD diodes have a 0.3V forward voltage (as is reasonably typical) we can say that the voltage across the diode will be 5V - 3.3V - 0.3V = 1.4V.
With a 4.7K resistor as a pullup the current through the diode will be 1.4V / 4700 = 298 microamps.
That's way below the limit you'd expect of a small internal ESD diode, let alone a larger external one.
The "frying" of an IO pin happens when too much current flows through the ESD diode and it blows, "failing short", causing the IO pin to be locked in either a HIGH or LOW state for the rest of eternity. Since the current is nowhere near the kind of levels needed for that it's not going to even remotely stress the diodes, and the inclusion of external diodes makes it even safer.