To be honest you are better off stepping down rather than stepping up. Simply because step down converters tend to be a higher efficiency and can deliver more current than a step up converter.
With step up converters you effectively convert current into voltage. With step down it's the other way around - voltage into current. Since button cells have very very little current capacity stepping up will reduce it even more and starve the ESP32 of power. By stepping down you increase the current.
Two 3V button cells in series will provide nominal 6V and about 40mA current. By adding a step down converter to 3.3V you are almost halving the voltage, but at the same time almost doubling the current. Adding a third cell in series will give you even more current output.
It is possible to get some incredibly tiny and high efficiency step-down modules. I myself make and sell one based around the TPS82130SILT which has the ability to run at as low as 3.7V with a 3.3V output (it has a "pass through" mode which allows a zero dropout voltage, but I include a protection diode with 0.4v drop). That means you get the entire 6V to 3.7V range out of your batteries before the voltage starts to droop (assuming the batteries still operate at 1.85V).