What are you trying to achieve? For how long is the board supposed to function? The simplest solution in this case is to simply get a 4.2V battery for the LilyGO board itself, most likely not just in terms of effort, but economically.
From your edits, I (now) assume you want to use the 5V pin on the board? I don't think that'll work the way you'd want it to, as it doesn't seem that it was intended for that (if it was, the VIN pin would've been brought out).
Q5 appears to disconnect the battery (VBAT) from +5V when the USB is connected.
Yeah, that makes sense to me as well.
Q4 also seems to disconnect the battery (BAT) from +5V when the USB is connected. What is the difference between VBAT and BAT ?
Yes, looking at the datasheet for the battery charger and the schematic for the LilyGO board, VBAT is the batteries voltage, while the BAT terminal is the one coming out of the charger and the one that usually supplies the current to both charge the battery and to possibly drive the system load.
VBUS (the USB power) is connected to +5V via a Zener diode. This appears to allow the USB to supply 5V, but prevent current flowing back to the PC. I think there could be an issue with high current through the Zener if my 5V regulator gave a voltage that was lower than the USB.
Yeah, it prevents the current going back into the USB port if you have the battery connected. You need a pretty substantial voltage to overcome that Schottky* barrier.
One issue you'll probably encounter is that if you connect a 5V source to that pin, the system will think it's powered on by a battery and you can damage the BAT terminal. If you have more than 1 of those boards and you're willing to experiment, you could try cutting the trace to BAT terminal altogether and connecting the 5V source and seeing if that works.
Another issue is that for safety reasons, it'd be prudent to switch that source off when you're connecting the USB; you'd also want to protect that 5V source from being damaged by the VUSB once it's on. There are probably ways you could keep the 5V on if you desolder the USBC connector and try to manually attach a probe to the datalines and ground the two together, but dealing with USBC is not easy and you're rapidly venturing into territory where it's just easier to get a different board and the screen separately or just use a LiPo battery to power the board in the first place (you could probably find a way to use multiple LiPo packs if you need your application to last longer as well).
Note: I didn't spend too much time looking at the schematic and it is a bit confusing, so a possible good exercise would be to simulate those two circuits in a SPICE program. And again, these types of questions are covered more-so by the EE SE. If you'd have posted it there, I'm sure you'd get more (and most likely, better) answers.