Rough block diagrams below show two ways this can be achieved - "Easiest" costs least but is inefficient. "Better" uses PV panel output more efficiently. Details of either can be discussed if of interest.
'Schottky diodes' should be used to minimise losses.
Connect PV panel to battery with diode.
Panel is clamped to battery voltage (+ diode drop).
This will not hurt the PV panel but wastes some of the available power.
[What is the formal PV panel specification. Is it 21V O/C or loaded or .... Do you have a datasheet link for it?].
Use 12V (nominal) to 5V converter to power load.
This could be
Use a buck regulator to charge the battery - ideally an MPPT (maximum power point transfer) controller. Details can be discussed if you are interested.
Use a second buck regulator to power the load. This must be capable of (at least) 11V to 22V input. Power this from a Schottky diode from the PV panel AND a Schottky diode from the battery. When the panel voltage is > battery voltage the panel will power the load.