The setup() function returns void so that's no help.
You wouldn't be able to get the return value anyway even if it did return something - it's not you calling the
setup() function, but
main(), which is not in your code or control.
This just doesn't seem to be an issue that is addressed in the official docs
That is because it is not within the remit of the API and its associated documentation.
Throw an exception (but where does that go?)
It goes in the bin. Exceptions are disabled. They make little sense in an embedded system. Yes, it is possible to use them, but not worth it.
Write a message to Serial and then go into an infinite loop or wait for a y/n to continue.
That kind of thing is what most people do - or variations on that kind of theme.
Set a setup_successful flag and then do if (!setup_successful) return; in the main loop()
That will just run
loop() again - after all, it's called
loop() for a reason.
You seem to be in the mindset of a PC programmer. Embedded programming is a very very different beast. You don't (usually) have an operating system to handle exceptions and errors and things like that. You don't have a keyboard and mouse, and a screen with pretty pictures on it. Every single thing that the board does is what you program. If you want it to raise an error when an error occurs then it is up to you to program a suitable mechanism for that to happen. Reporting that error and asking for feedback in some way is entirely governed by your system's design. If you have a TFT screen you could put an error message on that. If you have a serial connection you can use that to show error messages. If you don't have any of those you can blink an LED or use a buzzer to beep in a specific pattern (like a PC's POST beeps).