Basically, this is much more involved than it might appear on the surface. If you haven't already, I would make sure that I was comfortable with finding things in the datasheet (and it's a really good datasheet, at that). If you're still up for it, read on.
Bootloader
The way a bootloader works on the ATmega328 is that the memory on the chip is self-modifying. This means you can have a piece of code on the chip that modifies memory on the rest of the chip. A bootloader is one such piece of code that runs when the chip first starts up, and checks to see if there's code to be loaded. If there is, it writes that code to the rest of the memory, and then executes it. See section 12.2 of the datasheet.
The way you would program an actual bootloader is described in detail in section 30 "BTLDR - Boot Loader support". I haven't programmed a bootloader before, so you're on your own here, but there are plenty of resources to guide you. In fact, the datasheet has example code in there for it.
(Note that you must program your chip normally at least once, to put the bootloader on it.)
One thing to note is that bootloaders should be small. You don't want bootloaders to take up a lot of your memory, since they only run once on boot and are really just a convenience feature. There are competitions for writing the smallest/most efficient bootloader, and you'll almost certainly want to write your bootloader in assembly. Notice that the only example code given in the datasheet is in assembly. Which brings me to my next point:
Bluetooth
Bluetooth is very annoying to program for. You have to set up different profiles, or do GATT stuff if your module is actually BLE/"Smart" instead of classic bluetooth, etc. However, I'm assuming from your post that you just have one of those serial modules you can find on eBay for dirt cheap. Before I go on, I want to make this very clear: It is non-trivial to set up. It will take up lots of precious code space on your microcontroller that you would want to use on your actual program. Remember how bootloaders are written in assembly? Writing a bluetooth interface in assembly will probably take a while.
As for how your bluetooth module works; it is basically a serial port, but over-the-air. This means the bluetooth interface that you write would need to bit-bang everything if you don't use the built-in USART (say you need those pins for your actual program), and if you do use the built-in USART, setting that up will take even more time. (For a guide on how to setup the USART, see section 24 of the aforementioned datasheet. See? It's really handy.)
Once you actually have bluetooth working, you can do whatever you want. Since you write the bootloader, you can read data from your bluetooth serial link however you fancy. As long as you know what you're sending, all you have to do is read it back accordingly.
Bonus: Flashing
You are correct that a .hex
file is just a huge text file that you send to the chip. However, the way the ATmega is actually programmed is not simply using the serial port; instead, it's a slightly extended SPI protocol. (If you don't know what SPI is, you really shouldn't be doing this. If you're curious, Sparkfun has a nice tutorial on SPI.) This is called ICSP, or "in-circuit serial programming". The way to do this for AVRs is explained in an application note.
Of course, you cannot replicate this with just a bluetooth module. Instead, you would need a co-processor to program your chip for you, and read the data to-be-programmed from your bluetooth module.
TL;DR: This is probably not worth it. If you're planning on doing this, it will take significant resources, and you might even need a better chip to get anything done. You almost certainly should just buy an ESP8266 / ESP32 instead.