ESP8266 OTA update with different flash size settings?

Question

I have an ESP8266 in a project. It's specifically this ESP8266 SMT Module - ESP-12F item. It's been working great with OTA, but I realized belatedly that I didn't change the flash settings when I started compiling through the Arduino IDE several versions ago. I had been using the setting "1 MB (FS:64KB OTA:~470KB)", and I've updated OTA over multiple versions with the same basic structure of firmware and using the same server-side set-up.

I can no longer update with that setting because my compiled sketch is now over the OTA size limit. My chip actually has 4 MB, however, so this limit is artificial. I tried changing to "4 MB (FS:2 MB OTA:~1019KB)", and that works under the following test conditions:

1. I compile the "old" program with the 4-MB settings and I re-upload the old program via wired connection.
2. I compile the "new" program that's to come via OTA with those same 4-MB settings.

That's fine for a test, but I already have one of these chips "deployed" in a location where it's hard to access it. I'd like to update that deployed chip OTA while also changing the flash partition. I'm willing to lose any files stored outside of the program itself. So I'd like instead to have:

1. A deployed, OTA-enabled chip with firmware running on the 1-MB partition structure noted above:
2. (Any intermediate step needed)
3. OTA update to new firmware that has been compiled with the 4-MB partition structure.

I'm willing to upload a dummy sketch at "Any intermediate step needed" to work around the current size of the sketch, but, after some experimentation, OTA seems to fail when the partition structure is different regardless of the size of the binary.

That's consistent with #2 in the answer here: esp8266 OTA issue, but that older question took a different direction with the OP and the #2 point there is stated without any detail. I understand that in a broad sense, the prior answer covers my case, but I'm looking for more detail on the specific aspect of the flash partition settings. Especially for these related questions:

• Is there any work-around strategy available to change the flash partitioning between different versions served OTA?
• If no, is the limitation determined by the software in the current OTA framework in the public libraries, or is it somehow fundamental to how the program is laid out in the hardware memory?

In theory at least, it seems like I ought to be able to force the new program to be written OTA at the same location that it would have been written had the old program also had the 4-MB partition. (Again noting that I'm willing to lose any other data in the flash to make this update work.)

Answer 1

The basic problem is that when you upload over OTA you don't upload the partition structure.

The ESP32's flash is split into a number of "partitions" or areas. It might look something like:

+----+-+-------------+-------------+---------------------+
|BOOT|P|OTA Program 1|OTA Program 2| SPIFFS              |
+----+-+-------------+-------------+---------------------+

When you upload over UART you upload all that structure, and that structure can be changed. When you upload over OTA you only upload one of "OTA Program 1" or "OTA Program 2". One of those is active and the other is inactive. You upload to the inactive one and then at the next reboot that becomes the active one. This allows the system to confirm that the program has been written correctly before running it to reduce the chances of bricking a device in the field.

Because of this you can never upload anything that doesn't fit into one of the OTA program partitions.

One risky option which you would have to do thorough local testing of first, is to write a program which, when executed, will change the partitioning. You would have to make sure that it runs in the lower OTA partition so that the inactive one is the one that gets both moved and expanded, which may take a little trial and error.

Once that has happened you should be able to upload a larger OTA program into the new, bigger, inactive OTA segment.

But that's only supposition and it's not something I have ever done or heard of anyone doing.