Attempt to change an ESP8266's flashs layout over-the-air

Question

Backstory

Several Tasmota devices based on an ESP8285 have to be flashed with alien firmware. The devices' programming ports are not accessible (except for one dummy device which was cracked open for testing).
The Tasmota firmware currently installed is based on the Arduino core, uses ESP8266httpUpdate.h and allows to OTA-update from a http server.

Problem

The problem is that the Tasmota firmware for the ESP8266 (=8285) family always uses a 1MB partition scheme while my specific device is based on the ESP8285H16 which incorporates 2MB of flash. The second half of flash memory is therefore unused.

What I want to do

Using only over-the-air updates I want to update the Tasmota devices' flash layout so that the full 2MB of flash can be used and eventually upload a firmware that is bigger than what's currently possible.
Ideally the final flash layout uses a completely new partition scheme.

How can it be achieved?

I know this can be challenging and will probably require multiple intermediate firmware versions. This is okay because the upgrade process will not happen "in the field" (but still no physical access to the chips) and occasional fails are acceptable - the process will be tested beforehand.

There's already a related answer from @Majenko here, but it covers the ESP32. While I know where to start with an ESP32, I'm a bit puzzled with the ESP8266/8285:

I tried to read the whole flash images of the device to get a picture of what I'm dealing with and found the flash layout to be completely different from what I'm used to (ESP32).
Mainly, there wasn't a partition table at flash address 0x8000 while the Espressif documention specifies this exact address.

I found almost no documentation about the ESP8266 partition scheme as defined by the Arduino core:

• Where in flash is the partition table?
• If there is none, how does OTA work on ESP8266?
• Where in flash is the bootloader?
• Where does the filesystem begin (if there is one)?

Requirements

• The flash layout needs to be changed using only OTA updates
• Any information stored in the current flash or filesystem may be deleted
• Multi-step updates are no problem
• There's physical access to one device so the routine can be tested

Edit: Illustration

This is the normal OTA routine according to the docs:

and this is what I have in mind:

But for this I need to know the borders of each section, ideally during runtime.

Answer 1

As it turns out the update routine of ESPhttpUpdate.h works just fine when the flash layout changes after an OTA update. At least in my case where the second half of flash memory is unused. If there was a file system in the initial installation, it will be lost.
Though, it should be possible to even preserve the filesystem if it is copied to the right location before a firmware with different partition scheme is uploaded, but I have no need to follow that path now...

to summarize...

1. A Tasmota firmware was installed on a 2MB ESP8285H16. Since Tasmota firmware for ESP8266/8285 devices uses only 1MB of flash - maybe something like ~500kB sketch, ~400kB OTA, ~100kB FS).
   The second half of flash was unused and the maximum size of a firmware that could be directly uploaded over-the-air was therefore ~400kB .

2. An intermediate firmware was provided on a http server for the device to update.
   This firmware:

   • was built for a 2MB flash layout (1MB sketch, ~700kB OTA, ~256kB FS)
   • contains nothing but Wifi login and the OTA routine Upon success, this update will leave the flash in the new layout

3. With the additional 300kB of OTA memory, the device could now be updated to the final firmware. This firmware was built with the same partitioning like the intermediate firmware.

This is the platformio.ini which was used in my case:

[env:esp8285]
platform = espressif8266
framework = arduino
board = esp8285

board_build.filesystem = littlefs
board_build.ldscript = eagle.flash.2m256.ld
board_build.mcu=esp8285
board_upload.flash_size=2MB
board_build.flash_mode=dout

monitor_speed = 115200
upload_port = COM5

If there is only little space left for OTA in the original flash layout, the firmware binary can be gzip'ed to save some memory.