Is it possible to upload and run code from EEPROM during boot time?

Question

I am having problems while writing custom bootloader, so that it uploads code from EEPROM (for now internal, as I have no external memory in my hands) and writes into flash. After ~2 weeks of struggling I encountered these question and answers. After that, I realized that I made a mistake not checking the possibility of it. However, I am still confused about several things:

1. If Harvard Architecture limits what I want, how come optiboot can flash the memory from incoming UART data, which is not even a memory?

2. If UART reads and flashes memory accordingly, cant I do the same with EEPROM? Read bytes, erase flash, write bytes.

3. If it is not possible, given two answers in above link, then how does this work? I guess, it is the same logic as I want, using external memory and rewriting flash. Does it mean Harvard Architecture limits only internal memory exchange?

UPDATE: Here is what I have done for now:

In separate sketch, EEPROM is filled with binary code. Bootloader (edited optiboot v8):

  // Set up watchdog to trigger after desired timeout
  watchdogConfig(WDTPERIOD);
#if BIGBOOT

  if ((eeprom_read(0x000) == 0x0C) && (eeprom_read(0x001) == 0x94))
  {
    //while (EECR & (1 << EEPE)); //wait if previous data is still being written
    uint8_t i, j;

    for (i = 0; i < 8; ++i) // 8 pages total
    {
      uint8_t *code;
      code = buff.bptr;
      address.word = (i * 128);

      for (j = 0; j < 128; ++j)  // 128 words at a time
      {     
        if (j % 64 == 0)     
          LED_PORT ^= _BV(LED);

        *code++ = eeprom_read(address.word + j);
      }

#ifdef VIRTUAL_BOOT_PARTITION
      virtualBootPartition(buff, address);
#endif 

      writebuffer(0x46, buff, address, 128);
    }

    while(1); // wait for WDT
  }
#endif


#if (LED_START_FLASHES > 0) || LED_DATA_FLASH || LED_START_ON
  /* Set LED pin as output */
  LED_DDR |= _BV(LED);
#endif

while my function looks like this:

#if BIGBOOT 
static uint8_t eeprom_read(uint16_t addr)
{
  EEAR = addr; //read from that address
  EECR |= (1 << EERE); //enable reading

  return EEDR;
}
#endif

I use make atmega328 BIGBOOT=1 to generate hex file. It does not work + UART uploading also fails. If I set BIGBOOT to 0 and generate, UART uploading works fine.

P.S. virtualBootPartition is function which is called in STK_PROG_PAGE case. In order to not repeat the same piece of code, I collected that part into the function.

#ifdef VIRTUAL_BOOT_PARTITION
/*
 *              How the Virtual Boot Partition works:
 * At the beginning of a normal AVR program are a set of vectors that
 * implement the interrupt mechanism.  Each vector is usually a single
 * instruction that dispatches to the appropriate ISR.
 * The instruction is normally an rjmp (on AVRs with 8k or less of flash)
 * or jmp instruction, and the 0th vector is executed on reset and jumps
 * to the start of the user program:
 * vectors: jmp startup
 *          jmp ISR1
 *          jmp ISR2
 *             :      ;; etc
 *          jmp lastvector
 * To implement the "Virtual Boot Partition", Optiboot detects when the
 * flash page containing the vectors is being programmed, and replaces the
 * startup vector with a jump to te beginning of Optiboot.  Then it saves
 * the applications's startup vector in another (must be unused by the
 * application), and finally programs the page with the changed vectors.
 * Thereafter, on reset, the vector will dispatch to the beginning of
 * Optiboot.  When Optiboot decides that it will run the user application,
 * it fetches the saved start address from the unused vector, and jumps
 * there.
 * The logic is dependent on size of flash, and whether the reset vector is
 * on the same flash page as the saved start address.
 */

#if FLASHEND > 8192
/*
 * AVR with 4-byte ISR Vectors and "jmp"
 * WARNING: this works only up to 128KB flash!
 */
#if FLASHEND > (128*1024)
#error "Can't use VIRTUAL_BOOT_PARTITION with more than 128k of Flash"
#endif
  if (address.word == RSTVEC_ADDRESS) {
    // This is the reset vector page. We need to live-patch the
    // code so the bootloader runs first.
    //
    // Save jmp targets (for "Verify")
    rstVect0_sav = buff.bptr[rstVect0];
    rstVect1_sav = buff.bptr[rstVect1];

        // Add "jump to Optiboot" at RESET vector
        // WARNING: this works as long as 'main' is in first section
    buff.bptr[rstVect0] = ((uint16_t)pre_main) & 0xFF;
    buff.bptr[rstVect1] = ((uint16_t)pre_main) >> 8;

#if (SAVVEC_ADDRESS != RSTVEC_ADDRESS)
// the save_vector is not necessarilly on the same flash page as the reset
//  vector.  If it isn't, we've waiting to actually write it.
  } 
  else if (address.word == SAVVEC_ADDRESS) {
      // Save old values for Verify
      saveVect0_sav = buff.bptr[saveVect0 - SAVVEC_ADDRESS];
      saveVect1_sav = buff.bptr[saveVect1 - SAVVEC_ADDRESS];

      // Move RESET jmp target to 'save' vector
      buff.bptr[saveVect0 - SAVVEC_ADDRESS] = rstVect0_sav;
      buff.bptr[saveVect1 - SAVVEC_ADDRESS] = rstVect1_sav;
  }
#else 
        // Save old values for Verify
        saveVect0_sav = buff.bptr[saveVect0];
    saveVect1_sav = buff.bptr[saveVect1];

        // Move RESET jmp target to 'save' vector
        buff.bptr[saveVect0] = rstVect0_sav;
        buff.bptr[saveVect1] = rstVect1_sav;
}
#endif

#else
/*
 * AVR with 2-byte ISR Vectors and rjmp
 */
  if (address.word == rstVect0) {
    // This is the reset vector page. We need to live-patch
    // the code so the bootloader runs first.
    //
    // Move RESET vector to 'save' vector
      // Save jmp targets (for "Verify")
    rstVect0_sav = buff.bptr[rstVect0];
    rstVect1_sav = buff.bptr[rstVect1];
    addr16_t vect;
    vect.word = ((uint16_t)pre_main-1);
    // Instruction is a relative jump (rjmp), so recalculate.
    // an RJMP instruction is 0b1100xxxx xxxxxxxx, so we should be able to
    // do math on the offsets without masking it off first.
    // Note that rjmp is relative to the already incremented PC, so the
    //  offset is one less than you might expect.
    buff.bptr[0] = vect.bytes[0]; // rjmp to start of bootloader
    buff.bptr[1] = vect.bytes[1] | 0xC0;  // make an "rjmp"
#if (SAVVEC_ADDRESS != RSTVEC_ADDRESS)
  } 
  else if (address.word == SAVVEC_ADDRESS) {
    addr16_t vect;
    vect.bytes[0] = rstVect0_sav;
    vect.bytes[1] = rstVect1_sav;
    // Save old values for Verify
    saveVect0_sav = buff.bptr[saveVect0 - SAVVEC_ADDRESS];
    saveVect1_sav = buff.bptr[saveVect1 - SAVVEC_ADDRESS];

    vect.word = (vect.word-save_vect_num); //substract 'save' interrupt position
  // Move RESET jmp target to 'save' vector
  buff.bptr[saveVect0 - SAVVEC_ADDRESS] = vect.bytes[0];
  buff.bptr[saveVect1 - SAVVEC_ADDRESS] = (vect.bytes[1] & 0x0F)| 0xC0;  // make an "rjmp"
  }
#else

  // Save old values for Verify
  saveVect0_sav = buff.bptr[saveVect0];
  saveVect1_sav = buff.bptr[saveVect1];

  vect.bytes[0] = rstVect0_sav;
  vect.bytes[1] = rstVect1_sav;
  vect.word = (vect.word-save_vect_num); //substract 'save' interrupt position
  // Move RESET jmp target to 'save' vector
  buff.bptr[saveVect0] = vect.bytes[0];
  buff.bptr[saveVect1] = (vect.bytes[1] & 0x0F)| 0xC0;  // make an "rjmp"
  // Add rjmp to bootloader at RESET vector
  vect.word = ((uint16_t)pre_main-1); // (main) is always <= 0x0FFF; no masking needed.
  buff.bptr[0] = vect.bytes[0]; // rjmp 0x1c00 instruction
}
  
#endif
#endif // FLASHEND
#endif // VBP

Answer 1

If you have a sketch which can write its updated version into EEPROM of ATmega328p and then boootload from the EEPROM, then you can use the flash for the same purpose. The ATmega328p has 32kB of flash and only 1kB EEPROM.

For my ArduinoOTA library I developed a way to store the uploaded binary in upper half of the flash and then let the bootloader copy the binary to address 0 and reset the MCU.

The Optiboot 8 has do_spm() function. This function can be called from the sketch to write to flash memory.

To copy the binary stored in upper half of the flash to address 0, I wrote a new function for the Optiboot. It is called copy_flash_pages and has an extra boolean parameter to request a reset of MCU after copying the pages.

For network upload the sketch size is larger then half of the flash of 328p because of the networking library. So ArduinoOTA is only for ATmega with more then 64kB flash.

If you have a way how to get the binary of size less then half of the 328p flash, then you can write it to the upper half of the flash and then use my copy_flash_pages function to bootload it.

Resources:

• header file to access the Optiboot functions from sketch
• boards definition and bootloader for 328p with Optiboot with copy_flash_pages function in my_boards (note the changed fuses in boards.txt for Uno/Nano/Mini)
• the InternalStorageAVR class in ArduinoOTA library as example how to store and apply the binary

Example of InternalStorage use:

  if (!InternalStorage.open(length)) {
    Serial.println("There is not enough space to store the update. Can't continue with update.");
    return;
  }
  byte b;
  while (length > 0) {
    if (!source.readBytes(&b, 1)) // reading a byte with timeout
      break;
    InternalStorage.write(b);
    length--;
  }
  InternalStorage.close();
  if (length > 0) {
    Serial.print("Timeout downloading update file at ");
    Serial.print(length);
    Serial.println(" bytes. Can't continue with update.");
    return;
  }
  Serial.println("Sketch update apply and reset.");
  InternalStorage.apply(); // this doesn't return