How to install the application + its configuration to many devices of the same kind? (ESP32)

Question

Being the newbie in the Arduino area... Having more devices, how can I upload some configuration parameters after the code was uploaded? Is it possible?

Situation: I have the simple application for ESP32 with a simple sensor. It is to be used on many physical locations. Each location has its own Wi-Fi with its own SSID and password. An IT person on the location should be able to set the configuration parameters somehow. I would like to learn the possibilities -- to choose what is more suitable.

Separate compilation and upload: If I understand correctly, I can use the arduino-cli compile ... to compile the sketch to binary, and then arduino-cli upload ... to put the binary into the device. This way, I am able to prepare many devices with exactly the same binary content. (If I understand it correctly, then arduino-cli config ... is for setting the development environment, not for configuring the application on the board.)

Configuration: I have noticed the existence of Arduino Preferences Library (Preferences.h). I understand how to use it from inside the code of the application. Can the wanted preferences be uploaded to the board similarly as the sketch is? (I guess no, but asking to be sure.)

I have also noticed the existence of a lightweight file system (SPIFFS), and the ability to create (say) /config.json on the device from inside the application code. There is the plug-in to Arduino IDE to appear as a new tool (.../tool/esp32fs.jar). Then the data/ subdirectory is created in the sketch directory, and its content can be uploaded by another IDE command. That seems to be nice way to upload the configuration in my case; however, ...

Is there any command-line alternative to upload the configuration data that way? Can a single file be uploaded, or is it necessary to erase whole file system area, and upload whole content of the data/ like directory?

Answer 1

If you don't want to include the individual config data in the sketch for each location there are mainly two possibilities to handle that:

• As consumer IOT devices often do you can write your code to create a hotspot with a website, where the IT staff can then connect to and config the device by inputting the config data into the website.
• You can use SPIFFS/LittleFS (The second is the successor of the first) to create a config file, which you then can read at the start of your sketch. This file system is saved in flash and can be written with an extra tool.

I would go for the second option. I'm currently still using SPIFFS, so I don't know if the following will be different for LittleFS. Your paths might vary (depending on core/version/OS).

1. First create your data directory, where you then place your config.json file. The standard is a directory named "data" inside the sketch folder.

2. Then you need to invoke mkspiffs to create a binary image for your data folder. I'm doing it like this:

    ~/.arduino15/packages/esp32/tools/mkspiffs/0.2.3/mkspiffs -c ./data -d 5 -b 4096 -s 0x170000 -p 256 ${file_path}/spiffs_image.bin
   

   {file_path} is the path of the sketch folder for me. That way the image is placed directly besides the sketch. The options mean the following (I will also paste the help page of the command below):

    -d 5  -->  Debug Level at max (goes from 0 to 5)
    -b 4096  --> block size in bytes
    -s 0x170000  -->  image size in bytes
    -p 256       -->  page size in bytes
    -c ./data    -->  data directory
   

   You can get these information when you look at the verbose output in the Arduino IDE while uploading the filesystem over the tool. They depend on your board and the settings that you use (mostly the partition scheme).

• Then you can upload the image file via the esptool.py:

    ~/.arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 ~/.arduino15/packages/esp32/hardware/esp32/1.0.5/tools/esptool.py --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash 0x290000 ${file_path}/spiffs_image.bin
  

  0x290000 is the position in flash, where the data should be written. This also depends on your partition scheme and it should also appear in the Arduino IDE output, where you can copy it from. Note, that I'm invoking the python version here, that comes with the ESP32 core.

This works great for my ESP32 board (using the Wemos LOLIN32 with the "Standard" Partition scheme). Now you will have to think of a fitting workflow. You can have individual data directories and images for each location. Or you can change the file everytime you invoke the commands above (you can even do this in a script).

---

Here the help output of mkspiffs:

~/.arduino15/packages/esp32/tools/mkspiffs/0.2.3/mkspiffs --help

USAGE: 

   /home/christian/.arduino15/packages/esp32/tools/mkspiffs/0.2.3/mkspiffs 
                                        {-c <pack_dir>|-u <dest_dir>|-l|-i}
                                        [-d <0-5>] [-a] [-b <number>] [-p
                                        <number>] [-s <number>] [--]
                                        [--version] [-h] <image_file>


Where: 

   -c <pack_dir>,  --create <pack_dir>
     (OR required)  create spiffs image from a directory
         -- OR --
   -u <dest_dir>,  --unpack <dest_dir>
     (OR required)  unpack spiffs image to a directory
         -- OR --
   -l,  --list
     (OR required)  list files in spiffs image
         -- OR --
   -i,  --visualize
     (OR required)  visualize spiffs image


   -d <0-5>,  --debug <0-5>
     Debug level. 0 means no debug output.

   -a,  --all-files
     when creating an image, include files which are normally ignored;
     currently only applies to '.DS_Store' files and '.git' directories

   -b <number>,  --block <number>
     fs block size, in bytes

   -p <number>,  --page <number>
     fs page size, in bytes

   -s <number>,  --size <number>
     fs image size, in bytes

   --,  --ignore_rest
     Ignores the rest of the labeled arguments following this flag.

   --version
     Displays version information and exits.

   -h,  --help
     Displays usage information and exits.

   <image_file>
     (required)  spiffs image file

And for the esptool.py:

~/.arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 ~/.arduino15/packages/esp32/hardware/esp32/1.0.5/tools/esptool.py --help
usage: esptool [-h] [--chip {auto,esp8266,esp32,esp32s2beta}] [--port PORT]
               [--baud BAUD]
               [--before {default_reset,no_reset,no_reset_no_sync}]
               [--after {hard_reset,soft_reset,no_reset}] [--no-stub]
               [--trace] [--override-vddsdio [{1.8V,1.9V,OFF}]]
               [--connect-attempts CONNECT_ATTEMPTS]
               {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,version,get_security_info}
               ...

esptool.py v2.9-dev - ESP8266 ROM Bootloader Utility

positional arguments:
  {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,version,get_security_info}
                        Run esptool {command} -h for additional help
    load_ram            Download an image to RAM and execute
    dump_mem            Dump arbitrary memory to disk
    read_mem            Read arbitrary memory location
    write_mem           Read-modify-write to arbitrary memory location
    write_flash         Write a binary blob to flash
    run                 Run application code in flash
    image_info          Dump headers from an application image
    make_image          Create an application image from binary files
    elf2image           Create an application image from ELF file
    read_mac            Read MAC address from OTP ROM
    chip_id             Read Chip ID from OTP ROM
    flash_id            Read SPI flash manufacturer and device ID
    read_flash_status   Read SPI flash status register
    write_flash_status  Write SPI flash status register
    read_flash          Read SPI flash content
    verify_flash        Verify a binary blob against flash
    erase_flash         Perform Chip Erase on SPI flash
    erase_region        Erase a region of the flash
    version             Print esptool version
    get_security_info   Get some security-related data

optional arguments:
  -h, --help            show this help message and exit
  --chip {auto,esp8266,esp32,esp32s2beta}, -c {auto,esp8266,esp32,esp32s2beta}
                        Target chip type
  --port PORT, -p PORT  Serial port device
  --baud BAUD, -b BAUD  Serial port baud rate used when flashing/reading
  --before {default_reset,no_reset,no_reset_no_sync}
                        What to do before connecting to the chip
  --after {hard_reset,soft_reset,no_reset}, -a {hard_reset,soft_reset,no_reset}
                        What to do after esptool.py is finished
  --no-stub             Disable launching the flasher stub, only talk to ROM
                        bootloader. Some features will not be available.
  --trace, -t           Enable trace-level output of esptool.py interactions.
  --override-vddsdio [{1.8V,1.9V,OFF}]
                        Override ESP32 VDDSDIO internal voltage regulator (use
                        with care)
  --connect-attempts CONNECT_ATTEMPTS
                        Number of attempts to connect, negative or 0 for
                        infinite. Default: 7.

Answer 2

I usually use BLE to set the WiFi credentials.

After the WiFi connection is taken care of, I usually provision the rest of the configuration data by having a web application in the LAN that I put into "pairing" mode with the MAC address of the ESP. The ESP then downloads the config from the web application. Although I don't have super secret config data, for good measure the application ends the pairing mode after one download and will not serve further config requests.