How can Arduino source code be tested automatically (continuous integration)?

Question

I'd like manage my Arduino source code (projects and libraries) under source control with continuous testing. How can I automatically compile the code with continuous integration tools, to make sure each version compiles cleanly? In the best case one should be able to configure builds for multiple processors, optionally run unit tests and check for maximum binary size.

Answer 1

Newest version of Arduino ide has a command line interface to build and upload code. But you can obviously do it by makefile and avrdude. Now, you have compiled your code BUT you need testing. As simulator are complex,incomplete, expansive and.. Just a simulation, and because the chip are relatively cheap, building a board witch will make HW interaction AND check the results its the fastest and easiest way. On that "special" board you may upload something like an interpreter witch take a test from the PC and execute it, something similar to firmata for arduino. At least that is how I would build it. And as far as I know there are no know implementation of this, even if I'm quite sure many industries should and probably do that.

Answer 2

As the creator of PlatformIO I would recommend you to look into it. It's a cross-platform code builder and missing library manager. It can build the same code for the many popular embedded development platforms and boards.

PlatformIO can be be integrated with many popular Continuous Integration (CI) systems (or your own). See documentation with examples.

Let's look into .travis.yml config/template for Travis CI:

language: python
python:
    - "2.7"

env:
    - PLATFORMIO_CI_SRC=path/to/source/file.c
    - PLATFORMIO_CI_SRC=path/to/source/file.ino
    - PLATFORMIO_CI_SRC=path/to/source/directory

install:
    - python -c "$(curl -fsSL https://raw.githubusercontent.com/platformio/platformio/master/scripts/get-platformio.py)"

script:
    - platformio ci --board=TYPE_1 --board=TYPE_2 --board=TYPE_N

Example

Integration for USB_Host_Shield_2.0 project. The .travis.yml configuration file:

language: python
python:
    - "2.7"

env:
    - PLATFORMIO_CI_SRC=examples/acm/acm_terminal
    - PLATFORMIO_CI_SRC=examples/Bluetooth/WiiIRCamera PLATFORMIO_BUILD_FLAGS="-DWIICAMERA"
    - PLATFORMIO_CI_SRC=examples/ftdi/USBFTDILoopback
    - PLATFORMIO_CI_SRC=examples/Xbox/XBOXUSB
    # - ...

install:
    - python -c "$(curl -fsSL https://raw.githubusercontent.com/platformio/platformio/master/scripts/get-platformio.py)"

    # Libraries from PlatformIO Library Registry
    # http://platformio.org/#!/lib/show/416/TinyGPS
    # http://platformio.org/#!/lib/show/417/SPI4Teensy3
    - platformio lib install 416 417

script:
    - platformio ci --board=uno --board=teensy31 --board=due --lib="."

• Configuration file: https://github.com/felis/USB_Host_Shield_2.0/blob/master/.travis.yml
• Build History: https://travis-ci.org/felis/USB_Host_Shield_2.0

Answer 3

One example of setting up Jenkins continuous integration for Arduino project can be found here: Continuous integration for embedded systems

The example shows how to build and upload image to Arduino and execute Selenium web tests (the system under test is Arduino-based web-server).

Answer 4

I wrote this test unittest framework for Arduino, since i could not find proper existing. While i did not write it for CI, it would be good fit for CI job, since it requires no hardware but can be ran on PC.

Running the CI jobs without the hardware has good sides and bad sides, good sides beeing that

• No flashing, no hardware needed -> can be runned in parallel -> fast to check for example for every commit
• No hardware issues influencing the tests -> no need to worry did this test just fail because my XYZ-hardware component is not stable

On the downside there is:

• Its not testing the actual target code, for example your 'int' is 32 bits in your PC and 16bits in AVR.

Answer 5

I've just put together a decent size writeup as an answer to this related question about an Arduino CI/unit testing framework I wrote that's finally mature enough to start talking publicly about.

The arduino_ci ruby gem supports both local testing and Travis CI integration (e.g. this build job for the Adafruit FONA library).

Some examples of how it integrates:

• The DoSomething example library shown above, used to test arduino_ci itself
• A practical example, testing a Queue implementation
• A complex example, simulating a library that controls an interactive device over a SoftwareSerial connection as part of the Adafruit FONA library

I have an issue created for reporting sketch size but no work has been done on that. It's also possible that I could parallelize the builds someday, but at the moment I'm not precisely sure how I'd do that. Right now, the most time consuming build step on Travis CI is downloading the Arduino IDE... parallel testing won't put much of a dent in that if every job needs to do that step.

Answer 6

I would advice to use the arduino eclipse plugin named sloeber sloeber.io of which I'm the project lead.
It integrates with version control and allows building on multiple platforms as it supports multiple configurations.
Though it is not yet idiot proof I have documented and demonstrated using unit testing of arduino code on the local pc using the google test framework.
Here is a link to a blog containing a presentation of how you can do it. http://blog.baeyens.it/#post25