Nanaimo: Hardware-In-the-Loop Unit Testing

Note

Nanaimo is alpha software and will remain so until we bump its version to 1.0.0 or greater. We will not knowingly break compatibility within a minor revision but we will break compatibility a few more times between minor revisions until beta is declared. Because of this you should depend on a minor version explicitly. For example

nanaimo ~= 0.1

A delicious Python treat that makes on-target testing sweet and satisfying.

Nanaimo is a set of utilities and plugins designed to enable integration of hardware test apparatuses with pytest. This can allow on-target tests to run as part of continuous integration pipelines like Buildkite, Bamboo, or Jenkins.

Example of S32K dev boards attached to Raspberry PI CI workers running the Buildkite agent and using Nanaimo.

Nanaimo is designed to enable testing of software-defined, physical components in isolation to provide pre-integration verification of software interfaces and behavioral contracts. It adapts asynchronous control and monitoring of these components to fit familiar testing idioms (e.g. x-unit testing) using the popular python test framework, pytest.

Block diagram of Nanaimo’s relationship to other components of a typical software build and test pipeline.

Nanaimo is not a simulation framework and is not designed to support the complexity of a full hardware-in-the-loop platform. Instead it’s focused on testing small integrations with a few hardware components and instruments using concepts, syntax, and frameworks familiar to software engineers. Examples of these small integrations might include verifying a SPI driver for a microcontroller or ensuring the upload time for a serial bootloader meets expected Key-Performance-Indicators (KPIs). To do this Nanaimo abstractions provide async interfaces to hardware either directly using available communication protocols (e.g. serial or IP networks) or by invoking a CLI provided by the instrument vendor. Because of this latter use case some instruments will require additional programs be available in a test environment.

Example scenario using Nanaimo to test an I2C driver for a microcontroller.

This design is an amalgam of the TLYF (Test Like You Fly) methodology and the Swiss cheese model of failure analysis. Specifically; Nanaimo facilitates testing on actual or representative hardware for the first integration of software into a part or subassembly. Traditionally software engineers were responsible only for unit-testing and Software-In-the-Loop (SIL) simulation of their code. Nanaimo encourages software engineers to also provide hardware integration tests by enabling Hardware-In-the-Loop continuous-integration (HIL-CI, perhaps?).

Hierarchy of system testing. Nanaimo focuses on part and subassembly testing.

Note

Nanaimo is named after Nanaimo bars which are about the best things humans have ever invented.