Nanaimo: Hardware-In-the-Loop Unit Testing

Note

Nanaimo is alpha software and will remain so until we bump its version to 1.0.0 or greater. We will not knowingly break compatibility within a minor revision but we will break compatibility a few more times between minor revisions until beta is declared. Because of this you should depend on a minor version explicitly. For example

nanaimo ~= 0.1

Fig. 7 A delicious Python treat that makes on-target testing sweet and satisfying.

Nanaimo is a set of utilities and plugins designed to enable integration of hardware test apparatuses with pytest. This can allow on-target tests to run as part of continuous integration pipelines like Buildkite, Bamboo, or Jenkins.

Fig. 8 Example of S32K dev boards attached to Raspberry PI CI workers running the Buildkite agent and using Nanaimo.

Nanaimo is designed to enable testing of software-defined, physical components in isolation to provide pre-integration verification of software interfaces and behavioral contracts. It adapts asynchronous control and monitoring of these components to fit familiar testing idioms (e.g. x-unit testing) using the popular python test framework, pytest.

Fig. 9 Block diagram of Nanaimo’s relationship to other components of a typical software build and test pipeline.

Nanaimo is not a simulation framework and is not designed to support the complexity of a full hardware-in-the-loop platform. Instead it’s focused on testing small integrations with a few hardware components and instruments using concepts, syntax, and frameworks familiar to software engineers. Examples of these small integrations might include verifying a SPI driver for a microcontroller or ensuring the upload time for a serial bootloader meets expected Key-Performance-Indicators (KPIs). To do this Nanaimo abstractions provide async interfaces to hardware either directly using available communication protocols (e.g. serial or IP networks) or by invoking a CLI provided by the instrument vendor. Because of this latter use case some instruments will require additional programs be available in a test environment.

Fig. 10 Example scenario using Nanaimo to test an I2C driver for a microcontroller.

This design is an amalgam of the TLYF (Test Like You Fly) methodology and the Swiss cheese model of failure analysis. Specifically; Nanaimo facilitates testing on actual or representative hardware for the first integration of software into a part or subassembly. Traditionally software engineers were responsible only for unit-testing and Software-In-the-Loop (SIL) simulation of their code. Nanaimo encourages software engineers to also provide hardware integration tests by enabling Hardware-In-the-Loop continuous-integration (HIL-CI, perhaps?).

Fig. 11 Hierarchy of system testing. Nanaimo focuses on part and subassembly testing.

Note

Nanaimo is named after Nanaimo bars which are about the best things humans have ever invented.

License

The MIT License (MIT)

Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.