ansible-openwisp2-imagegenerator

This Ansible role allows you to create several OpenWISP2 firmware images for different organizations while managing their configurations.

NOTE: This role has not been tested in a production environment yet. If you want to use it, please test it first and if you encounter any issues, report them. Be ready to learn about the build process and how it works.

Required Role Variables

The following variables are necessary:

• openwisp2fw_source_dir: the directory containing the OpenWRT source files used during compilation.
• openwisp2fw_generator_dir: the directory where the image generators are set up.
• openwisp2fw_bin_dir: the directory where the final images will be saved.
• openwisp2fw_organizations: a list of organizations; check the example playbook.yml file in the create playbook section for its format.

Usage (Tutorial)

If you're unfamiliar with Ansible, don’t worry! This guide will walk you through the steps.

If you're already comfortable with Ansible, you can skip to the "Install this role" section.

Key Concepts:

• "Compilation Server" refers to the server that compiles the images.
• "Local Machine" refers to the host from which you run Ansible, such as your laptop or CI server.

Ansible automates configuration management by using SSH to connect to the compilation server and executing a set of commands.

1. Install Ansible

If you haven't done so already, install Ansible on your local machine. It's easiest to use the official Python package manager:

sudo pip install ansible

If you need help with pip, check Installing pip.

You can also install Ansible through other methods as long as you install a version from the 2.0.x series, which is the version we tested with this playbook.

2. Install This Role

To keep it simple, install this role on your local machine using ansible-galaxy:

sudo ansible-galaxy install openwisp.openwisp2-imagegenerator

3. Choose a Working Directory

Create a working directory on your local machine for the configuration files of your firmware images:

mkdir ~/my-openwisp2-firmware-conf
cd ~/my-openwisp2-firmware-conf

Version-controlling this directory is a great idea to track changes, roll back, and have continuous integration for automatic builds.

4. Create Inventory File

The inventory file defines the group of servers. In our simple case, we will have just one server.

Create a new file named hosts on your local machine with the following content:

[myserver]
mycompiler.mydomain.com ansible_user=<youruser> ansible_become_pass=<sudo-password>

Replace mycompiler.mydomain.com with your server's hostname or IP address. Enter your SSH username and sudo password in place of <youruser> and <sudo-password> respectively. These are needed for the Installation of dependencies step.

5. Create Playbook File

Create a new playbook file named playbook.yml on your local machine with the following content:

# playbook.yml
- hosts: your_host_here
  roles:
    - openwisp.openwisp2-imagegenerator
  vars:
    openwisp2fw_source_dir: /home/user/openwisp2-firmware-source
    openwisp2fw_generator_dir: /home/user/openwisp2-firmware-generator
    openwisp2fw_bin_dir: /home/user/openwisp2-firmware-builds
    openwisp2fw_source_targets:
        - system: ar71xx
          subtarget: generic
          profile: Default
        - system: x86
          subtarget: generic
          profile: Generic
    openwisp2fw_organizations:
        - name: snakeoil # organization name
          flavours: # supported flavours
            - standard
          luci_openwisp: # configuration for luci_openwisp
            username: "operator"
            password: "<CHANGE_ME>" # this will be encrypted
          openwisp: # configuration for openwisp
            url: "https://my-openwisp2-instance.com"
            shared_secret: "my-openwisp2-secret"
            unmanaged: "{{ openwisp2fw_default_unmanaged }}"
          root_password: "<CHANGE_ME>" # this will also be encrypted

This playbook will compile firmware images for an organization named snakeoil using the standard flavour for two architectures: ar71xx and x86.

At this point, your directory should look like this:

.
├── hosts
└── playbook.yml

6. Run the Playbook

Now it’s time to start compiling OpenWISP2 firmware.

Run the playbook from your local machine with this command:

ansible-playbook -i hosts playbook.yml -e "recompile=1 cores=4"

You can change cores=4 to match the number of available CPU cores.

When the playbook finishes, the generated images will be on the compilation server in the directory set in openwisp2fw_bin_dir, which is /home/user/openwisp2-firmware-builds in this example. The folder structure will be like this:

/home/user/openwisp2-firmware-builds
├── snakeoil/
├── snakeoil/2016-12-02-094316/ar71xx/standard/
├── snakeoil/2016-12-02-094316/x86/standard/
└── snakeoil/latest/ # latest version link

If everything worked well, you're ready to customize your configuration as needed!

Role Variables

You can customize many variables as needed; look at the defaults for a complete list.

Some of these variables are explained in the sections on Organizations and Flavours.

Organizations

If you’re using OpenWISP, you might need to compile images for various groups such as for-profit clients, non-profits, or any defined group known as an "organization".

You can freely define organizations in openwisp2fw_organizations.

Refer to the "example playbook.yml file" for guidance.

If you need to add specific files to each organization’s images, see "Adding files for specific organizations".

Flavours

A flavour is a mix of packages included in an image.

You might create different flavours based on your needs, such as:

• standard: the most common option
• minimal: for devices with limited storage
• mesh: for setups using mesh networking

By default, only a standard flavour is available.

To create your own flavours, define openwisp2fw_image_flavours. Check the default variables for its structure.

Build Process

The build process has several steps:

1. Install Dependencies

In this phase, necessary operating system dependencies for subsequent steps are installed or updated.

2. Compilation

The OpenWRT source is compiled to create an "Image Generator". This archive contains precompiled packages and a Makefile for generating customized images.

The source files are downloaded and compiled in the openwisp2fw_source_dir.

3. Preparation of Generators

During this step, image generators are prepared for building various images for different organizations and flavours.

The generators are prepared in the openwisp2fw_generator_dir.

4. Build Final Images

Here, a series of images is produced. Numerous images will be built for each architecture, organization, and flavour, which can create many files, so proceed cautiously.

For instance, if you pick 2 architectures (ar71xx and x86), 2 organizations (A and B), and 2 flavours (standard and mini), you will end up with 8 image groups.

The images will be saved in the openwisp2fw_bin_dir.

5. Upload Images to OpenWISP Firmware Upgrader

The final step is to upload images to the OpenWISP Firmware Upgrader module. This step is optional and disabled by default.

To enable it, configure openwisp2fw_uploader and openwisp2fw_organizations.categories as shown below:

- hosts:
    - myhost
  roles:
    - openwisp.openwisp2-imagegenerator
  vars:
    openwisp2fw_controller_url: "https://openwisp.myproject.com"
    openwisp2fw_organizations:
      - name: staging
        flavours:
          - default
        openwisp:
          url: "{{ openwisp2fw_controller_url }}"
          shared_secret: "xxxxx"
        root_password: "xxxxx"
        categories:
          default: <CATEGORY-UUID>
      - name: prod
        flavours:
          - default
        openwisp:
          url: "{{ openwisp2fw_controller_url }}"
          shared_secret: "xxxxx"
        root_password: "xxxxx"
        categories:
          default: <CATEGORY-UUID>
    openwisp2fw_uploader:
        enabled: true
        url: "{{ openwisp2fw_controller_url }}"
        token: "<REST-API-USER-TOKEN>"
        image_types:
            - ath79-generic-ubnt_airrouter-squashfs-sysupgrade.bin
            # Other image types...

Replace the placeholders with the appropriate values.

The upload script creates a new build object and uploads the firmware images listed in image_types that match identifiers defined in hardware.py file of OpenWISP Firmware Upgrader.

Important Points about upload_firmware.py

• The script reads version information from the .config file of the OpenWrt source code.
• It checks for existing builds with the same version and category to add new images, preventing duplication.

Adding Files to Images

You can add arbitrary files to every generated image by placing them in a files/ directory in your playbook folder.

Example structure:

.
├── hosts
├── playbook.yml
└── files/etc/profile

The files/etc/profile will be added to every image.

Adding Files for Specific Organizations

You can also add custom files to specific organizations’ images.

For instance, if you have an organization snakeoil and want to add a custom banner, create this structure:

.
├── hosts
├── playbook.yml
└── organizations/snakeoil/etc/banner

Since this is done just before final image building, you can replace any files created earlier.

Extra Parameters

You can provide extra parameters to ansible-playbook:

• recompile: repeat the compilation step.
• cores: number of cores for compilation.
• orgs: a list of organization names to limit image generation.

Examples

Recompile with 16 cores:

ansible-playbook -i hosts playbook.yml -e "recompile=1 cores=16"

Generate images for only organization foo:

ansible-playbook -i hosts playbook.yml -e "orgs=foo"

Generate images for organizations foo and bar:

ansible-playbook -i hosts playbook.yml -e "orgs=foo,bar"

Run Specific Steps

You can run individual steps using Ansible tags.

Example 1: run only the preparation of generators:

ansible-playbook -i hosts playbook.yml -t generator

Example 2: run both preparation and build steps:

ansible-playbook -i hosts playbook.yml -t generator,build

Targets Without Subtargets

To compile targets that don’t specify a subtarget (e.g., sunxi, ARMv8, QEMU), set openwisp2fw_source_targets like this:

openwisp2fw_source_targets:
    - system: sunxi
      profile: sun7i-a20-lamobo-r1
    - system: armvirt
      profile: Default

Support

For assistance, see OpenWISP Support Channels.