ansible-tap

Test Anything Protocol (TAP) producer for Ansible.

This callback plugin allows you to write TAP test suites as Ansible playbooks. Consider it an Ansible-only alternative to Serverspec and Testinfra.

Requirements

• Ansible 2
• Python 2.7+

Install

While you can't install this plugin directly using ansible-galaxy, you can use ansible-galaxy to download it:

ansible-galaxy install -p roles/ benwebber.tap

Navigate to the role directory and run make install:

cd roles/benwebber.tap
make install

This will copy the plugin to ~/.ansible/plugins/callback.

Usage

Configure Ansible to use this plugin as the standard output callback:

ANSIBLE_STDOUT_CALLBACK=tap ansible-playbook -i hosts test.yml -l hostname

You can also set it to be the default callback in ansible.cfg:

[defaults]
stdout_callback=tap

Writing Ansible tests

By default, Ansible will abort the play if any tasks fail. Set ignore_errors: true on all tests to disable this behaviour:

- name: check if service is running
  command: systemctl is-active service
  register: is_active
  tags: diagnostic

- name: assert that service is running
  assert:
    that: is_active.rc == 0
  ignore_errors: true

This will ensure Ansible executes the entire test suite, barring any unexpected failure.

If you have a lot of tests, you can set ignore_errors: true on a block:

- name: check if service is running
  command: systemctl is-active service
  register: is_active
  tags: diagnostic

- name: check if service is enabled
  command: systemctl is-enabled service
  register: is_enabled
  tags: diagnostic

- ignore_errors: true
  block:
  - name: assert that service is running
    assert:
      that: is_active.rc == 0

  - name: assert that service is enabled
    assert:
      that: is_enabled.rc == 0

If a task fails, the plugin will output troubleshooting information as an embedded YAML document:

not ok - assert: assert that variable is set
  ---
  _ansible_no_log: false
  _ansible_verbose_always: true
  assertion: status is defined
  changed: false
  evaluated_to: false
  failed: true
  invocation:
    module_args:
      that: status is defined
    module_name: assert
  ...

Excluding tasks from TAP stream

Often, the result of a test will depend on previous tasks. You will naturally want to exclude these setup tasks from the TAP stream.

To do so, simply tag setup tasks with the diagnostic tag:

- name: set up next test
  command: 'true'
  register: true_
  tags: diagnostic

- name: should always pass
  assert:
    that: true_.rc == 0
  ignore_errors: true

The callback plugin will print diagnostic lines instead of test lines:

# command: set up next test
ok - assert: should always pass

Unlike individual test cases, you probably do not want to ignore errors for this type of task. Failures would represent an error in the test suite and not a test failure.

Expected failures and unexpected successes

TAP supports a TODO directive to ignore tests for features that haven't been implemented yet.

If a test marked with TODO fails, TAP consumers will consider it an expected failure. Similarly, if a test marked with TODO passes, TAP consumers will consider it an unexpected success.

Tag expected failures with TODO:

- name: expected failure
  assert:
    that: false
  ignore_errors: true
  tags: TODO

This will output a # TODO directive in TAP stream:

not ok - assert: expected failure # TODO

If the test passes, you'll receive unexpected success output:

ok - assert: expected failure # TODO

Skipping tests

TAP also supports a SKIP directive to ignore specific tests.

This callback uses Ansible's when statement to control skipped tests:

- name: this is a skipped task
  assert:
    that: false
  ignore_errors: true
  when: false

The reason for skipping the test will appear in the test line:

ok - assert: skipped # SKIP Conditional check failed

Example

The tests/ directory contains an example test suite which produces all possible test results.

After installing the plugin, run the test suite with:

ANSIBLE_STDOUT_CALLBACK=tap ansible-playbook -i localhost, -c local tests/playbooks/test_multiple_with_failures.yml

You will receive the following TAP stream. You can pass this to any TAP consumer.

TAP version 13
# command: set up next test
ok - assert: pass
not ok - assert: failed
  ---
  _ansible_no_log: false
  _ansible_verbose_always: true
  assertion: false
  changed: false
  evaluated_to: false
  failed: true
  invocation:
    module_args:
      that: false
    module_name: assert
  ...
not ok - assert: expected failure # TODO
ok - assert: unexpected pass # TODO
ok - assert: skipped # SKIP Conditional check failed
1..5

Caveats

At present, this plugin only supports running tests against a single host at a time. The TAP specification does not describe a way to combine multiple output streams.

License

MIT