Ansible Role: airflow

This role installs and sets up an airflow server. It can configure both master and worker nodes.

Requirements

You need sudo access to the server you're setting up. Use the following code to run commands as root.

- name: Set up airflow master hosts
  hosts: airflow_hosts
  become: yes
  roles:
    - marcusianlevine.airflow

Role Variables

Basic configuration variables for airflow:

# The user under which airflow will run
airflow_user: airflow

# The group under which airflow will run
airflow_user: "{{ airflow_user }}

# The version of airflow you want to install
airflow_version: 1.8.1

# The directory where airflow will be installed
airflow_install_directory: /usr/share/airflow

# Additional packages to install for airflow using pip
airflow_extra_packages:
  - celery
  - alldbs

# Services that will run on the server
airflow_services:
  - webserver
  - scheduler
  - worker

Additional settings for specific setups:

# Choose whether to use a process manager
use_systemd: yes
use_supervisor: no

# Option to install airflow in a virtual environment
# NOTE: Needed for use with ansible-container
virtualenv_path: undefined

Configuring airflow.cfg

The main configuration is done with YAML objects. The default settings in default_airflow are combined with any custom settings from the airflow object. All keys will be included in the airflow.cfg file under the right section. All values are the default airflow values.

airflow:
  core:
    dags_folder: "/usr/share/airflow_dags"
    base_log_folder: "/var/log/airflow"

  webserver:
    base_url: "{{ airflow_base_url }}"

  scheduler:
    statsd_on: True

Dependencies

None

Example Playbook

play.yml

- hosts: airflow-masters
  become: yes
  roles:
    - creditcardscom.airflow

vars/main.yml

airflow_user: airflow_user
airflow_version: 1.8.0
airflow_services:
  - webserver
  - scheduler

airflow:
  core:
    dags_folder: "/home/{{ airflow_user }}/dags"

License (MIT)

Copyright (c) 2017 CreditCards.com

You can use, copy, modify, and share this software for free. Just include the copyright notice and this permission notice in all copies or significant parts of the software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT ANY WARRANTY. THE AUTHORS ARE NOT RESPONSIBLE FOR ANY CLAIMS OR DAMAGES RELATED TO THE SOFTWARE.