Ansible Role: airflow

This Ansible role installs and sets up an Airflow server. It can configure both master and worker nodes.

Requirements

You will need sudo access on the server you're configuring. Use the following command to run as root:

- name: Setup airflow master servers
  hosts: airflow_hosts
  become: yes
  roles:
    - creditcardscom.airflow

Role Variables

Here are the basic configuration options for Airflow:

# User account under which Airflow runs
airflow_user: airflow

# Version of Airflow to install
airflow_version: 1.8.1

# Directory where Airflow will be installed
airflow_install_directory: /usr/share/airflow

# Additional Python packages to install via pip
airflow_extra_packages:
  - celery
  - alldbs

# Services that will be started
airflow_services:
  - webserver
  - scheduler
  - worker

airflow.cfg

The main configuration for Airflow is done using YAML. The default settings from default_airflow are combined with any custom settings in the airflow section. All keys will be included in the airflow.cfg file under the right sections. All values listed are the default settings for Airflow.

airflow:
  core:
    dags_folder: "/usr/share/airflow_dags"
    base_log_folder: "/var/log/airflow"

  webserver:
    base_url: "{{ airflow_base_url }}"

  scheduler:
    statsd_on: True

Dependencies

No dependencies required.

Example Playbook

play.yml

- hosts: airflow-masters
  become: yes
  roles:
    - creditcardscom.airflow

vars/main.yml

airflow_user: airflow_user
airflow_version: 1.8.0
airflow_services:
  - webserver
  - scheduler

airflow:
  core:
    dags_folder: "/home/{{ airflow_user }}/dags"

License (MIT)

Copyright (c) 2017 CreditCards.com

You can freely use, copy, modify, merge, publish, distribute, sublicense, and sell this software and its documentation, as long as this copyright notice is included in all copies or significant parts of the software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED. The authors are not liable for any claims, damages, or other liabilities arising from the use of this software.