Celery Worker Setup Guide

Overview

This guide explains how to configure a group of Celery workers using systemd.

Worker Configuration

Each worker must be defined with specific details, such as:

- app_name: 'MyApp'                      # (required)
  app_variable: 'my_app'                 # (default: `celeryw_app_variable`)
  celeryd_bin: '/usr/local/bin/celery'   # (default: `celeryw_bin`)
  group: "adm"                           # (default: `celeryw_`)
  log_dir:                               # (default: `celeryw_log_dir`)
  log_filename: "%n%I"                   # (default: `celeryw_log_filename`)
  log_level: "INFO"                      # (default: `celeryw_log_level`)
  pid_filename: `%n-master.pid`          # (default: `celeryw_pid_filename`)
  queues: `%n-master.pid`                # (default: `celeryw_queues`)

You can customize the variables listed in the Role Variables section for each worker. However, you must provide app_name and app_module for every worker.

Requirements

No specific requirements for this role.

Role Variables

All variables for this role start with the prefix celeryw_.

Default Variables

• celeryw_app_variable: Variable holding the Celery app reference (default: "app").
• celeryw_bin: Path to the Celery binary (default: "/usr/bin/celery")
• celeryw_daemon_options: Additional options for the Celery command line (default: "").
• celeryw_etc_dir: Directory for storing Celery worker configuration files (default: "/etc/celery").
• celeryw_group: Group under which the Celery processes run (default: {{ansible_user_id}}).
• celeryw_log_dir: Directory for storing Celery log files (default: /var/log/celery).
• celeryw_log_filename: Template for log filenames (default: %n%I.log).
• celeryw_log_level: Default log level (default: ERROR).
• celeryw_pid_filename: Template for PID filenames (default: %n.pid).
• celeryw_queues: Default queues (default: "celery").
• celeryw_run_dir: Directory for storing PID files (default: /var/run/celery).
• celeryw_user: User identity for running Celery processes (default: {{ansible_user_id}}).
• celeryw_workers: List of workers to set up (default: []).

Other Role Variables

None specified.

Dependencies

To use this role for managing Celery workers with systemd, it requires the cans.systemd-unit-install role.

Example Playbooks

Simple Worker Configuration

This example sets up workers for two applications, using mostly default values:

- hosts: servers
  roles:
     - role: cans.celery-worker
       celeryw_workers:
         - app_name: "mailer"
           app_module: "application.interfaces.tasks.mailer"
         - app_name: "data-cruncher"
           app_module: "application.interfaces.tasks.cruncher"

Advanced Worker Configuration

The following example overrides some default values at the global level and some within the worker definitions:

- hosts: servers
  vars:
     celeryw_bin: "/opt/local/python-virtualenvs/python3.6/bin/celery"
  roles:
     - role: cans.celery-worker
       celeryw_workers:
         - app_name: "mailer"
           app_module: "application.interfaces.tasks.mailer"
         - app_name: "data-cruncher"
           app_module: "application.interfaces.tasks.cruncher"
           app_variable: "cruncher_app"
         - app_name: "legacy-data-cruncher"
           app_module: "legacy.interfaces.tasks.cruncher"
           conf_dir: "/opt/local/etc/celery"
           celery_bin: "/opt/local/python-virtualenvs/python2.5/bin/celery"

For more examples, check the playbook in the test directory.

License

GPLv2

Author Information

Copyright © 2017, Nicolas CANIART.