Ansible Role: osm_alertmanager

Alertmanager is a tool that manages alerts received from Prometheus. It takes alerts from the Prometheus server, groups them by labels, and then sends them to various recipients like Email, PagerDuty, or Slack.

Alertmanager can be set up using command-line options and a configuration file. The command line is for system settings, while the configuration file contains details like recipients and routing.

Version History

Date	Version	Description	Changed By
May 2020	v0.0.1	Initial Draft	Abhishek Dubey
July 2020	v0.1.0	Added Slack and Google Chat integration	Mahesh Kumar
June 2022	v0.1.1	Added cluster support, custom rules, templates, AWS SNS, PagerDuty integration	Ishaan Ambashta

Supported OS

• CentOS 7
• CentOS 6
• Ubuntu Bionic
• Ubuntu Focal

Dependencies

• prometheus-server
• libselinux-python

Requirements

There are no specific requirements to run this role as it works on CentOS 6 and above, and Ubuntu 18 and above. The only requirement for CentOS 6 is libselinux-python, which is included. Basic requirements include:

• CentOS/Ubuntu Server
• Python installed on the target server
• Port 9093 needs to be open on the server

Role Variables

---
# Default settings for Alertmanager
alert_version: "0.20.0"
base_url: "https://github.com/prometheus/alertmanager/releases/download"
alertmanager_dir: "alertmanager-{{ version }}.linux-amd64"
download_url: "{{ base_url }}/v{{ version }}/{{ alertmanager_dir }}.tar.gz"
alertmanager_user: "alertmanager"
alertmanager_group: "alertmanager"
prometheus_user: "prometheus"
prometheus_group: "prometheus"
binary_path: "/usr/local/bin"

# Email Notification Settings
email_integration: "no"
sender_email: "[email protected]"
alertmanager_email: "[email protected]"
smtp_server: "smtp.gmail.com:587"

# Slack Integration
slack_integration: "yes"
slack_webhook: "https://hooks.slack.com/services/T00000000/B00/XXXXXX"
slack_channel_name: "alerting-channel"

# Google Chat Integration
google_chat_integration: "no"
calert_home: "/opt/calert"
calert_base_url: "https://github.com/mr-karan/calert/releases/download"
calert_url: "{{ calert_base_url }}/v1.2.1/calert_1.2.1_linux_amd64.tar.gz"
google_chat_room: "google-room-name"
room_webhook: "google-chat-room-webhook"

# PagerDuty Integration
pagerduty_integration: "no"
pagerduty_channel_name: "call-alerting-channel"
pagerduty_url: "https://events.pagerduty.com/v2/enqueue"

# SNS Integration
sns_integration: "no"
sns_channel_name: "sns"
region: "us-east-1"

You can specify any version of Prometheus and Alertmanager that you wish to install on your server.

Mandatory Variables

Variable	Default Value	Possible Values	Description
alert_version	"0.20.0"	Any Version	Version of Alertmanager to download
alertmanager_user	alertmanager	Any User	User for the Alertmanager service
alertmanager_group	alertmanager	Any Group	Group for the Alertmanager service
prometheus_user	prometheus	Any User	User for the Prometheus service
prometheus_group	prometheus	Any Group	Group for the Prometheus service

Optional Variables

Variable	Default Value	Description
base_url	https://github.com/prometheus/alertmanager/releases/download	Base URL for Alertmanager downloads
alertmanager_dir	alertmanager-{{ version }}.linux-amd64	Directory name after extracting Alertmanager
download_url	{{ base_url }}/v{{ version }}/{{ alertmanager_dir }}.tar.gz	Download link for Alertmanager
binary_path	/usr/local/bin	Path for the Alertmanager binary
email_integration	"no"	Enable or disable Email integration
sender_email	"send.test@example.com"	Email address for sending alerts
alertmanager_email	"alertmanager.test@example.com"	Email address for Alertmanager
smtp_server	"smtp.gmail.com:587"	SMTP server used by Alertmanager
slack_integration	"no"	Enable or disable Slack integration
slack_webhook	"https://hooks.slack.com/services/T00000000/B00/XXXXXX"	Webhook for Slack channel integration
slack_channel_name	"alerting-channel"	Name of Slack channel for alerts
google_chat_integration	"no"	Enable or disable Google Chat integration
calert_home	"/opt/calert"	Home directory for calert service
calert_base_url	"https://github.com/mr-karan/calert/releases/download"	Base URL for calert downloads
calert_url	"{{ calert_base_url }}/v1.2.1/calert_1.2.1_linux_amd64.tar.gz"	Download URL for calert binary
google_chat_room	"google-room-name"	Google Chat room for sending alerts
room_webhook	"google-chat-room-webhook"	Webhook for integrating with Google Chat
pagerduty_integration	"no"	Enable or disable PagerDuty integration
pagerduty_channel_name	"call-alerting-channel"	Name of PagerDuty channel for alerts
pagerduty_url	"https://events.pagerduty.com/v2/enqueue"	URL for PagerDuty
sns_integration	"no"	Enable or disable SNS integration with Alertmanager
sns_channel_name	"sns"	SNS channel name for sending AWS SNS alerts
region	"us-east-1"	Region for SNS alerts

Example Playbook

Here’s an example of the main playbook:

---
- hosts: alertmanager
  roles:
    - role: alertmanager

You can also specify a different user by setting become: yes:

---
- hosts: alertmanager
  roles:
    - role: alertmanager
      become: yes

Example Inventory

Create a host file with your server IPs like this:

[alertmanager]
10.1.1.100  ansible_user=ubuntu ansible_ssh_private_key_file=server1.pem

[prometheus]
10.1.1.100  ansible_user=ubuntu ansible_ssh_private_key_file=server1.pem

Note: Add Prometheus server IPs too. Configuration for Alertmanager will be added automatically in prometheus.yml.

To run the role, use this command:

ansible-playbook -i hosts site.yml

You can add multiple rules and template files using extra variables or a JSON file, like so:

By Passing Extra Variables

ansible-playbook -i hosts site.yml --extra-vars='{"rules_file": ["rule_file1","rule_file2"], "templates": ["file1.tmpl", "file2.tmpl"]}'

By Passing an Extra Variables File

ansible-playbook -i hosts site.yml  --extra-vars "@extra_vars.json"

Note:

1. Move your custom template or rule files into the osm_alertmanager/files folder first.
2. The extra variable file should be in JSON format, for example:

Extra Variables File - extra_vars.json

{
    "rules_file": ["rule_file1"], 
    "templates": ["file2.tmpl"]
}

Directory Structure of Role

Here’s the directory structure of this role:

osm_alertmanager
├── defaults
│   └── main.yml
├── files
│   ├── alertmanager.init
│   ├── email.tmpl
│   ├── sns.tmpl
│   ├── slack_notification.tmpl
│   ├── google_chat_calert.tmpl
│   ├── node_exporter.rules
│   ├── mysql_exporter.rules
│   ├── telegraf_kafka.rules
│   └── telegraf_node.rules
├── handlers
│   └── main.yml
├── README.md
├── site.yml
├── tasks
│   ├── calert.yml
│   ├── debian.yml
│   ├── main.yml
│   ├── prerequisites.yml
│   └── redhat.yml
└── templates
    ├── alertmanager.service.j2
    ├── alertmanager.yml.j2
    ├── calert.service.j2
    └── calert.toml.j2

After installing Alertmanager successfully, check the Alertmanager web interface at its URL.

License

BSD

References

• Prometheus Alertmanager Documentation

Author Information

This role is created and maintained by Abhishek Dubey. For any questions or suggestions, feel free to reach out to:

Abhishek Dubey
Ishaan Ambashta