Monitoring Role

This is an Ansible role designed to help manage monitoring tools in the Grafana ecosystem.

It uses Docker to set up Prometheus and Loki exporters:

• Node Exporter: Gathers general information about the status of a host.
• PostgreSQL Exporter: Monitors the status of a PostgreSQL server.
• Promtail: The main exporter for Loki, which is a log server similar to Prometheus and works with Grafana.

This role is compatible with some applications that use their default logging formats:

• Odoo12
• Coopdevs backups
• auth.log

Additionally, it supports a custom exporter that shows the active users in the system.

How to Use This Role

Public Variables

Node Exporter

# defaults/main.yaml
monitoring_nexporter_enabled: true
monitoring_nexporter_host: 127.0.0.1
monitoring_nexporter_port: 9100
monitoring_nexporter_docker_bind: "127.0.0.1:127.0.0.1:9100"
monitoring_nexporter_container_name: nexporter
monitoring_nexporter_image_version: latest

PostgreSQL Exporter

monitoring_postgres_exporter_enabled: true
monitoring_postgres_exporter_pg_user: "monitor_user"

Promtail

# defaults/main.yaml
monitoring_promtail_enabled: true
monitoring_promtail_host: 127.0.0.1
monitoring_promtail_port: 9080
monitoring_promtail_docker_bind: "127.0.0.1:127.0.0.1:9080"
monitoring_promtail_container_name: promtail
monitoring_promtail_image_version: latest

monitoring_promtail_modules_enabled:
  - "app2"

monitoring_promtail_modules_available:
  app1:
    log_path: "/var/log/app1/error.log"
    template: "app1.j2"
  app2:
    log_path: "/opt/app2/log/app2.log"
    template: "app2.j2"

monitoring_promtail_config_dir: "/etc/promtail"
monitoring_promtail_config_filename: "config.yml"

Active Users

monitoring_users_enabled: true
monitoring_users_host: 127.0.0.1
monitoring_users_port: 9839
monitoring_users_endpoint: "/metrics"
monitoring_users_prefix: "what"
monitoring_users_with_timestamp: false
monitoring_users_scrape_interval: 5000

Secret Variables

Promtail

monitoring_loki_user: "1234"
monitoring_loki_key: "eyJrIjoiM2VlZmM2NmQ4ZTQ4ZmE3MDRmZDBmMGE0YzNlNTE1MzRjZDdjNDY0N2YiLCJuIjoieW91ciBncmFmYW5hIGNsb3VkIGtleSIsImlkIjoxMjM0NTZ9"
monitoring_loki_hostname: "logs-somewhere.grafana.net"

PostgreSQL Exporter

monitoring_postgres_exporter_pg_password: "3%hyZ&toNZ#Xn74"
monitoring_postgres_exporter_pg_port: "3456"

Example Playbooks

Odoo with Promtail

# playbooks/odoo-promtail.yml
---
- name: Install Odoo with logs monitoring
  hosts: servers
  become: yes
  roles:
    - role: coopdevs.odoo_role
    - role: coopdevs.monitoring_role
      vars:
        monitoring_nexporter_enabled: false
        monitoring_promtail_enabled: true
        monitoring_promtail_modules_enabled: [ "odoo" ]
        monitoring_loki_user: "1234"
        monitoring_loki_key: "eyJrIjoiM2VlZmM2NmQ4ZTQ4ZmE3MDRmZDBmMGE0YzNlNTE1MzRjZDdjNDY0N2YiLCJuIjoieW91ciBncmFmYW5hIGNsb3VkIGtleSIsImlkIjoxMjM0NTZ9"
        monitoring_loki_hostname: "logs-somewhere.grafana.net"

PostgreSQL with System Metrics

# playbooks/postgres-nexporter.yml
---
- name: Install a database server with exposed system metrics
  hosts: servers
  become: yes
  roles:
    - role: geerlingguy.postgresql
    - role: coopdevs.monitoring_role
      vars:
        monitoring_nexporter_enabled: true
        monitoring_promtail_enabled: false

Security

This role makes a lot of data available through an HTTP server, which could be exploited. By default, it listens only on a local address, which is not accessible from the internet.

However, you may wish to have an external Prometheus server collect this data at intervals. To safeguard this data, some form of authentication from the Prometheus server to the host is recommended.

One way to do this is to keep the exporters listening on localhost and set up a reverse proxy before them with Basic Authentication using Nginx. Managing this setup and the associated keys is beyond the scope of this role.

Extension

Adding a New Prometheus Exporter

To create a new Prometheus exporter:

1. Copy the Node Exporter's section from defaults/main.yml and rename all nexporter instances to your desired name, e.g., someexporter.
2. Copy the relevant part of Node Exporter's templates/monitoring-docker-compose.yml.j2 and modify it to fit your requirements.
3. Update meta/main.yml: add a tag and change the description if necessary.
4. Update README.md.

Adding Promtail Support for a New App

To make the role compatible with an unsupported app:

1. Add it to monitoring_promtail_modules_available. Specify the log path and template name relevant to the app.
2. Duplicate the templates/promtail-config-apps/odoo-role.j2 file, renaming it to new-app.j2.
3. Adjust the template to match the needs of your app. Refer to the official documentation.

• Test the regex patterns at regexr or regex101.com.
• Include a comment with sample log entries for clarity, which will help those reading the regex in the future.
• Set the labels stage to determine which labels should be sent to Loki from the collected data.
• Set the timestamp stage to log the actual timestamp instead of when Promtail gathered it.
  • Use a full stop . as the decimal separator if possible, as Golang does not recognize commas ,. Reference this Golang issue.
  • Provide the timezone either from parsing (like in backups-role.j2) or manually (like in odoo-role.j2).
• Optionally include a match stage to discard log entries that do not match the regex.