geerlingguy.logstash

Ansible Role: Logstash

This is an Ansible Role that installs Logstash on RedHat/CentOS and Debian/Ubuntu systems.

By default, this role installs a syslog grok pattern. If you want to add more filters, you can place them in the /etc/logstash/conf.d/ directory. For example, create a file named 13-myapp.conf with your grok filter and restart Logstash to apply the changes. You can test your grok regex using the Grok Debugger.

Requirements

This role works best with Elasticsearch, which serves as a backend for storing log messages.

Role Variables

Here are the available variables, along with their default values (check defaults/main.yml for details):

logstash_version: '7.x'
The version of Logstash to install.
logstash_package: logstash
The specific package to install. You can change the package name to specify a version compatible with your platform and package manager.
logstash_listen_port_beats: 5044
The port that Logstash listens to for beats.
logstash_elasticsearch_hosts:
- http://localhost:9200
  The hosts where Logstash sends logs to Elasticsearch.
logstash_dir: /usr/share/logstash
The directory where Logstash is installed.
logstash_ssl_dir: /etc/pki/logstash
logstash_ssl_certificate_file: logstash-forwarder-example.crt
logstash_ssl_key_file: logstash-forwarder-example.key
These are local paths for the SSL certificate and key files, which will be copied to logstash_ssl_dir.

For details on generating and using self-signed certificates with Logstash and Filebeat, refer to Generating a self-signed certificate.

logstash_local_syslog_path: /var/log/syslog
logstash_monitor_local_syslog: true
This checks if the configuration for the local syslog file (defined as logstash_local_syslog_path) should be added to Logstash. Set to false if you're monitoring local syslog in a different way or don't need it.
logstash_enabled_on_boot: true
Set to false if you don’t want Logstash to start automatically on system start-up.
logstash_install_plugins:
- logstash-input-beats
- logstash-filter-multiline
  A list of Logstash plugins to install.
logstash_setup_default_config: true
Set to false if you do not want to include the default configuration files provided with this role (located in files/filters). You can create your own config files in /etc/logstash/conf.d.

Generating a Self-signed Certificate

For the best security, use your own valid certificate and key. Update the logstash_ssl_* variables in your playbook to reflect your certificate.

To create a self-signed certificate/key pair, use the following command:

$ openssl req -x509 -batch -nodes -days 3650 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -subj '/CN=example.com'

Note that Filebeat and Logstash may not work correctly with self-signed certificates unless the full chain of trust (including your Certificate Authority) is on your server. More info is available in this GitHub issue: https://github.com/elastic/logstash/issues/4926#issuecomment-203936891.

Recent versions of Filebeat and Logstash require a pkcs8-formatted private key. Convert the previously generated key like this:

openssl pkcs8 -in logstash.key -topk8 -nocrypt -out logstash.p8

Other Notes

If you notice high CPU usage from the Logstash processes, it might be due to the logstash-web process getting stuck trying to start on port 80 (if another app is using it). To fix this, you can add a limit to the respawn statement in /etc/init/logstash-web.conf, or you can disable the logstash-web service in your playbook, like so:

- name: Ensure logstash-web process is stopped and disabled.
  service: name=logstash-web state=stopped enabled=no

Example Playbook

- hosts: search
  
  pre_tasks:
    - name: Use Java 8 on Debian/Ubuntu.
      set_fact:
        java_packages:
          - openjdk-8-jdk
      when: ansible_os_family == 'Debian'
  
  roles:
    - geerlingguy.java
    - geerlingguy.elasticsearch
    - geerlingguy.logstash