Ansible Role: Logstash

An Ansible Role that installs Logstash on RedHat/CentOS Debian/Ubuntu.

Note that this role installs a syslog grok pattern by default; if you want to add more filters, see the documentation for logstash_configuration_files and logstash_filter_files.

As an example, you could create a file named 13-myapp.conf with the appropriate grok filter and restart logstash to start using it. Test your grok regex using the Grok Debugger.

Requirements

Though other methods are possible, this role is made to work with Elasticsearch as a backend for storing log messages.

Role Variables

Available variables are listed below, along with default values (see defaults/main.yml):

logstash_listen_port_beats: 5044

The port over which Logstash will listen for beats.

logstash_version: 2.3

Default major version of Logstash to install.

logstash_configuration_files:
  - 01-lumberjack-input.conf
  - 30-lumberjack-output.conf

logstash_filter_files:
  - 10-syslog.conf
  - 11-nginx.conf
  - 12-apache.conf
  - 14-solr.conf
  - 15-drupal.conf
    logstash_elasticsearch_host: localhost

List of inputs/outputs (configuration files) and filter files to install. By default the lumberjack input/output and variable filter files are installed. You can specify your own configuration files by placing input/output files in your own templates directory, and filters in your own files directory. Then create your own list in the vars/main.yml directory, and vars_files in your wrapper role, and include this role to deploy Logstash with your own configuration. See Example Playbook below.

logstash_elasticsearch_hosts:
  - http://localhost:9200

The hosts where Logstash should ship logs to Elasticsearch.

logstash_ssl_dir: /etc/pki/logstash
logstash_ssl_certificate_file: logstash-forwarder-example.crt
logstash_ssl_key_file: logstash-forwarder-example.key

SSL configuration for Logstash to accept requests from logstash-forwarder running on remote hosts. Security note: On production or public-facing (e.g. any non-test) servers, you should create your own key/certificate pair and use that instead of the included default! You can use OpenSSL to create the key and certificate files, with a command like the following: openssl req -x509 -batch -nodes -days 3650 -newkey rsa:2048 -keyout logstash.key -out logstash.crt.

Note that filebeat and logstash may not work correctly with self-signed certificates unless you also have the full chain of trust (including the Certificate Authority for your self-signed cert) added on your server. See: https://github.com/elastic/logstash/issues/4926#issuecomment-203936891

For the logstash_ssl_certificate_file and logstash_ssl_key_file, you can provide a path relative to the role directory, or an absolute path to the file.

logstash_local_syslog_path: /var/log/syslog
logstash_monitor_local_syslog: true

Whether configuration for local syslog file (defined as logstash_local_syslog_path) should be added to logstash. Set this to false if you are monitoring the local syslog differently, or if you don't care about the local syslog file. Other local logs can be added by your own configuration files placed inside /etc/logstash/conf.d.

logstash_enabled_on_boot: yes

Set this to no if you don't want logstash to run on system startup.

logstash_install_plugins:
  - logstash-input-beats

A list of Logstash plugins that should be installed.

logstash_install_dir: /usr/share/logstash

Path to installation directory. Formerly "/opt/logstash" was hardcoded, now the default is "/usr/share/logstash".

Other Notes

If you are seeing high CPU usage from one of the logstash processes, and you're using Logstash along with another application running on port 80 on a platform like Ubuntu with upstart, the logstash-web process may be stuck in a loop trying to start on port 80, failing, and trying to start again, due to the restart flag being present in /etc/init/logstash-web.conf.

To avoid this problem, either change that line to add a limit to the respawn statement, or set the logstash-web service to enabled=no in your playbook, e.g.:

- name: Ensure logstash-web process is stopped and disabled.
  service: name=logstash-web state=stopped enabled=no

Example Playbook

- hosts: search
  vars_files:
    - vars/main.yml

  roles:
    - geerlingguy.elasticsearch
    - geerlingguy.logstash

License

MIT / BSD

Author Information

This role was created in 2014 by Jeff Geerling, author of Ansible for DevOps.