Ansible-ELK

This is an Ansible Playbook that sets up the ELK/EFK Stack and the Filebeat client on remote machines.

What does it do?

• It automatically installs a complete ELK or EFK stack (Elasticsearch, Logstash/Fluentd, Kibana)
  • Supports ELK versions 5.6 and 2.4 as separate branches, while the master branch currently supports version 6.x.
  • Uses Nginx as a reverse proxy for Kibana, with an option to use Apache instead by setting apache_reverse_proxy: true.
  • Generates SSL certificates for Filebeat or Logstash-forwarder.
  • Configures firewall rules if they are enabled, using either iptables or firewalld.
  • Adjusts Elasticsearch heap size to half of your memory, with a maximum of 32GB.
  • Deploys ELK clients using SSL and Filebeat for Logstash by default.
  • If Fluentd is used instead of Logstash, installs rsyslog to collect OpenStack logs from /var/log/*.
  • Service ports can be changed in install/group_vars/all.yml.
  • Optionally, installs curator and Elastic X-Pack Suite.
  • Available on Ansible Galaxy.

Requirements

• A RHEL7 or CentOS7 server/client without modifications.
• For ELK clients using Filebeat, you can also use RHEL7/CentOS7, Rocky, or Fedora.
• An ELK/EFK server with at least 8GB of RAM (you can try with less, but 5.x versions are resource-heavy; 2.4 series is better for limited resources).
• You might need to adjust vm.swappiness since ELK/EFK is resource-intensive and excessive swapping can slow down responsiveness.
  • Set it as follows:

  echo "vm.swappiness=10" >> /etc/sysctl.conf
  sysctl -p
  

Notes

• The current ELK version is 6.x, but you can use the 5.6 or 2.4 branches if needed.
• I will continue to update this playbook for new major ELK versions as time permits.
• The nginx htpasswd is initially set to admin/admin.
• The default nginx ports are 80/8080 for Kibana and SSL certificate retrieval (these can be changed).
• Uses OpenJDK for Java.
• The installation process is fairly quick, taking about 3 minutes on a test virtual machine.
• Fluentd can replace Logstash by setting logging_backend: fluentd in group_vars/all.yml.
• Curator can be installed by setting install_curator_tool: true in install/group_vars/all.yml.
• X-Pack Suite for Elasticsearch, LogStash, or Kibana can be installed by enabling:
  • install_elasticsearch_xpack: true
  • install_kibana_xpack: true
  • install_logstash_xpack: true
  • Note: X-Pack adds extra security and authentication, so Kibana will need its own credentials - default username: elastic, password: changeme.

ELK/EFK Server Instructions

1. Clone the repository and prepare your hosts file:

git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
sed -i 's/host-01/elkserver/' hosts
sed -i 's/host-02/elkclient/' hosts

2. If using a non-root user for Ansible (like ec2-user in AWS), set this in your variables:

ansible_system_user: ec2-user

3. Run the playbook:

ansible-playbook -i hosts install/elk.yml

4. (Check playbook messages)
5. Access ELK at http://host-01:80 (default, nginx) or http://host-01/kibana (if using apache).
6. Default login:
   • Username: admin
   • Password: admin

Create your Kibana Index Pattern

1. Login to Kibana and create your index pattern.

2. Sample data can be helpful, but you can add it later.

3. You're now ready to set up your clients to send data via Filebeat/SSL.

ELK Client Instructions

1. Run the client playbook against the elk_server variable you generated:

ansible-playbook -i hosts install/elk-client.yml --extra-vars 'elk_server=X.X.X.X'

2. Once finished, go back to your ELK instance and you should start seeing log results from ELK/EFK clients via Filebeat.

5.6 ELK/EFK (Deprecated)

• The 5.6 version of ELK/EFK is still available. To use it, switch to the 5.6 branch:

git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
git checkout 5.6

2.4 ELK/EFK (Deprecated)

• The 2.4 version of ELK/EFK is also available. To use it, switch to the 2.4 branch:

git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
git checkout 2.4

• You can watch a deployment video here:

File Hierarchy

.
├── hosts
├── install
│   ├── elk_client.yml
│   ├── elk.yml
│   ├── group_vars
│   │   └── all.yml
│   └── roles
│       ├── apache
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       ├── 8080vhost.conf.j2
│       │       └── kibana.conf.j2
│       ├── curator
│       │   ├── files
│       │   │   └── curator.repo
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       ├── curator-action.yml.j2
│       │       └── curator-config.yml.j2
│       ├── elasticsearch
│       │   ├── files
│       │   │   ├── elasticsearch.in.sh
│       │   │   └── elasticsearch.repo
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       └── elasticsearch.yml.j2
│       ├── elk_client
│       │   ├── files
│       │   │   └── elk.repo
│       │   └── tasks
│       │       └── main.yml
│       ├── filebeat
│       │   ├── meta
│       │   │   └── main.yml
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       ├── filebeat.yml.j2
│       │       └── rsyslog-openstack.conf.j2
│       ├── firewall
│       │   ├── handlers
│       │   │   └── main.yml
│       │   └── tasks
│       │       └── main.yml
│       ├── fluentd
│       │   ├── files
│       │   │   ├── filebeat-index-template.json
│       │   │   └── fluentd.repo
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       ├── openssl_extras.cnf.j2
│       │       └── td-agent.conf.j2
│       ├── heartbeat
│       │   ├── meta
│       │   │   └── main.yml
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       └── heartbeat.yml.j2
│       ├── instructions
│       │   └── tasks
│       │       └── main.yml
│       ├── kibana
│       │   ├── files
│       │   │   └── kibana.repo
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       └── kibana.yml.j2
│       ├── logstash
│       │   ├── files
│       │   │   ├── filebeat-index-template.json
│       │   │   └── logstash.repo
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       ├── 02-beats-input.conf.j2
│       │       ├── logstash.conf.j2
│       │       └── openssl_extras.cnf.j2
│       ├── metricbeat
│       │   ├── meta
│       │   │   └── main.yml
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       └── metricbeat.yml.j2
│       ├── nginx
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       ├── kibana.conf.j2
│       │       └── nginx.conf.j2
│       ├── packetbeat
│       │   ├── meta
│       │   │   └── main.yml
│       │   ├── tasks
│       │   │   └── main.yml
│       │   └── templates
│       │       └── packetbeat.yml.j2
│       └── xpack
│           └── tasks
│               └── main.yml
└── meta
    └── main.yml

This project consists of several directories and files necessary for setting up the ELK stack.