etcd

ansible-role-etcd

This Ansible role is used in Kubernetes the not so hard way with Ansible - etcd cluster. But it can be used without a Kubernetes cluster of course.

Installs a etcd cluster. HINT: This playbook does NOT reload or restart the etcd cluster processes after the systemd service file was changed! This is intentional! It would be a very bad idea to restart all etcd processes at the same time. So if the etcd.service file has changed restart/reload etcd by manually one node after the other and check log output if the node joined the cluster again afterwards! Of course this process can be automated too but it's currently not part of this role. As a side node: The script will issue a systemctl daemon-reload after the etcd service file was changed so that at least systemd is aware of the changed file and you don't take care about that. So a reboot of a etcd node would also active the new configuration.

Upgrading a etcd cluster which was installed by this role is described in here.

Versions

I tag every release and try to stay with semantic versioning. If you want to use the role I recommend to checkout the latest tag. The master branch is basically development while the tags mark stable releases. But in general I try to keep master in good shape too. A tag 13.1.1+3.5.13 means this is release 13.1.1 of this role and it's meant to be used with etcd version 3.5.13 (but should work with newer versions also). If the role itself changes X.Y.Z before + will increase. If the etcd version changes X.Y.Z after + will increase. This allows to tag bugfixes and new major versions of the role while it's still developed for a specific etcd release.

Changelog

see CHANGELOG.md

Requirements

This role requires that you already created some certificates for etcd (see Kubernetes the not so hard way with Ansible - Certificate authority (CA) and Ansible role kubernetes_ca). The playbook searches the certificates in etcd_ca_conf_directory on the host this playbook runs. Of course you can create the certificates on your own (see Generate self-signed certificates - Git repository is archived but information is still valid).

Role Variables

# The directory from where to copy the etcd certificates. By default this
# will expand to user's LOCAL $HOME (the user that run's "ansible-playbook ..."
# plus "/etcd-certificates". That means if the user's $HOME directory is e.g.
# "/home/da_user" then "etcd_ca_conf_directory" will have a value of
# "/home/da_user/etcd-certificates".
etcd_ca_conf_directory: "{{ '~/etcd-certificates' | expanduser }}"

# etcd Ansible group
etcd_ansible_group: "k8s_etcd"

# etcd version
etcd_version: "3.5.13"

# Port where etcd listening for clients
etcd_client_port: "2379"

# Port where etcd is listening for it's peer's
etcd_peer_port: "2380"

# Interface to bind etcd ports to
etcd_interface: "tap0"

# Run etcd daemon as this user.
#
# Note 1: If you want to use an "etcd_peer_port" < 1024 you most probably need
# to run "etcd" as user "root".
# Note 2: If the user specified in "etcd_user" does not exist then the role
# will create it. Only if the user already exists the role will not create it
# but it will adjust it's UID/GID and shell if specified (see settings below).
# Additionally if "etcd_user" is "root" then this role wont touch the user
# at all.
etcd_user: "etcd"

# UID of user specified in "etcd_user". If not specified the next available
# UID from "/etc/login.defs" will be taken (see "SYS_UID_MAX" setting).
# etcd_user_uid: "999"

# Shell for specified user in "etcd_user". For increased security keep
# the default.
etcd_user_shell: "/bin/false"

# Specifies if the user specified in "etcd_user" will be a system user (default)
# or not. If "true" the "etcd_user_home" setting will be ignored. In general
# it makes sense to keep the default as there should be no need to login as
# the user that runs "etcd".
etcd_user_system: true

# Home directory of user specified in "etcd_user". Will be ignored if
# "etcd_user_system" is set to "true". In this case no home directory will
# be created. Normally not needed.
# etcd_user_home: "/home/etcd"

# Run etcd daemon as this group
#
# Note: If the group specified in "etcd_group" does not exist then the role
# will create it. Only if the group already exists the role will not create it
# but will adjust GID if specified in "etcd_group_gid" (see setting below).
etcd_group: "etcd"

# GID of group specified in "etcd_group". If not specified the next available
# GID from "/etc/login.defs" will be take (see "SYS_GID_MAX" setting).
# etcd_group_gid: "999"

# Specifies if the group specified in "etcd_group" will be a system group (default)
# or not.
etcd_group_system: true

# Directory for etcd configuration
etcd_conf_dir: "/etc/etcd"

# Permissions for directory for etcd configuration
etcd_conf_dir_mode: "0750"

# Owner of directory specified in "etcd_conf_dir"
etcd_conf_dir_user: "root"

# Group owner of directory specified in "etcd_conf_dir"
etcd_conf_dir_group: "{{ etcd_group }}"

# Directory to store downloaded etcd archive
# Should not be deleted to avoid downloading over and over again
etcd_download_dir: "/opt/etcd"

# Permissions for directory to store downloaded etcd archive
etcd_download_dir_mode: "0755"

# Owner of directory specified in "etcd_download_dir"
etcd_download_dir_user: "{{ etcd_user }}"

# Group owner of directory specified in "etcd_download_dir"
etcd_download_dir_group: "{{ etcd_group }}"

# Directory to store etcd binaries
#
# IMPORTANT: If you use the default value for "etcd_bin_dir" which is
# "/usr/local/bin" then the settings specified in "etcd_bin_dir_mode",
# "etcd_bin_dir_user" and "etcd_bin_dir_group" are ignored. This is
# done to prevent that the permissions of "/usr/local/bin" are changed.
# This directory normally exists already on every Linux installation
# and should not be changed.
# So please be careful if you specify a directory like "/usr/bin" or
# "/bin" as "etcd_bin_dir" as this will change the permissions of
# these directories and this is something you normally do not want.
etcd_bin_dir: "/usr/local/bin"

# Permissions for directory to store etcd binaries
etcd_bin_dir_mode: "0755"

# Owner of directory specified in "etcd_bin_dir"
etcd_bin_dir_user: "{{ etcd_user }}"

# Group owner of directory specified in "etcd_bin_dir"
etcd_bin_dir_group: "{{ etcd_group }}"

# etcd data directory (etcd database files so to say)
etcd_data_dir: "/var/lib/etcd"

# Permissions for directory to store etcd data
etcd_data_dir_mode: "0700"

# Owner of directory specified in "etcd_data_dir"
etcd_data_dir_user: "{{ etcd_user }}"

# Group owner of directory specified in "etcd_data_dir"
etcd_data_dir_group: "{{ etcd_group }}"

# Architecture to download and install
etcd_architecture: "amd64"

# Only change this if the architecture you are using is unsupported
# For more information, see this:
# https://github.com/etcd-io/website/blob/main/content/en/docs/v3.5/op-guide/supported-platform.md
etcd_allow_unsupported_archs: false

# By default etcd tarball gets downloaded from official
# etcd repository. This can be changed to some custom
# URL if needed. For more information which protocols
# can be used see:
# https://docs.ansible.com/ansible/latest/collections/ansible/builtin/get_url_module.html
# It's only important to keep the filename naming schema:
# "etcd-v{{ etcd_version }}-linux-{{ etcd_architecture }}.tar.gz"
etcd_download_url: "https://github.com/etcd-io/etcd/releases/download/v{{ etcd_version }}/etcd-v{{ etcd_version }}-linux-{{ etcd_architecture }}.tar.gz"

# By default the SHA256SUMS file is used to verify the
# checksum of the tarball archive. This can also be
# changed to your needs.
etcd_download_url_checksum: "sha256:https://github.com/coreos/etcd/releases/download/v{{ etcd_version }}/SHA256SUMS"

# Options for [Service] section. For more information see:
# https://www.freedesktop.org/software/systemd/man/systemd.service.html#Options
# The options below "Type=notify" are mostly security/sandbox related settings
# and limit the exposure of the system towards the unit's processes.
# https://www.freedesktop.org/software/systemd/man/systemd.exec.html
etcd_service_options:
  - User={{ etcd_user }}
  - Group={{ etcd_group }}
  - Restart=on-failure
  - RestartSec=5
  - Type=notify
  - ProtectHome=true
  - PrivateTmp=true
  - ProtectSystem=full
  - ProtectKernelModules=true
  - ProtectKernelTunables=true
  - ProtectControlGroups=true
  - CapabilityBoundingSet=~CAP_SYS_PTRACE

etcd_settings:
  "name": "{{ ansible_hostname }}"
  "cert-file": "{{ etcd_conf_dir }}/cert-etcd-server.pem"
  "key-file": "{{ etcd_conf_dir }}/cert-etcd-server-key.pem"
  "trusted-ca-file": "{{ etcd_conf_dir }}/ca-etcd.pem"
  "peer-cert-file": "{{ etcd_conf_dir }}/cert-etcd-peer.pem"
  "peer-key-file": "{{ etcd_conf_dir }}/cert-etcd-peer-key.pem"
  "peer-trusted-ca-file": "{{ etcd_conf_dir }}/ca-etcd.pem"
  "advertise-client-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_client_port }}"
  "initial-advertise-peer-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_peer_port }}"
  "listen-peer-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_peer_port }}"
  "listen-client-urls": "{{ 'https://' + hostvars[inventory_hostname]['ansible_' + etcd_interface].ipv4.address + ':' + etcd_client_port + ',https://127.0.0.1:' + etcd_client_port }}"
  "peer-client-cert-auth": "true"            # Enable peer client cert authentication
  "client-cert-auth": "true"                 # Enable client cert authentication
  "initial-cluster-token": "etcd-cluster-0"  # Initial cluster token for the etcd cluster during bootstrap.
  "initial-cluster-state": "new"             # Initial cluster state ('new' or 'existing')
  "data-dir": "{{ etcd_data_dir }}"          # etcd data directory (etcd database files so to say)
  "wal-dir": ""                              # Dedicated wal directory ("" means no separated WAL directory)
  "auto-compaction-retention": "0"           # Auto compaction retention in hour. 0 means disable auto compaction.
  "snapshot-count": "100000"                 # Number of committed transactions to trigger a snapshot to disk
  "heartbeat-interval": "100"                # Time (in milliseconds) of a heartbeat interval
  "election-timeout": "1000"                 # Time (in milliseconds) for an election to timeout. See tuning documentation for details
  "max-snapshots": "5"                       # Maximum number of snapshot files to retain (0 is unlimited)
  "max-wals": "5"                            # Maximum number of wal files to retain (0 is unlimited)
  "quota-backend-bytes": "0"                 # Raise alarms when backend size exceeds the given quota (0 defaults to low space quota)
  "logger": "zap"                            # Specify ‘zap’ for structured logging or ‘capnslog’.
  "log-outputs": "systemd/journal"           # Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd
  "enable-v2": "true"                        # enable v2 API to stay compatible with previous etcd 3.3.x (needed for flannel e.g.)
  "discovery-srv": ""                        # Discovery domain to enable DNS SRV discovery, leave empty to disable. If set, will override initial-cluster.

# Certificate authority and certificate files for etcd
etcd_certificates:
  - ca-etcd.pem               # certificate authority file
  - ca-etcd-key.pem           # certificate authority key file
  - cert-etcd-peer.pem        # peer TLS cert file
  - cert-etcd-peer-key.pem    # peer TLS key file
  - cert-etcd-server.pem      # server TLS cert file
  - cert-etcd-server-key.pem  # server TLS key file

The etcd default settings defined in etcd_settings can be overridden by defining a variable called etcd_settings_user. You can also add additional settings by using this variable. E.g. to override the default value for log-output setting and add a new setting like grpc-keepalive-min-time add the following settings to group_vars/k8s.yml:

etcd_settings_user:
  "log-output": "stdout"
  "grpc-keepalive-min-time": "10s"

Example Playbook

- hosts: k8s_etcd
  roles:
    - githubixx.etcd

Testing

This role has a small test setup that is created using Molecule, libvirt (vagrant-libvirt) and QEMU/KVM. Please see my blog post Testing Ansible roles with Molecule, libvirt (vagrant-libvirt) and QEMU/KVM how to setup. The test configuration is here.

Afterwards Molecule can be executed: