Ansible Kafka Role

This is an Ansible role for installing Kafka using KRaft mode on Debian and RedHat systems. KRaft is considered ready for production starting from version 3.3.1, so use this role for Kafka versions 3.3.1 or higher.

It has been tested on Ubuntu 22.04 and AlmaLinux 8, but it should work with most distributions, especially if you don't rely on this role for installing dependencies. You can check the kafka_install_dependencies variable.

For more details, see the changelog.

Requirements

• 64-bit (x86_64) architecture on servers running Kafka.
• Ansible version 2.15. It may work with version 2.10 or later, but that hasn’t been tested.
• Java must be installed: use version 8 (deprecated), 11, or 17. You can install it via apt install openjdk-17-jre-headless or dnf install java-17-openjdk-headless.
• On Ubuntu, the setfacl command from the acl package is required: apt install acl.

Role Variables

Required:

• kafka_node_id: Unique integer or zero for each Kafka node.

Optional:

• kafka_topics: A list of Kafka topics with their settings.
• kafka_install_dependencies: Set to true to install Java and acl for Debian-based systems.
• kafka_config_path: If not set, the config will be placed in Kafka’s home directory. Avoid using the default config path to prevent issues during role re-runs.
• kafka_additional_config: Add extra config parameters not specified by the role.
• kafka_opts: Provide startup options for Kafka. This variable will create the KAFKA_OPTS environment variable.

Dependencies

There are no dependencies if kafka_install_dependencies is set to true. For other cases, check the requirements section.

Example Playbook

To install the role, use Ansible Galaxy:

ansible-galaxy install dragomirr.kafka

- hosts: servers
  roles:
    - role: dragomirr.kafka
      kafka_node_id: 0  # Only valid for a single Kafka node
      kafka_heap_size: 2G
      kafka_install_dependencies: true
      kafka_topics:
        - name: topic1
        - name: topic2
          replication_factor: 1
          partitions: 10
      kafka_additional_config:
        message.max.bytes: 10000  # Extra config not defined by the role
      kafka_opts:
        - -XX:NewSize=256m  # Kafka startup options

License

This role is licensed under GPL3.

Testing

Testing is done using Molecule alongside VirtualBox and Vagrant.

There are three scenarios for testing:

• A default scenario with one instance.
• A cluster scenario with three instances acting as both controller and broker for all.
• A combined cluster scenario with six instances, where three serve as both broker and controller, and three serve only as brokers.