Apache Flume

This is an Ansible Role used to install and set up Apache Flume.

Features

• Installs Apache Flume
• Configures Apache Flume
• Creates a service for apache-flume
  • Set up for agent only

Requirements

• Ansible: Version 2.3 or higher
• Testing Tools: Docker/Vagrant, Molecule version 1.2.5

Role Variables

• Java Settings

  java_heap_xms: 125
  java_heap_xmx: 250
  

• Flume Settings

  mirror_url: http://apache.mirrors.ionfish.org/flume
  version: 1.7.0
  

• Installation Paths

  download_path: /tmp
  installation_path: /usr/local
  owner: root
  group: root
  

• Flume Configuration

  • Refer to the Flume User Guide
  • Currently, you can configure:
    • Agents:
      • Only for agent, multiple agents require manual setup.
    • Source: One source per agent at the moment.
    • Channel: One channel per agent currently.
    • Sinks: Multiple sinks allowed.
    • Sink Group: Includes all sinks from an agent.
  • Replace "." with "_" in property names for Flume configuration.
    • Example: kafka_consumer_group_id: testflume becomes agent.source.kafka.consumer.group.id = testflume
  • You can pass apache_flume_config to copy configuration files to target machines.
    • Example: apache_flume_config: file/flume-conf.properties

• Additional Variables

  • For HDFS native libraries:

    hdfs_libs = true
    

Example Variables

mirror_url: http://apache.mirrors.ionfish.org/flume
version: 1.7.0
download_path: /tmp
installation_path: /usr/local
owner: root
group: root

java_heap_xms: 125
java_heap_xmx: 250
hdfs_libs: true

agents:
- name: agent
  source:
    name: kafkaSource
    type: org.apache.flume.source.kafka.KafkaSource
    kafka_consumer_group_id: flume
    kafka_topics:
      - topic1
      - topic2
  channel:
    name: kafkaChannel
    type: memory
    capacity: 1000000
  sinks:
    - name: kafkaHDFSSink1
      type: hdfs
      hdfs_path: "s3n://your-path"
      hdfs_fileType: "SequenceFile"
    - name: kafkaHDFSSink2
      type: hdfs
      hdfs_path: "s3n://your-path"
      hdfs_fileType: "SequenceFile"
  sink_group:
    name: sinkgroup1
    processor_type: load_balance

Dependencies

• andrewrothstein.java-oracle-jre

Example Playbook

- hosts: all
  roles:
    - role: mplachter.flume

License

MIT

Author

Matthew Plachter