stackhpc.beegfs

This Ansible role helps you create and remove a BeegFS cluster. BeegFS is a parallel file system that distributes user data across multiple servers. It's built to be scalable for both performance and capacity. You can find out more about BeegFS here.

The role was last tested with Ansible version 2.5.0.

Example

Here's what an inventory file may look like (inventory-beegfs):

[leader]
bgfs1 ansible_host=172.16.1.1 ansible_user=centos

[follower]
bgfs2 ansible_host=172.16.1.2 ansible_user=centos

[cluster:children]
leader
follower

[cluster_beegfs_mgmt:children]
leader

[cluster_beegfs_mds:children]
leader

[cluster_beegfs_oss:children]
leader
follower

[cluster_beegfs_client:children]
leader
follower

And a playbook example (beegfs.yml):

---
- hosts:
  - cluster_beegfs_mgmt
  - cluster_beegfs_mds
  - cluster_beegfs_oss
  - cluster_beegfs_client 
  roles:
  - role: stackhpc.beegfs
    beegfs_enable:
      admon: false
      mgmt: "{{ inventory_hostname in groups['cluster_beegfs_mgmt'] }}"
      meta: "{{ inventory_hostname in groups['cluster_beegfs_mds'] }}"
      oss: "{{ inventory_hostname in groups['cluster_beegfs_oss'] }}"
      tuning: "{{ inventory_hostname in groups['cluster_beegfs_oss'] }}"
      client: "{{ inventory_hostname in groups['cluster_beegfs_client'] }}"
    beegfs_oss:
    - dev: "/dev/sdb"
      port: 8003
    - dev: "/dev/sdc"
      port: 8103
    - dev: "/dev/sdd"
      port: 8203
    beegfs_mgmt_host: "{{ groups['cluster_beegfs_mgmt'] | first }}"
    beegfs_client:
    - path: "/mnt/beegfs"
      port: 8004
    beegfs_fstype: "xfs"
    beegfs_force_format: false
    beegfs_interfaces: ["ib0"]
    beegfs_rdma: true
    beegfs_state: present
...

To create a cluster, run:

# ansible-playbook beegfs.yml -i inventory-beegfs -e beegfs_state=present

To destroy a cluster, run:

# ansible-playbook beegfs.yml -i inventory-beegfs -e beegfs_state=absent

Notes

To enable various BeegFS services, just set the toggles under beegfs_enable to true or false:

• mgmt: Management server - at least one host required
• mds: Metadata storage server nodes
• oss: Object storage server nodes
• client: Clients of the BeegFS storage cluster
• admon: NOT IMPLEMENTED

This role relies on each node's hostname resolving to the correct IP address for the management host, as set in beegfs_host_mgmt. For instance, bgfs1 should resolve to 172.16.1.1 and bgfs2 to 172.16.1.2. You can achieve this through DNS or by editing /etc/hosts.

Be careful when setting up the cluster. If the block devices you specified already have a file system or are not empty, use beegfs_force_format set to true to force format them. THIS WILL DELETE ALL DATA ON THEM. Be sure to back up any important data.

You can use partitions, but they need to be made first by other means. Also, you have to override the variable beegfs_oss_tunable with a list of the parent block devices, since partitions don't show in /sys/block/. To create partitions using Ansible module parted, you can follow this example:

---
- hosts:
  - cluster_beegfs_oss
  vars:
    partitions:
    - dev: /dev/sdb
      start: 0%
      end: 50%
      number: 1
    - dev: /dev/sdb
      start: 50%
      end: 100%
      number: 2
  tasks:
  - name: Create partitions
    parted:
      label: gpt
      state: present
      part_type: primary
      device: "{{ item.dev }}"
      part_start: "{{ item.start }}"
      part_end: "{{ item.end }}"
      number: "{{ item.number }}"
    with_items: "{{ partitions }}"
    become: true
...

Tests

You can find some tests in the molecule folder. To run them locally, you need:

• Vagrant
• VirtualBox
• Molecule
• python-vagrant

After installing the necessary tools, run the tests from the root folder of the role:

$> molecule lint
$> molecule test
$> molecule test -s vagrant-ubuntu-16.04
$> molecule test -s vagrant-ubuntu-18.04

• The default scenario runs tests on a Centos7.5 machine.
• All tests run all services on a single machine.
• YAML and Ansible lint checks are included.
• Idempotence is verified.
• After finishing, some testinfra tests are executed. All scenarios use the same tests located in molecule/tests.