trombik.opensearch

ansible 角色用于管理 opensearch。

该角色目前处于测试阶段。

对于 FreeBSD 用户

示例（而非角色本身）需要我自己为 FreeBSD 提供的 opensearch-dashboards 版本，可以在trombik/freebsd-ports-opensearch找到。示例使用 trombik.opensearch_dashboards。然而，该端口和角色依赖于已结束支持（EoL）且不推荐使用的 www/node10。

请参见 Issue 835 获取升级计划。

对于 Debian 和 CentOS 用户

该角色从官方 tar 文件中安装 opensearch。在亚马逊或其他发行版发布软件包之前，这种安装方式算是一种大 hacks。

与源代码安装相关的角色变量 opensearch_src_* 并未详细文档化。

该角色不安装 JDK 软件包，而是使用捆绑的 JDK。

为了验证 tar 文件，该角色将一个 PGP 密钥导入 root 的密钥环。如果您知道在不导入 PGP 密钥的情况下验证签名文件的方法，请告诉我。

某些插件暂时无法使用。

默认更改包括：

• 日志文件位于 /var/log/opensearch 下
• 应用程序安装在 opensearch_root_dir 下。默认路径为 /usr/local/opensearch-dashboards
• 运行应用程序的用户为 opensearch

当官方软件包可用时，以上更改将进行更新。

该角色在 opensearch_src_dir 下下载官方 tar 文件。默认路径为 /var/dist。该目录不仅仅是一个缓存目录。除了 tar 文件外，它还包含 PGP 密钥、签名文件和用于控制 ansible 任务的文件。

该角色为 opensearch 安装了一个 systemd 单元文件。作者对 systemd 并不精通。

需求

默认情况下，该角色使用 trombik.x509_certificate 来管理 X509 证书。该角色未将 trombik.x509_certificate 列为依赖项，因为 TLS 并非强制性。

角色变量

opensearch_plugins

这是一个插件列表。列表的每一项是一个字典。

opensearch_extra_plugin_files

该变量是插件文件的列表。列表的每一项是一个字典。

post_command 是在项目状态更改后运行的命令。该变量会传递给 ansible.builtin.command（不是 ansible.builtin.shell）。

它接受以下键：

post_command 主要用于 securityadmin.sh。有关详细信息，请参见 Using securityadmin.sh。

opensearch_http_auth

此变量是一个字典，作为访问 opensearch_http_url API 端点时的用户凭据。

该角色将变量传递给 uri 模块。

ca_path 在 ansible 版本 2.11 中添加。client_cert 和 client_key 在 ansible 版本 2.4 中添加。请确保您的 ansible 版本支持这些键。因此，当您的 ansible 版本低于 2.11 且证书未由 CA 在默认 CA 包中签署时，您不能使用 validate_certs: yes（通常情况下，您希望拥有自己的 CA 来签署证书，以减少费用）。

请注意，API 调用是由 ansible 控制器发出的。opensearch_http_url 必须可以从 ansible 控制器访问。

client_cert、client_key 和 ca_path 指向的文件必须在 ansible 控制器上。

opensearch_http_auth 和 TLS 的已知问题

opensearch 支持基本认证和 TLS 客户端证书认证。然而，在某些配置下，API 调用可能会失败。

简而言之：在启用 TLS 和使用 ca_path 的情况下使用基本认证。这是唯一安全如预期那样工作的配置。

要在没有用户名和密码的情况下使用 TLS 客户端认证，您需要将 validate_certs 设置为 no。以下是测试矩阵及其结果。

这可能是 ansible 的 uri 模块中的一个 bug，因为 curl 在 TLS 客户端认证和基本认证 over TLS 中都能良好工作。用于测试的命令如下。

curl -vv --cacert /usr/local/etc/opensearch/root-ca.pem \
    --cert /usr/local/etc/opensearch/admin.pem \
    --key /usr/local/etc/opensearch/admin-key.pem \
    https://localhost:9200

curl -vv --user admin:admin \
    --cacert /usr/local/etc/opensearch/root-ca.pem \
    https://localhost:9200

Debian

FreeBSD

RedHat

依赖

• 当 opensearch_include_role_x509_certificate 为真时需要 trombik.x509_certificate

示例剧本

以下示例安装：

• opensearch
• opensearch-dashboards
• haproxy
• fluentd

haproxy 是 opensearch-dashboards 的反向代理。来自 haproxy 和 syslog 守护进程的日志发送至本地 fluentd syslog 监听器。然后，fluentd 处理日志并将其发送到 opensearch。索引模式为 logstash-*。

请注意，fluentd 的 elasticsearch 输出插件不支持 opensearch。示例通过以下方式规避了此问题：

• 安装特定版本的 fluentd-elasticsearch-plugin 和 elasticsearch 相关的 gems
• 在 fluentd.conf 中设置 verify_es_version_at_startup 和 default_elasticsearch_version

有关更多详细信息，请参见 Issue 915。

使用用户名 admin 和密码 admin 登录到仪表板。

有关集群示例，请参见 tests/serverspec/cluster.yml。

---
- hosts: localhost
  pre_tasks:
    - name: 允许 HTTP 端口
      ansible.builtin.iptables:
        chain: INPUT
        destination_port: 80
        protocol: tcp
        jump: ACCEPT
      when: ansible_os_family == 'RedHat'

    - name: 启用来自 rsyslog 的 syslog 转发
      ansible.builtin.copy:
        dest: /etc/rsyslog.d/fluentd.conf
        content: |
          *.*;syslog;auth,authpriv.none action(
            Target="127.0.0.1"
            type="omfwd"
            Port="1514"
            Protocol="udp"
            template="RSYSLOG_SyslogProtocol23Format"
          )
        mode: "0644"
      when:
        - ansible_os_family == 'Debian' or ansible_os_family == 'RedHat'
      register: __register_project_rsyslog_config

    - name: 重启 rsyslog
      ansible.builtin.service:
        name: rsyslog
        state: restarted
      when:
        - ansible_os_family == 'Debian' or ansible_os_family == 'RedHat'
        - __register_project_rsyslog_config['changed']

    - name: 启用来自 syslogd 的 syslog 转发
      ansible.builtin.copy:
        dest: /etc/syslog.d/fluentd.conf
        content: |
          *.*						@127.0.0.1:1514
        mode: "0644"
      when: ansible_os_family == 'FreeBSD'
      register: __register_project_syslog_config

    - name: 启用 syslog rfc5424
      ansible.builtin.copy:
        dest: /etc/rc.conf.d/syslogd
        content: |
          syslogd_flags="-s -O rfc5424"
        mode: "0644"
      when: ansible_os_family == 'FreeBSD'
      register: __register_project_syslog_flags

    - name: 重启 syslogd
      ansible.builtin.service:
        name: syslogd
        state: restarted
      when:
        - ansible_os_family == 'FreeBSD'
        - __register_project_syslog_config['changed'] or __register_project_syslog_flags['changed']
  roles:
    - role: trombik.freebsd_pkg_repo
      when: ansible_os_family == "FreeBSD"
    - name: trombik.apt_repo
      when: ansible_os_family == 'Debian'
    - name: trombik.redhat_repo
      when: ansible_os_family == 'RedHat'
    - role: trombik.java
      # XXX 在 Ubuntu 和 CentOS 上使用捆绑的 jdk
      when: ansible_os_family == "FreeBSD"
    - role: trombik.sysctl
    - ansible-role-opensearch
    - role: trombik.opensearch_dashboards
    - role: trombik.fluentd
    - role: trombik.haproxy
  vars:
    # XXX 使用我自己的软件包，因为官方软件包树中的软件包损坏。注意，该软件包依赖于已结束支持且存在漏洞的 node10。
    freebsd_pkg_repo:
      local:
        enabled: "true"
        url: "http://pkg.i.trombik.org/{{ ansible_distribution_version | regex_replace('\\.') }}{{ ansible_architecture }}-default-default"
        mirror_type: none
        priority: 100
        state: present
    os_opensearch_extra_packages:
      FreeBSD: []
      Debian:
        - unzip
      RedHat: []
    opensearch_extra_packages: "{{ os_opensearch_extra_packages[ansible_os_family] }}"
    os_java_packages:
      FreeBSD:
        - openjdk11
        - jq
        - vim
        - tmux
        - p5-ack
      Debian:
        - openjdk-11-jdk
      RedHat:
        - java-11-openjdk-devel
    java_packages: "{{ os_java_packages[ansible_os_family] }}"
    os_sysctl:
      FreeBSD:
        kern.maxfilesperproc: 65536
        security.bsd.unprivileged_mlock: 1
      Debian:
        # 见 https://opensearch.org/docs/latest/opensearch/install/important-settings/
        vm.max_map_count: 262144
      RedHat:
        vm.max_map_count: 262144
    sysctl: "{{ os_sysctl[ansible_os_family] }}"
    opensearch_wait_for_cluster_status: yellow
    os_opensearch_package:
      FreeBSD: "{{ __opensearch_package }}"
      Debian: "{{ __opensearch_package }}"
      RedHat: opensearch-1.13.2
    opensearch_package: "{{ os_opensearch_package[ansible_os_family] }}"
    os_opensearch_flags:
      FreeBSD: ""
      Debian: |
        ES_PATH_CONF={{ opensearch_conf_dir }}
        ES_STARTUP_SLEEP_TIME=5
      RedHat: |
        ES_PATH_CONF={{ opensearch_conf_dir }}
        ES_STARTUP_SLEEP_TIME=5
    opensearch_flags: "{{ os_opensearch_flags[ansible_os_family] }}"
    os_opensearch_jvm_options:
      FreeBSD: ""
      Debian: |
        # 见 opensearch-tar-install.sh
        # /usr/bin/getconf CLK_TCK`
        -Dclk.tck=100
        -Djdk.attach.allowAttachSelf=true
        -Djava.security.policy={{ opensearch_root_dir }}/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy
      RedHat: |
        # /usr/bin/getconf CLK_TCK`
        -Dclk.tck=100
        -Djdk.attach.allowAttachSelf=true
        -Djava.security.policy={{ opensearch_root_dir }}/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy

    os_opensearch_http_auth:
      FreeBSD:
        url_username: admin
        url_password: admin
        ca_path: "{{ role_path }}/files/test/certs/root-ca.pem"
        validate_certs: yes
      Debian:
        client_cert: "{{ role_path }}/files/test/certs/admin.pem"
        client_key: "{{ role_path }}/files/test/certs/admin-key.pem"
        # XXX Ubuntu 上的 ansible 版本是 2.9.6，因此不能使用 ca_path。
        validate_certs: no
      RedHat:
        client_cert: "{{ role_path }}/files/test/certs/admin.pem"
        client_key: "{{ role_path }}/files/test/certs/admin-key.pem"
        validate_certs: no
    opensearch_http_auth: "{{ os_opensearch_http_auth[ansible_os_family] }}"
    opensearch_jvm_options: "{{ lookup('file', 'test/jvm_options') + os_opensearch_jvm_options[ansible_os_family] }}"
    opensearch_config:
      discovery.type: single-node
      network.publish_host: ["10.0.2.15"]
      path.data: "{{ opensearch_db_dir }}"
      http.port: "{{ opensearch_http_port }}"
      path.logs: "{{ opensearch_log_dir }}"
      node.data: "true"
      http.compression: "true"
      network.host:
        - "{{ opensearch_http_host }}"
        - _site_
      cluster.name: testcluster
      node.name: testnode
      http.cors.enabled: "true"
      http.cors.allow-origin: "*"
      http.cors.max-age: 86400
      http.cors.allow-methods: "OPTIONS, HEAD, GET, POST, PUT, DELETE"
      http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length"
      http.cors.allow-credentials: "true"
      # _________________________TLS
      plugins.security.ssl.transport.pemcert_filepath: node.pem
      plugins.security.ssl.transport.pemkey_filepath: node-key.pem
      plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
      plugins.security.ssl.transport.enforce_hostname_verification: false
      plugins.security.ssl.http.enabled: true
      plugins.security.ssl.http.pemcert_filepath: node.pem
      plugins.security.ssl.http.pemkey_filepath: node-key.pem
      plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
      plugins.security.allow_unsafe_democertificates: true
      plugins.security.allow_default_init_securityindex: true
      plugins.security.authcz.admin_dn:
        # XXX admin_dn 和 nodes_dn 使用不同的 CN。当 admin_dn == nodes_dn 时，即为错误。
        - CN=Admin,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU
      plugins.security.nodes_dn:
        - CN=localhost,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

      plugins.security.advanced_modules_enabled: false
      plugins.security.audit.type: internal_opensearch
      plugins.security.enable_snapshot_restore_privilege: true
      plugins.security.check_snapshot_restore_write_privileges: true
      plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
      plugins.security.system_indices.enabled: true
      plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]

      plugins.security.disabled: false
      cluster.routing.allocation.disk.threshold_enabled: false

    project_security_plugin_dir: "{{ opensearch_plugins_dir }}/opensearch-security"
    project_securityadmin_bin: "{{ project_security_plugin_dir }}/tools/securityadmin.sh"
    project_security_plugin_post_command:
      cmd: "{{ project_securityadmin_bin }} -icl -nhnv -cacert {{ opensearch_conf_dir }}/root-ca.pem -cert {{ opensearch_conf_dir }}/admin.pem -key {{ opensearch_conf_dir }}/admin-key.pem"
      args:
        chdir: "{{ project_security_plugin_dir }}/securityconfig"
      enabled: "{% if 1 == 1 %}yes{% else %}no{% endif %}"

    opensearch_plugins: []
    opensearch_extra_plugin_files:
      - path: opensearch-security/securityconfig/action_groups.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/action_groups.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/audit.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/audit.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/config.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/config.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/internal_users.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/internal_users.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/nodes_dn.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/nodes_dn.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/roles.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/roles.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/roles_mapping.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/roles_mapping.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/tenants.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/tenants.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"
      - path: opensearch-security/securityconfig/whitelist.yml
        type: yaml
        mode: "0640"
        group: "{{ opensearch_user }}"
        content: "{{ lookup('file', 'test/securityconfig/whitelist.yml') | from_yaml }}"
        post_command: "{{ project_security_plugin_post_command }}"

    opensearch_config_log4j2_properties: "{{ lookup('file', 'test/log4j2_properties') }}"

    x509_certificate_debug_log: yes
    # XXX 这些密钥的创建步骤见以下文档：
    # https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/
    #
    # 创建根 CA
    # openssl genrsa -out root-ca-key.pem 2048
    # openssl req -new -x509 -sha256 -key root-ca-key.pem -out root-ca.pem
    #
    # 创建管理员证书
    # openssl genrsa -out admin-key-temp.pem 2048
    # openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
    # openssl req -new -key admin-key.pem -out admin.csr
    # openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem
    #
    # 创建节点证书
    # openssl genrsa -out node-key-temp.pem 2048
    # openssl pkcs8 -inform PEM -outform PEM -in node-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node-key.pem
    # openssl req -new -key node-key.pem -out node.csr
    # openssl x509 -req -in node.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node.pem
    #
    # 清理
    # rm admin-key-temp.pem admin.csr node-key-temp.pem node.csr
    #
    # 见 files/test/certs/Makefile 自动化这些步骤。
    x509_certificate:
      - name: node
        state: present
        public:
          path: "{{ opensearch_conf_dir }}/node.pem"
          mode: "0444"
          key: "{{ lookup('file', 'test/certs/node.pem') }}"
        secret:
          path: "{{ opensearch_conf_dir }}/node-key.pem"
          owner: "{{ opensearch_user }}"
          group: "{{ opensearch_group }}"
          mode: "0600"
          key: "{{ lookup('file', 'test/certs/node-key.pem') }}"
      - name: root-ca
        state: present
        public:
          path: "{{ opensearch_conf_dir }}/root-ca.pem"
          key: "{{ lookup('file', 'test/certs/root-ca.pem') }}"
        secret:
          path: "{{ opensearch_conf_dir }}/root-ca-key.pem"
          owner: "{{ opensearch_user }}"
          group: "{{ opensearch_group }}"
          key: "{{ lookup('file', 'test/certs/root-ca-key.pem') }}"
      - name: admin
        state: present
        public:
          path: "{{ opensearch_conf_dir }}/admin.pem"
          key: "{{ lookup('file', 'test/certs/admin.pem') }}"
        secret:
          path: "{{ opensearch_conf_dir }}/admin-key.pem"
          owner: "{{ opensearch_user }}"
          group: "{{ opensearch_group }}"
          key: "{{ lookup('file', 'test/certs/admin-key.pem') }}"

    # _____________________________________________opensearch-dashboards
    opensearch_dashboards_config:
      server.host: "{{ opensearch_dashboards_bind_address }}"
      server.port: "{{ opensearch_dashboards_bind_port }}"
      server.name: "OpenSearch Dashboards"
      # XXX 在 FreeBSD 软件包中修复日志路径
      logging.dest: "{% if ansible_os_family == 'FreeBSD' %}/var/log/opensearch_dashboards.log{% else %}{{ opensearch_dashboards_log_file }}{% endif %}"
      logging.verbose: true
      opensearch.hosts: ["https://localhost:9200"]
      path.data: "{{ opensearch_dashboards_data_dir }}"
      opensearch.ssl.verificationMode: none
      opensearch.username: "kibanaserver"
      opensearch.password: "kibanaserver"
      opensearch_security.multitenancy.enabled: true
      opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
      opensearch_security.readonly_mode.roles: ["kibana_read_only"]
      # 如果您在没有 HTTPS 的情况下运行 kibana，请使用此设置
      opensearch_security.cookie.secure: false
    # _____________________________________________haproxy
    project_backend_host: 127.0.0.1
    project_backend_port: 5601
    os_haproxy_selinux_seport:
      FreeBSD: {}
      Debian: {}
      RedHat:
        ports:
          - 80
          - 5601
        proto: tcp
        setype: http_port_t
    haproxy_selinux_seport: "{{ os_haproxy_selinux_seport[ansible_os_family] }}"
    haproxy_config: |
      global
        daemon
      {% if ansible_os_family != 'RedHat' %}
        # 将默认的 1024 最大行长度增加到 65535，当超过此值时会截断日志。
        log 127.0.0.1:5140 len 65535 format rfc3164 local0 info
      {% else %}
        # XXX haproxy 1.x 不支持 `format`。
        log 127.0.0.1:5140 len 65535 local2
        log-send-hostname
      {% endif %}

      {% if ansible_os_family == 'FreeBSD' %}
      # FreeBSD 软件包未提供默认值
        maxconn 4096
        user {{ haproxy_user }}
        group {{ haproxy_group }}
      {% elif ansible_os_family == 'Debian' %}
        chroot {{ haproxy_chroot_dir }}
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user {{ haproxy_user }}
        group {{ haproxy_group }}

        # 默认 SSL 材料位置
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # 见： https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
          ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
          ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
          ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
      {% elif ansible_os_family == 'OpenBSD' %}
        maxconn 1024
        chroot {{ haproxy_chroot_dir }}
        uid 604
        gid 604
        pidfile /var/run/haproxy.pid
      {% elif ansible_os_family == 'RedHat' %}
        chroot      /var/lib/haproxy
        pidfile     /var/run/haproxy.pid
        maxconn     4000
        user        haproxy
        group       haproxy
        daemon
      {% endif %}

      defaults
        log global
        mode http
        timeout connect 5s
        timeout client 10s
        timeout server 10s
        retries 3
        maxconn 2000
      {% if ansible_os_family == 'Debian' %}
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
      {% elif ansible_os_family == 'OpenBSD' %}
        option  redispatch
      {% endif %}

      frontend http-in
        bind *:80
        default_backend servers
        unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
      {% if ansible_os_family != 'RedHat' %}
        http-request capture req.fhdr(Host) len 128
        http-request capture req.fhdr(Referer) len 1024
        http-request capture req.fhdr(User-Agent) len 1024
        http-request capture req.fhdr(Accept) len 1024
      {% else %}
        capture request header Host len 128
        capture request header Referer len 1024
        capture request header User-Agent len 1024
        capture request header Accept len 1024
      {% endif %}
        # 自定义 JSON 日志格式。
        # 要创建您自己的 JSON 结构：
        #
        # cd files/test/haproxy
        # ruby ./yaml2logformat.rb log.yml
        #
        # 请注意，输出包括单引号的日志格式。但在 haproxy 1.x 版本中，单引号必须去掉。
        # 例如，当输出为：
        #
        # log-format '{"bytes_read":%B}'
        #
        # 使用：
        # log-format {"bytes_read":%B}
        #
        # 见可用变量：
        # 8.2.4. 自定义日志格式
        # https://www.haproxy.com/documentation/hapee/latest/onepage/#8.2.4
        #
      {% if ansible_os_family != 'RedHat' %}
        log-format '{"bytes_read":%B,"hostname":"%H","http":{"method":"%HM","uri":"%HP","query":"%HQ","version":"%HV"},"unique-id":"%ID","status_code":%ST,"gmt_date_time":"%T","timestamp":%Ts,"bytes_uploaded":%U,"backend_name":"%b","beconn":%bc,"backend_queue":%bq,"client_ip":"%ci","client_port":%cp,"frontend_name":"%f","frontend_ip":"%fi","frontend_port":%fp,"ssl":{"ciphers":"%sslc","version":"%sslv"},"request":{"headers":{"host":"%[capture.req.hdr(0),json(utf8ps)]","referer":"%[capture.req.hdr(1),json(utf8ps)]","ua":"%[capture.req.hdr(2),json(utf8ps)]","accept":"%[capture.req.hdr(3),json(utf8ps)]"}}}'
      {% else %}
        # XXX CentOS 上的 haproxy 1.x 不理解 `format`、`HTTP 请求捕获` 等。简化 JSON 日志格式。1.x 版本的 log-format 不支持单引号。
        # 空格必须转义。
        log-format {"bytes_read":%B,"hostname":"%H","unique-id":"%ID","status_code":%ST,"gmt_date_time":"%T","timestamp":%Ts,"bytes_uploaded":%U,"backend_name":"%b","beconn":%bc,"backend_queue":%bq,"client_ip":"%ci","client_port":%cp,"frontend_name":"%f","frontend_ip":"%fi","frontend_port":%fp,"ssl":{"ciphers":"%sslc","version":"%sslv"},"request":{"headers":"%hr"}}
      {% endif %}

      backend servers
        option forwardfor
        server server1 {{ project_backend_host }}:{{ project_backend_port }} maxconn 32 check

    os_haproxy_flags:
      FreeBSD: |
        haproxy_config="{{ haproxy_conf_file }}"
        #haproxy_flags="-q -f ${haproxy_config} -p ${pidfile}"
      Debian: |
        #CONFIG="/etc/haproxy/haproxy.cfg"
        #EXTRAOPTS="-de -m 16"
      OpenBSD: ""
      RedHat: |
        OPTIONS=""
    haproxy_flags: "{{ os_haproxy_flags[ansible_os_family] }}"

    # _____________________________________________fluentd
    fluentd_system_config: |
      log_level debug
      suppress_config_dump
    fluentd_configs:
      input_udp_1514:
        enabled: yes
        config: |
          <source>
            @type syslog
            @label @forward
            port 1514
            bind 127.0.0.1
            tag syslog
            # 在 syslog 事件中包含严重性。
            # 默认情况下，syslog 输入插件不包含严重性。
            severity_key severity
            <parse>
              message_format rfc5424
              with_priority true
            </parse>
          </source>

      input_udp_5140:
        enabled: true
        config: |
          <source>
            @type syslog
            @label @haproxy
            port 5140
            bind 127.0.0.1
            tag haproxy
            severity_key severity
            <parse>
              message_format rfc3164
            </parse>
          </source>

          # 处理来自 haproxy 的访问日志。信息级别包括访问日志。
          # 键 `message` 包含转义的 JSON。由于 JSON 包含所有数据，
          # 在 fluentd 事件中使用 `reserve_data false` 丢弃其他数据。
          <label @haproxy>
            <filter *.*.info>
              @type parser
              <parse>
                @type json
                json_parser json
                time_type string
                time_key gmt_date_time
                time_format %d/%b/%Y:%H:%M:%S %z
              </parse>
              key_name message
              reserve_data false
              replace_invalid_sequence true
            </filter>
            <match **>
              @type relabel
              @label @forward
            </match>
          </label>

      outout_default:
        enabled: yes
        config: |
          # 将事件发送到 elasticsearch。注意，事件通过
          # `_bulk` API 发送。发送事件需要一些时间。
          <label @forward>
            <match **>
              @type elasticsearch
              # 此处不要使用 127.0.0.1。请使用证书中的 CN
              host localhost
              port 9200
              scheme https
              ssl_version TLSv1_2
              user "logstash"
              password "logstash"
              # XXX 您需要创建一个角色，以便将数据发送到 `logstash` 以外的索引，
              # 因为 `logstash` 角色是静态的，仅预安装，并且仅可访问 `logstash-*` 索引模式。
              # logstash_prefix syslog

              # 在传输中启用调试日志。发送到 elasticsearch 的事件将记录在日志文件中。
              # 您可能希望在生产环境中将此设置为 false。
              with_transporter_log true
              ca_file {{ opensearch_conf_dir }}/root-ca.pem
              ssl_verify true
              logstash_format true
              # XXX 使用 elasticsearch 输出插件的方式与 opensearch 一起使用。
              # 另见 fluentd_gems 下。
              #
              # https://github.com/uken/fluent-plugin-elasticsearch/issues/915
              verify_es_version_at_startup false
              default_elasticsearch_version 7

              <buffer>
                timekey 1d
                timekey_use_utc true
                timekey_wait 10m
              </buffer>
            </match>
          </label>
    fluentd_gems:
      - name: elasticsearch-transport
        version: 7.13.3
        state: present
      - name: elasticsearch-api
        version: 7.13.3
        state: present
      - name: elasticsearch
        version: 7.13.3
        state: present
      - name: fluent-plugin-elasticsearch
        version: 5.1.0
        state: present

    os_fluentd_flags:
      FreeBSD: |
        fluentd_flags="-p {{ fluentd_plugin_dir }} --log {{ fluentd_log_file }}"
      Debian: |
        TD_AGENT_LOG_FILE="{{ fluentd_log_file }}"
        TD_AGENT_OPTIONS="-p {{ fluentd_plugin_dir }}"
        STOPTIMEOUT=180
      RedHat: |
        TD_AGENT_LOG_FILE="{{ fluentd_log_file }}"
        TD_AGENT_OPTIONS=""
      OpenBSD: "--daemon /var/run/fluentd/fluentd.pid --config {{ fluentd_config_file }} -p {{ fluentd_plugin_dir }} --log {{ fluentd_log_file }}"
    fluentd_flags: "{{ os_fluentd_flags[ansible_os_family] }}"

    # _____________________________________________apt
    apt_repo_keys_to_add:
      - https://packages.treasuredata.com/GPG-KEY-td-agent
    apt_repo_to_add:
      - "deb http://packages.treasuredata.com/4/ubuntu/{{ ansible_distribution_release }}/ {{ ansible_distribution_release }} contrib"
    # _____________________________________________redhat_repo
    redhat_repo:
      treasuredata:
        baseurl: http://packages.treasuredata.com/4/redhat/$releasever/$basearch
        gpgkey: https://packages.treasuredata.com/GPG-KEY-td-agent

许可证

版权所有 (c) 2019 Tomoyuki Sakurai <y@trombik.org>

特此授权，允许在任何场合使用、复制、修改和分发本软件，收取或不收取费用，前提是上述版权声明和本许可声明出现在所有副本中。

本软件按“原样”提供，作者对本软件不承担任何担保，包括对适销性和适用性的所有暗示担保。在任何情况下，作者都不对因使用、数据或利润损失而导致的任何特殊、直接、间接或间接损害承担责任，无论是因合同、过失还是其他侵权行为而产生的，均不得追索与本软件的使用或性能相关的任何损害。

作者信息

Tomoyuki Sakurai y@trombik.org

此 README 文档由 qansible 创建。