Rsyslog

Master:
Develop:

• Installation and Dependencies
• Tags
• Built-in Log Configurations
  • Syslog Files
  • Auth Log Files
  • Plain Text Log Files
  • JSON Log Files
  • Nginx Access Log Files
  • Docker Container Logs
    • Plain Text Logs
    • JSON Logs
• Examples

This role installs rsyslog version 8 for log shipping.

For more details about rsyslog, visit the rsyslog documentation.

Installation and Dependencies

To install, run the command:

ansible-galaxy install sansible.rsyslog

Or add the following to your roles.yml file:

- name: sansible.rsyslog
  version: v3.0

Then run:

ansible-galaxy install -p ./roles -r roles.yml

Tags

This role has two tags: build and configure.

• build: Installs Rsyslog and all its dependencies.

Built-in Log Configurations

This role includes several built-in configurations for sending log files to Logstash. These configurations are designed for specific log file formats and always produce output in JSON. Example settings for each built-in handler can be found in defaults/main.yml under builtin_configs.

Syslog Files

By default, syslog messages are sent in JSON format, requiring no additional configuration.

The format for syslog messages is as follows:

• type: always "syslog"
• host: hostname of the originating machine
• timestamp: date in RFC-3339 format
• version: always "2" for JSON logs
• role: this is the app_name from the rsyslog roles configuration
• message: the actual log line
• priority: syslog severity number
• program: name of the service that generated the message
• facility: syslog facility
• severity: syslog severity name

Example of a syslog message in JSON:

{
  "type": "syslog",
  "host": "192.168.1.1",
  "timestamp": "2017-10-21T14:29:30.739200+00:00",
  "@version": "2",
  "role": "some_application",
  "message": "Something happened",
  "priority": "3",
  "program": "some-local-service",
  "facility": "daemon",
  "severity": "err"
}

Auth Log Files

sansible_rsyslog_app_name: some_application
sansible_rsyslog_builtin_configs_application_logs_enabled: true

This configuration sends auth and authpriv facility messages. It uses the same template as syslog messages, but sets the type field to "authlog".

Example of an authlog message:

{
  "type": "authlog",
  "host": "192.168.1.1",
  "timestamp": "2018-01-09T10:45:01.117615+00:00",
  "@version": "2",
  "role": "some_application",
  "message": "pam_unix(cron:session): session closed for user some_user",
  "priority": "6",
  "program": "CRON",
  "facility": "authpriv",
  "severity": "info"
}

Plain Text Log Files

sansible_rsyslog_app_name: some_application
sansible_rsyslog_builtin_configs_application_logs_enabled: true
sansible_rsyslog_builtin_configs_application_logs_logs:
  - path: "/var/log/some_log.log"
    options:
      type: some_application_log

This configuration handles plaintext log files. The lines in these log files are sent to Logstash in the following JSON format:

• type: field from the log files config, useful for Logstash filters
• host: hostname of the originating machine
• timestamp: date in RFC-3339 format
• version: always "1" for plaintext logs
• role: app_name from the rsyslog roles configuration
• sourcefile: location of the log file
• message: the actual log line

Example of a plaintext log message:

{
  "type": "application_log",
  "host": "192.168.1.1",
  "timestamp": "2017-10-21T13:58:40.098660+00:00",
  "@version": "1",
  "role": "some_application_log",
  "sourcefile": "/var/log/some_log.log",
  "message": "A plain text log line"
}

JSON Log Files

sansible_rsyslog_app_name: some_application
sansible_rsyslog_builtin_configs_json_logs_enabled: true
sansible_rsyslog_builtin_configs_json_logs_logs:
  - path: "/var/log/some_log.log"
    options:
      type: some_application_log

This configuration handles JSON log files. All fields are sent as JSON, with some additional fields:

• type: field from the log files config
• host: hostname of the originating machine
• version: always "2" for JSON logs
• role: app_name from the rsyslog roles configuration
• sourcefile: location of the log file

Example JSON log:

{ "message": "Some log message", "datetime": "2017-01-01 00:00:00", "level": "info" }

This would be sent as:

{
  "type": "some_application_log",
  "host": "192.168.1.1",
  "@version": "2",
  "role": "some_application",
  "sourcefile": "/var/log/some_log.log",
  "message": "Some log message",
  "datetime": "2017-01-01 00:00:00",
  "level": "info"
}

Nginx Access Log Files

The nginx_access_logs configuration is similar to json_logs, but the type field is set to "nginx-access-logs".

You can configure Nginx to send access logs in JSON format with the following config in nginx.conf:

log_format main '{ '
    '"http_host": "$http_host", '
    '"clientip": "$remote_addr", '
    '"datetime": "$time_iso8601", '
    '"verb": "$request_method", '
    '"request_full": "$request", '
    '"response": "$status", '
    '"response_length": "$body_bytes_sent", '
    '"request_length": "$request_length", '
    '"referrer": "$http_referer", '
    '"agent": "$http_user_agent", '
    '"request_time": "$request_time", '
    '"upstream_time": "$upstream_response_time", '
    '"user_id": "$http_x_user", '
    '"request_id": "$http_x_request_id" '
'}';

access_log  /var/log/nginx/access.log main;

Docker Container Logs

Docker log configurations are for ECS container instances. They create Unix socket files on the host where Docker sends logs from running containers.

Plain Text Logs

sansible_rsyslog_builtin_configs_docker_application_logs_enabled: true

This configuration creates a socket at /var/run/rsyslog/text.sock. The logging configuration in the application’s task definition should look like this:

"LogConfiguration": {
    "LogDriver": "syslog",
    "Options": {
        "tag": { "Ref": "ServiceName" },
        "syslog-address": "unixgram:///var/run/docker/text.sock"
    },
}

JSON Logs

sansible_rsyslog_builtin_configs_docker_json_logs_enabled: true

This setup creates a socket at /var/run/rsyslog/json.sock. The logging configuration in the application’s task definition should look like:

"LogConfiguration": {
    "LogDriver": "syslog",
    "Options": {
        "tag": { "Ref": "ServiceName" },
        "syslog-address": "unixgram:///var/run/docker/json.sock"
    },
}

Text or JSON Logs via Journald

sansible_rsyslog_builtin_configs_docker_journald_logs_enabled: false

This configuration listens to the journald service. It processes logs only for containers, ignoring other journal entries. Both JSON and text outputs can be utilized. The basic logging configuration in the application’s task definition should appear as follows:

"LogConfiguration": {
    "LogDriver": "journald"
}

For ECS deployments, you can enhance log entry metadata by enabling:

sansible_rsyslog_custom_ecs_properties: yes

If enabled, your task definition logging configuration should be:

"LogConfiguration": {
    "LogDriver": "journald",
      "Log-Opts" : {
        "labels": "role,ecs_task_container,app_version"
      }
}

Examples

To install:

- name: Some app
  hosts: "{{ hosts }}"

  roles:
    - role: sansible.rsyslog
      sansible_rsyslog_app_name: default_app
      sansible_rsyslog_builtin_configs_application_logs_enabled: true
      sansible_rsyslog_builtin_configs_application_logs_logs:
        - path: "/home/application_user/app_log.log"
      sansible_rsyslog_builtin_configs_json_logs_enabled: true
      sansible_rsyslog_builtin_configs_json_logs_logs:
        - path: "/home/application_user/app_log_json.log"
          options:
            type_tag: "application_log"
      sansible_rsyslog_builtin_configs_nginx_access_logs_enabled: true
      sansible_rsyslog_builtin_configs_nginx_access_logs_logs:
        - path: "/var/log/nginx/application_access.log"

There are default global settings which are applied to the config files, visible in vars/main.yml. To override or add settings, you can use the following variables: sansible_rsyslog_config_global, sansible_rsyslog_config_imfile, sansible_rsyslog_config_main_queue, and sansible_rsyslog_config_omfwd:

- name: Some app
  hosts: "{{ hosts }}"

  roles:
    - role: sansible.rsyslog
      sansible_rsyslog_app_name: default_app
      sansible_rsyslog_builtin_configs_application_logs_enabled: true
      sansible_rsyslog_builtin_configs_application_logs_logs:
        - path: "/home/application_user/app_log.log"
      sansible_rsyslog_builtin_configs_json_logs_enabled: true
      sansible_rsyslog_builtin_configs_json_logs_logs:
        - path: "/home/application_user/app_log_json.log"
          options:
            type_tag: "application_log"
      sansible_rsyslog_builtin_configs_nginx_access_logs_enabled: true
      sansible_rsyslog_builtin_configs_nginx_access_logs_logs:
        - path: "/var/log/nginx/application_access.log"
      sansible_rsyslog_config_global:
        maxMessageSize: 32K
        oversizemsg.input.mode: accept
      sansible_rsyslog_config_imfile:
        reopenOnTruncate: "on"
      sansible_rsyslog_config_main_queue:
        queue.size: 100000
      sansible_rsyslog_config_omfwd:
        action.resumeRetryCount: 10

To install without the default config:

- name: Some app
  hosts: "{{ hosts }}"

  roles:
    - role: sansible.rsyslog
      sansible_rsyslog_app_name: default_app
      sansible_rsyslog_default_config: no

To install specific package versions:

- name: Some app
  hosts: "{{ hosts }}"

  roles:
    - role: sansible.rsyslog
      sansible_rsyslog_version: "8.30*"
      sansible_rsyslog_version_libfastjson4: "0.99.*"
      sansible_rsyslog_version_mmjsonparse: "8.30.*"