23.21. filebeat - Filebeat Forwarder

The following documentation is for Filebeat Forwarder (filebeat) content package at version v4.6.0-beta01.47+gfb4d9d119b24c3ffbb0ec412c8495528d8f92159.

This plugin is used to send events to Filebeat so that it can forward the event to logstash or elasticsearch.

23.21.1. Installation / Configuration Filebeat

23.21.1.1. Install filebeat

The quick start installation for filebeat can be found here

The following is the example for RPM-based systems.

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.9.2-x86_64.rpm
rpm -vi filebeat-7.9.2-x86_64.rpm

23.21.1.2. Configure filebeat

There are two modes of operation of the filebeat plugin. The first mode, file, is the original method for transferring events to filebeat from the filebeat plugin. The other method, tcp, uses tcp sockets to send messages.

Regardless of mode, you will need to configure your output section. See the logstash or elasticsearch as needed.

23.21.1.2.1. file mode

Inside the /etc/filebeat/filebeat.yml file, you will need to add a filebeat.inputs stanza.

This looks like:

- type: log
  paths:
  - /var/log/filebeat-plugin/json.log
  fields:
    drp_id: test1
  fields_under_root: true
  json.keys_under_root: true

The two parts that the user needs to configure are hte filename of the intermediate logging file. Also, it is helpful to add a field to the event that indicates which DRP endpoint the event came from. This is done by changing test1 to the DRP endpoint id.

23.21.1.2.2. tcp mode

Inside the /etc/filebeat/filebeat.yml file. You will need to add a filebeat.inputs stanza.

- type: tcp
  enabled: true
  host: "127.0.0.1:9000"
  max_message_size: 10MiB
  fields:
    drp_id: test1
  fields_under_root: true
  json.keys_under_root: true

Additionally, a couple processors needs to be added to handle the JSON conversion. The log directory already handles this.

- decode_json_fields:
    fields: ["message"]
    process_array: false
    max_depth: 1
    target: ""
    overwrite_keys: true
    add_error_key: true
- drop_fields:
    fields:
      - message

These processors cause the system to convert the TCP message data into a json blob stored at the top of the event and drop the message field to remove duplicate data.

23.21.2. Configuration DRP Filebeat Plugin

The default plugin for the filebeat plugin will use the file mode with a filename of /var/log/filebeat-plugin/filebeat.json. This plugin will be added automatically on startup if not already created. These are the defaults for the starting system.

If the path needs to be changed, add the filebeat/path parameter to match the input file path in the filebeat yaml file.

To change modes, add the filebeat/mode parameter to the plugin and set it to tcp. The default is file.

In tcp mode, the default tcp connection string is 127.0.0.1:9000. This should match the tcp filebeat input stanza exactly.

23.21.3. Object Specific Documentation

23.21.3.1. params

The content package provides the following params.

23.21.3.1.1. filebeat/mode

Filebeat connection mode to attack to filebeat.

This can be: * file - uses filebeat/path to define file * tcp - uses filebeat/tcp to define connection string

23.21.3.1.2. filebeat/path

Filebeat log file path. The plugin writes the events to this log file. This file should be under log rotate.

23.21.3.1.3. filebeat/tcp

Filebeat tcp connection string in the form of ip:port.

23.21.3.1.4. filebeat/elasticsearch

Filebeat elasticsearch connection string in the form of ip:port.

23.21.3.1.5. filebeat/events

This parameter defines the DRP events that should be registered for and forwarded to Filebeat.

The default is:

*.*.*

This is all events. The general form is scope.action.specific.

23.21.3.2. profiles

The content package provides the following profiles.

23.21.3.2.1. bootstrap-filebeat

Profile to bootstrap file beat.

23.21.3.3. tasks

The content package provides the following tasks.

23.21.3.3.1. filebeat-setup

A task to install and setup filebeat.