Graylog Collector Sidecar

Required Graylog version: 2.0 and later + installed graylog-plugin-collector (this plugin available in install package - folder plugins)
The Graylog Collector Sidecar is a supervisor process for 3rd party log collectors like NXLog. The Sidecar program is able to fetch configurations from a Graylog server and render them as a valid configuration file for various log collectors. You can think of it like a centralized configuration management system for your log collectors.

Download Graylog Collector Sidecar at
https://github.com/Graylog2/collector-sidecar/releases

Installation

Download a package and install it on the target system.

Ubuntu

Install the NXLog package from the offical download page

  $ sudo /etc/init.d/nxlog stop
  $ sudo update-rc.d -f nxlog remove
  $ sudo gpasswd -a nxlog adm

  $ sudo dpkg -i collector-sidecar_0.0.8-1_amd64.deb
Edit /etc/graylog/collector-sidecar/collector_sidecar.yml, you should set at least the correct URL to your Graylog server and proper tags. The tags are used to define which configurations the host should receive.

  $ sudo graylog-collector-sidecar -service install
  $ sudo start collector-sidecar

CentOS

  $ sudo service nxlog stop
  $ sudo chkconfig --del nxlog
  $ sudo gpasswd -a nxlog root

  $ sudo rpm -i collector-sidecar-0.0.8-1.x86_64.rpm
Activate the Sidecar as a system service

  $ sudo graylog-collector-sidecar -service install
  $ sudo systemctl start collector-sidecar

Windows

Install the NXLog package from the offical download page

  $ C:\Program Files (x86)\nxlog\nxlog -u

  $ graylog_collector_sidecar_installer.exe
It's also possible to run the installer in silent mode with

  $ graylog_collector_sidecar_installer.exe /S
Edit C:\Program Files (x86)\graylog\collector-sidecar\collector_sidecar.yml, you should set at least the correct URL to your Graylog server and proper tags.

  $ C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe -service install
  $ C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe -service start
You can also run the Sidecar in foreground for debugging purposes. Simply call it like this and watch for error messages:

  $ graylog-collector-sidecar -c /etc/graylog/collector-sidecar/collector_sidecar.yml

Configuration

There are a couple of configuration settings for the Sidecar:
ParameterDescription
server_urlURL to the Graylog API, e.g. http://127.0.0.1:12900
tls_skip_verifyIgnore errors when the REST API was started with a self-signed certificate
node_idName of the Sidecar instance, will also show up in the web interface
collector_idUnique ID (UUID) of the instance. This can be an ID string or a path to an ID file
tagsList of configuration tags. All configurations on the server side that match the tag list will be fetched and merged by this instance
log_pathA path to a directory where the Sidecar can store the output of each running collector backend
log_rotation_timeRotate the stdout and stderr logs of each collector after X seconds
log_max_ageDelete rotated log files older than Y seconds
update_intervalThe interval in seconds the sidecar will fetch new configurations from the Graylog server
backendsA list of collector backends the user wants to run on the target host
Each backend can be enabled/disabled and should point to a binary of the actual collector and a path to a configuration file the Sidecar can write to:
ParameterDescription
nameThe type name of the collector
enabledWeather this backend should be started by the Sidecar or not
binary_pathPath to the actual collector binary
configuration_pathA path for this collector configuration file Sidecar can write to

Sample configuration

server_url: http://10.0.10.10:12900
node_id: client-hostname
collector_id: file:/etc/graylog/collector-sidecar/collector-id
tags: linux
update_interval: 10
log_path: /var/log/graylog/collector-sidecar
backends:
    - name: nxlog
      enabled: true
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf

Use the Graylog web interface to configure remote collectors

Navigate to System → Collectors → Manage configurations, this is the entry point for all Sidecar configurations. Multiple configurations can be created. Because not all connected Sidecars should fetch all configurations, it’s essential to provide tags for each configuration. Every Sidecar is only fetching the configuration with the tag it was started with. See also the tags parameter in the section before. Each configuration can hold parts for multiple collector backends.

So you can create one configuration with the tag linux and this include e.g. an input section for a NXlog collector and one for a Filebeat collector. The Sidecar will then pick the right parts based on the backends that are enabled for the host system.


Outputs, Inputs and Snippets

In the example above, Sidecar is instructing NXlog to create a GELF output that writes log messages back to Graylog. The two inputs are for reading in /var/log/syslog as a file input and listening on the UDP port 514 for incoming syslog messages. Both inputs route their messages to the GELF output.

There are three sections in a configuration: Outputs, Inputs and Snippets.

Step 1: Create outputs - Once data is collected by NXLog, the data is transmitted to this IP or address and port. You need to configure a GELF “Input” (System->Inputs) to capture data on the port.

Step 2: Create inputs - Data collected by NXLog. Think of this as a source of log data. For example, it could be a file or a syslog.
In this step we will assign the Collector Output in step 1 we have created.

Step 3: Snippets - Snippets can be used to represent more complicated collector configurations. Simply paste the whole content of your NXlog configuration into a snippet or use it as an extension to the inputs and outputs defined before. All snippets will be copied directly to the generated collector configuration, no matter if there inputs or outputs defined.

Step 4: Fill the tag of Collector, this step will apply the configure we doing to the Collector Sidecar. So we can manage which client should use which configuration.
tags: linux -> this tag is linux.

Step 5: Click on Update tags to push the configure to client.

Configure Collector get log from Syslog services

In Collector Configuration > Configure Collector Inputs > Create new Input
Name: Syslog
Forward to: Output in Step 1
Type: [NXLog] UDP Syslog listener
Host: 127.0.0.1
Port: 514

Next I will configure Rsyslog on Client machine, which running Graylog collector sidecar.
vi /etc/rsyslog.conf
Update remote host then restart rsyslog services
*.* @127.0.0.1:514

Issues

Log folder (centos 7): /var/log/graylog/collector-sidecar
Log file: nxlog_stderr.log nxlog_stdout.log nxlog.log

2016-07-04 05:57:06 ERROR Failed to load module from /usr/lib/nxlog/modules/extension/xm_gelf.so, /usr/lib/nxlog/modules/extension/xm_gelf.so: cannot open shared object file

Resolve:
- Find missing file
# rpm -ql nxlog-ce | grep xm_gelf
/usr/libexec/nxlog/modules/extension/xm_gelf.so

Create symbolink
# ln -s /usr/libexec/nxlog/modules/* /usr/lib/nxlog/modules/.

Nhận xét

Đăng nhận xét

Bài đăng phổ biến từ blog này

CLEANING UP THE ZABBIX DATABASE

My Zabbix database size increased a lot in the last few months and since my disk was running out space, I decided to clean up the old events from the database. Looking at the database tables, the biggest one was  history_uint , which holds the items history data – over 400 millions of records and over 30 Gb of disk space before the clean up. Since deleting the old records from this table directly would be a very slow process, I decided to create a new table and insert the latest records from the  history_uint  table and then just replace the old table with a new one. Since this is not an offical procedure, use it at your own risk. Environment: Zabbix v2.2 MySql 5.1 – InnoDB with  innodb_file_per_table=ON Step 1 – Stop the Zabbix server Step 2 – Open your favourite MySQL client and create a new table CREATE TABLE  history_uint_new  LIKE  history_uint; Step 3 – Insert the latest records from the  history_uint  table to the  history_uint_new table First you need to de
Đọc thêm

Configuring DHCP Relay service on the FortiGate unit

Configuring DHCP Relay service on the FortiGate unit If the clients are configured to obtain a IP address using DHCP relay, configure the FortiGate server as below: To configure DHCP relay on the FortiGate unit   1. Go to System > Network > Interfaces and select Interface want to configure DHCP relay.   2. Enable DHCP Server in the interface and choose Advanced   3. For Mode, select Relay.   4. In Type select Regular.   5. Select OK.   6. If a router is installed between the FortiGate unit and the DHCP server, define a static route to the DHCP server. Note : Sometimes it is required to specify more than one DHCP relay IP, to allow for the coverage of additional LAN subnets. Although the Web Based Manager (GUI) does not provide an option to configure this, it is possible to set up to 8 IPs from the CLI.       config system interface             edit <REQUIRED_INTERFACE_HERE>               set dhcp-relay-ip <FIRST_IP> <SECOND_IP> ....<EIGHTH_IP&
Đọc thêm

Stuxnet Trojan - Memory Forensics with Volatility | Part I

Hình ảnh
Stuxnet could be the first advanced malware. It is thought that it was developed by the United States and Israel to attack Iran's nuclear facilities. It attacked Windows systems using a zero-day exploit and It was focused on SCADA systems in order to  affect critical infrastructures... Also, it may be spread from USB drivers. It is necessary a squad of highly capable programmers with depth of knowledge of industrial processes and an interest in attacking industrial infrastructure to develop this malware. Kaspersky Lab concluded that the sophisticated attack could only have been conducted "with nation-state support" and a study of the spread of Stuxnet by Symantec says that it was spread to Iran (58.85%), Indonesia (18.22%), India (8.31%), Azerbaijan (2.57%).... Thanks to Malware Cookbook we can download a memory dump from an infected host with this malware in the URL below:  http://www.jonrajewski.com/data/Malware/stuxnet.vmem.zip Ok, let’s go. We are going to analyze i
Đọc thêm