Peter Davies

Ramblings

ELK on Debian

I wanted one place to search syslog and auth logs from a handful of web servers. The first version of this, a couple of years ago, was Elasticsearch 1.7, a Kibana 4 tarball with a borrowed init script, Logstash 1.5 and logstash-forwarder on each server. All of that has moved on. These are my notes from rebuilding it on the Elastic 5.x packages, which put all three parts in one APT repository and swap logstash-forwarder for Filebeat.

The log server is a small Debian 9 VPS. The same steps work on Ubuntu 16.04.

Java

Elasticsearch 5 and Logstash 5 need Java 8. On Debian 9 and Ubuntu 16.04 the default JRE is OpenJDK 8, so there is no need for the old Oracle Java PPA:

apt-get install default-jre
java -version

You do not need to set JAVA_HOME for the packaged services; they find Java themselves.

Add the Elastic repository

apt-get install apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-5.x.list
apt-get update

Elasticsearch

apt-get install elasticsearch

Keep it on localhost. In /etc/elasticsearch/elasticsearch.yml:

network.host: localhost

Elasticsearch 5 starts with a 2 GB heap. On a 1 GB or 2 GB VPS it dies on start-up, and the only clue is an hs_err_pid*.log in /tmp and nothing listening on 9200. Set the heap to half the RAM or less in /etc/elasticsearch/jvm.options, with both values the same:

-Xms512m
-Xmx512m

On a box that small I also added a 1 GB swap file so the rest of the system has somewhere to go. Elastic recommend no swap for real clusters; for a single small log server it stopped the out-of-memory kills.

fallocate -l 1G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl vm.swappiness=10
echo 'vm.swappiness=10' >> /etc/sysctl.conf

Enable and start it, then check it answers:

systemctl enable elasticsearch
systemctl start elasticsearch
curl -XGET 'localhost:9200/?pretty'

You should get a JSON block with the cluster name, the version number and the tagline "You Know, for Search". If not, netstat -tulpn and /var/log/elasticsearch/ are the places to look.

Kibana

apt-get install kibana

In /etc/kibana/kibana.yml, again localhost only, since Apache will sit in front of it:

server.host: "localhost"

systemctl enable kibana
systemctl start kibana

Logstash

apt-get install logstash

Filebeat on the other servers will talk to Logstash over TLS. Make a self-signed certificate whose CN is the name the clients will use for the log server:

mkdir -p /etc/pki/tls/certs /etc/pki/tls/private
cd /etc/pki/tls
openssl req -subj '/CN=logs.example.com/' -x509 -days 3650 -batch -nodes \
  -newkey rsa:2048 -keyout private/logstash.key -out certs/logstash.crt

The certificate (not the key) gets copied to every server that ships logs.

I split the pipeline into three files in /etc/logstash/conf.d/, which Logstash reads in name order. The input, 02-beats-input.conf:

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
    ssl_key => "/etc/pki/tls/private/logstash.key"
  }
}

A syslog filter, 10-syslog-filter.conf. It splits a traditional syslog line into timestamp, host, program, PID and message, decodes the priority, and uses the log's own timestamp instead of the time Logstash received it:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

The two date formats are there because syslog pads single-digit days with a space.

The output, 30-elasticsearch-output.conf:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Test the config before starting it:

/usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
systemctl enable logstash
systemctl start logstash

Apache in front of Kibana

Kibana 5 has no login of its own, so it must not be open to the world. I put it behind an Apache virtual host that proxies to port 5601 and only allows the office IP. Enable the proxy modules first:

a2enmod proxy proxy_http

Then in the virtual host:

<VirtualHost *:443>
    ServerName logs.example.com

    ProxyRequests Off
    ProxyPass / http://localhost:5601/
    ProxyPassReverse / http://localhost:5601/

    <Location />
        Require ip 203.0.113.10
    </Location>

    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/logs.example.com.crt
    SSLCertificateKeyFile /etc/ssl/private/logs.example.com.key
</VirtualHost>

Add basic authentication as well if more than one fixed address needs access.

Firewall

The only ports that need to be reachable are SSH and HTTPS from the office, and 5044 from the servers that send logs. I use a chain for "trusted sources" and drop everything else on those ports. In /etc/iptables.up.rules:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:TRUSTED - [0:0]
:SHIPPERS - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p tcp --dport 22 -j TRUSTED
-A INPUT -p tcp --dport 80 -j TRUSTED
-A INPUT -p tcp --dport 443 -j TRUSTED
-A INPUT -p tcp --dport 5044 -j SHIPPERS
-A TRUSTED -s 203.0.113.10 -j ACCEPT
-A TRUSTED -j DROP
-A SHIPPERS -s 198.51.100.21 -j ACCEPT
-A SHIPPERS -s 198.51.100.22 -j ACCEPT
-A SHIPPERS -j DROP
COMMIT

Load it with iptables-restore < /etc/iptables.up.rules, from a second SSH session so you can back out if you lock yourself out. Elasticsearch on 9200 and Kibana on 5601 are bound to localhost, so they are not reachable from outside anyway.

If the box has a mail server installed for system mail, set Postfix to listen on loopback only so it does not become another open port:

inet_interfaces = loopback-only

Filebeat on each server

On every server that sends logs, add the same Elastic repository and key as above, then:

apt-get install filebeat

Copy the certificate from the log server to /etc/pki/tls/certs/logstash.crt. Then replace /etc/filebeat/filebeat.yml with:

filebeat.prospectors:
  - input_type: log
    paths:
      - /var/log/syslog
      - /var/log/auth.log
    document_type: syslog

output.logstash:
  hosts: ["logs.example.com:5044"]
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]

document_type: syslog is what sets the type field the Logstash filter checks for. Filebeat only allows one output, so if you edit the stock file instead of replacing it, comment out the output.elasticsearch section.

systemctl enable filebeat
systemctl start filebeat

Seeing the data

In Kibana, create an index pattern of logstash-* with @timestamp as the time field. If nothing turns up, check /var/log/filebeat/filebeat on the sending server for TLS errors (usually the hostname in hosts not matching the certificate CN) and /var/log/logstash/ on the log server.

Today

The 5.x packages are long out of support. Since 8.0, Elasticsearch ships with security on by default and its own JDK, so the Java step and the "keep it on localhost because there is no login" approach both change. apt-key is deprecated on current Debian; put the key in /usr/share/keyrings/ and reference it with signed-by= in the source list. In Filebeat, prospectors became inputs in 6.3 and document_type was removed in 6.0.