--- title: "ELK on Debian" date: "2017-11-06" path: "/184-elk-on-debian.html" section: "/ramblings" description: "Building a small Elasticsearch, Logstash and Kibana log server on Debian with the Elastic 5.x packages, Filebeat on the servers it watches, and access locked down." --- I wanted one place to search syslog and auth logs from a handful of web servers. The first version of this, a couple of years ago, was Elasticsearch 1.7, a Kibana 4 tarball with a borrowed init script, Logstash 1.5 and logstash-forwarder on each server. All of that has moved on. These are my notes from rebuilding it on the Elastic 5.x packages, which put all three parts in one APT repository and swap logstash-forwarder for Filebeat. The log server is a small Debian 9 VPS. The same steps work on Ubuntu 16.04. ## Java Elasticsearch 5 and Logstash 5 need Java 8. On Debian 9 and Ubuntu 16.04 the default JRE is OpenJDK 8, so there is no need for the old Oracle Java PPA: ```bash apt-get install default-jre java -version ``` You do not need to set `JAVA_HOME` for the packaged services; they find Java themselves. ## Add the Elastic repository ```bash apt-get install apt-transport-https wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add - echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-5.x.list apt-get update ``` ## Elasticsearch ```bash apt-get install elasticsearch ``` Keep it on localhost. In `/etc/elasticsearch/elasticsearch.yml`: ```yaml network.host: localhost ``` Elasticsearch 5 starts with a 2 GB heap. On a 1 GB or 2 GB VPS it dies on start-up, and the only clue is an `hs_err_pid*.log` in `/tmp` and nothing listening on 9200. Set the heap to half the RAM or less in `/etc/elasticsearch/jvm.options`, with both values the same: ```text -Xms512m -Xmx512m ``` On a box that small I also added a 1 GB swap file so the rest of the system has somewhere to go. Elastic recommend no swap for real clusters; for a single small log server it stopped the out-of-memory kills. ```bash fallocate -l 1G /swapfile chmod 600 /swapfile mkswap /swapfile swapon /swapfile echo '/swapfile none swap sw 0 0' >> /etc/fstab sysctl vm.swappiness=10 echo 'vm.swappiness=10' >> /etc/sysctl.conf ``` Enable and start it, then check it answers: ```bash systemctl enable elasticsearch systemctl start elasticsearch curl -XGET 'localhost:9200/?pretty' ``` You should get a JSON block with the cluster name, the version number and the tagline "You Know, for Search". If not, `netstat -tulpn` and `/var/log/elasticsearch/` are the places to look. ## Kibana ```bash apt-get install kibana ``` In `/etc/kibana/kibana.yml`, again localhost only, since Apache will sit in front of it: ```yaml server.host: "localhost" ``` ```bash systemctl enable kibana systemctl start kibana ``` ## Logstash ```bash apt-get install logstash ``` Filebeat on the other servers will talk to Logstash over TLS. Make a self-signed certificate whose CN is the name the clients will use for the log server: ```bash mkdir -p /etc/pki/tls/certs /etc/pki/tls/private cd /etc/pki/tls openssl req -subj '/CN=logs.example.com/' -x509 -days 3650 -batch -nodes \ -newkey rsa:2048 -keyout private/logstash.key -out certs/logstash.crt ``` The certificate (not the key) gets copied to every server that ships logs. I split the pipeline into three files in `/etc/logstash/conf.d/`, which Logstash reads in name order. The input, `02-beats-input.conf`: ```text input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash.crt" ssl_key => "/etc/pki/tls/private/logstash.key" } } ``` A syslog filter, `10-syslog-filter.conf`. It splits a traditional syslog line into timestamp, host, program, PID and message, decodes the priority, and uses the log's own timestamp instead of the time Logstash received it: ```text filter { if [type] == "syslog" { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } syslog_pri { } date { match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } ``` The two date formats are there because syslog pads single-digit days with a space. The output, `30-elasticsearch-output.conf`: ```text output { elasticsearch { hosts => ["localhost:9200"] } } ``` Test the config before starting it: ```bash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t systemctl enable logstash systemctl start logstash ``` ## Apache in front of Kibana Kibana 5 has no login of its own, so it must not be open to the world. I put it behind an Apache virtual host that proxies to port 5601 and only allows the office IP. Enable the proxy modules first: ```bash a2enmod proxy proxy_http ``` Then in the virtual host: ```apache ServerName logs.example.com ProxyRequests Off ProxyPass / http://localhost:5601/ ProxyPassReverse / http://localhost:5601/ Require ip 203.0.113.10 SSLEngine on SSLCertificateFile /etc/ssl/certs/logs.example.com.crt SSLCertificateKeyFile /etc/ssl/private/logs.example.com.key ``` Add basic authentication as well if more than one fixed address needs access. ## Firewall The only ports that need to be reachable are SSH and HTTPS from the office, and 5044 from the servers that send logs. I use a chain for "trusted sources" and drop everything else on those ports. In `/etc/iptables.up.rules`: ```text *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :TRUSTED - [0:0] :SHIPPERS - [0:0] -A INPUT -i lo -j ACCEPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT -A INPUT -p tcp --dport 22 -j TRUSTED -A INPUT -p tcp --dport 80 -j TRUSTED -A INPUT -p tcp --dport 443 -j TRUSTED -A INPUT -p tcp --dport 5044 -j SHIPPERS -A TRUSTED -s 203.0.113.10 -j ACCEPT -A TRUSTED -j DROP -A SHIPPERS -s 198.51.100.21 -j ACCEPT -A SHIPPERS -s 198.51.100.22 -j ACCEPT -A SHIPPERS -j DROP COMMIT ``` Load it with `iptables-restore < /etc/iptables.up.rules`, from a second SSH session so you can back out if you lock yourself out. Elasticsearch on 9200 and Kibana on 5601 are bound to localhost, so they are not reachable from outside anyway. If the box has a mail server installed for system mail, set Postfix to listen on loopback only so it does not become another open port: ```text inet_interfaces = loopback-only ``` ## Filebeat on each server On every server that sends logs, add the same Elastic repository and key as above, then: ```bash apt-get install filebeat ``` Copy the certificate from the log server to `/etc/pki/tls/certs/logstash.crt`. Then replace `/etc/filebeat/filebeat.yml` with: ```yaml filebeat.prospectors: - input_type: log paths: - /var/log/syslog - /var/log/auth.log document_type: syslog output.logstash: hosts: ["logs.example.com:5044"] ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"] ``` `document_type: syslog` is what sets the `type` field the Logstash filter checks for. Filebeat only allows one output, so if you edit the stock file instead of replacing it, comment out the `output.elasticsearch` section. ```bash systemctl enable filebeat systemctl start filebeat ``` ## Seeing the data In Kibana, create an index pattern of `logstash-*` with `@timestamp` as the time field. If nothing turns up, check `/var/log/filebeat/filebeat` on the sending server for TLS errors (usually the hostname in `hosts` not matching the certificate CN) and `/var/log/logstash/` on the log server. ## Today The 5.x packages are long out of support. Since 8.0, Elasticsearch ships with security on by default and its own JDK, so the Java step and the "keep it on localhost because there is no login" approach both change. `apt-key` is deprecated on current Debian; put the key in `/usr/share/keyrings/` and reference it with `signed-by=` in the source list. In Filebeat, `prospectors` became `inputs` in 6.3 and `document_type` was removed in 6.0.