Logging the real client IP behind a load balancer

16 September 2019

Put a site behind a load balancer, a CDN or a Varnish box and the access log changes overnight: every request now comes from the same handful of addresses, those of the proxy. The visitor's address is still there, in the X-Forwarded-For header the proxy adds, but Apache's standard combined format logs the address of whatever opened the TCP connection.

That matters more than tidy logs. fail2ban, log-based analytics, abuse reports and any "who did this" question after an incident all need the real address.

Quick fix: a second log format

The first thing I did, on a Virtualmin server with some sites behind a balancer and some not, was log the header when it's there and the connection address when it isn't. In the virtual host:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

SetEnvIf X-Forwarded-For "^." forwarded

CustomLog /var/log/virtualmin/example.com_access_log combined env=!forwarded
CustomLog /var/log/virtualmin/example.com_access_log proxy env=forwarded
ErrorLog /var/log/virtualmin/example.com_error_log

How it works:

proxy is combined with %h (remote host) replaced by %{X-Forwarded-For}i (the value of that request header).
SetEnvIf sets an environment variable called forwarded on any request that has the header at all.
Two CustomLog lines write to the same file, each conditional on that variable, so every request is logged exactly once in a format that suits it.

It works, and both formats keep the same field layout so log tools don't notice. But it has problems:

The header is just text the client can send. Anyone hitting the server directly can put any address they like in it, and that's what gets logged.
When a request passes through more than one proxy the header is a list, 203.0.113.7, 198.51.100.20, and the whole list goes into the first field. Some log parsers choke on that.
Only the log changes. PHP's $_SERVER['REMOTE_ADDR'], Apache's own Require ip rules and the error log all still see the proxy.

Better fix: mod_remoteip

Apache 2.4 includes mod_remoteip, which rewrites the client address itself, for every part of Apache and for the application, but only when the request came from a proxy you trust.

a2enmod remoteip

Then in the main config or a conf-available file:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8

Use RemoteIPInternalProxy for balancers on a private network, as in this example where the balancer sits inside a VPC on 10.0.0.0/8. For a proxy on a public address, such as a CDN, use RemoteIPTrustedProxy with its published ranges instead.

mod_remoteip reads X-Forwarded-For from right to left. It strips addresses only while they belong to a trusted proxy, and the first one that doesn't is the client. A forged address added by the visitor sits to the left of their real one, so it's never reached. A request that didn't come from a trusted proxy keeps its connection address and the header is ignored.

Now change the log format to use %a, the client address after mod_remoteip has done its work:

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Don't rely on %h here; %a is the field documented to follow mod_remoteip. If you want to keep the balancer's address as well, %{c}a is the address of the actual connection.

With that in place the single standard CustomLog line is enough again, and the conditional pair can go.

The error log needs its own format if you want the client there too:

ErrorLogFormat "[%t] [%l] [pid %P] [client %a] %M"

Checking it

Restart Apache, load a page through the balancer and look at the end of the log:

apachectl configtest && systemctl restart apache2
tail -f /var/log/virtualmin/example.com_access_log

The first field should be your own public address, not the balancer's. A quick check in PHP confirms the application sees it too:

echo '<?php echo $_SERVER["REMOTE_ADDR"];' > /home/example/public_html/ip.php

Load it, check it and delete it.

One more thing

If the backend servers can be reached directly as well as through the balancer, lock that down. Allow port 80 and 443 only from the balancer's subnet in the firewall or security group. Otherwise someone can skip the balancer, and any WAF or rate limiting on it, entirely.