---
title: "Logging the real client IP behind a load balancer"
date: "2019-09-16"
path: "/195-logging-the-real-client-ip-behind-a-load-balancer.html"
section: "/ramblings"
description: "Two ways to get the visitor's address into Apache 2.4 logs when every request arrives from a load balancer or proxy: conditional log formats and mod_remoteip."
---

Put a site behind a load balancer, a CDN or a Varnish box and the access log changes overnight: every request now comes from the same handful of addresses, those of the proxy. The visitor's address is still there, in the `X-Forwarded-For` header the proxy adds, but Apache's standard `combined` format logs the address of whatever opened the TCP connection.

That matters more than tidy logs. fail2ban, log-based analytics, abuse reports and any "who did this" question after an incident all need the real address.

## Quick fix: a second log format

The first thing I did, on a Virtualmin server with some sites behind a balancer and some not, was log the header when it's there and the connection address when it isn't. In the virtual host:

```apache
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

SetEnvIf X-Forwarded-For "^." forwarded

CustomLog /var/log/virtualmin/example.com_access_log combined env=!forwarded
CustomLog /var/log/virtualmin/example.com_access_log proxy env=forwarded
ErrorLog /var/log/virtualmin/example.com_error_log
```

How it works:

- `proxy` is `combined` with `%h` (remote host) replaced by `%{X-Forwarded-For}i` (the value of that request header).
- `SetEnvIf` sets an environment variable called `forwarded` on any request that has the header at all.
- Two `CustomLog` lines write to the same file, each conditional on that variable, so every request is logged exactly once in a format that suits it.

It works, and both formats keep the same field layout so log tools don't notice. But it has problems:

- The header is just text the client can send. Anyone hitting the server directly can put any address they like in it, and that's what gets logged.
- When a request passes through more than one proxy the header is a list, `203.0.113.7, 198.51.100.20`, and the whole list goes into the first field. Some log parsers choke on that.
- Only the log changes. PHP's `$_SERVER['REMOTE_ADDR']`, Apache's own `Require ip` rules and the error log all still see the proxy.

## Better fix: mod_remoteip

Apache 2.4 includes `mod_remoteip`, which rewrites the client address itself, for every part of Apache and for the application, but only when the request came from a proxy you trust.

```bash
a2enmod remoteip
```

Then in the main config or a `conf-available` file:

```apache
RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.0.0/8
```

Use `RemoteIPInternalProxy` for balancers on a private network, as in this example where the balancer sits inside a VPC on 10.0.0.0/8. For a proxy on a public address, such as a CDN, use `RemoteIPTrustedProxy` with its published ranges instead.

`mod_remoteip` reads `X-Forwarded-For` from right to left. It strips addresses only while they belong to a trusted proxy, and the first one that doesn't is the client. A forged address added by the visitor sits to the left of their real one, so it's never reached. A request that didn't come from a trusted proxy keeps its connection address and the header is ignored.

Now change the log format to use `%a`, the client address after `mod_remoteip` has done its work:

```apache
LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
```

Don't rely on `%h` here; `%a` is the field documented to follow `mod_remoteip`. If you want to keep the balancer's address as well, `%{c}a` is the address of the actual connection.

With that in place the single standard `CustomLog` line is enough again, and the conditional pair can go.

The error log needs its own format if you want the client there too:

```apache
ErrorLogFormat "[%t] [%l] [pid %P] [client %a] %M"
```

## Checking it

Restart Apache, load a page through the balancer and look at the end of the log:

```bash
apachectl configtest && systemctl restart apache2
tail -f /var/log/virtualmin/example.com_access_log
```

The first field should be your own public address, not the balancer's. A quick check in PHP confirms the application sees it too:

```bash
echo '<?php echo $_SERVER["REMOTE_ADDR"];' > /home/example/public_html/ip.php
```

Load it, check it and delete it.

## One more thing

If the backend servers can be reached directly as well as through the balancer, lock that down. Allow port 80 and 443 only from the balancer's subnet in the firewall or security group. Otherwise someone can skip the balancer, and any WAF or rate limiting on it, entirely.