--- title: "Anatomy of a compromise: a wp-login brute force on Elastic Beanstalk" date: "2020-10-06" path: "/203-anatomy-of-a-compromise-a-wp-login-flood.html" section: "/ramblings" description: "Reading wp-login.php brute force attempts in Apache logs behind an AWS load balancer, why IP blocking on the instance misses them, and what a successful break-in left on another WordPress site." --- This one is two WordPress sites in the autumn of 2020. The first was on AWS Elastic Beanstalk, being worked over by a password guesser. The second, a few weeks earlier, was on an ordinary server where someone had already got in. Together they show both ends of the same story. ## The Beanstalk site I was looking into 503 errors on the site, so I logged into one of the instances and filtered the Apache log for them: ```bash cd /var/log/httpd tail -f access_log | grep " 503 " ``` What came back was a steady run of POSTs to `wp-login.php`: ```text 172.31.10.20 (203.0.113.77) - - [08/Sep/2020:09:01:50 +0000] "POST /wp-login.php HTTP/1.1" 503 18250 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" 172.31.40.30 (203.0.113.77) - - [08/Sep/2020:09:02:58 +0000] "POST /wp-login.php HTTP/1.1" 503 18250 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" 172.31.40.30 (203.0.113.77) - - [08/Sep/2020:09:04:21 +0000] "POST /wp-login.php HTTP/1.1" 503 18250 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" 172.31.10.20 (203.0.113.77) - - [08/Sep/2020:09:06:47 +0000] "POST /wp-login.php HTTP/1.1" 503 18250 "-" "Mozilla/5.0 (X11; Linux i686; rv:2.0.1) Gecko/20100101 Firefox/4.0.1" ``` ## Reading the lines **Two addresses per line.** Beanstalk's Apache log format puts the connecting address first and the `X-Forwarded-For` header in brackets after it, along the lines of: ```apache LogFormat "%h (%{X-Forwarded-For}i) %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined ``` The first address is in the VPC's private range: it is the load balancer, not the visitor. Two different ones appear because the load balancer has a node in each availability zone. The address in brackets is the real client, and it is the same on every line. **Every attempt is a POST** with no referrer. A person logging in loads the form first, so you would see a GET of `wp-login.php` and a referrer on the POST. A script just posts the form fields. **The user agent** is an ancient Firefox on 32-bit Linux. Tools copy a user agent string once and never update it. A browser that old visiting in 2020 is a bot. **One every minute or two** from one address. That is slow on purpose, to stay under simple rate limits and fail2ban-style thresholds. Each login POST makes WordPress boot fully, look up the user and check a password hash. That is far more work than serving a cached page, which is why login brute forcing hurts a small site out of proportion to the number of requests. ## Why blocking on the instance does not work The obvious fix is `Deny from` the address in `.htaccess`. Behind a load balancer it does nothing, because Apache sees every request coming from the load balancer. Block that and you block everyone. Apache needs to be told to trust the load balancer and take the client address from the header. That is `mod_remoteip`: ```apache RemoteIPHeader X-Forwarded-For RemoteIPInternalProxy 172.31.0.0/16 ``` `172.31.0.0/16` is the default VPC range; use your own VPC's CIDR. On Beanstalk, ship this as part of the application so every new instance gets it: through `.ebextensions` on the Amazon Linux AMI platforms, or as a file under `.platform/httpd/conf.d/` on Amazon Linux 2. Once it is in place, change `%h` to `%a` in the log format, or the logs keep showing the load balancer. I wrote up [logging the real client IP behind a load balancer](/195-logging-the-real-client-ip-behind-a-load-balancer.html) separately. With the real address available, you can restrict the login page in `.htaccess`. If only a known office or VPN logs in, allow just that: ```apache Require ip 198.51.100.0/24 ``` That needs `AllowOverride` to include `AuthConfig` for the directory. Blocking single addresses is a losing game, though, because the next run comes from somewhere else. The better place is in front of the instances, before the requests cost anything. AWS WAF on the load balancer can do that; I went through it in [blocking bots, from .htaccess to AWS WAF](/201-blocking-bots-from-htaccess-to-aws-waf.html). A rate-based rule scoped to POSTs to `/wp-login.php` catches the noisy runs, but rate-based rules have a minimum threshold of 100 requests per five minutes per address, and a guesser this slow stays well under it. For that, a WAF rule that only lets known addresses POST to the login page does the job. Do the same for `xmlrpc.php`, which accepts login attempts too, often many per request. If nothing uses it, block it outright. ## What it looks like when one gets in A few weeks earlier, on a different WordPress site on a conventional Linux server, someone had already got in. I do not know how; the files were what I had to go on. The directory of one of the default themes had picked up some new neighbours: - Ten executable binaries with short generic names that looked like terminal devices, all created within a minute of each other. - Two more binaries, added a month later, whose names made it plain they were XMRig coin miners for 64-bit Linux. - A PHP file with a 32-character hex name. Random hex names are a favourite for webshells, because they do not collide with anything and do not mean anything. - `header.php` had grown to over 70 KB against a couple of KB for the stock theme, and it had the same modification time as the hex-named PHP file. Whatever wrote one, wrote the other. Everything belonged to the site's own user, which tells you the code was running as the web user, not root. To find everything that user owned anywhere on the system, not just in the web root: ```bash find / -group example 2>/dev/null | grep -v '^/home/example/public_html/' ``` And to check WordPress itself against the published checksums: ```bash wp core verify-checksums wp plugin verify-checksums --all ``` Themes have no checksum command, so reinstall the theme from wordpress.org or the vendor rather than trying to tell which of its files are original. Clean-up there followed the same pattern as the rest of this series: kill the miners, take a copy, replace core, plugins and theme with clean downloads, check `wp_users` for admin accounts you do not recognise, rotate every password and the salts in `wp-config.php`, and check every crontab on the box. ## Prevention - Strong, unique passwords and two-factor authentication on every WordPress admin account. Brute forcing only works against guessable passwords. - Rate limit or restrict `wp-login.php` and `xmlrpc.php` at the edge, not on the instance. - On a load-balanced site, set up `mod_remoteip` from day one so your logs and rules see real addresses. - Keep WordPress, plugins and themes updated, and delete the ones you do not use. - Make the code read-only to the web user where you can (`DISALLOW_FILE_EDIT` and `DISALLOW_FILE_MODS` in `wp-config.php` stop the admin editor and installer), so a stolen admin login cannot become a webshell in one click. ## Other parts - [A PHP link-injection backdoor](/178-anatomy-of-a-compromise-php-link-injection-backdoor.html) - [Forensics on a breached shop server](/190-anatomy-of-a-compromise-shop-server-forensics.html) - [Coin miners: the probe that missed](/192-anatomy-of-a-compromise-thinkphp-exploit-coin-miner.html) - [A Magento skimmer hiding behind an image URL](/193-anatomy-of-a-compromise-magento-image-post-backdoor.html) - [The JavaScript injection that kept coming back](/200-anatomy-of-a-compromise-the-javascript-injection-that-kept-coming-back.html) - [A wp-login brute force on Elastic Beanstalk](/203-anatomy-of-a-compromise-a-wp-login-flood.html)