--- title: "Blocking bots: from .htaccess to AWS WAF" date: "2020-09-08" path: "/201-blocking-bots-from-htaccess-to-aws-waf.html" section: "/ramblings" description: "Blocking unwanted crawlers and login attacks by user agent in Apache, then moving the same rules to AWS WAF with rate limiting and managed rule groups." --- Bot traffic comes in two kinds. There are crawlers that announce themselves honestly but which you don't want, usually SEO tools and foreign search engines indexing a site that doesn't sell in their market. And there are scripts hammering `wp-login.php` or probing for vulnerable plugins, which say whatever they like about themselves. This is how I've handled both, first on a single Apache server and then on sites that moved behind an AWS load balancer. ## Start with robots.txt The honest crawlers read `robots.txt`, and asking politely costs nothing: ```text User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / ``` That deals with the well-behaved ones, but not the ones that ignore it. ## Blocking by user agent in Apache The old pattern I had in a lot of `.htaccess` files: ```apache BrowserMatchNoCase "Baiduspider" bots BrowserMatchNoCase "SemrushBot" bots Order Allow,Deny Allow from ALL Deny from env=bots ``` `BrowserMatchNoCase` sets an environment variable when the `User-Agent` matches, and the access rules refuse anything with that variable set. `Order`/`Allow`/`Deny` is Apache 2.2 syntax, which only works on 2.4 through `mod_access_compat`. The 2.4 version: ```apache BrowserMatchNoCase "Baiduspider" bad_bot BrowserMatchNoCase "SemrushBot" bad_bot BrowserMatchNoCase "AhrefsBot" bad_bot BrowserMatchNoCase "MJ12bot" bad_bot Require all granted Require not env bad_bot ``` Blocked requests get a 403. Be careful how short the patterns are: they match anywhere in the string, and a short one such as `Bing` will also catch Microsoft's link preview fetcher as well as its search crawler. Decide whether you really want a whole search engine gone before adding it. On a busy site I'd rather this sat in the virtual host than in `.htaccess`, since Apache rereads `.htaccess` on every request. ## Why that stops being enough User agent blocking only stops bots that tell the truth. The login attacks I was seeing were a steady stream of `POST` requests to `wp-login.php` from thousands of addresses, all claiming to be the same ancient browser. That's easy to match but it's still reaching PHP on every request, and when the site moved behind an Application Load Balancer it made sense to stop it there. AWS WAF attaches to the load balancer (or CloudFront) and drops requests before they reach the servers. ## A WAF rule: one fake browser, one URL, one method This is a WAFv2 rule in the JSON form the console's rule editor and the API both use. It blocks `POST` requests to the login page from one exact user agent string: ```json { "Name": "BlockWpLoginBot", "Priority": 6, "Action": { "Block": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "BlockWpLoginBot" }, "Statement": { "AndStatement": { "Statements": [ { "ByteMatchStatement": { "FieldToMatch": { "Method": {} }, "PositionalConstraint": "EXACTLY", "SearchString": "POST", "TextTransformations": [{ "Type": "NONE", "Priority": 0 }] } }, { "ByteMatchStatement": { "FieldToMatch": { "UriPath": {} }, "PositionalConstraint": "STARTS_WITH", "SearchString": "/wp-login.php", "TextTransformations": [{ "Type": "LOWERCASE", "Priority": 0 }] } }, { "ByteMatchStatement": { "FieldToMatch": { "SingleHeader": { "Name": "user-agent" } }, "PositionalConstraint": "EXACTLY", "SearchString": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)", "TextTransformations": [{ "Type": "NONE", "Priority": 0 }] } } ] } } } ``` Reading it: - `AndStatement` means all three conditions must match. Nobody real is logging into WordPress from Internet Explorer 6 on Windows 2000, so the risk of blocking a person is close to zero. - The path match lowercases the URI first so `/WP-LOGIN.PHP` can't slip past. - `Priority` sets the order rules are checked in, lowest first. Put cheap, specific blocks like this before the managed groups. - `VisibilityConfig` gives you a CloudWatch metric and a sample of matched requests, which is how you see whether it's working. Use `"Count": {}` instead of `"Block": {}` for the first day. The rule then only counts what it would have blocked, and you can check the sampled requests before turning it on. ## Rate limiting the login page The exact-match rule only works until the botnet changes its user agent. A rate-based rule doesn't care what the client claims to be: ```json { "Name": "RateLimitWpLogin", "Priority": 7, "Action": { "Block": {} }, "VisibilityConfig": { "SampledRequestsEnabled": true, "CloudWatchMetricsEnabled": true, "MetricName": "RateLimitWpLogin" }, "Statement": { "RateBasedStatement": { "Limit": 100, "AggregateKeyType": "IP", "ScopeDownStatement": { "ByteMatchStatement": { "FieldToMatch": { "UriPath": {} }, "PositionalConstraint": "STARTS_WITH", "SearchString": "/wp-login.php", "TextTransformations": [{ "Type": "LOWERCASE", "Priority": 0 }] } } } } } ``` The limit counts requests per IP address over a rolling five minutes, and 100 is the lowest WAF allows. The scope-down statement means only login requests count towards it, so a busy visitor browsing the site is never affected. ## Managed rule groups AWS also maintains rule groups you can add to a web ACL. The ones worth turning on for a PHP site: ```text AWSManagedRulesCommonRuleSet general OWASP-style protections, including XSS AWSManagedRulesKnownBadInputsRuleSet request patterns tied to known exploits AWSManagedRulesSQLiRuleSet SQL injection, including sleep and benchmark tricks AWSManagedRulesPHPRuleSet PHP function names and code injection in requests AWSManagedRulesWordPressRuleSet WordPress-specific exploit paths ``` These cover the sort of detections people used to add ModSecurity for: script tags in parameters, `SLEEP()` and `BENCHMARK()` probes for blind SQL injection, PHP function names in query strings. Add them in Count mode too. The common rule set in particular has size limits on request bodies that will block large uploads in some admin areas; when it does, override that single rule to Count rather than removing the group. ## Applying changes The console is the easiest way to edit a web ACL. From the command line, `aws wafv2 update-web-acl` replaces the whole rule list and needs the lock token from a fresh `get-web-acl`, so fetch, edit and put back the full JSON rather than trying to add one rule. None of this replaces keeping WordPress and its plugins up to date. A WAF buys time and cuts noise; it doesn't fix a vulnerable plugin.