Blocking bots: from .htaccess to AWS WAF

8 September 2020

Bot traffic comes in two kinds. There are crawlers that announce themselves honestly but which you don't want, usually SEO tools and foreign search engines indexing a site that doesn't sell in their market. And there are scripts hammering wp-login.php or probing for vulnerable plugins, which say whatever they like about themselves. This is how I've handled both, first on a single Apache server and then on sites that moved behind an AWS load balancer.

Start with robots.txt

The honest crawlers read robots.txt, and asking politely costs nothing:

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12bot
Disallow: /

That deals with the well-behaved ones, but not the ones that ignore it.

Blocking by user agent in Apache

The old pattern I had in a lot of .htaccess files:

BrowserMatchNoCase "Baiduspider" bots
BrowserMatchNoCase "SemrushBot" bots

Order Allow,Deny
Allow from ALL
Deny from env=bots

BrowserMatchNoCase sets an environment variable when the User-Agent matches, and the access rules refuse anything with that variable set. Order/Allow/Deny is Apache 2.2 syntax, which only works on 2.4 through mod_access_compat. The 2.4 version:

BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "SemrushBot" bad_bot
BrowserMatchNoCase "AhrefsBot" bad_bot
BrowserMatchNoCase "MJ12bot" bad_bot

<RequireAll>
    Require all granted
    Require not env bad_bot
</RequireAll>

Blocked requests get a 403. Be careful how short the patterns are: they match anywhere in the string, and a short one such as Bing will also catch Microsoft's link preview fetcher as well as its search crawler. Decide whether you really want a whole search engine gone before adding it.

On a busy site I'd rather this sat in the virtual host than in .htaccess, since Apache rereads .htaccess on every request.

Why that stops being enough

User agent blocking only stops bots that tell the truth. The login attacks I was seeing were a steady stream of POST requests to wp-login.php from thousands of addresses, all claiming to be the same ancient browser. That's easy to match but it's still reaching PHP on every request, and when the site moved behind an Application Load Balancer it made sense to stop it there. AWS WAF attaches to the load balancer (or CloudFront) and drops requests before they reach the servers.

A WAF rule: one fake browser, one URL, one method

This is a WAFv2 rule in the JSON form the console's rule editor and the API both use. It blocks POST requests to the login page from one exact user agent string:

{
  "Name": "BlockWpLoginBot",
  "Priority": 6,
  "Action": { "Block": {} },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "BlockWpLoginBot"
  },
  "Statement": {
    "AndStatement": {
      "Statements": [
        {
          "ByteMatchStatement": {
            "FieldToMatch": { "Method": {} },
            "PositionalConstraint": "EXACTLY",
            "SearchString": "POST",
            "TextTransformations": [{ "Type": "NONE", "Priority": 0 }]
          }
        },
        {
          "ByteMatchStatement": {
            "FieldToMatch": { "UriPath": {} },
            "PositionalConstraint": "STARTS_WITH",
            "SearchString": "/wp-login.php",
            "TextTransformations": [{ "Type": "LOWERCASE", "Priority": 0 }]
          }
        },
        {
          "ByteMatchStatement": {
            "FieldToMatch": { "SingleHeader": { "Name": "user-agent" } },
            "PositionalConstraint": "EXACTLY",
            "SearchString": "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)",
            "TextTransformations": [{ "Type": "NONE", "Priority": 0 }]
          }
        }
      ]
    }
  }
}

Reading it:

AndStatement means all three conditions must match. Nobody real is logging into WordPress from Internet Explorer 6 on Windows 2000, so the risk of blocking a person is close to zero.
The path match lowercases the URI first so /WP-LOGIN.PHP can't slip past.
Priority sets the order rules are checked in, lowest first. Put cheap, specific blocks like this before the managed groups.
VisibilityConfig gives you a CloudWatch metric and a sample of matched requests, which is how you see whether it's working.

Use "Count": {} instead of "Block": {} for the first day. The rule then only counts what it would have blocked, and you can check the sampled requests before turning it on.

The exact-match rule only works until the botnet changes its user agent. A rate-based rule doesn't care what the client claims to be:

{
  "Name": "RateLimitWpLogin",
  "Priority": 7,
  "Action": { "Block": {} },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "RateLimitWpLogin"
  },
  "Statement": {
    "RateBasedStatement": {
      "Limit": 100,
      "AggregateKeyType": "IP",
      "ScopeDownStatement": {
        "ByteMatchStatement": {
          "FieldToMatch": { "UriPath": {} },
          "PositionalConstraint": "STARTS_WITH",
          "SearchString": "/wp-login.php",
          "TextTransformations": [{ "Type": "LOWERCASE", "Priority": 0 }]
        }
      }
    }
  }
}

The limit counts requests per IP address over a rolling five minutes, and 100 is the lowest WAF allows. The scope-down statement means only login requests count towards it, so a busy visitor browsing the site is never affected.

Managed rule groups

AWS also maintains rule groups you can add to a web ACL. The ones worth turning on for a PHP site:

AWSManagedRulesCommonRuleSet       general OWASP-style protections, including XSS
AWSManagedRulesKnownBadInputsRuleSet   request patterns tied to known exploits
AWSManagedRulesSQLiRuleSet         SQL injection, including sleep and benchmark tricks
AWSManagedRulesPHPRuleSet          PHP function names and code injection in requests
AWSManagedRulesWordPressRuleSet    WordPress-specific exploit paths

These cover the sort of detections people used to add ModSecurity for: script tags in parameters, SLEEP() and BENCHMARK() probes for blind SQL injection, PHP function names in query strings. Add them in Count mode too. The common rule set in particular has size limits on request bodies that will block large uploads in some admin areas; when it does, override that single rule to Count rather than removing the group.

Applying changes

The console is the easiest way to edit a web ACL. From the command line, aws wafv2 update-web-acl replaces the whole rule list and needs the lock token from a fresh get-web-acl, so fetch, edit and put back the full JSON rather than trying to add one rule.

None of this replaces keeping WordPress and its plugins up to date. A WAF buys time and cuts noise; it doesn't fix a vulnerable plugin.