Load testing a site from its sitemap with Siege

Hitting the home page a thousand times tells you how fast the home page is when it's cached. Real visitors land on category pages, product pages and old blog posts, many of which aren't. The quickest way to get a realistic list of URLs is the XML sitemap the site already publishes, and Siege will work through a list like that happily. I first wrote this up for Magento stores in 2012; I still use the same approach, but the URL extraction needed redoing.

Only do this to sites you're responsible for, and tell the hosting provider first if it isn't your own server. From the outside a load test looks exactly like a denial of service attack.

Install

apt-get install siege
siege --version

Run it from a machine other than the web server, ideally one with a decent connection. Running it on the server itself means Siege and the site compete for the same CPU.

Build the URL list

My old version split the sitemap on <url> tags with sed and kept lines whose priority was 0.5 or 1.0, which matched how Magento 1 laid out its sitemap. It broke on any sitemap that didn't use those exact priorities or put everything on one line. Pulling out every <loc> is simpler and works on any sitemap:

curl -s https://www.example.com/sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' > urls.txt
wc -l urls.txt

grep -o prints only the matching part, and the (?<=<loc>) lookbehind (Perl regex, hence -P) starts the match just after the opening tag.

Many sites now publish a sitemap index instead, a sitemap of sitemaps. WordPress 5.5 does this at /wp-sitemap.xml, and SEO plugins and Magento do it on larger sites. Fetch each child sitemap in turn:

curl -s https://www.example.com/wp-sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | while read -r map; do
      curl -s "$map" | grep -oP '(?<=<loc>)[^<]+'
    done > urls.txt

If the sitemap is gzipped (sitemap.xml.gz), add --compressed to curl, or pipe it through gunzip.

A big store can give you tens of thousands of URLs. For a load test a random sample is enough:

shuf -n 500 urls.txt > sample.txt

Run the test

siege -c 50 -t 120S -i -f sample.txt

• -c 50 runs 50 simulated users at once.
• -t 120S runs for 120 seconds. The unit is required: S, M or H.
• -i ("internet" mode) makes each user pick URLs at random from the file instead of going through it in order, which is closer to real traffic.
• -f is the URL file.

By default each simulated user waits a random delay of up to a second between requests (-d changes it). Add -b for benchmark mode, which drops the delay and sends requests as fast as possible. That's useful for finding the ceiling, less so for modelling real visitors.

Siege prints the summary on standard error, so siege ... > result.txt gives you an empty file. Capture both:

siege -c 50 -t 120S -i -f sample.txt 2>&1 | tee siege-result.txt

Reading the summary

The figures that matter:

• Availability: the percentage of requests that succeeded. Anything under 100% needs explaining before you look at anything else.
• Response time: the average seconds per request.
• Transaction rate: requests per second the site sustained.
• Longest transaction: the slowest single request. A big gap between this and the average usually means some pages are uncached or a database query is slow.
• Failed transactions: responses with a status of 400 or above, plus socket errors and timeouts. Use -v on a short run to see which URLs are failing and why.

Start low and raise -c in steps (10, 25, 50, 100), running each for a couple of minutes. The interesting number is the concurrency where response time stops being flat and starts climbing. While it runs, watch the server with top or htop, and if there's a cache in front, varnishstat shows whether requests are being served from cache.

Cache warming

The same list makes a good cache warmer after a deploy or a full cache flush. One user, working through every URL in order, with each request shown:

siege -c 1 -r 1 -v -f urls.txt

-r 1 means one pass through the file and then stop. -v prints each request with its status code and time, so you can also spot any URLs in the sitemap that return 404 or redirect, which shouldn't be in a sitemap at all.

Settings file

Siege keeps its defaults in ~/.siege/siege.conf (run siege.config to create one). Two lines are worth checking before a test: accept-encoding, so Siege asks for compressed pages the way a browser does, and user-agent, so you can pick out the test traffic in the access log afterwards. Extra request headers can also be added for a single run with -H:

siege -c 25 -t 60S -i -H "Accept-Encoding: gzip" -f sample.txt