Peter Davies

Ramblings

A Varnish 4 VCL you can actually read

Over the years I've collected a lot of VCL. Most of it started from someone else's file, grew a return (pass) for every awkward URL anyone ever reported, and ended up at 300 to 400 lines that nobody wanted to touch. Moving those sites from Varnish 3 to 4 was a good excuse to start again with one file I could explain line by line.

This is that file. It sits in front of Apache or Nginx on port 8080 and suits a WordPress site or a Magento 1 store. For Magento 2, use the VCL Magento generates for you (Stores, Configuration, Advanced, System, Full Page Cache, Export VCL for Varnish 4) instead; it knows about Magento's own cache tags.

I wrote about installing Varnish 3 a while back. If your VCL is from that era, these are the changes that will stop it compiling:

Varnish 3                 Varnish 4
req.request               req.method
req.backend               req.backend_hint
vcl_fetch                 vcl_backend_response (uses bereq/beresp)
error 403 "..."           return (synth(403, "..."))
vcl_error                 vcl_synth
return (lookup)           return (hash) in vcl_recv
remove                    unset
beresp.saintmode          gone; use probes and grace instead

The backend and ACLs

vcl 4.0;

import std;

backend web {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 5s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 60s;
    .probe = {
        .url = "/health.php";
        .interval = 5s;
        .timeout = 2s;
        .window = 5;
        .threshold = 3;
    }
}

acl purge {
    "127.0.0.1";
    "192.0.2.0"/24;
}

acl debug {
    "198.51.100.25";
}

The long first_byte_timeout is there for slow admin pages and report exports; ten minutes is generous but stops Varnish returning a 503 halfway through an export.

The probe matters more than it looks. Grace mode (below) only serves stale pages when Varnish knows the backend is unhealthy, and the probe is how it knows. Point it at a tiny PHP file that returns "OK", so it tests PHP as well as the web server without loading the whole application every five seconds.

vcl_recv: deciding what to cache

sub vcl_recv {
    set req.backend_hint = web;

    if (req.url == "/varnish-health") {
        return (synth(200, "OK"));
    }

    if (req.method == "PURGE") {
        if (!client.ip ~ purge) {
            return (synth(405, "Not allowed"));
        }
        return (purge);
    }

    if (req.method == "BAN") {
        if (!client.ip ~ purge) {
            return (synth(405, "Not allowed"));
        }
        ban("obj.http.X-Host == " + req.http.host + " && obj.http.X-Url ~ " + req.url);
        return (synth(200, "Banned"));
    }

    if (req.http.X-Forwarded-Proto != "https") {
        return (synth(750, ""));
    }

    if (req.method !~ "^(GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE|PATCH)$") {
        return (pipe);
    }

    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    if (req.http.Authorization) {
        return (pass);
    }

    if (req.url ~ "^/(wp-admin|wp-login\.php|xmlrpc\.php|wp-cron\.php|admin|checkout|customer|wishlist)") {
        return (pass);
    }

    if (req.url ~ "[?&](utm_[a-z]+|gclid|fbclid)=") {
        set req.url = regsuball(req.url, "(?<=[?&])(utm_[a-z]+|gclid|fbclid)=[^&]*&?", "");
        set req.url = regsub(req.url, "[?&]+$", "");
    }

    if (req.url ~ "^[^?]*\.(css|js|gif|jpe?g|png|svg|ico|webp|woff2?|ttf|eot|pdf)(\?.*)?$") {
        unset req.http.Cookie;
        return (hash);
    }

    if (req.http.Cookie ~ "wordpress_logged_in_|wp-postpass_|comment_author_|NO_CACHE") {
        return (pass);
    }
    unset req.http.Cookie;

    if (req.http.User-Agent ~ "(?i)mobile|android|iphone|ipod|blackberry|opera mini") {
        set req.http.X-Device = "mobile";
    } else {
        set req.http.X-Device = "desktop";
    }

    return (hash);
}

Walking through it:

I don't normalise Accept-Encoding any more. Varnish 4 handles gzip itself, and it adds the client to X-Forwarded-For before vcl_recv runs, so the blocks that did both in Varnish 3 VCL can go.

vcl_hash: what makes a page unique

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    hash_data(req.http.X-Forwarded-Proto);
    if (req.http.X-Device) {
        hash_data(req.http.X-Device);
    }
    return (lookup);
}

URL and host as standard, plus the protocol so an HTTP and HTTPS version are never mixed up, plus the device group.

vcl_hit: grace mode

sub vcl_hit {
    if (obj.ttl >= 0s) {
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 10s > 0s) {
            return (deliver);
        }
    } else {
        if (obj.ttl + obj.grace > 0s) {
            return (deliver);
        }
    }
    return (fetch);
}

A fresh object is delivered. An object that expired less than ten seconds ago is delivered while Varnish fetches a new copy in the background. If the probe says the backend is down, anything still inside its grace period is served. This is what keeps the site up during a database restart.

vcl_backend_response: what to store and for how long

sub vcl_backend_response {
    set beresp.http.X-Url = bereq.url;
    set beresp.http.X-Host = bereq.http.host;
    set beresp.grace = 6h;

    if (beresp.status >= 500) {
        set beresp.uncacheable = true;
        set beresp.ttl = 10s;
        return (deliver);
    }

    if (bereq.url ~ "^[^?]*\.(css|js|gif|jpe?g|png|svg|ico|webp|woff2?|ttf|eot|pdf)(\?.*)?$") {
        unset beresp.http.Set-Cookie;
        set beresp.ttl = 4d;
    }

    if (beresp.http.Set-Cookie) {
        set beresp.uncacheable = true;
        set beresp.ttl = 120s;
        return (deliver);
    }

    if (beresp.http.Vary ~ "User-Agent") {
        set beresp.http.Vary = regsub(beresp.http.Vary, ",? *User-Agent *", "");
        set beresp.http.Vary = regsub(beresp.http.Vary, "^, *", "");
        if (beresp.http.Vary == "") {
            unset beresp.http.Vary;
        }
    }

    if (beresp.http.Content-Type ~ "text/html" && !beresp.http.Cache-Control) {
        set beresp.ttl = 1h;
    }

    if (beresp.http.Content-Type ~ "text|javascript|json|xml|svg") {
        set beresp.do_gzip = true;
    }

    return (deliver);
}

vcl_deliver and vcl_synth

sub vcl_deliver {
    unset resp.http.X-Url;
    unset resp.http.X-Host;

    if (client.ip ~ debug) {
        if (obj.hits > 0) {
            set resp.http.X-Cache = "HIT";
            set resp.http.X-Cache-Hits = obj.hits;
        } else {
            set resp.http.X-Cache = "MISS";
        }
    } else {
        unset resp.http.X-Varnish;
        unset resp.http.Via;
        unset resp.http.X-Powered-By;
        unset resp.http.Server;
    }
}

sub vcl_synth {
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.Location = "https://" + req.http.host + req.url;
        return (deliver);
    }
}

sub vcl_pipe {
    set bereq.http.Connection = "close";
}

Your office gets X-Cache headers; everyone else gets fewer clues about the software stack. If Varnish sits behind a load balancer, client.ip is the balancer, not the visitor, and the ACL checks need to use std.ip() on the first address in X-Forwarded-For instead.

Status 750 is a private code: vcl_recv uses it to say "redirect this", and vcl_synth turns it into a real 301.

Testing it

Compile before loading. A syntax error in a live reload leaves the old VCL running, but it's still better to know first:

varnishd -C -f /etc/varnish/default.vcl > /dev/null
systemctl reload varnish

Watch what it does with one URL:

varnishlog -g request -q 'ReqURL ~ "^/blog/"'
varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss

varnishlog shows every step of the request, including which return was taken. If you need more, std.log("...") lines appear as VCL_Log entries in the same output.