Peter Davies

Ramblings

Lessons from GOV.UK PaaS (Cloud Foundry)

For a few years a good share of my hosting work ran on GOV.UK PaaS, the Cloud Foundry platform the UK government ran for public sector services. It's closing at the end of 2024, so I've spent the last few months moving things off it. These are the things I'd want to know if I were starting on any Cloud Foundry platform, and the exit steps I wish I'd planned from the start.

Names below are made up: an app called web in an org called example-org.

Orgs, spaces and targeting

Cloud Foundry groups everything into orgs (roughly, a customer or a team) and spaces inside them. I used one space per environment, so staging and production held the same apps with different data and services.

cf login -a https://api.london.cloud.service.gov.uk --sso
cf target -o example-org -s staging

--sso gives you a one-time code from the browser instead of typing a password into the terminal. Get into the habit of reading cf target before every destructive command; the one time you forget is the time you're in production.

A Python app

The platform builds your app with a buildpack. For Python it looks for three files next to your code: requirements.txt, a runtime.txt naming the Python version, and a Procfile saying how to start it.

The app I moved most often was a Dash dashboard. Dash wraps a Flask server, and gunicorn needs to be pointed at that server object, not the Dash app:

import os

from dash import Dash, html

app = Dash(__name__)
app.layout = html.Main([html.H1("Dashboard")])

server = app.server

if __name__ == "__main__":
    app.run(port=int(os.getenv("PORT", 8080)), debug=False)

web: gunicorn app:server

python-3.11.x

Check which versions the buildpack supports before you pin one; each buildpack release only carries a few patch versions per minor release, and old ones drop off. Cloud Foundry also passes the port in $PORT, and gunicorn binds to it automatically under the buildpack.

Then a manifest.yml, so cf push with no arguments does the same thing every time:

applications:
  - name: web
    instances: 2
    memory: 512M
    disk_quota: 1G
    timeout: 180
    buildpacks:
      - python_buildpack
    health-check-type: http
    health-check-http-endpoint: /
    routes:
      - route: web-staging.london.cloudapps.digital
    env:
      DEPLOYMENT: "PAAS"

health-check-type: http means the platform only sends traffic to an instance once that URL answers, rather than as soon as the process is listening. timeout is how long it waits for that during start-up; heavy Python imports can take a while.

cf push
cf logs web --recent

Autoscaling

The autoscaler is a service like any other: create an instance, bind it to the app and attach a policy.

cf create-service autoscaler autoscaler-free-plan web-autoscaler
cf bind-service web web-autoscaler
cf attach-autoscaling-policy web policy.json
cf restage web

{
  "instance_min_count": 2,
  "instance_max_count": 6,
  "scaling_rules": [
    { "metric_type": "cpu", "operator": ">=", "threshold": 70, "breach_duration_secs": 120, "cool_down_secs": 300, "adjustment": "+1" },
    { "metric_type": "cpu", "operator": "<", "threshold": 25, "breach_duration_secs": 300, "cool_down_secs": 300, "adjustment": "-1" }
  ]
}

To see it work, put some load on it and watch the history:

siege -v -t10M -c100 https://web-staging.london.cloudapps.digital/
cf autoscaling-history web

Run that against staging, never against a production route, and tell the platform team before you load test a shared platform.

Backing services

Redis, PostgreSQL, MySQL, S3 buckets: all created from the marketplace and bound to the app. The plan decides size and whether it's highly available, so staging can be small and production HA. cf marketplace lists the plans:

cf marketplace
cf create-service redis <ha-plan-name> web-redis
cf bind-service web web-redis
cf restage web
cf env web

Binding puts the connection details into the app's environment as JSON in VCAP_SERVICES. Read them from there rather than copying hosts and passwords into config; that way the same code works in every space.

Logs

Platform logs only keep a short window. To keep them, send them to your own log service through a user-provided service with a syslog drain URL:

cf create-user-provided-service web-log-drain -l syslog-tls://logs.example.com:6514
cf bind-service web web-log-drain
cf restage web

The router logs include an x_forwarded_for field holding a comma-separated chain of addresses. The first entry is the client; the rest are the CDN and the platform's load balancers. Split it in your log pipeline, or every request appears to come from the same few addresses.

Custom domains and the CDN route

A custom domain went through the cdn-route service, which put a CloudFront distribution and a certificate in front of the app:

cf create-service cdn-route cdn-route web-cdn -c '{"domain": "example.com,www.example.com"}'
cf service web-cdn

cf service then shows two sets of DNS records to create: a CNAME for each domain pointing at the distribution, and a validation CNAME for each domain for the certificate. Provisioning sits at "create in progress" until DNS is right, which can take a while. Then map the routes to the app:

cf map-route web example.com
cf map-route web example.com --hostname www

By default the CDN forwards few request headers to the app. Applications that depend on headers (some CMSs, anything with sessions) need them forwarded:

cf update-service web-cdn -c '{"headers": ["Authorization", "Accept-Language"]}'

Forwarding all headers with ["*"] works too, but it effectively turns off caching, because every request looks unique.

Internal networking

For monitoring I ran Prometheus and Grafana as apps in the same space. They don't need public routes; apps can talk over the container network on an internal domain once a network policy allows it:

cf map-route prometheus apps.internal --hostname example-prometheus
cf add-network-policy grafana prometheus --protocol tcp --port 8080

Nothing reaches an app over the internal network without a policy, so each hop needs one: Grafana to Prometheus, Prometheus to the exporter, and so on.

The exporter that turned platform metrics into Prometheus format was itself an app that logged in to the Cloud Foundry API, so it needed credentials, set with cf set-env. Use a dedicated account with read-only (auditor) roles for that, never your own login, and remember that anyone with access to the space can read them back with cf env.

Getting out

This is the part I'd plan on day one next time. Before deleting anything, get the data.

Databases aren't reachable from outside the platform. The conduit plugin opens a temporary tunnel and runs your local client through it:

cf install-plugin conduit
cf conduit web-db -- mysqldump --single-transaction --set-gtid-purged=OFF --no-tablespaces --column-statistics=0 --result-file backup.sql web-db

The extra flags stop a recent mysqldump from asking for privileges a managed database doesn't give you, or querying tables it doesn't have.

S3 buckets created through the platform can be given an external service key, which hands you credentials to use with the normal AWS CLI from anywhere:

cf create-service-key web-media web-media-export -c '{"allow_external_access": true}'
cf service-key web-media web-media-export
aws s3 sync s3://<bucket-name-from-key> ./media

Treat those credentials like any other secret and delete the key when you're done (cf delete-service-key).

Then take things down in order: move DNS away first, unmap routes, unbind and delete services, and finally the space.

cf unbind-service web web-redis
cf delete-service web-redis
cf delete-space staging -o example-org

cf delete-space deletes every app, route and service instance in it, without a list of what's going. Run cf apps and cf services in that space first and read them.

What I took from it

The parts that made leaving easy were the twelve-factor habits the platform forced on us: config from the environment, a manifest in the repository, logs going somewhere else and no reliance on local disk. The parts that made it hard were the platform-specific services (the CDN route, the autoscaler) and data sitting in services we could only reach through the platform's own tooling. Whatever you run on, know how you'd get your data out before you put it in.