---
title: "Lessons from GOV.UK PaaS (Cloud Foundry)"
date: "2024-01-10"
path: "/220-lessons-from-gov-uk-paas-cloud-foundry.html"
section: "/ramblings"
description: "Working with the cf CLI on a hosted Cloud Foundry platform: spaces, manifests, a Python app under gunicorn, autoscaling, backing services, log drains, custom domains, internal networking and getting your data out when the platform closes."
---

For a few years a good share of my hosting work ran on GOV.UK PaaS, the Cloud Foundry platform the UK government ran for public sector services. It's closing at the end of 2024, so I've spent the last few months moving things off it. These are the things I'd want to know if I were starting on any Cloud Foundry platform, and the exit steps I wish I'd planned from the start.

Names below are made up: an app called `web` in an org called `example-org`.

## Orgs, spaces and targeting

Cloud Foundry groups everything into orgs (roughly, a customer or a team) and spaces inside them. I used one space per environment, so `staging` and `production` held the same apps with different data and services.

```bash
cf login -a https://api.london.cloud.service.gov.uk --sso
cf target -o example-org -s staging
```

`--sso` gives you a one-time code from the browser instead of typing a password into the terminal. Get into the habit of reading `cf target` before every destructive command; the one time you forget is the time you're in production.

## A Python app

The platform builds your app with a buildpack. For Python it looks for three files next to your code: `requirements.txt`, a `runtime.txt` naming the Python version, and a `Procfile` saying how to start it.

The app I moved most often was a Dash dashboard. Dash wraps a Flask server, and gunicorn needs to be pointed at that server object, not the Dash app:

```python
import os

from dash import Dash, html

app = Dash(__name__)
app.layout = html.Main([html.H1("Dashboard")])

server = app.server

if __name__ == "__main__":
    app.run(port=int(os.getenv("PORT", 8080)), debug=False)
```

```text
web: gunicorn app:server
```

```text
python-3.11.x
```

Check which versions the buildpack supports before you pin one; each buildpack release only carries a few patch versions per minor release, and old ones drop off. Cloud Foundry also passes the port in `$PORT`, and gunicorn binds to it automatically under the buildpack.

Then a `manifest.yml`, so `cf push` with no arguments does the same thing every time:

```yaml
applications:
  - name: web
    instances: 2
    memory: 512M
    disk_quota: 1G
    timeout: 180
    buildpacks:
      - python_buildpack
    health-check-type: http
    health-check-http-endpoint: /
    routes:
      - route: web-staging.london.cloudapps.digital
    env:
      DEPLOYMENT: "PAAS"
```

`health-check-type: http` means the platform only sends traffic to an instance once that URL answers, rather than as soon as the process is listening. `timeout` is how long it waits for that during start-up; heavy Python imports can take a while.

```bash
cf push
cf logs web --recent
```

## Autoscaling

The autoscaler is a service like any other: create an instance, bind it to the app and attach a policy.

```bash
cf create-service autoscaler autoscaler-free-plan web-autoscaler
cf bind-service web web-autoscaler
cf attach-autoscaling-policy web policy.json
cf restage web
```

```json
{
  "instance_min_count": 2,
  "instance_max_count": 6,
  "scaling_rules": [
    { "metric_type": "cpu", "operator": ">=", "threshold": 70, "breach_duration_secs": 120, "cool_down_secs": 300, "adjustment": "+1" },
    { "metric_type": "cpu", "operator": "<", "threshold": 25, "breach_duration_secs": 300, "cool_down_secs": 300, "adjustment": "-1" }
  ]
}
```

To see it work, put some load on it and watch the history:

```bash
siege -v -t10M -c100 https://web-staging.london.cloudapps.digital/
cf autoscaling-history web
```

Run that against staging, never against a production route, and tell the platform team before you load test a shared platform.

## Backing services

Redis, PostgreSQL, MySQL, S3 buckets: all created from the marketplace and bound to the app. The plan decides size and whether it's highly available, so staging can be small and production HA. `cf marketplace` lists the plans:

```bash
cf marketplace
cf create-service redis <ha-plan-name> web-redis
cf bind-service web web-redis
cf restage web
cf env web
```

Binding puts the connection details into the app's environment as JSON in `VCAP_SERVICES`. Read them from there rather than copying hosts and passwords into config; that way the same code works in every space.

## Logs

Platform logs only keep a short window. To keep them, send them to your own log service through a user-provided service with a syslog drain URL:

```bash
cf create-user-provided-service web-log-drain -l syslog-tls://logs.example.com:6514
cf bind-service web web-log-drain
cf restage web
```

The router logs include an `x_forwarded_for` field holding a comma-separated chain of addresses. The first entry is the client; the rest are the CDN and the platform's load balancers. Split it in your log pipeline, or every request appears to come from the same few addresses.

## Custom domains and the CDN route

A custom domain went through the `cdn-route` service, which put a CloudFront distribution and a certificate in front of the app:

```bash
cf create-service cdn-route cdn-route web-cdn -c '{"domain": "example.com,www.example.com"}'
cf service web-cdn
```

`cf service` then shows two sets of DNS records to create: a CNAME for each domain pointing at the distribution, and a validation CNAME for each domain for the certificate. Provisioning sits at "create in progress" until DNS is right, which can take a while. Then map the routes to the app:

```bash
cf map-route web example.com
cf map-route web example.com --hostname www
```

By default the CDN forwards few request headers to the app. Applications that depend on headers (some CMSs, anything with sessions) need them forwarded:

```bash
cf update-service web-cdn -c '{"headers": ["Authorization", "Accept-Language"]}'
```

Forwarding all headers with `["*"]` works too, but it effectively turns off caching, because every request looks unique.

## Internal networking

For monitoring I ran Prometheus and Grafana as apps in the same space. They don't need public routes; apps can talk over the container network on an internal domain once a network policy allows it:

```bash
cf map-route prometheus apps.internal --hostname example-prometheus
cf add-network-policy grafana prometheus --protocol tcp --port 8080
```

Nothing reaches an app over the internal network without a policy, so each hop needs one: Grafana to Prometheus, Prometheus to the exporter, and so on.

The exporter that turned platform metrics into Prometheus format was itself an app that logged in to the Cloud Foundry API, so it needed credentials, set with `cf set-env`. Use a dedicated account with read-only (auditor) roles for that, never your own login, and remember that anyone with access to the space can read them back with `cf env`.

## Getting out

This is the part I'd plan on day one next time. Before deleting anything, get the data.

Databases aren't reachable from outside the platform. The `conduit` plugin opens a temporary tunnel and runs your local client through it:

```bash
cf install-plugin conduit
cf conduit web-db -- mysqldump --single-transaction --set-gtid-purged=OFF --no-tablespaces --column-statistics=0 --result-file backup.sql web-db
```

The extra flags stop a recent `mysqldump` from asking for privileges a managed database doesn't give you, or querying tables it doesn't have.

S3 buckets created through the platform can be given an external service key, which hands you credentials to use with the normal AWS CLI from anywhere:

```bash
cf create-service-key web-media web-media-export -c '{"allow_external_access": true}'
cf service-key web-media web-media-export
aws s3 sync s3://<bucket-name-from-key> ./media
```

Treat those credentials like any other secret and delete the key when you're done (`cf delete-service-key`).

Then take things down in order: move DNS away first, unmap routes, unbind and delete services, and finally the space.

```bash
cf unbind-service web web-redis
cf delete-service web-redis
cf delete-space staging -o example-org
```

`cf delete-space` deletes every app, route and service instance in it, without a list of what's going. Run `cf apps` and `cf services` in that space first and read them.

## What I took from it

The parts that made leaving easy were the twelve-factor habits the platform forced on us: config from the environment, a manifest in the repository, logs going somewhere else and no reliance on local disk. The parts that made it hard were the platform-specific services (the CDN route, the autoscaler) and data sitting in services we could only reach through the platform's own tooling. Whatever you run on, know how you'd get your data out before you put it in.