---
title: "Two-node MySQL storage with DRBD"
date: "2020-11-03"
path: "/206-two-node-mysql-storage-with-drbd.html"
section: "/ramblings"
description: "Mirroring a MySQL data directory between two Debian servers with DRBD 8.3: the resource file, first sync, moving the data directory, manual failover and split-brain recovery."
---

DRBD mirrors a block device from one server to another over the network, like RAID 1 with a cable in the middle. Put a filesystem on top, put MySQL's data directory on that, and if the first server dies the second has an identical copy of the disk, up to the last committed write, ready to mount.

These notes are from a pair of Debian cloud servers running DRBD 8.3, each with a public interface and a private one. Replication runs over the private network. I'm writing them up because the one mistake in my first config is worth knowing about.

## Install and check

On both nodes:

```bash
apt-get install drbd8-utils
modprobe drbd
cat /proc/drbd
```

```text
version: 8.3.8 (api:88/proto:86-94)
```

If `/proc/drbd` exists, the kernel module is loaded. Note each node's private address with `ip addr show` and its hostname with `uname -n`. DRBD matches the `on` sections in the config against the hostname, so they have to be exactly the same.

## A disk to replicate

DRBD wants a block device of its own, the same size on both nodes. On a cloud server that means attaching a separate volume, which shows up as something like `/dev/xvdb`. Partition it and leave the partition unformatted; DRBD puts the filesystem on later.

For a test with no spare volume, a file on a loop device will do:

```bash
dd if=/dev/zero of=/srv/drbd-r0.img bs=1M count=10240
losetup /dev/loop0 /srv/drbd-r0.img
```

That gives you 10 GB. Don't run production like this: DRBD on a loop device can deadlock under memory pressure, and the loop device has to be set up again before DRBD starts at every boot.

## The resource file

`/etc/drbd.d/r0.res`, identical on both nodes:

```text
resource r0 {
  protocol C;

  startup {
    wfc-timeout 30;
    degr-wfc-timeout 15;
  }

  disk {
    on-io-error detach;
  }

  syncer {
    rate 40M;
  }

  net {
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
  }

  on db-a {
    device    /dev/drbd0;
    disk      /dev/xvdb1;
    address   10.10.10.1:7788;
    meta-disk internal;
  }

  on db-b {
    device    /dev/drbd0;
    disk      /dev/xvdb1;
    address   10.10.10.2:7788;
    meta-disk internal;
  }
}
```

`protocol C` is synchronous: a write isn't reported complete until both disks have it. It's the slowest of DRBD's three protocols and the only one I'd use for a database, because the whole point is that the second copy is never behind.

`wfc-timeout` and `degr-wfc-timeout` stop a node waiting forever at boot for its partner. Without them, a server whose twin is down sits at the DRBD prompt and never finishes booting.

`on-io-error detach` drops the local disk and carries on from the peer if the disk underneath fails. My first config said `panic`, which reboots the node instead; detach keeps the service up.

`rate` caps the background resync so a full sync doesn't starve normal traffic. Set it to around a third of the link's real throughput. I started at 100M, which is more than a shared cloud private network gives you.

`meta-disk internal` keeps DRBD's metadata at the end of the same device, which is simplest.

### The mistake

My first version of this file also had these lines:

```text
startup { become-primary-on both; }
net     { allow-two-primaries; }
```

That's dual-primary mode: both nodes can write at once. It exists for cluster filesystems such as OCFS2 or GFS2, which coordinate locking between the nodes. ext4 doesn't, and nor does MySQL. Mount an ext4 DRBD device on both nodes and write to it, and you corrupt it. For a database, one node is primary and the other is a standby. Leave both lines out.

## First sync

On both nodes, write the metadata and bring the resource up:

```bash
drbdadm create-md r0
drbdadm up r0
cat /proc/drbd
```

Both will show `Secondary/Secondary` and `Inconsistent/Inconsistent`, because neither has ever been declared the good copy. On the node that will start as primary, and only that one:

```bash
drbdadm -- --overwrite-data-of-peer primary r0
watch cat /proc/drbd
```

The status moves to `SyncSource` with a progress bar. You can carry on while it syncs. When it finishes, both sides say `UpToDate/UpToDate`.

## Moving MySQL onto it

On the primary:

```bash
mkfs.ext4 /dev/drbd0
service mysql stop
mount /dev/drbd0 /mnt
rsync -a /var/lib/mysql/ /mnt/
umount /mnt
mount /dev/drbd0 /var/lib/mysql
service mysql start
```

Install the same MySQL version on the standby, but don't let it start at boot and don't mount anything there. Its own `/var/lib/mysql` is just a mount point.

One Debian catch: the system tables now live on the DRBD disk, including the `debian-sys-maint` user that Debian's init script uses. Copy `/etc/mysql/debian.cnf` from the primary to the standby, or the init script on the standby won't be able to shut MySQL down cleanly after a failover.

## Failing over by hand

On the current primary:

```bash
service mysql stop
umount /var/lib/mysql
drbdadm secondary r0
```

On the other node:

```bash
drbdadm primary r0
mount /dev/drbd0 /var/lib/mysql
service mysql start
```

Point the application at the new node, or move a floating IP. Doing this automatically when a node dies is the job of a cluster manager (Heartbeat, later Pacemaker with Corosync); it's a separate subject, and DRBD is the easy half.

## Split brain

If the link between the nodes drops and both become primary, each side has writes the other doesn't. That's split brain. The three `after-sb` policies decide what DRBD does when the nodes reconnect:

- `after-sb-0pri discard-zero-changes`: neither is primary now; if only one side changed anything, take its data automatically.
- `after-sb-1pri discard-secondary`: one is primary; throw away the secondary's changes.
- `after-sb-2pri disconnect`: both are primary; don't guess, stay disconnected and wait for a person.

To recover by hand, pick the node whose data you're throwing away and run on it:

```bash
drbdadm secondary r0
drbdadm -- --discard-my-data connect r0
```

Then on the node you're keeping:

```bash
drbdadm connect r0
```

It resyncs from the survivor. Anything written to the other node during the split is gone, so look at it first.

## Today

DRBD 9 moved the `syncer` options into the `disk` section (`resync-rate`) and uses `drbdadm primary --force r0` for the first sync. For MySQL specifically I'd now reach for MySQL's own replication, Group Replication or Galera, or a managed database, before shared block storage.