Peter Davies

Ramblings

Two-node MySQL storage with DRBD

DRBD mirrors a block device from one server to another over the network, like RAID 1 with a cable in the middle. Put a filesystem on top, put MySQL's data directory on that, and if the first server dies the second has an identical copy of the disk, up to the last committed write, ready to mount.

These notes are from a pair of Debian cloud servers running DRBD 8.3, each with a public interface and a private one. Replication runs over the private network. I'm writing them up because the one mistake in my first config is worth knowing about.

Install and check

On both nodes:

apt-get install drbd8-utils
modprobe drbd
cat /proc/drbd

version: 8.3.8 (api:88/proto:86-94)

If /proc/drbd exists, the kernel module is loaded. Note each node's private address with ip addr show and its hostname with uname -n. DRBD matches the on sections in the config against the hostname, so they have to be exactly the same.

A disk to replicate

DRBD wants a block device of its own, the same size on both nodes. On a cloud server that means attaching a separate volume, which shows up as something like /dev/xvdb. Partition it and leave the partition unformatted; DRBD puts the filesystem on later.

For a test with no spare volume, a file on a loop device will do:

dd if=/dev/zero of=/srv/drbd-r0.img bs=1M count=10240
losetup /dev/loop0 /srv/drbd-r0.img

That gives you 10 GB. Don't run production like this: DRBD on a loop device can deadlock under memory pressure, and the loop device has to be set up again before DRBD starts at every boot.

The resource file

/etc/drbd.d/r0.res, identical on both nodes:

resource r0 {
  protocol C;

  startup {
    wfc-timeout 30;
    degr-wfc-timeout 15;
  }

  disk {
    on-io-error detach;
  }

  syncer {
    rate 40M;
  }

  net {
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
  }

  on db-a {
    device    /dev/drbd0;
    disk      /dev/xvdb1;
    address   10.10.10.1:7788;
    meta-disk internal;
  }

  on db-b {
    device    /dev/drbd0;
    disk      /dev/xvdb1;
    address   10.10.10.2:7788;
    meta-disk internal;
  }
}

protocol C is synchronous: a write isn't reported complete until both disks have it. It's the slowest of DRBD's three protocols and the only one I'd use for a database, because the whole point is that the second copy is never behind.

wfc-timeout and degr-wfc-timeout stop a node waiting forever at boot for its partner. Without them, a server whose twin is down sits at the DRBD prompt and never finishes booting.

on-io-error detach drops the local disk and carries on from the peer if the disk underneath fails. My first config said panic, which reboots the node instead; detach keeps the service up.

rate caps the background resync so a full sync doesn't starve normal traffic. Set it to around a third of the link's real throughput. I started at 100M, which is more than a shared cloud private network gives you.

meta-disk internal keeps DRBD's metadata at the end of the same device, which is simplest.

The mistake

My first version of this file also had these lines:

startup { become-primary-on both; }
net     { allow-two-primaries; }

That's dual-primary mode: both nodes can write at once. It exists for cluster filesystems such as OCFS2 or GFS2, which coordinate locking between the nodes. ext4 doesn't, and nor does MySQL. Mount an ext4 DRBD device on both nodes and write to it, and you corrupt it. For a database, one node is primary and the other is a standby. Leave both lines out.

First sync

On both nodes, write the metadata and bring the resource up:

drbdadm create-md r0
drbdadm up r0
cat /proc/drbd

Both will show Secondary/Secondary and Inconsistent/Inconsistent, because neither has ever been declared the good copy. On the node that will start as primary, and only that one:

drbdadm -- --overwrite-data-of-peer primary r0
watch cat /proc/drbd

The status moves to SyncSource with a progress bar. You can carry on while it syncs. When it finishes, both sides say UpToDate/UpToDate.

Moving MySQL onto it

On the primary:

mkfs.ext4 /dev/drbd0
service mysql stop
mount /dev/drbd0 /mnt
rsync -a /var/lib/mysql/ /mnt/
umount /mnt
mount /dev/drbd0 /var/lib/mysql
service mysql start

Install the same MySQL version on the standby, but don't let it start at boot and don't mount anything there. Its own /var/lib/mysql is just a mount point.

One Debian catch: the system tables now live on the DRBD disk, including the debian-sys-maint user that Debian's init script uses. Copy /etc/mysql/debian.cnf from the primary to the standby, or the init script on the standby won't be able to shut MySQL down cleanly after a failover.

Failing over by hand

On the current primary:

service mysql stop
umount /var/lib/mysql
drbdadm secondary r0

On the other node:

drbdadm primary r0
mount /dev/drbd0 /var/lib/mysql
service mysql start

Point the application at the new node, or move a floating IP. Doing this automatically when a node dies is the job of a cluster manager (Heartbeat, later Pacemaker with Corosync); it's a separate subject, and DRBD is the easy half.

Split brain

If the link between the nodes drops and both become primary, each side has writes the other doesn't. That's split brain. The three after-sb policies decide what DRBD does when the nodes reconnect:

To recover by hand, pick the node whose data you're throwing away and run on it:

drbdadm secondary r0
drbdadm -- --discard-my-data connect r0

Then on the node you're keeping:

drbdadm connect r0

It resyncs from the survivor. Anything written to the other node during the split is gone, so look at it first.

Today

DRBD 9 moved the syncer options into the disk section (resync-rate) and uses drbdadm primary --force r0 for the first sync. For MySQL specifically I'd now reach for MySQL's own replication, Group Replication or Galera, or a managed database, before shared block storage.