DBRD mirrored disk

So today I'm gonna talk about how to set up a replication between two servers. In my case I've created one LVM on both servers that is a RAID10 and now I want to use DRBD to keep the block devices synchronized and therefore protect me against a server failure.

We start with installing the DRBD packages. I've found the easiest to just use the elrepo repository: http://elrepo.org/ follow the instructions there to set up the repository in Yum.

You then install the package and kernel module. If you have a recent enough kernel the kmod package may not be necessary. Look it up from drbd website.

yum install -y drbd84-utils kmod-drbd84

Next up we need to configure the drbd resource that we want to mirror. In our case we want to keep /home mirrored so we create a new resource file in /etc/drbd.d/ called home.res with the following contents:


resource home {
  device /dev/drbd1;
  disk /dev/vg0/home;
  meta-disk internal;
  on se3 {
    address 192.168.1.243:7789;
  }
  on se5 {
    address 192.168.1.245:7789;
  }
}

What it says is that it will use the local device /dev/vg0/home and create a new block device /dev/drbd1 that you should then use. In addition we ask to use internal metadata, which means that the metainfo is kept on the same device as data. This however means that you will have to re-create the filesystem on the device. If you want to create a drbd mirror of an already existing filesystem, then you have to use external metadata and I suggest you consule DRBD website Chapter 17 DRBD internals that discusses the metadata usage. There are downsides to using metadata on the same block device if the block device is not a raid volume, but a single disk so do take this into consideration.

So the above configuration describes a drbd resource called home that has two servers with their respective IP-s given and the port on which to talk to the other host. It is preferred to use a separate heartbeat channel if possible, but it works quite well over a single network interface that is also used for other things.

Now that we have described the resource we have to enable it the first time. Choose one of the nodes and ask drbd to create the resources metadata:

# drbdadm create-md home

as we use internal metadata it'll warn you that you are possibly destroying existing data. Remember that you are indeed if you had anything previously stored. In that case you better consult drbd manual how to set up with external metadata.

Now that we have the metadata created we can bring the resource up. For it we first load the dbrd kernel module and then ask the resource to be brought up:

# modprobe drbd
# dbrdadm up home

We can now check the status:


[root@se5 drbd.d]# cat /proc/drbd
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by dag@Build64R5, 2012-09-06 08:15:57

 1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----s
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2147483608


So right now it shows that it's waiting for connection to the other host (we've not yet installed/configured it there) and it considers itself inconsistent, which is normal at this point.

Now let's install drbd on the other server and copy over the configuration so that we can also bring up the other part of our distributed mirror.

rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://elrepo.org/elrepo-release-5-4.el5.elrepo.noarch.rpm
yum install -y drbd84-utils kmod-drbd84
scp se5:/etc/drbd.d/home.res /etc/drbd.d/home.res
drbdadm create-md home
modprobe drbd
drbdadm up home

Now that we've repeated the steps the two resources should be in communication, but still unable to decide who's the master from which to do the initial synchronization:


[root@se3 ~]# cat /proc/drbd
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by dag@Build64R5, 2012-09-06 08:15:57

 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:2147483608

So now let's pick one server (in our case se3) that we call the primary and force it to be considered it:

drbdadm primary --force home

Now drbd knows from whom to copy the necessary data over (even though we've not really even created any data on it) and starts the synchronization:


[root@se3 ~]# cat /proc/drbd
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by dag@Build64R5, 2012-09-06 08:15:57

 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:4025152 nr:0 dw:0 dr:4025152 al:0 bm:245 lo:1 pe:13 ua:0 ap:0 ep:1 wo:f oos:2143460504
[>....................] sync'ed:  0.2% (2093220/2097148)M
finish: 5:59:58 speed: 99,236 (82,104) K/sec

You can now already create a filesystem on the new device and mount it as well as start using, but remember, that it's still under sync so until that is not completed the performance might be degraded.

What I have not covered here is what the default settings are and what they mean. I recommend reading up on drbd website, but basically the default protocol that drbd uses requires synchronized writes. So if the devices are connected any write to the primary device will also initiate a write on the secondary and the OK will be given back only once both have completed. It can be set up as asynchronous write if needed, but that requires changing the protocol. Also, one can set up an active-active (i.e. primary/primary) setup allowing both devices to be used at the same time, but caution should be observed in such setups to avoid split brain situations and issues. In our case we want an active-passive configuration with NFS server failover, but that we will discuss in a separate post.