A brief summary of setting up a CentOS 5.4 DRBD Cluster in Primary/Primary

Marius — Sat, 14 Nov 2009 20:10:45 +0000

Clustering is cool and fun when it’s working, but for getting to such status it mostly takes like a dozen hours, coffee and many nerves. To simplify it a bit, I’d like to share some information on that topic, especially the stones I stumbled over.

First of all, we defined our scenario. Let’s assume that we’d like to have a cluster of two servers, both running the latest CentOS (5.4), both up-to-date, both using the very same partitioning and both using DRBD and GFS(2). On most SuSE or Debian systems, the installation would be pretty straight-forward: You install the base system, set up the DRBD, format it with some OCFS and make Heartbeat monitor everything. So far so good. On RHEL/CentOS it seems to work a bit different, due to the different tools they’re using. When installing the installation-group “Cluster Storage” for example, yum fetches packages named openais and cman – tools you’ve probably never heard of, when you come form the Debian corner (as I do). But before I describe those in detail, let’s just configure our plain base-installation.

What do we need to do first? What’s one of the most important things on two systems that should run “symmetrically” and have the very same data available, with every change that’s being made every second? Exactly, the time would be one of those things. We need to assure that both systems use the very same time. Mostly, you’ll be using some x86_64 hardware for such setups, where the problems start: On 64-bit hardware, the timekeeping with TSC doesn’t run that perfect, which is why we should just deactivate it and leave HPET do its job alone. After installing ntpd we need to open our grub.conf and add the notsc option to our kernels. It should looks something like this:

===================================================================
RCS file: /etc/grub.conf,v
retrieving revision 1.1
diff -u -r1.1 /etc/grub.conf
— /etc/grub.conf 2009/11/13 13:30:26 1.1
+++ /etc/grub.conf 2009/11/13 13:32:26
@@ -13,9 +13,9 @@
hiddenmenu
title CentOS (2.6.18-164.6.1.el5)
root (hd0,0)
- kernel /vmlinuz-2.6.18-164.6.1.el5 ro root=LABEL=/
+ kernel /vmlinuz-2.6.18-164.6.1.el5 ro root=LABEL=/ notsc
initrd /initrd-2.6.18-164.6.1.el5.img
title CentOS (2.6.18-164.el5)
root (hd0,0)
- kernel /vmlinuz-2.6.18-164.el5 ro root=LABEL=/
+ kernel /vmlinuz-2.6.18-164.el5 ro root=LABEL=/ notsc
initrd /initrd-2.6.18-164.el5.img

Now, we can shutdown the ntpd and set its drift-file to 0.000. After that, simply reboot and check the dmesg for the HPET-lines and of course also check the time on both systems to be identically.

Now, what else should we configure until we start building our actual cluster? Probably, we should take a look into the system-config-securitylevel-tui tool. Depending on what environment you plan your cluster to run in, you either want to open each port by port manually in the firewall and configure your SELinux to allow CMAN/OpenAIS and DRBD to work properly – or you simply turn of those “toys” and configure the network-segment to be secure by itself. It depends to you and I’m not going to write how to reconfigure the firewall or your SELinux-environment. For my tests, I simply turned both off. Especially the combination of CMAN/OpenAIS and SELinux can become pretty tricky, when SELinux runs in any other mode than “Disabled”.

Now, let’s finally please our inner kid and install some software:

# yum groupinstall “Cluster Storage”
…
# yum install drbd83 kmod-drbd83
…

I’ve chosen to use drbd83 since it’s the next stable release and already obsoletes drbd82 in CentOS 5.4 – and drbd is simply just too old. Of course, upgrades might become tricky when using explicitly versioned packages, but on DRBD it’s always a bit tricky, since there could be configuration changes which would have to be implemented manually on future versions.

However, now let’s create the infamous and poorly documented /etc/cluster/cluster.conf. For testing, we could simply use something like this:

Configuring OpenAIS this way isn’t actually the best way… it’s not even “good”. But for testing (and understanding how stuff works) it should be enough. Those rules expect manual intervention when one of the two server should become unavailable and needs to be brought back into the cluster.

The domain “.cross” is expected to be a hostname.domainname entry within the /etc/hosts of each server and defines the direct cross-cable-connection from one server to another. We need this connection to shrink down network latency and provide a way for OpenAIS and (in this example) also DRBD to directly communicate with each other. A better setup would be to set the heartbeat on top of a serial-line, since it would be most fault-tolerant.

Okay, next. What’s left? Exactly, the actual DRBD – so let’s set it up:

global { usage-count yes; }
common { syncer { rate 100M; } }
resource the-disk {
protocol C;
startup {
wfc-timeout 20;
degr-wfc-timeout 10;
# become-primary-on both; # Uncomment this only after tested!
}
net {
cram-hmac-alg sha1;
shared-secret “i4m501337″;
allow-two-primaries;
}
on server-1 {
device /dev/drbd1;
disk /dev/sdb;
address 10.100.0.1:7789;
meta-disk internal;
}
on server-2 {
device /dev/drbd1;
disk /dev/sdb;
address 10.100.0.2:7789;
meta-disk internal;
}
disk {
fencing resource-and-stonith;
}
handlers {
#outdate-peer “/sbin/handler”;
}
}

This configuration defines our two servers and tells DRBD to use /dev/sdb on both for the actual data. Our meta-disk will be internal and with address we defined – guess what? – the IP addresses of our two servers. Those are the .cross-domain addresses!

Next, we initialize our meta-disks (on both nodes), set our generation identifier, start the actual DRBD service and check the roles it currently runs in:

# drbdadm create-md the-disk
…
# drbdadm — 6::::1 set-gi the-disk
…
# service drbd start
…
# drbdadm role all
Secondary/Secondary

If all those steps succeed, we can try to promote both nodes to primary:

# drbdadm primary all
# drbdadm role all
Primary/Primary

And if this now worked out properly, we can enable the automatic promotion from within our drbd.conf:

# rcsdiff -u /etc/drbd.conf
===================================================================
RCS file: /etc/drbd.conf,v
retrieving revision 1.2
diff -u -r1.2 /etc/drbd.conf
— /etc/drbd.conf 2009/11/13 10:34:23 1.2
+++ /etc/drbd.conf 2009/11/13 15:16:26
@@ -9,7 +9,7 @@
startup {
wfc-timeout 20;
degr-wfc-timeout 10;
- # become-primary-on both; # Uncomment this only after tested!
+ become-primary-on both; # Uncomment this only after tested!
}
net {
cram-hmac-alg sha1;

Great. So we’re set up now? Nope. We’re not. Next, we need to change DRBDs boot order in order for it to work properly with the GFS auto-mounting on boot:

# rcsdiff -u /etc/init.d/drbd
===================================================================
RCS file: /etc/init.d/drbd,v
retrieving revision 1.1
diff -u -r1.1 /etc/init.d/drbd
— /etc/init.d/drbd 2009/11/13 10:57:15 1.1
+++ /etc/init.d/drbd 2009/11/13 10:58:15
@@ -1,6 +1,6 @@
#!/bin/bash
#
-# chkconfig: 345 70 08
+# chkconfig: 345 22 75
# description: Loads and unloads the drbd module
#
# Copright 2001-2008 LINBIT Information Technologies

And let it run on boot:

# chkconfig –level 345 drbd on

Great! So, now we are set up, right? Nope, wrong. We have a running DRBD setup now, but we still lack of a cluster-able file-system. GFS2 is a pretty good choice for such a task, so let’s try to format the DRBD-device on one of our nodes with it:

# mkfs.gfs2 -p lock_dlm -t cluster-setup:mycluster /dev/drbd1 -j 2

Before we can try to mount the device, we need to have OpenAIS/CMAN running, in order to manage our GFS consistency. Let’s start the cman service therefor (on both nodes!):

# service cman start

Starting cman and starting fenced could take several seconds, be patient. If you installed both servers identically and followed this documentation step by step everything should work out just fine.

At last, we can now mount our DRBD device into some folder (on both nodes) and start playing around with our fresh setup:

# mount -t gfs2 /dev/drbd1 /mnt/somefolder

I hope everything worked out for you and I also hoped that this brief summary helped you getting a bit easier into the actual setup of such a cluster setup. Feel free to ask any questions or provide feedback in any form.

Enjoy!

marius » Cluster

A brief summary of setting up a CentOS 5.4 DRBD Cluster in Primary/Primary