DRBD 3 machines stacked setup

This is copy/paste from http://www.howtoforge.com/drbd-8.3-third-node-replication-with-debian-etch plus a split-brain fixes.

WARNING: DO NOT do this setup, unless you'r OK with the speed to remote node. The max. speed you will get from drbd device is the speed you can push data to 3rd node.
--------------




DRBD 8.3 Third Node Replication With Debian Etch


Installation and Set Up Guide for DRBD 8.3 + Debian Etch


The Third Node Setup


by Brian Hellman


The recent release of DRBD 8.3 now includes The Third Node feature as a freely available component. This document will cover the basics of setting up a third node on a standard Debian Etch installation. At the end of this tutorial you will have a DRBD device that can be utilized as a SAN, an iSCSI target, a file server, or a database server.



Note: LINBIT support customers can skip Section 1 and utilize the package repositories.


LINBIT has hosted third node solutions available, please contact them at sales_us at linbit.com for more information.


 


Preface:



The setup is as follows:



  • Three servers: alpha, bravo, foxtrot

  • alpha and bravo are the primary and secondary local nodes

  • foxtrot is the third node which is on a remote network

  • Both alpha and bravo have interfaces on the 192.168.1.x network (eth0) for external connectivity.

  • A crossover link exists on alpha and bravo (eth1) for replication using 172.16.6.10 and .20

  • Heartbeat provides a virtual IP of 192.168.5.2 to communicate with the disaster recovery node located in a geographically diverse location


 


Section 1: Installing The Source


These steps need to be done on each of the 3 nodes.



Prerequisites:



  • make

  • gcc

  • glibc development libraries

  • flex scanner generator

  • headers for the current kernel


Enter the following at the command line as a privileged user to satisfy these dependencies:


apt-get install make gcc libc6 flex linux-headers-`uname -r` libc6-dev linux-kernel-headers


Once the dependencies are installed, download DRBD. The latest version can always be obtained at http://oss.linbit.com/drbd/. Currently, it is 8.3.



cd /usr/src/

wget http://oss.linbit.com/drbd/8.3/drbd-8.3.0.tar.gz


After the download is complete:



  • Uncompress DRBD

  • Enter the source directory

  • Compile the source

  • Install DRBD



tar -xzvf drbd-8.3.0.tar.gz

cd /usr/src/drbd-8.3.0/

make clean all

make install


Now load and verify the module:



modprobe drbd

cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11


Once this has been completed on each of the three nodes, continue to next section.



 


Section 2: Heartbeat Configuration


Setting up a third node entails stacking DRBD on top of DRBD. A virtual IP is needed for the third node to connect to, for this we will set up a simple Heartbeat v1 configuration. This section will only be done on alpha and bravo.


Install Heartbeat:



apt-get install heartbeat


Edit the authkeys file:


vi /etc/ha.d/authkeys


auth 1
1 sha1 yoursupersecretpasswordhere

Once the file has been created, change the permissions on the file. Heartbeat will not start if this step is not followed.


chmod 600 /etc/ha.d/authkeys


Copy the authkeys file to bravo:


scp /etc/ha.d/authkeys bravo:/etc/ha.d/


Edit the ha.cf file:


vi /etc/ha.d/ha.cf


debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 1
deadtime 10
warntime 5
initdead 60
udpport 694
ucast eth0 192.168.1.10
ucast eth0 192.168.1.20
auto_failback off
node alpha
node bravo

Copy the ha.cf file to bravo:


scp /etc/ha.d/ha.cf bravo:/etc/ha.d/


Edit the haresources file, the IP created here will be the IP that our third node refers to.


vi /etc/ha.d/haresources


alpha IPaddr::192.168.5.2/24/eth0

Copy the haresources file to bravo:


scp /etc/ha.d/haresources bravo:/etc/ha.d/


Start the heartbeat service on both servers to bring up the virtual IP:


alpha:/# /etc/init.d/heartbeat start


bravo:/# /etc/init.d/heartbeat start


Heartbeat will bring up the new interface (eth0:0).


Note: It may take heartbeat up to one minute to bring the interface up.



alpha:/# ifconfig eth0:0


eth0:0 Link encap:Ethernet HWaddr 00:08:C7:DB:01:CC

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1


 


Section 3: DRBD Configuration


Configuration for DRBD is done via the drbd.conf file. This needs to be the same on all nodes (alpha, bravo, foxtrot). Please note that the usage-count is set to yes, which means it will notify Linbit that you have installed DRBD. No personal information is collected. Please see this page for more information :


global { usage-count yes; }

resource data-lower {
protocol C;
net {
shared-secret "LINBIT";
}
syncer {
rate 12M;
}

on alpha {
device /dev/drbd1;
disk /dev/hdb1;
address 172.16.6.10:7788;
meta-disk internal;
}

on bravo {
device /dev/drbd1;
disk /dev/hdd1;
address 172.16.6.20:7788;
meta-disk internal;
}
}

resource data-upper {
protocol A;
syncer {
after data-lower;
rate 12M;
al-extents 513;
}
net {
shared-secret "LINBIT";
}
stacked-on-top-of data-lower {
device /dev/drbd3;
address 192.168.5.2:7788; # IP provided by Heartbeat
}

on foxtrot {
device /dev/drbd3;
disk /dev/sdb1;
address 192.168.5.3:7788; # Public IP of the backup node
meta-disk internal;
}
}

 


Section 4: Preparing The DRBD Devices


Now that the configuration is in place, create the metadata on alpha and bravo.



alpha:/usr/src/drbd-8.3.0# drbdadm create-md data-lower


Writing meta data...

initializing activity log

NOT initialized bitmap

New drbd meta data block successfully created.



bravo:/usr/src/drbd-8.3.0# drbdadm create-md data-lower


Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block successfully created.


Now start DRBD on alpha and bravo:


alpha:/usr/src/drbd-8.3.0# /etc/init.d/drbd start


bravo:/usr/src/drbd-8.3.0# /etc/init.d/drbd start


Verify that the lower level DRBD devices are connected:



cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11

0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:19530844


Tell alpha to become the primary node:


NOTE: As the command states, this is going to overwrite any data on bravo: Now is a good time to go and grab your favorite drink.


alpha:/# drbdadm -- --overwrite-data-of-peer primary data-lower

alpha:/# cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11

0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---

ns:3088464 nr:0 dw:0 dr:3089408 al:0 bm:188 lo:23 pe:6 ua:53 ap:0 ep:1 wo:b oos:16442556

[==>.................] sync'ed: 15.9% (16057/19073)M

finish: 0:16:30 speed: 16,512 (8,276) K/sec


After the data sync has finished, create the meta-data on data-upper on alpha, followed by foxtrot.


Note the resource is data-upper and the --stacked option is on alpha only.



alpha:~# drbdadm --stacked create-md data-upper


Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block successfully created.

success



foxtrot:/usr/src/drbd-8.3.0# drbdadm create-md data-upper


Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block sucessfully created.


Bring up the stacked resource, then make alpha the primary of data-upper:


alpha:/# drbdadm --stacked adjust data-upper


foxtrot:~# drbdadm adjust data-upper

foxtrot:~# cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@foxtrot, 2009-02-02 10:28:37

1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent A r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:19530208


alpha:~# drbdadm --stacked -- --overwrite-data-of-peer primary data-upper

alpha:~# cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11

0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---

ns:19532532 nr:0 dw:1688 dr:34046020 al:1 bm:1196 lo:156 pe:0 ua:0 ap:156 ep:1 wo:b oos:0

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r---

ns:14512132 nr:0 dw:0 dr:14512676 al:0 bm:885 lo:156 pe:32 ua:292 ap:0 ep:1 wo:b oos:5018200

[=============>......] sync'ed: 74.4% (4900/19072)M

finish: 0:07:06 speed: 11,776 (10,992) K/sec


Drink time again!


After the sync is complete, access your DRBD block device via /dev/drbd3. This will write to both local nodes and the remote third node. In your Heartbeat configuration you will use the "drbdupper" script to bring up your /dev/drbd3 device. Have fun!



DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.






If you ever get a split-brain (two nodes are in StandAlone and won't want to connect or one is WFConnection the other is StandAlone - it's splitbrain!)
On the node that is outdated do:

drbdadm secondary
drbdadm -- --discard-my-data connect

on the node that has fresh data:
drbdadm --stacked connect

DRBD 3 machines stacked setup

This is copy/paste from http://www.howtoforge.com/drbd-8.3-third-node-replication-with-debian-etch plus a split-brain fixes.

WARNING: DO NOT do this setup, unless you'r OK with the speed to remote node. The max. speed you will get from drbd device is the speed you can push data to 3rd node.
--------------




DRBD 8.3 Third Node Replication With Debian Etch


Installation and Set Up Guide for DRBD 8.3 + Debian Etch


The Third Node Setup


by Brian Hellman


The recent release of DRBD 8.3 now includes The Third Node feature as a freely available component. This document will cover the basics of setting up a third node on a standard Debian Etch installation. At the end of this tutorial you will have a DRBD device that can be utilized as a SAN, an iSCSI target, a file server, or a database server.



Note: LINBIT support customers can skip Section 1 and utilize the package repositories.


LINBIT has hosted third node solutions available, please contact them at sales_us at linbit.com for more information.


 


Preface:



The setup is as follows:



  • Three servers: alpha, bravo, foxtrot

  • alpha and bravo are the primary and secondary local nodes

  • foxtrot is the third node which is on a remote network

  • Both alpha and bravo have interfaces on the 192.168.1.x network (eth0) for external connectivity.

  • A crossover link exists on alpha and bravo (eth1) for replication using 172.16.6.10 and .20

  • Heartbeat provides a virtual IP of 192.168.5.2 to communicate with the disaster recovery node located in a geographically diverse location


 


Section 1: Installing The Source


These steps need to be done on each of the 3 nodes.



Prerequisites:



  • make

  • gcc

  • glibc development libraries

  • flex scanner generator

  • headers for the current kernel


Enter the following at the command line as a privileged user to satisfy these dependencies:


apt-get install make gcc libc6 flex linux-headers-`uname -r` libc6-dev linux-kernel-headers


Once the dependencies are installed, download DRBD. The latest version can always be obtained at http://oss.linbit.com/drbd/. Currently, it is 8.3.



cd /usr/src/

wget http://oss.linbit.com/drbd/8.3/drbd-8.3.0.tar.gz


After the download is complete:



  • Uncompress DRBD

  • Enter the source directory

  • Compile the source

  • Install DRBD



tar -xzvf drbd-8.3.0.tar.gz

cd /usr/src/drbd-8.3.0/

make clean all

make install


Now load and verify the module:



modprobe drbd

cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11


Once this has been completed on each of the three nodes, continue to next section.



 


Section 2: Heartbeat Configuration


Setting up a third node entails stacking DRBD on top of DRBD. A virtual IP is needed for the third node to connect to, for this we will set up a simple Heartbeat v1 configuration. This section will only be done on alpha and bravo.


Install Heartbeat:



apt-get install heartbeat


Edit the authkeys file:


vi /etc/ha.d/authkeys


auth 1
1 sha1 yoursupersecretpasswordhere

Once the file has been created, change the permissions on the file. Heartbeat will not start if this step is not followed.


chmod 600 /etc/ha.d/authkeys


Copy the authkeys file to bravo:


scp /etc/ha.d/authkeys bravo:/etc/ha.d/


Edit the ha.cf file:


vi /etc/ha.d/ha.cf


debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility local0
keepalive 1
deadtime 10
warntime 5
initdead 60
udpport 694
ucast eth0 192.168.1.10
ucast eth0 192.168.1.20
auto_failback off
node alpha
node bravo

Copy the ha.cf file to bravo:


scp /etc/ha.d/ha.cf bravo:/etc/ha.d/


Edit the haresources file, the IP created here will be the IP that our third node refers to.


vi /etc/ha.d/haresources


alpha IPaddr::192.168.5.2/24/eth0

Copy the haresources file to bravo:


scp /etc/ha.d/haresources bravo:/etc/ha.d/


Start the heartbeat service on both servers to bring up the virtual IP:


alpha:/# /etc/init.d/heartbeat start


bravo:/# /etc/init.d/heartbeat start


Heartbeat will bring up the new interface (eth0:0).


Note: It may take heartbeat up to one minute to bring the interface up.



alpha:/# ifconfig eth0:0


eth0:0 Link encap:Ethernet HWaddr 00:08:C7:DB:01:CC

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1


 


Section 3: DRBD Configuration


Configuration for DRBD is done via the drbd.conf file. This needs to be the same on all nodes (alpha, bravo, foxtrot). Please note that the usage-count is set to yes, which means it will notify Linbit that you have installed DRBD. No personal information is collected. Please see this page for more information :


global { usage-count yes; }

resource data-lower {
protocol C;
net {
shared-secret "LINBIT";
}
syncer {
rate 12M;
}

on alpha {
device /dev/drbd1;
disk /dev/hdb1;
address 172.16.6.10:7788;
meta-disk internal;
}

on bravo {
device /dev/drbd1;
disk /dev/hdd1;
address 172.16.6.20:7788;
meta-disk internal;
}
}

resource data-upper {
protocol A;
syncer {
after data-lower;
rate 12M;
al-extents 513;
}
net {
shared-secret "LINBIT";
}
stacked-on-top-of data-lower {
device /dev/drbd3;
address 192.168.5.2:7788; # IP provided by Heartbeat
}

on foxtrot {
device /dev/drbd3;
disk /dev/sdb1;
address 192.168.5.3:7788; # Public IP of the backup node
meta-disk internal;
}
}

 


Section 4: Preparing The DRBD Devices


Now that the configuration is in place, create the metadata on alpha and bravo.



alpha:/usr/src/drbd-8.3.0# drbdadm create-md data-lower


Writing meta data...

initializing activity log

NOT initialized bitmap

New drbd meta data block successfully created.



bravo:/usr/src/drbd-8.3.0# drbdadm create-md data-lower


Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block successfully created.


Now start DRBD on alpha and bravo:


alpha:/usr/src/drbd-8.3.0# /etc/init.d/drbd start


bravo:/usr/src/drbd-8.3.0# /etc/init.d/drbd start


Verify that the lower level DRBD devices are connected:



cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11

0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:19530844


Tell alpha to become the primary node:


NOTE: As the command states, this is going to overwrite any data on bravo: Now is a good time to go and grab your favorite drink.


alpha:/# drbdadm -- --overwrite-data-of-peer primary data-lower

alpha:/# cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11

0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---

ns:3088464 nr:0 dw:0 dr:3089408 al:0 bm:188 lo:23 pe:6 ua:53 ap:0 ep:1 wo:b oos:16442556

[==>.................] sync'ed: 15.9% (16057/19073)M

finish: 0:16:30 speed: 16,512 (8,276) K/sec


After the data sync has finished, create the meta-data on data-upper on alpha, followed by foxtrot.


Note the resource is data-upper and the --stacked option is on alpha only.



alpha:~# drbdadm --stacked create-md data-upper


Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block successfully created.

success



foxtrot:/usr/src/drbd-8.3.0# drbdadm create-md data-upper


Writing meta data...

initialising activity log

NOT initialized bitmap

New drbd meta data block sucessfully created.


Bring up the stacked resource, then make alpha the primary of data-upper:


alpha:/# drbdadm --stacked adjust data-upper


foxtrot:~# drbdadm adjust data-upper

foxtrot:~# cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@foxtrot, 2009-02-02 10:28:37

1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent A r---

ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:19530208


alpha:~# drbdadm --stacked -- --overwrite-data-of-peer primary data-upper

alpha:~# cat /proc/drbd


version: 8.3.0 (api:88/proto:86-89)

GIT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by root@alpha, 2009-02-05 10:36:11

0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r---

ns:19532532 nr:0 dw:1688 dr:34046020 al:1 bm:1196 lo:156 pe:0 ua:0 ap:156 ep:1 wo:b oos:0

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent A r---

ns:14512132 nr:0 dw:0 dr:14512676 al:0 bm:885 lo:156 pe:32 ua:292 ap:0 ep:1 wo:b oos:5018200

[=============>......] sync'ed: 74.4% (4900/19072)M

finish: 0:07:06 speed: 11,776 (10,992) K/sec


Drink time again!


After the sync is complete, access your DRBD block device via /dev/drbd3. This will write to both local nodes and the remote third node. In your Heartbeat configuration you will use the "drbdupper" script to bring up your /dev/drbd3 device. Have fun!



DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.






If you ever get a split-brain (two nodes are in StandAlone and won't want to connect or one is WFConnection the other is StandAlone - it's splitbrain!)
On the node that is outdated do:

drbdadm secondary
drbdadm -- --discard-my-data connect

on the node that has fresh data:
drbdadm --stacked connect