In this tutorial we will set up a highly available server providing ATA-over-Ethernet (AoE) targets to AoE initiators. Should a server become unavailable, services provided by our cluster will continue to be available to client systems.

Our highly available system will resemble the following:

aoe.jpg

AoE server1: node1.home.local IP address: 10.10.1.251
AoE server2: node2.home.local IP address: 10.10.1.252

To begin, set up two Ubuntu 9.04 (Jaunty Jackalope) systems. In this guide, the servers will be set up in a virtual environment using KVM-84. Using a virtual environment will allow us to add additional disk devices and NICs as needed.

The following partition scheme will be used for the Operating System installation:

/dev/vda1 -- 10 GB / (primary' jfs, Bootable flag: on)
/dev/vda5 -- 1 GB swap (logical)

After the installation of a minimal Ubuntu install on both servers, we will install packages required to configure a bonded network interface, and in-turn assign static IP addresses to bond0 of node1 and node2. Using a bonded interface will prevent a single point of failure should the client accessible network fail.

Install ifenslave

apt-get -y install ifenslave

Append the following to /etc/modprobe.d/aliases.conf:

alias bond0 bonding
options bond0 mode=0 miimon=100 downdelay=200 updelay=200 max_bonds=2

Modify our network configuration and assign eth0 and eth1 as slaves of bond0.

Example /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The interfaces that will be bonded
auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

# The target-accessible network interface
auto bond0
iface bond0 inet static
        address 10.10.1.251
        netmask 255.255.255.0
        broadcast 10.10.1.255
        network 10.10.1.0
        gateway 10.10.1.1
        up /sbin/ifenslave bond0 eth0
        up /sbin/ifenslave bond0 eth1

We do not need to define eth0 or eth1 in /etc/network/interfaces as they will be brought up when the bond comes up. I have included them for documentation purposes.

Please note: AoE does not use TCP/IP for communication, it instead uses raw Ethernet frames to carry ATA commands and data. We are assigning an IP address so we can administer the nodes on the public interface.

Review the current status of the bonded interface.

cat /proc/net/bonding/bond0 
Example output:
Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 54:52:00:6d:f7:4d

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 54:52:00:11:36:cf

Please note: A bonded network interface supports multiple modes. In this example eth0 and eth1 are in an round-robin configuration.

Shutdown both servers and add additional devices. We will add additional disks to contain the DRBD meta data and the data that is mirrored between the two servers. We will also add an isolated network for the two servers to communicate and transfer the DRBD data.

The following partition scheme will be used for the DRBD data:

/dev/vdb1 -- 1 GB unmounted (primary) DRBD meta data
/dev/vdc1 -- 1 GB umounted (primary) DRBD device used for AoE configuration files
/dev/vdd1 -- 10 GB unmounted (primary) DRBD device used as the AoE target

Sample output from fdisk -l:

Disk /dev/vda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000d570a

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           1        1244     9992398+  83  Linux
/dev/vda2            1245        1305      489982+   5  Extended
/dev/vda5            1245        1305      489951   82  Linux swap / Solaris

Disk /dev/vdb: 1073 MB, 1073741824 bytes
root@node1:~# fdisk -l

Disk /dev/vda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000d570a

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           1        1244     9992398+  83  Linux
/dev/vda2            1245        1305      489982+   5  Extended
/dev/vda5            1245        1305      489951   82  Linux swap / Solaris

Disk /dev/vdb: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0xba6f1cad

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1               1        2080     1048288+  83  Linux

Disk /dev/vdc: 1073 MB, 1073741824 bytes
16 heads, 63 sectors/track, 2080 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0xdbde4889

   Device Boot      Start         End      Blocks   Id  System
/dev/vdc1               1        2080     1048288+  83  Linux

Disk /dev/vdd: 10.7 GB, 10737418240 bytes
16 heads, 63 sectors/track, 20805 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0xf505afa1

   Device Boot      Start         End      Blocks   Id  System
/dev/vdd1               1       20805    10485688+  83  Linux

The isolated network between the two servers will be:

AoE server1: node1-private IP address: 10.10.2.251
AoE server2: node2-private IP address: 10.10.2.252

We will again bond these two interfaces. If our server is to be highly available, we should eliminate all single points of failure.

Append the following to /etc/modprobe.d/aliases.conf:

alias bond1 bonding
options bond0 mode=0 miimon=100 downdelay=200 updelay=200

Example /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The interfaces that will be bonded
auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

# The initiator-accessible network interface
auto bond0
iface bond0 inet static
        address 10.10.1.251
        netmask 255.255.255.0
        broadcast 10.10.1.255
        network 10.10.1.0
        gateway 10.10.1.1
        up /sbin/ifenslave bond0 eth0
        up /sbin/ifenslave bond0 eth1

# The isolated network interface
auto bond1
iface bond1 inet static
        address 10.10.2.251
        netmask 255.255.255.0
        broadcast 10.10.2.255
        network 10.10.2.0
        up /sbin/ifenslave bond1 eth2
        up /sbin/ifenslave bond1 eth3

Ensure that /etc/hosts on both nodes contains the names and IP addresses of the two servers.

Example /etc/hosts:

127.0.0.1       localhost
10.10.1.251     node1.home.local    node1
10.10.1.252     node2.home.local    node2
10.10.2.251     node1-private
10.10.2.252     node2-private

Install NTP to ensure both servers have the same time.

apt-get -y install ntp

You can verify the time is in sync with the date command.

At this point, you can either modprobe the second bond, or restart both servers.

Install drbd and heartbeat.

apt-get -y install drbd8-utils heartbeat

As we will be using heartbeat with drbd, we need to change ownership and permissions on several DRBD related files on both servers.

chgrp haclient /sbin/drbdsetup
chmod o-x /sbin/drbdsetup
chmod u+s /sbin/drbdsetup
chgrp haclient /sbin/drbdmeta
chmod o-x /sbin/drbdmeta
chmod u+s /sbin/drbdmeta

Using /etc/drbd.conf as an example create your resource configuration. We will define two resources.

  1. The drbd device that will contain our AoE configuration files
  2. The drbd device that will become our AoE target

Example /etc/drbd.conf:

resource aoe.config {
        protocol C;
 
        handlers {
        pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
        pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
        local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
        outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";      
        }

        startup {
        degr-wfc-timeout 120;
        }

        disk {
        on-io-error detach;
        }

        net {
        cram-hmac-alg sha1;
        shared-secret "password";
        after-sb-0pri disconnect;
        after-sb-1pri disconnect;
        after-sb-2pri disconnect;
        rr-conflict disconnect;
        }

        syncer {
        rate 100M;
        verify-alg sha1;
        al-extents 257;
        }

        on node1 {
        device  /dev/drbd0;
        disk    /dev/vdc1;
        address 10.10.2.251:7788;
        meta-disk /dev/vdb1[0];
        }

        on node2 {
        device  /dev/drbd0;
        disk    /dev/vdc1;
        address 10.10.2.252:7788;
        meta-disk /dev/vdb1[0];
        }
}

resource aoe.target.0 {
        protocol C;
 
        handlers {
        pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
        pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
        local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
        outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";      
        }

        startup {
        degr-wfc-timeout 120;
        }

        disk {
        on-io-error detach;
        }

        net {
        cram-hmac-alg sha1;
        shared-secret "password";
        after-sb-0pri disconnect;
        after-sb-1pri disconnect;
        after-sb-2pri disconnect;
        rr-conflict disconnect;
        }

        syncer {
        rate 100M;
        verify-alg sha1;
        al-extents 257;
        }

        on node1 {
        device  /dev/drbd1;
        disk    /dev/vdd1;
        address 10.10.2.251:7789;
        meta-disk /dev/vdb1[1];
        }

        on node2 {
        device  /dev/drbd1;
        disk    /dev/vdd1;
        address 10.10.2.252:7789;
        meta-disk /dev/vdb1[1];
        }
}

Duplicate the DRBD configuration to the other server.

scp /etc/drbd.conf root@10.10.1.252:/etc/

Initialize the meta-data disk on both servers.

[node1]drbdadm create-md aoe.config
[node1]drbdadm create-md aoe.target.0
[node1]drbdadm create-md aoe.config
[node2]drbdadm create-md aoe.target.0

We could have initialized the meta-data disk for both resources with:

[node1]drbdadm create-md all
[node2]drbdadm create-md all

If a reboot was not performed post-installation of drbd, the module for DRBD will not be loaded.

Start the drbd service (which will load the module).

[node1]/etc/init.d/drbd start
[node2]/etc/init.d/drbd start

Decide which server will act as a primary for the DRBD device that will contain the AoE configuration files and initiate the first full sync between the two servers.

We will execute the following on node1:

drbdadm -- --overwrite-data-of-peer primary aoe.config

Review the current status of DRBD.

cat /proc/drbd 
Example output:
IT-hash: 9ba8b93e24d842f0dd3fb1f9b90e8348ddb95829 build by ivoks@ubuntu, 2009-01-17 07:49:56
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---
    ns:761980 nr:0 dw:0 dr:769856 al:0 bm:46 lo:10 pe:228 ua:256 ap:0 ep:1 wo:b oos:293604
        [=============>......] sync'ed: 72.3% (293604/1048292)K
        finish: 0:00:13 speed: 21,984 (19,860) K/sec
 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r---
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:10485692

I prefer to wait for the initial sync to complete before proceeding, however, waiting is not a requirement.

Once completed, format /dev/drbd0 and mount it.

[node1]mkfs.jfs /dev/drbd0
[node1]mkdir -p /srv/data
[node1[mount /dev/drbd0 /srv/data

To ensure replication is working correctly, create data on node1 and then switch node2 to be primary.

dd if=/dev/zero of=/srv/data/test.zeros bs=1M count=100

Switch to node2 and make it the Primary DRBD device:

On node1:
[node1]umount /srv/data
[node1]drbdadm secondary aoe.config
On node2:
[node2]mkdir -p /srv/data
[node2]drbdadm primary aoe.config
[node2[mount /dev/drbd0 /srv/data

You should now see the 100MB file in /srv/data on node2. We will now delete this file and make node1 the primary DRBD server to ensure replication is working in both directions.

Switch to node1 and make it the Primary DRBD device:

On node2:
[node2]rm /srv/data/test.zeros
[node2]umount /srv/data
[node2]drbdadm secondary aoe.config
On node1:
[node1]drbdadm primary aoe.config
[node1]mount /dev/drbd0 /srv/data

Performing an ls /srv/data on node1 will verify the file is now removed and synchronization successfully occured in both directions.

Decide which server will act as a primary for the DRBD device that will be the AoE target and initiate the first full sync between the two servers. We will execute the following on node1:

drbdadm -- --overwrite-data-of-peer primary aoe.target.0

We could have initiated the full sync for both resources with:

drbdadm -- --overwrite-data-of-peer primary all

Next we will install AoE target package. The plan is to have heartbeat control the service instead of init, thus we will prevent AoE from starting with the normal init routines. We will then place the AoE target configuration files on the DRBD device so both servers will have the information available when they are the primary DRBD device.

Install AoE target package on node1 and node2. [node1]sudo apt-get -y install vblade [node2]sudo apt-get -y install vblade

The init scripts for vblade as of 4/22/2009 need to be modified.

  1. The init script attempts to create a PID file in a directory that does not exist
  2. The init script attempts to place a null value in the PID file.

Working /etc/init.d/vblade:
#!/bin/bash

### BEGIN INIT INFO
# Provides:             vblade
# Required-Start:       $network $local_fs $remote_fs
# Required-Stop:
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    virtual AoE blade emulator
### END INIT INFO

set -e

# /etc/init.d/vblade start and stop the vblade daemon

test -x /usr/sbin/vblade || exit 0

. /lib/lsb/init-functions

test -d /var/run/vblade || mkdir -p /var/run/$prog

RETVAL=0
prog=vblade

test -d /var/run/vblade || mkdir -p /var/run/$prog

start_vblade() {
   ALLOWMACS=""
   [ -n "$5" ] && ALLOWMACS="-m $5"
   ID="$1-e$2.$3"
   PID_FILE=/var/run/$prog/${ID}.pid
   if [ -f $PID_FILE ]; then
        log_daemon_msg "The PID for $ID exists."
        continue
   fi
   $prog $ALLOWMACS $2 $3 $1 $4 >> /var/log/$prog.log 2>&1 &
   pid=$!
   echo $pid > $PID_FILE
   echo -n $"$4 (e$2.$3@$1) [pid $pid]"
   [ "$RETVAL" = 0 ] && log_end_msg 0 || log_end_msg 1
   echo
}

start() {
   log_daemon_msg "Starting vblade daemons" "vblade"
   sed /^#/d /etc/$prog.conf | sed /^$/d | while read line
   do
         start_vblade $line
   done
}

stop() {
   log_daemon_msg "Stopping vblade daemons" "vblade"
   log_progress_msg "vblade"
   if ! ls /var/run/vblade/*.pid 2> /dev/null; then
        log_daemon_msg "No vblade to stop"
   else
      for pidfile in `ls /var/run/$prog/*.pid`
         do
         kill -9 `cat $pidfile`
         rm -f $pidfile
       done
   fi
  echo
}


case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart|force-reload)
                stop
                start
                ;;
        reload)
                stop
                start
                ;;
        *)
        echo $"Usage: $0 {start|stop|restart|reload|restart|force-reload}"
        RETVAL=1
esac

exit 0

Temporarily stop vblade

/etc/init.d/vblade stop

Remove vblade from the init scripts.

update-rc.d -f vblade remove

Relocate vblade configuration to /srv/data/aoe:

[node1]mkdir -p /srv/data/aoe
[node1]mv /etc/vblade.conf /srv/data/aoe
[node1]ln -s /srv/data/aoe/vblade.conf /etc/vblade.conf
[node2]rm /etc/vblade.conf
[node2]ln -s /srv/data/aoe/vblade.conf /etc/vblade.conf

Define our AoE target.

Vblade defines AoE targets in /etc/vblade.conf

Example /etc/vblade.conf:

# network_device shelf slot file/disk/partition mac[,mac[,mac]]
bond0 0 1 /dev/drbd1

The above example:

  1. Will use bond0 as out network interface
  2. Defines our AoE target as shelf 0 slot 1
  3. Defines the device associated with the AoE target

Last but not least configure heartbeat to failover AoE in case a node fails.

On node1, define the cluster within /etc/heartbeat/ha.cf.

Example /etc/heartbeat/ha.cf:

logfacility     local0
keepalive 2
deadtime 30
warntime 10
initdead 120
bcast bond0
bcast bond1
node node1
node node2

On node1, define the authentication mechanism within /etc/heartbeat/authkeys the cluster will use.

Example /etc/heartbeat/authkeys:

auth3
3 md5 password

Change the permissions of /etc/heartbeat/authkeys.

chmod 600 /etc/heartbeat/authkeys

On node1, define the resources that will run on the cluster within /etc/heartbeat/haresources. We will define the master node for the resource, the Virtual IP address, the file systems used, and the service to start.

Example /etc/heartbeat/haresources:

node1 drbddisk::aoe.config Filesystem::/dev/drbd0::/srv/data::jfs
node1 drbddisk::aoe.target.0 vblade

Copy the cluster configuration files from node1 to node2.

[node1]scp /etc/heartbeat/ha.cf root@10.10.1.252:/etc/heartbeat/
[node1]scp /etc/heartbeat/authkeys root@10.10.1.252:/etc/heartbeat/
[node1]scp /etc/heartbeat/haresources root@10.10.1.252:/etc/heartbeat/

At this point you can either:

  1. Unmount /srv/data, make node1 secondary for drbd, and start heartbeat
  2. Reboot both servers

To test connectivity to our new AoE target, configure an additional system to be an initiator.

I will use Ubuntu 9.04 (Jaunty Jackalope) for this as well.

Install the AoE userland tools.

apt-get -y install aoetools

The default configuration does not automatically start AoE communication. Modify /etc/default/aoetools to indicate which interface AoE should communicate on.

sed -i 's/INTERFACES="none"/INTERFACES="eth0"/' /etc/default/aoetools

Start the aoetools init script. This will cause a discovery to start.

Review the status of AoE discovery.

aoe-stat
Example output:
      e0.1        10.737GB   eth0 up

Once the available LUNs are discovered, as expected, we have a new disk.

Example fdisk -l sample output:

Disk /dev/vda: 10.4 GB, 10485760000 bytes
255 heads, 63 sectors/track, 1274 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000478c5

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           1        1214     9751423+  83  Linux
/dev/vda2            1215        1274      481950    5  Extended
/dev/vda5            1215        1274      481918+  82  Linux swap / Solaris

Disk /dev/etherd/e0.1: 10.7 GB, 10737345024 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/etherd/e0.1 doesn't contain a valid partition table

Create a partition and file-system on our new AoE device.

fdisk /dev/etherd/e0.1
Command (m for help):                                                   <--- n
Command action
   e   extended
   p   primary partition (1-4)                                          <--- p
Partition number (1-4):                                                 < ---1 
First cylinder (1-10239, default 1):                                    <--- enter
Last cylinder, +cylinders or +size{K,M,G} (1-10239, default 10239):     <--- enter
Command (m for help):                                                   <--- w
mkfs.jfs -q /dev/etherd/e01.p1

Create a mount point for the new file-system.

mkdir -p /mnt/aoe

We can either use /etc/fstab to mount our file-system at boot time, or we can add it to /etc/default/aoetools. To keep all the disks in one location, we will place an entry in /etc/fstab. Modern distros prefer to use the disk's UUID for mounting in fstab, referring to the device "old school" still works as well.

The aoetools script is run after fstab is parsed and file systems are loaded. We can either move the init scripts to support AoE discovery before /etc/fstab is parsed, or we can use the "old school" enty in /etc/fstab. The aoetools init script will parse /etc/fstab and mount file-systems with entries for /dev/etherd.

Add our new AoE disk to /etc/fstab.

printf "/dev/etherd/e01.p1\t/mnt/aoe\tjfs\tnoatime\t0\t0\n" >> /etc/fstab

Mount our new AoE block device.

mount /mnt/aoe

Create data on the initiator node and test failover of the target. I prefer using a movie or a sound file as this will help show latency. Once you have the test data availble, play the movie or the mp3, and instruct node1 it is no longer a member of the cluster. This can be done by simply shutting down heartbeat.

[node1]/etc/init.d/heartbeat stop

Once you have tested the latency of the data transfer when node1 fails, start heart beat on node1, this will in-turn move the resources back to node1

[node1]/etc/init.d/heartbeat start

An alternative test would be to failover the nodes while writing data.

HighlyAvailableAoETarget (last edited 2009-04-28 03:06:52 by shelbywill)