Remote Server Upgrade to Software RAID
Introduction
|
This is a guide to installing software RAID 1 (mirroring) on a remote Ubuntu server with no access to external media. The purpose is to provide a step-by-step method for mirroring a running system onto a new second physical disk that has been installed.
I performed this installation over SSH on a remote server running Ubuntu 9.04 (Jaunty Jackalope) with legacy grub (version 0.97). The server had 2 250GB hard disks installed, although only one was partitioned and mounted with the running operating system. The server also had a lights-out board installed with remote KVM access. This is an environment typically offered by UK hosting company Fasthosts.co.uk.
This guide comes with no warranty of success. As file partitions are being manipulated, there is a high risk of data loss. You must back up any and all data that you cannot afford to lose.
Overview
The plan is as follows:
- Partition the new disk
- Create degraded RAID partitions involving the new disk
- Copy data from the original disk into the RAID partitions
- Configure the system to boot from the new disk on the mounted RAID partitions
- Reboot and add the original disk partitions to the RAID partitions
The following assumptions are made:
- The new disk is at least the same size as the original disk, and is installed and visible in the BIOS
- The hardware and BIOS are capable of booting from the new disk
- You are able to reboot the server and enter the BIOS
Mirror Original Disk onto the New Disk
Preparation
- Ensure any hardware RAID (or FakeRAID) is disabled in the BIOS. Details on how to do this are specific to each BIOS, though the setting is usually found in advanced peripheral settings.
- Update the package lists and install mdadm
sudo apt-get update sudo apt-get install mdadm
Partition Organisation
Throughout this guide, the original disk (hd0) is /dev/sda, and the new disk (hd1) is /dev/sdb.
For the purpose of this guide, /dev/sda has the following partitions and mount points:
- /dev/sda1 swap partition
- /dev/sda2 Extended partition
- /dev/sda5 mounted at /boot
- /dev/sda6 mounted at /
- /dev/sda7 mounted at /var
- /dev/sda8 mounted at /home
Configure the New Disk
Copy partition configuration of /dev/sda to /dev/sdb
sudo sfdisk -d /dev/sda | sfdisk --force /dev/sdb
Change the system for each partition on /dev/sdb to FD (Linux RAID autodetect)
sudo for partition in 1 5 6 7 8; do sfdisk --change-id /dev/sdb $partition fd; done
Remove any RAID meta data from the new disk. This would be the case if you had previously attempted to install RAID on this disk.
sudo for partition in 1 5 6 7 8; do mdadm --zero-superblock /dev/sdb$partition; done
If there was no meta data on the new disk, then you will see an error similar to the following which you do not need to worry about:
mdadm: Unrecognised md component device - /dev/sdb1
Note, there will be no output if meta data was actually zero'd.
Create degraded RAID arrays, one per partition.
sudo for partition in 1 5 6 7 8; do mdadm --create /dev/md$partition --level=1 --raid-devices=2 missing /dev/sdb$partition; done
Create file systems on the RAID partitions.
sudo mkswap /dev/md1 sudo for partition in 5 6 7 8; do mkfs.ext3 /dev/md$partition; done
Copy Data to New Disk
Update the mdadm configuration with our new RAID devices
sudo mdadm -Es >>/etc/mdadm/mdadm.conf
Ensure raid1 and md are loaded by the kernel at boot time, and load raid1 now to save a reboot.
sudo echo raid1 >> /etc/modules sudo echo md >> /etc/modules sudo modprobe raid1
Beginner Tip: It will be useful to have all your UUID's in a local text file for easy copy/paste. Open a text editor such as gedit locally and list all the UUID's with the following command:
sudo blkid
Then highlight all the output, and paste your highlighted text to your text editor by clicking your middle mouse button in that window. In the following steps, you can then easily highlight a UUID and paste it into your PuTTY or SSH client window with your middle mouse button.
Add an option to the grub boot menu to enable us to boot from the RAID partitions.
sudo nano /boot/grub/menu.lst
Scroll to the bottom, and create a copy of the first stanza that you find after the line
## ## End Default Options ## title Ubuntu 9.04 kernel 2.6.28-19-generic uuid 6b27b795-ff0e-4d1e-a631-851d6bdc63d5 kernel /vmlinuz-2.6.28-19-generic root=UUID=b6cb3dae-a029-4457-8807-dd45070622e9 ro quiet splash initrd /initrd.img-2.6.28-19-generic quiet title Ubuntu 9.04 kernel 2.6.28-19-generic uuid 6b27b795-ff0e-4d1e-a631-851d6bdc63d5 kernel /vmlinuz-2.6.28-19-generic root=UUID=b6cb3dae-a029-4457-8807-dd45070622e9 ro quiet splash initrd /initrd.img-2.6.28-19-generic quiet
The reason we make a copy is so that if there is an issue with booting onto the RAID partitions, you can recover to boot from the original disk. In the copied stanza block, make the following changes and save:
- Prepend the title with RAID
- Change uuid to that of /dev/md5, the partition that will be mounted at /boot from the list above (it should currently be set to the uuid of /dev/sda5)
- Change the kernel root uuid to that of /dev/mdb6, the partition that will be mounted at / (it should currently be set to the uuid of /dev/sda6)
- Optional: Comment hiddenmenu so that the menu is displayed at boot without need to hit Escape
- Optional: increase the timeout from 3 to 10, to give you time to make a selection
## ## End Default Options ## title Ubuntu 9.04 kernel 2.6.28-19-generic uuid 6b27b795-ff0e-4d1e-a631-851d6bdc63d5 kernel /vmlinuz-2.6.28-19-generic root=UUID=b6cb3dae-a029-4457-8807-dd45070622e9 ro quiet splash initrd /initrd.img-2.6.28-19-generic quiet title RAID Ubuntu 9.04 kernel 2.6.28-19-generic uuid 82987988:99a015c2:92160f38:875c2921 kernel /vmlinuz-2.6.28-19-generic root=UUID=8f6ac739:0afbba20:92160f38:875c2921 ro quiet splash initrd /initrd.img-2.6.28-19-generic quiet
Now we need to ensure that our RAID partitions are mounted instead of /dev/sdax. First make a copy of your current partition mounts, again so that if we fail to boot onto the RAID partition, we have a chance to recover.
sudo cp /etc/fstab /tmp
Then update the UUIDs for each mount point to the corresponding RAID partition UUIDs in /etc/fstab
sudo nano /etc/fstab
Mount the raid partitions and copy all data from /dev/sda.
sudo mkdir /raid
sudo mount /dev/md6 /raid
sudo mkdir /raid/{boot,var,home}
sudo mount /dev/md5 /raid/boot
sudo mount /dev/md7 /raid/var
sudo mount /dev/md8 /raid/home
sudo rsync -auHxv / /raid/
sudo rsync -auHxv /boot/ /raid/boot/
sudo rsync -auHxv /var/ /raid/var/
sudo rsync -auHxv /home/ /raid/home/Create a new initramfs on the RAID partition so that grub can find it at boot time.
cd /raid sudo chroot /raid sudo mount /proc sudo update-initramfs -u exit
Install Grub on New Disk and Prepare to Reboot
First of all, launch grub and check if the boot files have been copied to the new disk where grub can find them. If the data was copied successfully above, then you should see the output along the following lines:
sudo grub grub> find /grub/stage1 (hd0,4) (hd1,4)
In the output above, hd0 refers to the original disk, and 4 refers to the partition where the files exist. Partitions are zero-based in grub, so 4 implies /dev/sda5.
Now map (hd0) in grub to our new disk and install the bootloader:
grub> device (hd0) /dev/sdb grub> root (hd0,4) grub> setup (hd0) grub> quit
Finally, before rebooting, change the system for each partition on /dev/sda to FD (Linux RAID autodetect):
sudo for partition in 1 5 6 7 8; do sfdisk --change-id /dev/sda $partition fd; done
Reboot into RAID
Ensure you have your remote KVM access available and reboot the server. You need to enter the BIOS and prioritise the second disk over the first disk in the boot sequence. When the grub boot menu appears, select the RAID entry. If all is well, Ubuntu will load.
Login, and add the /dev/sdax partitions to their corresponding RAID devices:
sudo for partition in 1 5 6 7 8; do mdadm --add /dev/md$partition /dev/sda$partition; done
You can monitor progress of the synchronisation with the following command (hit Ctrl-C to exit):
sudo watch cat /proc/mdstat
If Things Go Wrong
If the server failed to boot onto the RAID partitions, you will end up at an initramfs prompt. To recover, you need to mount your /boot partition on the original disk /dev/sda, copy /tmp/fstab back to /etc/fstab, reboot, and select the non RAID option from the menu (or prioritise the first disk for boot in the BIOS).
Once you've logged in, you will need to remove all traces of the RAID from your new disk so that you can start again. For some reason, I found my RAID partitions had been renamed, so you should check what your RAID partitions are named as using any of the following commands:
sudo fdisk -l sudo cat /proc/mdstat sudo parted -l
Now stop all the RAID partitions. Modify these commands according to how your RAID partitions are named. In my case, they were renamed to /dev/md_d1, /dev/md_d5, etc)
sudo for partition in 1 5 6 7 8; do mdadm --stop /dev/md_d$partition; done
Remove the RAID meta data from the disk
sudo for partition in 1 5 6 7 8; do mdadm --zero-superblock /dev/sdb$partition; done
I found that parted -l was showing unrecognised disk label against the RAID partition references. To remove them, delete them from /dev
sudo rm -rf /dev/md_d*
Finally, remove the RAID partition configuration from the mdadm configuration file. There will be one line per partition at the end of /etc/mdadm/mdadm.conf that all must be removed:
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=172e1028:a1b301c3:92160f38:875c2921 ARRAY /dev/md5 level=raid1 num-devices=2 UUID=82987988:99a015c2:92160f38:875c2921 ARRAY /dev/md6 level=raid1 num-devices=2 UUID=8f6ac739:0afbba20:92160f38:875c2921
When you are ready to try again, reboot and be sure to jump into the BIOS and prioritise disk 0 in the boot order.