Ubuntu software raid5 and raid6 with mdadm

Oh the joys of mdadm

So I have time and time again either seen or heard about someone losing all their data.  When asked, did you have a backup of your stuff.  The answer is then a depressing no.  Well I have way to many important photos, important documents, music, etc… that I am certainly not willing to just lose because of a hard drive failure.  Roughly  two years ago I decided I couldn’t take the stress of the possibility of me losing my wives pictures or my pictures.  Pictures are the number one reason I wanted a secure backup.  I looked into hardware raid and software raid.  If I were to go hardware RAID the cards that I would be choosing from are in the $400 range.  I simple could not afford that on top of the price of hard drives.  So I researched a lot into software raid within ubuntu using mdadm.   The cost would only be for the hard drives and also my time of learning how to get it up and running.   I thought of ways to sell this idea to my wife because it was going to cost around $300 to get three 1TB hard drives.  Doesn’t seem like much but at the time we were broke college students and I didn’t really see her saying ok to that price.   Well I told her the idea of how I wanted to have a raid setup so we wouldn’t just lose everything from a stupid hard drive failure.  Of course I married a frikin sweet chica because she was all for it.  I was super excited because I was going to be buying not one, not two, but three 1TB hard drives.  At that time the was a lot of space, even though I knew I would be losing one drive to parity bits.  The price to pay for not being stressed out of losing data.

The sweet part about software raid with mdadm is that it is transferable.  I have changed out my motherboard and boom in one command line I have my raid up with all new components.  With mdadm I have the ability to expand existing RAID-5 arrays or even convert a RAID-5 to a RAID-6.

So to setup my RAID-5 system I bought 3 samsung 1TB hard drives and installed them.

I then needed to prepare the disks.

As root, issue an “fdisk -l” to show a list of available disks.

I had three, 1TB hard drives…  sdb1, sdc1,sdd1,  I wanted to use these for my RAID-5.  I had a drive sda1 as my main operating system drive.  The disks were unformatted so I then formatted them ext3 set the RAID flag in gParted.  gParted is “Gnome Partition Editor” and is extra piece of software to install to easily change partition information visually.  To install gParted simple type in:

sudo apt-get install gparted

In gParted you have to select each disk separately in the upper right hand corner.  With each drive I formatted it to ext3 and then right clicked on each new volume and selected the “Manage Flags” and checked the “RAID” flag.

Next had to create the md Device.

I installed mdadm “Mirrored Device Admin” by typing in:

sudo apt-get install mdadm

To create the RAID volume I typed:

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1

The “–verbose” is useful for debugging any issues because it displays more information on the screen as the RAID is being created.  The “–level=5″ decides that it create a RAID-5 volume.  The “–raid-devices=3″ tells it that there are three devices to the raid array.  Then the /dev/sdb1 /dev/sdc1 /dev/sdd1  are the three disks I wanted to use in the RAID.

On a side note, the man pages for mdadm is very useful just type in:

man mdadm

When I initiated the command for the raid to be created I was able to monitor the building of the array by typing:

sudo watch cat /proc/mdstat

Before I created a file system on the raid, I had to create my mdadm.conf file first.  The mdm daemon will need the information in this file on boot to re-assemble my array to be ready for mounting.  I created the file by typing:

sudo echo "DEVICE partitions" > /etc/mdadm/mdadm.conf

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

After that file was created, I then needed to create the filesystem and mount point.  To create the file system I formatted the new RAID to ext3 by typing:

sudo mke2fs -j /dev/mdo

This might take some time.  I then edited my fstab to make the RAID-5 volume automatically mount on boot.  I had my mount in /mnt/raid.  I added this line to my /etc/fstab file:

/dev/mdo /var/media auto defaults 0 3

I then mounted the array with is at /dev/md0.

So after all that and the time to let it re-sync, I had a RAID-5.

Later I changed all my drives over to 2TB drives.  The drives I went to are the Western Digital Green drives.  The model of the drive I got were the WD20EARS and they have the 4KB Physical Sectors.  I had to prepare the drives to make sure that each partition is aligned to begin on the 64th sector / 4KB boundary.  You can check the alignment of your drive with :

sudo fdisk -l -u

If any of the drives listed that you’re planning to use with a RAID-5/RAID-6 have a start lised at 63, you need to change it to 64.  Here is how to change it:

Run fdisk - "sudo fdisk -u /dev/sda"
Delete existing partitions - "d"
Create new partitions - "n"
Tell it primary partition - "p"
Tell it partition 1 = "1"
Tell it to align at sector 64 - "64"
Change the partition type (if you want the drive as part of your mdadm raid array) - "t"
Select Linux raid autodetect fs - "fd"
Write & Quit - "w"

Repeat that procedure with all your disks that you are going to be using in your raid.

I had three 2 TB drives and I just bought another.  I think I have an sickness.  Well I had plenty of space with the raid 5 and I am way to paranoid of losing data.  With the 4th drive I just bought I wanted to convert my RAID-5 into a RAID-6.  I had to update to my MDADM software in order for me to convert it with only one drive.  In order to convert I had to add the extra drive as a spare to the raid:

mdadm --add /dev/md0 /dev/sde1

Then I used the “–grow” command to use that 4th drive and convert the raid into a RAID-6.

mdadm --grow /dev/md0 --level=6 --raid-devices=4 --backup-file=/root/md0.backup

I can not stress how important it is to have the “–backup-file” in the command.

If –backup-file was used to grow the number of raid-devices in a RAID5, and the system crashed during the critical section, then the same –backup-file must be presented to –assemble to allow possibly corrupted data to be restored.

Pretty much that saved my butt.  I was in the middle of converting the raid and I went out that evening.  When I came back I found out the power went out.  I was sweating bullets!!!  I wasn’t sure if I should try the grow command or assemble.  I went with assemble.

sudo mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 --backup-file=/root/md0.backup

mdadm: restoring critical section
mdadm: /dev/md0 has been started with 3 drives (out of 4).

I was more than relieved when I saw that and mounted my degraded raid and saw all my data perfectly fine.  It was rebuilding after that and after a long wait I had a RAID-6 array.  It was a beautiful sight.  Now I can have up to any 2 of my drives fail and I still have my data.

I just bought  new drive again. (I know it is a sickness)  Well this time I just growing the space of the raid and keeping it a raid 6.  So I add the drive to the array as a spare:

mdadm --add /dev/md0 /dev/sdf1

Then I give the grow command.

mdadm --grow /dev/md0 --raid-devices=5

After that was done, which took about 2 full days, the filesystem then needed to be expanded to fill up the new space.  We check the filesystem then resize it.

fsck.ext3 /dev/md0

resize2fs /dev/md0

Then boom… More space.

Hopefully it wasn’t too confusing and I hope it was informational.

Disclaimer:  RAID is no perfect replacement for having good backups!  Has been said one needs at least have an offsite backup as well.

You may also like...

5 Responses

  1. Andrew says:

    Great post Grant. Really enjoyed reading about you backup solution. Thanks for sharing.

  2. Simon says:

    wow, that’s cool. i love how clever it is. how you used the mdadm, that’s awesome

  3. The Man says:

    I love this: “In gParted you have to select each disk separately in the upper right hand corner”. Next time I’m in a jam with gParted I’ll be sure to remember this nugget of knowledge!.

    Ok, so your raid disks are sdb,c,d, why didn’t you raid in your main os drive (sda)? Four disk RAID-5?? wowzers. Nah, three disk RAID-5 is still teh kool.

    Dude, watch cat is the bomb.

    Nevermind on the grief giving of the four disk raid-5. I read the entire post. When you converted over to 2TB drives, did you have to recreate the mdadm.conf??

    Next time, post pics!!

  4. David says:

    I appreciate the info regarding the backup file, especially since trusting all my data to RAID still makes me a little nervous…

    I’m trying to grow my software array (4×1.5TB – RAID5) into a 5 disk, RAID6. Your instructions mirror many others that I’ve found online, but I constantly get the following error, even after unmounting the filesystem…

    “mdadm: Cannot set device size/shape for /dev/md0: Device or resource busy”

    Is there a force command or something that I’m missing? MDADM needs the array to be running or it doesn’t perceive “/dev/md0″ to be anything at all, so I don’t know how to make the array any less busy than that… Any insight would be greatly appreciated!

    (Ubuntu 10.04 LTS w/MDADM 2.6.7.1)

    • gswarfield says:

      David,

      It has been a long time, but I’m pretty sure I had about the same problem. At that time I was using Ubuntu 10.04 LTS. But with the older version of MDADM, The developers didn’t have the code in it to convert a raid-5 to raid-6 with only one extra drive present. It could be done with 2 extra drives. I think it had something to do with the copying/overwriting of data if you only had one extra drive. But they allowed it in the newer version. So honestly what fixed my problem and headache was upgrading to the newer version of MDADM. I think at the time I think I upgraded to version 3.1.2. I had to download it and make the install file. But once that was installed. I’m pretty sure that I just used the same command, that was making me beat my head against the wall, and it worked fine.

      http://neil.brown.name/blog/20090817000931
      If you read a bit from here and some comments on that site, it explains a little about the grow and newer version (3.1 was kinda beta-ish at time of article). But good info there.

      Hopefully that helps you out. If you have any other questions, just ask.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>