RAID Setup

RAID Setup

This is not a comprehensive guide. These are just my notes.

Goals

Construct a RAID-5 array using software RAID utilities on Linux. The RAID should be fully encrypted with LVM2 on top to manage volumes.

Hardware

3 x 1TB Western Digital Digital Caviar Green Power SATA hard drives
SATA-supporting motherboard with ample available ports

Use fdisk (or cfdisk) to partition each drive. In my case, the drives were in the ports numbered 2-4 and had device names /dev/sd{b,c,d}. Set each partition to be the same size (I used one partition per drive) and set the types to Linux raid autodetect (type fd).

Create the array

optimus / # mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sd{b,c,d}1
mdadm: array /dev/md0 started.

optimus / # mdadm --detail md0
md0:
        Version : 00.90.03
  Creation Time : Tue Aug 26 15:39:48 2008
     Raid Level : raid5
     Array Size : 1953519872 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Aug 26 17:00:15 2008
          State : clean, degraded, recovering
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 0% complete

           UUID : 9058245a:1c485926:a177f5ae:4e2e1d42
         Events : 0.18

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       3       8       49        2      spare rebuilding   /dev/sdd1

The rebuild process will take quite a while (several hours).

Add the array to /dev/mdadm.conf

ARRAY /dev/md0 level=raid5 num-devices=3 UUID=9058245a:1c485926:a177f5ae:4e2e1d42

Set up encryption

Set up encryption:

cryptsetup -y --cipher aes-cbc-essiv:sha256 --key-size 256 luksFormat /dev/md0

Open to /dev/mapper/raid:

cryptsetup luksOpen /dev/md0 raid

Set up LVM

Initialize:

vgscan
vgchange -ay

Create the physical volume:

pvcreate /dev/mapper/raid

Create the volume group:

vgcreate -s 64M raid /dev/mapper/raid

Create the first logical volume, call it media and put it in the raid volume group:

optimus / # lvcreate -L500G -nmedia raid
  Logical volume "media" created

This logical volume is accessible through the device /dev/raid/media now.

Create filesystems

mkreiserfs /dev/raid/media

Quick tests

optimus ~ # hdparm -tT /dev/md0 /dev/mapper/raid /dev/raid/media

/dev/md0:
 Timing cached reads:   9092 MB in  2.00 seconds = 4551.54 MB/sec
 Timing buffered disk reads:  338 MB in  3.02 seconds = 112.08 MB/sec

/dev/mapper/raid:
 Timing cached reads:   8116 MB in  2.00 seconds = 4062.16 MB/sec
 Timing buffered disk reads:  190 MB in  3.03 seconds =  62.74 MB/sec

/dev/raid/media:
 Timing cached reads:   7760 MB in  2.00 seconds = 3883.65 MB/sec
 Timing buffered disk reads:  186 MB in  3.03 seconds =  61.47 MB/sec

optimus /mnt/media # time bonnie++ -u 1000 -s 8096
Using uid:1000, gid:1000.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
optimus.int.q 8096M   455  94 51598  13 26584   8  1193  99 63854  12 399.4  13
Latency               397ms    1798ms     155ms   20144us   47330us    2176ms
Version 1.93c       ------Sequential Create------ --------Random Create--------
optimus.int.quadpoi -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 15560  31 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency                70us      67us   11269us     868us      14us      98us

Fixing boot-time issues

For Gentoo, set the correct volume order on boot in /etc/conf.d/rc:

RC_VOLUME_ORDER="raid dm dm-crypt lvm evms"

You may have to follow the steps regarding the ${myservice} variable in these two bug reports if the problem hasn't been fixed (i.e., if you haven't switched to baselayout-2 yet):