MDADM - How can I add a disk and expand my array?

[-Sk3tChY-]

I tried following this tutorial, but had some troubles and noticed it was from 2008. Plus I want to actually know what's going on a little better.

A brief run-down on the setup:

• HP Microserver
• Ubuntu Server
• MDADM - RAID5 Array - /dev/md0
• Array consists of 4 x 2TB drives - sdb1, sdc1, sdd1, sde1

What I'd like to achieve: Add a 5th 2TB drive and expand the array to include this drive.

Hyrax helped me build this thing a long time ago and I've done very little to it ever since, so I've forggoten exactly how all the drives/array is partitioned.

From memory though I think we set it up in such a way that we separated /boot, /home the page file and then storage.

1. Is there anyway I can take a look at how these were setup?

I think maybe we created the array and the partitioned it?

Anyway, I've physically inserted the new drive into the server and powered everything on and if I do:

sudo fdisk -l

It appears to show the drive as sda and shows it has 1 partition sda1.

2. What prep work do I need to do to the drive before adding it to the array? Do I need to wipe it? Format it a particular way? In the aformentioned tutorial it uses FDISK to do a few things.

My next assumption is that I would issue:

mdadm --add /dev/md0 /dev/sda1

mdadm --grow /dev/md0 --raid-devices=5

And then use:

cat /proc/mdstat

To periodically check the build progress.

Is there something I am missing?

[-Sk3tChY-]

Thanks IAC.

I will issue some of these commands when I get home and post up the results!

Should be around 7:30pm. Appreciated the help!

[-Sk3tChY-]

Ok IAC, I did the commands you mentioned as here's how things look:

admin@server:~$ df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/mapper/Array-Root
                       19G  1.3G   17G   8% /
none                  431M  212K  431M   1% /dev
none                  436M     0  436M   0% /dev/shm
none                  436M  500K  436M   1% /var/run
none                  436M     0  436M   0% /var/lock
none                  436M     0  436M   0% /lib/init/rw
/dev/mapper/Array-Home
                      5.4T  4.1T  1.3T  78% /home

admin@server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda1[4](S) sdd1[2] sde1[3] sdc1[1] sdb1[0]
      5860535808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

unused devices: <none>

As I said earlier, I tried following the tutorial above and managed to add the disk to the array which is why it mentioned sda1 there. Not sure what (S) means though? Spare?

admin@server:~$ mount
/dev/mapper/Array-Root on / type ext4 (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/dev/mapper/Array-Home on /home type ext4 (rw,noatime)

admin@server:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/mapper/Array-Root /               ext4    noatime,errors=remount-ro 0       1
/dev/mapper/Array-Home /home           ext4    noatime         0       2
/dev/mapper/Array-Swap none            swap    sw              0       0

A lot of it is beyond me!

From memory though I think we created the array, then partitioned the array and installed the OS.

Anyway, as I said earlier I tried following the tutorial I linked above and got as far as putting in the drive and running the FDISK stuff it mentions and by the looks of it the drive is partitioned correctly?

The disk in question is /dev/sda:

admin@server:~$ sudo fdisk -l
[sudo] password for admin:

Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xb2ed4813

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      243202  1953512448   fd  Linux RAID autodetect

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00088060

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      243201  1953512001   fd  Linux RAID autodetect

Disk /dev/sde: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000df4dd

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1   *           1      243201  1953512001   fd  Linux RAID autodetect

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000be37c

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1      243201  1953512001   fd  Linux RAID autodetect

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a4aaf

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1      243201  1953512001   fd  Linux RAID autodetect

Disk /dev/md0: 6001.2 GB, 6001188667392 bytes
2 heads, 4 sectors/track, 1465133952 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 196608 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

However the End and Blocks sections differ? Perhaps I need to format properly in Linux or something? I've forgotten what Start, End and Block signifies and how it works unfortunately.

[Sp0tteh]

Hi -Sk3tChY-,

Looks like you have added the drive to the array but haven't ran the "grow" command yet?

It's been a while since I have played with mdadm and as you have noticed the tutorial is a little out of date now and I should probably update it.

Read though the comments as well, as a few people have posted extra use full information that might help you.

Yes (S) means a spare device

[HyRax1]

Basically:

1. Add the new disk to the array (will be added as a spare).
2. Tell the array to "grow" into the newly added drive. You will need to wait for it to be fully striped before you can proceed to the next step, which may take several hours. Monitor progress with the watch cat /proc/mdstat command.
3. If you're using LVM, now expand your volume to use the newly created space on the array.
4. Now unmount the filesystem(s) you're wanting to expand and expand it to use the newly created space on the array/volume. If you are expanding the root filesystem, you will need to reboot your PC on a Live CD to do this. If you're expanding /home or something, then drop back into a text-only console, or switch to singleuser mode and unmount there before expanding.
5. Don't forget to update your /etc/mdadm/mdadm.conf file to reflect the new total number of disks in your array.
6. Pat yourself on the back.

Other than that, that tutorial link you posted has everything you need.

[-Sk3tChY-]

I issued sudo mdadm --grow /dev/md0 --raid-devices=5

admin@server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda1[4] sdd1[2] sde1[3] sdc1[1] sdb1[0]
      5860535808 blocks super 0.91 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  reshape =  0.0% (1209984/1953511936) finish=5346.4min speed=6085K/sec

unused devices: <none>

So thus far, everything seems to look ok?

It looks like it's going to take a damn long time to add the drive to the array!

This actually happened when I first built the array too, from memory it took like 2 days to build!

[HyRax1]

If the filesystems on that array are unmounted, it will be quicker.

I'm currently replacing two failed 2TB drives on a RAID6 array and it's going to take 13 hours total to re-stripe them both with all the filesystems unmounted (booted a Live CD to do all the work so root could be unmounted).

[-Sk3tChY-]

Originally Posted by HyRax1

If the filesystems on that array are unmounted, it will be quicker.

I'm currently replacing two failed 2TB drives on a RAID6 array and it's going to take 13 hours total to re-stripe them both with all the filesystems unmounted (booted a Live CD to do all the work so root could be unmounted).

I don't want to touch anything for fear of breaking it!

Hypothetical here, if a drive failed right now during the expansion process, would I still be covered? I vaguely remember reading somewhere that if a drive does during a time like this you lose the lot.

Also, without getting ahead of myself, when this completes how do I go about checking/expanding LVM?

[HyRax1]

Originally Posted by -Sk3tChY-

Hypothetical here, if a drive failed right now during the expansion process, would I still be covered? I vaguely remember reading somewhere that if a drive does during a time like this you lose the lot.

If you lost the new drive being striped, then nothing. If you lost one of the existing members while the new drive was being striped, nothing but you will be degraded, because the new drive will not be part of the array until the striping has finished (you can see the mdadm detail showing no new space until the striping has finished). Once the new drive has finished striping and you lost any two of any member including the new one, yes - you're stuffed.

Originally Posted by -Sk3tChY-

Also, without getting ahead of myself, when this completes how do I go about checking/expanding LVM?

Gooooooogle pvresize and lvresize commands.

[-Sk3tChY-]

Originally Posted by HyRax1

Gooooooogle pvresize and lvresize commands.

Thanks big man, will do. I'm sure I'll report back here though with some questions soon after!

An update:

admin@server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda1[4] sdd1[2] sde1[3] sdc1[1] sdb1[0]
      5860535808 blocks super 0.91 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [====>................]  reshape = 21.0% (411832064/1953511936) finish=4854.6min speed=5292K/sec

unused devices: <none>

It's not even 1/4 completed!

Does it normally take this long adding a new disk to an existing array:

1. With mdadm?
2. With other software RAID?
3. With hardware RAID?

It just seems rather slow to me, at this rate it would take like 4 full days to add the disk to the array!

Could it be the fact my server isn't very powerful? (It's a HP Microserver with 1GB RAM)

[CAPT-Irrelevant]

I built a 3x2TB RAID5 array with mdadm on Fedora. It took ~8 hours to initialise, but then, I'm using the onboard sata controller. See my 'PHYRAXUS' spec in my sig.

[-Sk3tChY-]

Originally Posted by CAPT-Irrelevant

I built a 3x2TB RAID5 array with mdadm on Fedora. It took ~8 hours to initialise, but then, I'm using the onboard sata controller. See my 'PHYRAXUS' spec in my sig.

What do you mean by using on the onboard? That it's slower?

I'm using the onboard controller too, it's a HP micro server.

[CAPT-Irrelevant]

Originally Posted by -Sk3tChY-

What do you mean by using on the onboard? That it's slower?

I'm using the onboard controller too, it's a HP micro server.

Speed of controller may be subjective but I brought it up because to not use a dedicated expansion RAID card may imply reduction in speed performance.

I think the HP microservers were designed to have good read/write speeds.

[elvis]

Originally Posted by -Sk3tChY-

Does it normally take this long adding a new disk to an existing array:

1. With mdadm?
2. With other software RAID?
3. With hardware RAID?

Yes, changing a RAID configuration is slow. You're not just initialising new disk here. You're rebalancing all of the existing data, and doing so in a safe way (the process is recoverable/resumable if it crashes).

This is perfectly normal.

[-Sk3tChY-]

Originally Posted by CAPT-Irrelevant

Speed of controller may be subjective but I brought it up because to not use a dedicated expansion RAID card may imply reduction in speed performance.

Thought so, thanks for the clarification.

Originally Posted by elvis

Yes, changing a RAID configuration is slow. You're not just initialising new disk here. You're rebalancing all of the existing data, and doing so in a safe way (the process is recoverable/resumable if it crashes).

This is perfectly normal.

Makes sense, thanks for that.

Well an update for those of you who might be interested:

admin@server:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda1[4] sdd1[2] sde1[3] sdc1[1] sdb1[0]
      5860535808 blocks super 0.91 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [========>............]  reshape = 40.4% (790951628/1953511936) finish=4100.8min speed=4724K/sec

unused devices: <none>

Almost halfway!!