Disk mirroring

[bmwmay]

Hopefully this is the right forum for this question. I have been running an Amahi server since spring '09 and today (Merry F'ing Christmas;) my 500 gb external HD connected to the server died. Probably shit quality (Fantom) more than anything but I want to avoid this from happening again. I just bought two identical 1 tb seagate externals and want to mirror them so I can avoid having to deal with this going forward. I haven't found much on instructions for this so I'm hoping a linux wiz can help me out with this. I'm sure I could figure it out but want some advice before proceeding any further.

Thanks
Mike

[lou1z]

mirroring.....
when you mention this, most people think raid 1. the fact you have external drives (usb or esata?) makes using raid a little harder. that is, if you mean raid.
personally, i think raid for amahi is a little over the top. not just for amahi but for a lot of things unless they are mission critical. an exception would probably be virtualisation where i would certainly have raid going there. but for a standalone amahi, i could probably withstand some downtime.
i look at installing amahi onto its own partition (possibly a small sata flash drive http://linitx.com/viewcategory.php?cati ... 0,129,1007) and then have all data stored elsewhere. greyhole would be my choice here although i would also have an image of the os by itself should the worst happen.
as for backups, that would be up to your self on how much you value your data vs how much you spend on it.

just remember, with true mirroring, a corrupted file can be mirrored so there's no replacement for true backups as you've probably realised.

[moredruid]

If you want to use mirroring there are 2 mechanisms you can use in Linux:
mdraid or LVm mirroring.
mdraid is a bit more complicated IMO and depending on the situation it can be faster.
furthermore LVM mirroring can be "tacked on" later if need be. I find it to be more flexible.

I'll post a quick tutorial in my next post taken from my website here: http://bonabo.org/moredruid/tiki-index. ... +mirroring

Since I'm completely reconfiguring my VPS soon I'm not sure how long this will be up.

[moredruid]

This topic describes LVM (Logical Volume Management) mirroring in Linux in easy to follow steps.
difficulty: medium/hard

The starting point of this information is that you already have a working install of a Linux distribution.
You already have LVM set up on 1 disk, and you're using the full disk capacity in a single logical volume.

Now let's say you want a little more redundancy, and you don't have the option of creating the logical volume from scratch with Linux software RAID. You buy a new disk with at least the same capacity as your old disk.

In this tutorial I will use the volume group name "vgdata" and logical volume name "lvdata"

Adding a new disk to an existing Volume Group
warning: I will explain adding a __3rd__ hard drive (/dev/sdc) and we will have the following settings:
* /dev/sda is partitioned for the linux installation
* /dev/sdb is partitioned and used for LVM for a single datavolume (/dev/vgdata/lvdata)
* /dev/sdc is to be added to volume group vgdata to make sure there is enough space for mirroring

After physically adding the disk we need to add it to the volume group.
login as root
list your current partitions with:

Code: Select all

fdisk -l

now let's create a partition on the 3rd hard drive:

Code: Select all

fdisk /dev/sdc => you will enter a menu n => create new partition p => make partition primary 1 => this is the partition number, tap <enter> twice to create a partition that fills your complete hard drive t => this is the filesystem type, choose or type 8e for linux extended (LV) w => write changes to disk

now we need to create a physical volume:

Code: Select all

pvcreate /dev/sdc1

next we need to add this physical volume to the volume group vgdata:

Code: Select all

vgextend vgdata /dev/sdc1

with the vgdisplay command you can see the details of your volumegroup:

Code: Select all

vgdisplay -v vgdata

You should see that you have the added disk space in the volume group.
Now to get to the mirroring:

Code: Select all

lvconvert -m1 --mirrorlog core /dev/vgdata/lvdata

=> this will create a mirror with 1 spare disk (-m1) with the mirrorlog kept in memory (--mirrorlog core) on the selected volume group. The process will (depending on your distro) display it's progress on your terminal.

If you have a large volume this will take a while (my 1.6TB on 2 primary SATA2 disks with 2 SATA2 spares took around 8 hours to complete).

If you still want to monitor the progress (or state) of the LVM mirror you can issue the following command:

Code: Select all

lvs -ao +devices

which should give you a similar output to this:

Code: Select all

LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices lvdata vgdata mwi-ao 1.0T 100.00 lvdata_mimage_0(0),lvdata_mimage_1(0) [lvdata_mimage_0] vgdata iwi-ao 1.0T /dev/sdb1(0) [lvdata_mimage_1] vgdata iwi-ao 1.0T /dev/sdc1(0)

The Copy% column will display the % of data that is mirrored, in the above case this is 100%.

Now if one of the disks fails LVM will silently convert the mirrored volume to a so called linear (normal) volume so your data will still be accessible.

[bigfoot65]

This looks like good stuff for the wiki. Would you mind adding it? We could also add a link to the wiki main page so others will see it easily.

LVM has always been one of those areas folks ask about. If it is already in the wiki and I just missed it, please let me know.

[moredruid]

Actually it's taken from my own wiki... Added it to http://wiki.amahi.org/index.php/RAID_1 (at the bottom, though the RAID stuff might warrant a complete rewrite (it's factually correct but the page is not set up logically).

[bigfoot65]

Thanks, that is awesome. I am not very familiar with RAID. If you want to rearrange the page, that would be greatly appreciated.

I am trying to get the wiki updated so users can better help themselves. The response time is much quicker too

[lou1z]

looking at the op's request, my guess is that the drives are usb. so i went on a hunt to see if usb could be raided as i've never heard or seen it done.
http://eclug.homelinux.org/content/view/158/59/
so it can and i've learnt something new although i'm not sure whether its common practice. i may even have a play with the concept with 2 usb sticks to see what gives.

[moredruid]

Of course that can be done, Linux doesn't care about the connection, it just wants a block device of similar (or larger) size.

Basically you could have a USB disk and a iSCSI disk and mirror one to the other. Or the storage on your phone (e.g. android phones can be used as a usb stick) which you mirror to a tape backup device (I definately wouldn't recommend that but it _is_ possible).

[lou1z]

Of course that can be done, Linux doesn't care about the connection, it just wants a block device of similar (or larger) size.

Basically you could have a USB disk and a iSCSI disk and mirror one to the other. Or the storage on your phone (e.g. android phones can be used as a usb stick) which you mirror to a tape backup device (I definately wouldn't recommend that but it _is_ possible).

i always wondered as i knew the above but has it ever been done and what were the results? has anybody played with this?
i wonder what sort of load the system would come under with usb being utilised like this?