System Migration help

malderon
Posts: 5
Joined: Thu Dec 16, 2010 12:12 pm

System Migration help

Postby malderon » Mon Apr 25, 2011 4:37 am

Hi all. First off I want to say thanks so much for Amahi it has been fantastic.

I wasn't really sure what category this fitted into, I suppose it mostly concerns Greyhole but it was slightly more general than that so here it is! Feel free to move it.

My current system is has the following drives;

500GB Partitioned as 50GB root install with 450GB given to the storage pool
500GB all as storage pool
500GB all as storage pool
2TB all as storage pool.

The storage is starting to fill up and I'm left with a few problems/options.

My current motherboard only has 4 SATA ports so adding further drives is not straightforward. As I understand it, I could replace the 2 500GB storage drives with 2TB drives. As I understand it I would accomplish this by first running

Code: Select all

greyhole --going=
. However I haven't found many details on this command. Surely it requires that I have sufficient space / redundancy on the remaining drives to reallocate files currently stored on the drive to be removed. Which could be a problem. (although as far as redundancy go I could temporarily reduce the number of copies for a share potentially?)

I've been thinking about building a completely new system dedicated to amahi, rather than the cobbled together job I currently have. I notice that now the current release of amahi uses Fedora 14, ie more recent than the version I'm using.

I found this;
Reconnect the Greyhole storage pool after re-install of Fedora / Amahi

Disconnect all data drives before you reinstall. This will prevent you from destroying them by mistake during the Fedora/Amahi install!
Install Amahi per the regular instructions.
Connect the data drives, and use hda-diskmount to remount all of them.

Note that if you can, it would be faster if you can re-mount the drives in the same paths they used to have.

Example: If you have one 1TB and one 2TB, and the 1TB was mounted as /var/hda/files/drives/drive1, and the 2TB was mounted as /var/hda/files/drives/drive2, it would be better to re-mount that like that on your new HDA. If you mount them otherwise, it will still work, but Greyhole will work a while to re-build the links in your share before they can be usable.

Select the drives in the Amahi Dashboard: Setup > Shares > Storage Pool
Re-create your shares in the Amahi Dashboard: Setup > Shares.

Make sure to select 'Use Pool', and select the correct number of extra copies. If you don't remember all the shares you had, just check inside the 'gh' folder found at the root of your drives to get a reminder!

Example:

ls -1 /var/hda/files/drives/*/gh/ | grep -v : | sort -u

Launch a fsck to have Greyhole rebuild your shares.

greyhole --fsck

Until this completes, some of your files will be missing from your shares.
But it is talking about a reinstall on the same system. If I were to build a new system, install Fedora 14 / Amahi, connect the drives that currently make up my storage pool, and follow the above instructions - would this work? It seems so but I get understandably nervous when doing anything drastic to what basically constitutes all my storage. I try to touch the Amahi system as little as possible and it just sits there on 24/7 happily storing stuff.

Or if anyone has any other suggestions for what could be good things for me to do going forward then please, I'm all ears.

User avatar
sgtfoo
Posts: 419
Joined: Sun Jul 18, 2010 8:27 pm

Re: System Migration help

Postby sgtfoo » Mon Apr 25, 2011 10:14 am

I would suggest starting out by mounting the shares locally to the server so you can access them there, rather than across your network and tying that up. Buy a fresh drive (ie 2tb) and hook it up to the server via an external enclosure of any kind and offload any non-redundant shares to the 2TB drive. Everything is safe on there until you're done.

find out which drives contain your redundant copies and pull them out too. Any remaining storage drives can be formatted clean, which is handy. Pull out the rest of your storage drives. Then begin a fresh install of F14/Amahi6. Once finished, add back your redundant pair of drives and then your 2tb drive. Add them all to the greyhole pool and make it do a greyhole -f -o -l. At that point, add back shared drives or other shares at your discretion. To make the most of the few SATA ports, I would suggest grabbing 2tb drives for replacing each 500gb drive. Either that or buy a quick PCI/PCI-X SATA controller addon cart to give you more ports.
Once Amahi6/F14 are all setup and your other drives with shares on them are back, offload the 2TB you used as a lifeboat and then format it and add it to the pool and you're 2TB boosted!

Greyhole will ideally find and account for all the file in your previous server setup and everything will migrate well
See my process below..
I more or less went thru this already just this past week.

My system was running F12 with Amahi5:
Motherboard with 6 SATA connectors
1x 320gb hdd for OS
1x 1.5TB hdd
2x 1TB hdd

all of the non-OS drives had greyhole storage at some point, but I had removed the 1.5tb drive from the pool about a week prior to upgrading to Amahi6 / F14. I did it a week in advance so I could live-boot another distro so I could look at ALL the files on all 3 drives (including hidden folders/files) to see what was where in case of loss of data.
I was sure to not delete anything though because Greyhole has it's own complex way of keeping files between drives.
I had a single folder in my pool that existed in redundancy. It was spanned across the two 1tb drives.

For my system upgrade I did a fresh install to a 74GB WD Raptor I had found and added my drives each one at a time AFTER the new F14/amahi installation.

So now it looks like this...
1x 74GB hdd (this drive for for OS only.. /boot, swap, /)
1x 1.5tb hdd for storage, IN POOL
2x 1tb hdd for storage, IN POOL

After adding each drive, one at a time, I finished up by initiating greyhole with the checkmark next to each drive in the dashboard: all drives have the "use in pool" checked. All my shares are also checked for "uses pool" and so greyhole manages ALL my shares, redundant or not.

After everything was added I did 3 runs of greyhole in the terminal:

Code: Select all

greyhole -f --find-orphaned-files greyhole -f -a greyhole -f -l
Those commands do the FSCK each and every time, while finding any orphaned files, clearing the attic, and balancing shares across the drives. None of my files were gone at the end of this, and I had even found some lost files from back when I removed a drive from the pool.

I learned that it's generally to our benefit to use greyhole entirely on all drives, rather than somewhat confusingly on only a couple of many.
It also helps to mount your drives in a logical manner, although apparently it doesn't matter if it's identical to the setup before in terms of...
/var/hda/files
/var/hda/files/drives/drive1
/var/hda/files/drives/drive2
(this is what I have right now, with the 1.5tb drive mounted as the /var/hda/files)
It definitely helps to re-create all the identical share folders and folder locations once all your drives are back in. Then any subsequent drive additions are

My next step is to add another 2TB drive to the pool for extra space, and then I should be good for a year or so. After adding this 2TB drive I'll do another

Code: Select all

greyhole -f -l
to fsck and balance files across the drives and things should be pretty sane after that.

I hope this gives some insight at least, as I'm no expert, but I lost NOTHING in the process.
SgtFoo
HDA: VM inside oVirt FX-8300 95w (2 cores for HDA), 32GB RAM (2GB for HDA)
My PC: FX-8300, 16GB RAM, 3x 1TB HDDs, Radeon HD6970 2GB video; Win10 Pro x64
Other: PC, Asus 1215n (LXLE), Debian openZFS server (3x(2x2tb) mirrors)
Modem&Network: Thomson DCM475; Asus RT-AC66U; HP 1800-24G switch

Who is online

Users browsing this forum: Google [Bot] and 2 guests