Efficient way for migrating to new filesystem and greyhole?

[fumsli]

Hi there!

I'd like to migrate my windows based storage solution to an Amahi based one. And for the purposes of redundancy and ease of access (merged shares across disks/partitions/volumes) like to use greyhole.

I've got the following disks

Code: Select all

750GB NTFS (full) 1,5TB NTFS (full) 2TB NTFS (full) 3TB - (brand new)

and like to create a pool with the biggest three - the 750GB disk might be failing and will be removed from my setup (its data still needs to be imported though).

Do I have to buy another disk just for the migration?
If it is possible without the extra purchase: How do I do this efficiently, that is, with as little copy-overhead as possible?

As to the filesystem - I'm not sure whether to use ext3,4 or zfs. They all seem sound in theory - are there any advantages/drawbacks in combination with amahi/greyhole?

[bigfoot65]

See the guidance in the wiki.

http://wiki.amahi.org/index.php/Greyhole

[sgtfoo]

The cleanest way I've approached this with 3 HDAs already is by having everything go through locally mounted shares at the HDA, or through a client PC with the shares mounted.

Greyhole and Samba need to catch everything before planting it into the pool, using the Landing zone.

A friend of mine has more data (about 5TB) and his best bet was to load up the size of his LZ (1TB)(/var/hda/files) and then force a 'greyhole -f' and when it was done, repeat for another 1TB.

Took him 2 days.. er.. just the 3 hours he had after work each evening. But in the end, everything is done properly, and for the first transfer, it's best.

I'd also suggest to be rid of the bad drive immediately, and if it's not too old, check the manufacturer's website for any RMA options. I've RMA'd a WD and a Seagate each in the past 16 months without problems, and the best part is it only costs shipping.

[fumsli]

Let's see if I got it right in principle:

• put in the biggest empty disk
  partition it as one giant space
  format it minding 4kb sectors 'issue' for big hdds)
  move the landing zone to it
  set the share path using the webgui
  A: add it to the greyhole pool (using the gui?)
  put in an occupied disk
  copy up to $(free_size of 3tb disk) data from the full disk to the share path folders (=landing zone)
  force greyhole flush (greyhole /f)
  format the now empty drive
  while disks_left goto A

Now, assuming the above is correct:

• - do I have to worry about the growing pool-size in terms of long-time side effects?
  - am I correct to assume that it is best to add the drives in order of decreasing size?
  - what will happen if I set the number of copies for a shared folder to a number bigger than my current number of drives? and what will happen if the number is finally reached?
  - why do I need to flush the landing zone manually? the documentation states that this will be done in the background anyway.
  - could I speed this up by using a ram-drive? or would I run out of space without the manual flushes because the daemon does not move the files quickly enough? (that would be inconvenient since this cache partition would necessarily be rather small)
  - which filesystem should I use?

The failing drive will be gone as soon as I figure out the procedure to do it properly.

[sgtfoo]

I'm not sure about the method you're posting about, as it seems like it's mixing between the 2 methods available.
(note: Before you use the 3Tb drive, your BIOS needs to be UEFI or otherwise 3TB capable.)
The simplest method I see for what you have without buying an additional drive, assuming you have a UEFI BIOS would be to setup amahi with your empty 3tb drive with 2 partitions...
1x ?GB for LZ (LZ size is up to you) -- LZ partition -- /var/hda/files
1x remaining GB (remaining size of the drive) -- pooled partition -- /var/hda/files/drives/drive01
This would allow you to offload your smallest drives first and then once they've been offloaded, add them to the pool as well, allowing for the rest of your data to be put in the pool.
After the big transfer is done, you can use gparted from a USB-key and resize the LZ to something smaller if you like.

- do I have to worry about the growing pool-size in terms of long-time side effects

not many people have reported any issues aside from normal hard drive usage degradation.

- am I correct to assume that it is best to add the drives in order of decreasing size?

It doesn't really matter what size the drives are. You can mount them to accommodate your usage choice.

- what will happen if I set the number of copies for a shared folder to a number bigger than my current number of drives? and what will happen if the number is finally reached?

Not entirely sure about this but I imagine greyhole won't let you copy to more than the available drives. When you choose all the available drives, it does copy the files to all available drives.

- why do I need to flush the landing zone manually? the documentation states that this will be done in the background anyway.

Manual LZ -f (flush or file system check) is just the expedited way to do what happens at midnight daily anyways. Doing it manually avoids waiting until midnight before your LZ is free for more files again.

- could I speed this up by using a ram-drive? or would I run out of space without the manual flushes because the daemon does not move the files quickly enough? (that would be inconvenient since this cache partition would necessarily be rather small)

You can only work as fast as your drives can receive and then off-load from the LZ to the pool. I get the feeling a ram-drive would be an un-necessary added step.

- which filesystem should I use?

I use ext4 for all my drives. I have ...
/var/hda/files (1TB, my LZ & my non-pooled storage)
/var/hda/files/drives/drive1 (1TB) - (recently removed & RMA'd due to bad sectors)
/var/hda/files/drives/drive2 (1.5TB)
/var/hda/files/drives/drive3 (2TB)
It's up to you based on what the amahi Wiki says are usable filesystems.
With the scale of storage I have, I see no use for zfs yet as it's just too young.
It's really your own choice depending on the risk you want to take or features you wish to have

[fumsli]

Thank you for taking the time to answer me in such detail!

The simplest method I see ... would be to setup amahi with your empty 3tb drive with 2 partitions...
1x ?GB for LZ (LZ size is up to you) -- LZ partition
1x remaining GB (remaining size of the drive) -- pooled partition

Why do I need to separate the LZ and pooled partition? Greyhole shouldn't concern itself with files that were not put in it's share folders. Therefore creating a new directory that is not shared by greyhole should be fine - did I get that wrong?
As far as I can see the only thing that makes the process such an effort is the necessity to add empty disks. Again - did I get that wrong?

Manual LZ -f (flush or file system check) is just the expedited way to do what happens at midnight daily anyways. Doing it manually avoids waiting until midnight before your LZ is free for more files again.

(Where) can I set other intervals for flushing the LZ? (using a separate cron job doesn't feel clean)
One more thing - I have some folders that I want to store redundantly. Do I have to declare at pool-creation which ones they are and how many copies I want? Or can I do that when the whole thing is complete?

[sgtfoo]

Search around for a thread in the storage pooling section of the forums regarding having the LZ within the pool. We had a discussion about it and there's nothing wrong with it except that the pool may duplicate files un-necessarily and somehow use more space than required.

Having a drive that greyhole uses allows it to use the drive regardless of other things on it. It tends to get confusing. Once you see the folder structures you will understand.

Have you read this stuff? : https://github.com/gboudreau/Greyhole/wiki

Regarding manual fsck's, I think crons are the best way to add to the schedule. There may be a thread where this has been covered. Please search.

Regarding the folders to duplicate... it's done one a per-share-folder basis. So for example I have one folder share in the dashboard that is part of the grey pool AND is set to duplicate 1 time (2 copies across drives) and it's called "dual-backups"

Files are duplicated at each 'greyhole -f' at midnight or manually.

Also, go read this: http://revxatlarge.blogspot.com/2011/04 ... e-and.html