Transfer from landing to Pool Insanely slow

[gboudreau]

I am running Greyhole version 0.7.5-1. I don't know if I actually need to run the migration script from SQLite to MySQL, but I'm going to run that script and then re-run the test again after erasing the two hard drives.

Yes, you need to run the migration script.
Without running that, there is almost no change that would help you between that version and older versions.

[uslacker]

Try upgrading to the latest version:
(as root)

Code: Select all

yum -y install php-mbstring; rpm -Uvh http://greyhole.pommepause.com/releases/hda-greyhole-0.7.5-1.`uname -i`.rpm

Then execute the following migration script (as root again):

Code: Select all

/usr/share/greyhole/db_migration-sqlite2mysql.sh

Guillaume,

I followed this and at the end of the install got a message there is now a greyhole.conf.rpmnew.
the "new" conf file seems very different than the existing one. Should I do something to migrate it in place of the current file?

\\Greg

[gboudreau]

Don't touch. Amahi handles that file for you.

[nncs20]

Would it be possible to use one of the drives that is in the storage pool also as a landing for files being transferred in? I'm still having some trouble with a bit of overrun with faster data transfers over the network then greyhole is able to process and move them around, even with no duplicates being created.

[VirtualMe]

WHS V1 did this in a power pack, Power Pack 2 maybe? When transferring files, instead of sending them to the landing zone, the files got sent directly to their destination. From there they were duplicated if requested. I too would be interested in seeing if we can get the same functionality with Greyhole. I've been thinking about it but figured Guillaume has too much other stuff to do for right now.

[gboudreau]

Greyhole can't save the files directly at their destination at this time.
That would require the Samba module to be much smarter than it is right now, which would also mean much slower, which would mean slower file transfers.
But you can move the LZ to anywhere you'd like, even on a drive already in your pool.

For example, if you added /var/hda/files/drives/drive1 to your pool, Amahi in fact created a 'gh' folder there, and that's the folder Greyhole uses for the pool. You can create other folders next to 'gh' (NOT inside it!), and change your LZ to those directories. You can even set the LZ per-share. To do so, go in your Amahi Dashboard, in Setup > Shares, and change the 'Location' of your share to something else than the default /var/hda/files/share_name, for example, /var/hda/files/drives/drive1/share_name.

You'll need to manually launch a 'greyhole --fsck' after you changed the LZ, to re-create the missing links from your LZ, or you can manually move them from their old folder to the new one (you'll need to do that on the HDA itself, not using the shares). Example:

Code: Select all

mv /var/hda/files/share_name/* /var/hda/files/drives/drive1/share_name/

[untg99]

Having similar issues with insanely slow speeds copying from the.
I'm using a 60GB SSD for my LZ (also boot drive), so theoretically it should be really really fast but transfers are quite slow, in the order of kbytes per second, (this is also reflected in the greyhole logs).
I've changed the priority of the process with 'renice' and it sped the files up a bit (maybe from 100 per minute to 150-200 per minute).

Apart from this, copying files from my Mac, the transfer after about 5 minutes stalls and won't continue.

I've converted to using MySQL.

One other thing I tried was using the non-ssd drive as the LZ for the shares (it's possible it could be the SSD doing something funny because that SSD hasn't been 100% reliable in the past).
That seemed to speed it up but I haven't had enough of a chance to test to say if it's fixed the issue.

I am on a 1GB network so transferring to the Amahi box is very fast.

[gboudreau]

copying from the.

what ?

this is also reflected in the greyhole logs

How so? Paste something.

maybe from 100 per minute to 150-200 per minute

200 files per minute? It won't go much faster than that, because of the overhead that Greyhole creates to check for file locks, check free space, and create & save metadata.
If GH only transfered files from one drive to another, you could expect much higher files/second being transfered, but it does much more than just that.
If you just transfer small files, it's normal for Greyhole to be slow 'transferring' those files.
Try with big files, and see how fast they are transfered (greyhole.log says "Copying ... to ..." when the actual transfer is ongoing).

Apart from this, copying files from my Mac, the transfer after about 5 minutes stalls and won't continue.

Does that happen from only one client computer, or all clients?
That should be unrelated to Greyhole speeds, since file transfers are handled by Samba, not Greyhole.
See if you can reproduce on a share that is not in the pool to confirm.
When you do, you can always check what the server is doing when the transfer stalls. Use 'top', 'iotop' (yum install iotop), or 'iostat' to see if something takes all the resources, which would prevent Samba from being able to continue the transfer. Also look in the samba logs (/var/log/samba/*) for errors.

I am on a 1GB network so transferring to the Amahi box is very fast.

So it's fast, but stalls after 5 minutes ?

[untg99]

Huge reply <snip>

Ok, wrote a long reply but the more I think about it the more I realise it could be my dodgy Landing Zone drive (SSD) causing the problems.
I've since created a new share and put the Landing Zone, 'Location' on another one of the Pool drives so I will continue to monitor and see how it goes.

Apparently there is a faster way to copy batches of files initially straight onto the shares or something, I think I was reading one of your other posts or on the Wiki or something?

Thanks

[gboudreau]

You can indeed copy the files directly into the landing zone on the HDA, then run a "greyhole --fsck" to have Greyhole process those new files.