Its all a big mess. Pending testing

[Vertigo500]

So, the FSCK finally finished. I not looked at everything and checked yet, but i have noticed some things:

1. All the files i moved around during the FSCK and AFTER the deletion of the graveyards are in the trash. Probably stupid to do so, but i had a lot of unneded files and big mess on many of the shares, and was running out of space so i paniced, and deleted some and tried to organize some.

2. A lot of the files i had moved and renamed and otherwise organized during FSCK wich had been going on for weeks or months but LONG BEFORE the deletion of the graveyards are in the trash(or maybe missing i dont know, a lot of files..).

3. During the FSCK, a lot of the times when i checked the "tail" i noticed there where a lot of "more than 90 000 tasks queued, wont queue more". I dont know whats going on here, but i got a feeling this was going on at least 50% of the time. I felt sometimes like it was using 1 hour of that, and then 10 minutes of actual tasks before starting again. Is there room for improvement here?

4. Anytime i have a problem with the FSCK i get questions about what i have on the shares. I have movies, divx, blureay, pictures, documents and a lot of each type. Is that so unusual? I only have around 5 Gb, and a lot of that is duplicates. Why do i feel i have the only computer that takes a week to run a FSCK? Not everytime, but when i do a lot of transfers or there is a big backlog. Do i have more files than the average user?

5. If i had the time and knowhow, and not run in to problems with the installation, problems with landing zones and other things, i would have 10-20 GB in the shares at this point, and probably a lot more movies and other things, is Greyhole ready to handle it?

I am dissapoint with nr 2, why did this happen?

[lrevxl]

1. I'm not sure what you mean by files that you moved around are in the trash. When you deleted the tombstones and ran an fsck with the option to find orphaned files it creates tombstones for the files where and how they are in the pool directories. It will not sending anything to the trash, not unless it began processing actions which were deletes for those.

2. Again not sure what you mean by in the trash. But it sounds like your Greyhole hasn't been processing tasks for a long while, meaning that the renames/moves/modifications were never processed. This means that all you really did was move and rename symlinks. (When Greyhole processes the corresponding file operations it persists those changes to the pool drives.) When you deleted and recreated the tombstones all the files were picked up in their unmodified states so the symlinks that were recreated reflected the unmodified states.

3. Greyhole has a maximum of 100,000 queued tasks. It will keep queuing tasks from the spool directory until there are no more spool files or the db has 90% of the maximum tasks. (So 90,000 tasks loaded into the db.) It constantly checks for new spool files and loads them into the database as it runs through tasks. This operation is fast, all it's doing is reading a file, loading a corresponding task into the database, and deleting the file.

4. I have ~9tb of files, over 50,000 in total and I have no issues. I've lost no data, and my pool is not backlogged with tasks. It should not take a week to run an fsck. It takes around 20 minutes, perhaps a bit less, on my server. When you do a lot of transfers / moves / renames, you will create a lot of tasks, that's the nature of Greyhole, but your results are not typical, at least not in my experience.

5. Again, I have over 9,000gb of data, and Greyhole has been handling it without incident.

A friend of mine runs Greyhole as well and had an odd issue with his setup where Greyhole was processing tasks extremely slowly. It was running, and the tasks were going through, but it would only get through a few tasks a minute at the maximum. (Which is pretty slow.) He never figured out what was causing the slowness, but he ended up reinstalling Fedora and starting fresh. After that his Greyhole ran smoothly and up to speed. I do not know what caused it, but I do not think it was actually related to Greyhole, but rather an OS / installation issue. It's possible you're hitting something similar.

Is your queued task count going down? How quickly are tasks being processed in your logs?

[Vertigo500]

1. I'm not sure what you mean by files that you moved around are in the trash. When you deleted the tombstones and ran an fsck with the option to find orphaned files it creates tombstones for the files where and how they are in the pool directories. It will not sending anything to the trash, not unless it began processing actions which were deletes for those.

I mean that i had a lot of cartoons, under movies/tegnefilmer. I moved them to other subdirectories, and renamed then so Boxee would recongnize them. A very high percentage of them (ca 90%) was now almost empty directories that only had the cover and maybe a info file, but the movie was found in the trash, i hope, but maybe under the original name, i think but i am not sure. But it was not found in the right folder, but where i originally had them

So a lot of work i did have to be redone in the best case, worst case i lost a lot of files. I have not had the energy or time to check this out, i am almost paralyzed with how much work i have to redo.

2. Again not sure what you mean by in the trash. But it sounds like your Greyhole hasn't been processing tasks for a long while, meaning that the renames/moves/modifications were never processed. This means that all you really did was move and rename symlinks. (When Greyhole processes the corresponding file operations it persists those changes to the pool drives.) When you deleted and recreated the tombstones all the files were picked up in their unmodified states so the symlinks that were recreated reflected the unmodified states.

I love howard stern and i have a lot of files, mp3 content mostly. After deletion of the graveyards, i tried to orginize some of the files, and these files are easy to find again if anything goes wrong, so i moved them around. By that, i mean sorting them from random folders, to the year they where from. I made folders 2006, 2007 etc. I moved all files i could find into these folders, under Books/Howard stern/20xx. All of these folders where empty except for the folders contained in them. By that i mean that an mp3 could be directly in the 2006 folder, or under 02.04.2006 folder for that day. But that folder would also be empty. All these files where found in the trash. I moved them back, not much work here. The strange thing is, that i did it yesterday, and now i found a lot of them back in the trash. What the ????

3. Greyhole has a maximum of 100,000 queued tasks. It will keep queuing tasks from the spool directory until there are no more spool files or the db has 90% of the maximum tasks. (So 90,000 tasks loaded into the db.) It constantly checks for new spool files and loads them into the database as it runs through tasks. This operation is fast, all it's doing is reading a file, loading a corresponding task into the database, and deleting the file.

Ok. I dont know if i understand it all, or if its just as simple that i think it is, but ok

4. I have ~9tb of files, over 50,000 in total and I have no issues. I've lost no data, and my pool is not backlogged with tasks. It should not take a week to run an fsck. It takes around 20 minutes, perhaps a bit less, on my server. When you do a lot of transfers / moves / renames, you will create a lot of tasks, that's the nature of Greyhole, but your results are not typical, at least not in my experience.

I probably have more photos and documents. It doesent usually take a week to run an FSCK, maybe 20 minutes to a couple of hours, i am not sure. But after renaming a lot and organizing it takes a looong time, and i just dont understand why. Should i only move files to the server after all renaming and organizing is done, if so, thats no good for me. If that is the case, game over for me.

A friend of mine runs Greyhole as well and had an odd issue with his setup where Greyhole was processing tasks extremely slowly. It was running, and the tasks were going through, but it would only get through a few tasks a minute at the maximum. (Which is pretty slow.) He never figured out what was causing the slowness, but he ended up reinstalling Fedora and starting fresh. After that his Greyhole ran smoothly and up to speed. I do not know what caused it, but I do not think it was actually related to Greyhole, but rather an OS / installation issue. It's possible you're hitting something similar.

You may be on to something here. I only wish to organize files a bit more before reinstall, but that may not be wise.

Is your queued task count going down? How quickly are tasks being processed in your logs?

It finished yesterday. It took forever to get up to 1 500 000 spooled, then it was finished the day after i think. I think it goes pretty fast. I redid the command with the rebuild of the files, to be sure, and it may or may not lead to more problems. We shall see.

[Vertigo500]

I did an FSCK so i could time it. It took about 7 hours! If yours takes 20 minutes, thats strange.

I randomly counted the speed, and it processed 20 lines in a second, thats 72 000 files pr hour, and i looked at it several times at it looked as it was racing, not using long time on one task. I am now running treesize to check how many files i have, and here are the results:

Code: Select all

[size=50]TreeSize Professional Report, 30. juli 2011 [Current Drive] Size: 5600964 MB Used: 4227857 MB Free: 1373107 MB 1024 Bytes per Cluster (NTFS) Name Size Allocated Files Wasted % of Parent Last Change Last Access Compr. Owner Books (\\HDA) 112 489,1 MB 112 495,7 MB 13616 6,6 MB 100,00 % 29.07.2011 29.07.2011 0,00 % Unknown Pictures (\\HDA) 281 867,5 MB 281 902,4 MB 72749 35,0 MB 100,00 % 28.07.2011 29.07.2011 0,00 % Unknown Movies (\\HDA) 912 104,2 MB 912 122,3 MB 38897 18,1 MB 100,00 % 29.07.2011 30.07.2011 0,00 % Unknown Music (\\HDA) 49 834,7 MB 49 837,6 MB 6276 2,9 MB 100,00 % 16.07.2011 29.07.2011 0,00 % Unknown Docs (\\HDA) 465 785,3 MB 465 887,7 MB 210615 102,4 MB 100,00 % 20.06.2011 29.07.2011 0,00 % Unknown Software (\\HDA) 330 968,8 MB 330 986,4 MB 34236 17,7 MB 100,00 % 28.07.2011 29.07.2011 0,00 % Unknown Drop (\\HDA) 41 635,1 MB 41 650,4 MB 31715 15,3 MB 100,00 % 25.04.2011 17.07.2011 0,00 % Unknown Serie (\\HDA) 194 202,7 MB 194 222,6 MB 38865 19,9 MB 100,00 % 20.04.2011 30.07.2011 0,00 % Unknown Greyhole Trash (\\HDA) 643 329,5 MB 643 367,1 MB 77623 37,6 MB 100,00 % 29.07.2011 30.07.2011 0,00 % Unknown

Thats 524 592 files, and if i divide that by 72 000 files pr hour, i get 7.3 hours to process. So i have to fint out what all the small files are, and try to get rid of them.

[Vertigo500]

Not to insult anyone, but now i am having problems that is not acceptable. Something is really wrong. It is of course possible that its me, but how special am I? Is Greyhole not for normal users?

There is several things i dont understand, and if any of my concerns is not my fault, there better be a good explenation for this.

The FSCK is done.

1. I noticed that when i deleted a lot of files, they ended up in the trash. When i deleted some of them from trash, i noticed an error, i dont remember word for word, but it did not find the file. I was thinking, that maybe i had to wait for an FSCK, but i tried that, and still got the errors when deleting files. To clarify , this does not apply to all files, just some.

2. I have a folder named "tegnefilmer", wich conatins cartoons for the kids. I use this with Boxee. The files have to be correcly named according to IMDB to work correctly. I have done this before, but I had lost most of my renames at the start of this tread. I spent several hours yesterday redoing it, renaming files, and moving them to separate folders. I ran an FSCK after that, so i was sure it would stick this time. I checked today, and i find files in the trash, renames undone, and files moved back to where i started. WTF?

3. There is also a lot of files missing in another folder named "serier" wich contains series and some movies.

4. I have moved around a lot of other files and deleted others, and we are talking at least 200 000 files, and i dread even to check.

5. I issued the command going on Disk 7, as i have freed up enough space to remove it. I discover that during the FSCK it it used as part of the pool. Is there a reason for this? when am i supposed to remove it? How do i know it contains only duplicates of a file, and not the only copy?

6 Should i really encounter ANY of the problems above?

7. I am beginning to think, as i have asked before in this tread, and did not get an answer to, that all renames and organizing of files should only take place BEFORE moving files to greyhole, since it seems that it handles renames and moving of files , really badly in my system. How could it not be a warning about this? I have read a lot about it, but not read anything about this. Can i get a clear answer about this? Somebody must know. I said earlyer in this tread, that this was not acceptable to me, and i got no warning about this.


When i move 100 000 pictures, and spend several days to get this right, i dont want surprises, and it will be impossible to repair if it has gone wrong.

8 There are probably other things, but i have not had the guts to check it all.

[gboudreau]

I don't have time to read 3 pages worth of long posts, and lrevxl is already helping you, and being a Greyhole developer, he should know what's he talking about. But I'll say only a couple of things...

1. Make sure you're running the latest version of greyhole. That would be 0.9.9. You can check with: rpm -q hda-greyhole
See other threads in this forum on how to update to the latest version manually, since it's not available in Amahi automatically.

2. When you rename, delete, or touch in any way any of your files, you must do so using a mounted share (remotely, or on a locally mount share), NEVER in /var/hda/files/share_name on the HDA itself. If you do that, the next Greyhole fsck will undo most of what you did in there, and put back what it thinks should be there (per the gh folders metadata, i.e. the old filenames & files).

3. It looks like you're having a lot of issues, most of them probably related, but maybe not. What I'd do, if I were you: find all the file copies you care about, and don't have a backup of, and copy those to extra drives (borrow some, or buy some, if you don't have spares).
Connect those drives on another computer (not necessarily all at the same time), and re-install your HDA to start from a clean slate, then start from the beginning, slowly this time. Copy your data one directory at a time from the remote machine, checking that all files are where they should before moving on to the next. If you make sure to always work with the latest version of greyhole (greyhole.net to find it), and always modify your files using mounted shares, you should not have any problem.

Good luck.

[Vertigo500]

1. Well i updated Greyhole, but i cannot get the command for updating HDA to work, it just says 404 error. Any tips? The command i am running is : rpm -Uvh http://dl.getdropbox.com/u/364883/Amahi ... noarch.rpm

And i am guessing that i need a version number?

2. I did acces and move from a share from a win7 computer.

3. Not possible for me at this time. What makes you thinjk i need a new install? Also what happens to my 1 000 GB in the trash, and possibly 100 000 files that i have problems with? Will it just be discovered by my new install or are they lost forever? This is feeling like a nighmare at this point.

4. I have lost access to all the shares after updating Greyhole. This is A BIG PROBLEM!!!!!


I am feeling so lost now. I am so frustrated.

I read in the post above that i should always transfer files via network shares, and i will never have problems, but thats what i did, and i am having nothing but troubles.

Are there not anyone here who can help me? This is taking a lot of time, and i want it to work, but its getting harder to see the end of this. This may not be a useful solution when the shit hits the fan, like it does in the real world sometimes. If its one thing Greyhole has to do right, its not to loose files, and mess around with putting them in the trash arbitrarily.

And why is nobody taking this seriously, and anwering my question about if Greyhole can handle renames and moving around files, or if everything has to be static?

And on top of that i get a suggestion on reinstalling the whole system, when i have 1000 Gb in the trash, and about 100 000 files that may or may not have been deleted for good?

Also lrevxl has been a good help, but as he keeps pointing out, he does not use Amahi, and it would be nice to get some help from someone who knows about it, at least now, when all my access to the shares over the network are gone.

[Vertigo500]

So i still dont have access to the shares, would that not be a Amahi problem? Or Samba? Looks to me that the Samba server is running.

I cannot belive that the only solution is to reinstall Amahi. There got to be another way.

Somebody should be able to explain what will happen with the files in the trash if i do a reinstall. Will they show up again or will they be lost? This is kind of important for be to be able to recover my files.

I was stating in the beginning of this tread that i was not convinced of the stability of Ext3, and wanted to go back to NTFS, but i was told that it was an excellent system, and i was only user permissions that was the problem. I dunno, seem so me i have had problems i would NEVER see on a windows machine. Who has heard of renamed or moved files that suddenly appears in the trash on windows?

[jbmia]

This is probably the fix to your missing shares:
http://forums.amahi.org/viewtopic.php?f=39&t=3000

(edit: Above applies greyhole 0.9.7... Latest update to 0.9.9.1 shouldn't require this... old post.. just a heads up.. I'd installed 0.9.7 the day before yesterday and just found 0.9.9.1 just now... sorry for the confusion.. http://www.greyhole.net/releases... make sure you pick one of the hda-greyhole releases..)

Anybody remember the Radio Shack and Heathkit radio projects? Ever try to put one of those together? Did it work just by following directions like clockwork the first go around? Or.. did you have to whip out the schematics (with little to no knowledge of how to read them) and your volt / amp meter and go through and test things and find out you soldered something backwards or whatever?? That's kinda like what solutions like Amahi are... You've gotta roll up your sleeves a little and figure things out...

Lemme tell you.. if you think this is bad... go try to set up ZoneMinder... I just spent about 250 hours getting that thing up and running.. I've installed it like 8 times already... But, it works and it's cool...



jbmia

[Vertigo500]

This is probably the fix to your missing shares:
http://forums.amahi.org/viewtopic.php?f=39&t=3000

Lemme tell you.. if you think this is bad... go try to set up ZoneMinder... I just spent about 250 hours getting that thing up and running.. I've installed it like 8 times already... But, it works and it's cool...
jbmia

Thanks, that worked great! Death sentence on Greyhole in my system postponed, until more testing is done.....

Zoneminder looks cool, i am going to look in to that.