Spooled 1 110 522. Finally decreasing

[Vertigo500]

Update: after a loong time, the spool has started to decrease. What i still dont understand is what greyhole is doing to deleted files, as described below. Also it seems to me that i will get this same problem when cleaning out duplicates, but it will resolve itself, given enough time.


Several days ago, i copyed about 150Gb of data to a share called "Docs". The share is set ut to have 2 copies. Some of the data, was music and software, so i moved them to other shares and deleted some duplicates and unwanted files. I may have caused problems by moving duplicates of files and useless files to trash, running fsck, emtying trash, emtying attic without waiting for previous task to finish. I dont know.

I also for good measure installed dropbox, with about 40Gb to download, and 136 000 files left to download, that seems to cause much activity, since i dont understand, freezing of directories, looks like temporary files cause unneccecary activity. In a previous attempt i uninstalled when i got a error message about 90 000 files in queue.

The problem is, if it is a problem, that the server have been working on somethings since. I checked the stats, and the tasks are only increasing, the spool especially, not decrasing. When tailing activity, it seems like it is doing something with deleted files. It is using about 5 minutes per file, wich means several weeks will go by, before working as normal. What is happening?

Code: Select all

Write Delete Rename Books 0 0 0 Docs 0 2484 2 Drop 1352 1026 0 Movies 5 2 0 Music 0 0 0 Pictures 0 0 0 Software 1095 0 0 ======== Total 2452 + 3512 + 2 = 5966 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 181563

Some of my "tail"

Code: Select all

Mar 31 18:02:13 7 rename: Saving 1 tombstones for Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-dj splash - party time (original_version).mp3 Mar 31 18:02:13 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-dj splash - party time (original_version).mp3 Mar 31 18:02:13 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-dj splash - party time (original_version).mp3 Mar 31 18:02:13 7 rename: Changing tombstone: /var/hda/files/drives/drive3/gh/Docs/.Trash-500/files/Music/New Folder/50 Best Trance Techno songs vol.2/29 linkan - better days (orginal_mix).mp3 -> /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/29 linkan - better days (orginal_mix).mp3 Mar 31 18:08:11 7 rename: Saving 1 tombstones for Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/29 linkan - better days (orginal_mix).mp3 Mar 31 18:08:11 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/29 linkan - better days (orginal_mix).mp3 Mar 31 18:08:12 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/29 linkan - better days (orginal_mix).mp3 Mar 31 18:08:12 7 rename: Changing tombstone: /var/hda/files/drives/drive3/gh/Docs/.Trash-500/files/Music/New Folder/1_The_Dave_Ramsey_Show/01 The Dave Ramsey Show - 11142008 1.mp3 -> /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/1_The_Dave_Ramsey_Show/01 The Dave Ramsey Show - 11142008 1.mp3 Mar 31 18:14:06 7 rename: Saving 1 tombstones for Docs/.Trash-500/expunged/1169137466/New Folder/1_The_Dave_Ramsey_Show/01 The Dave Ramsey Show - 11142008 1.mp3 Mar 31 18:14:07 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Docs/.Trash-500/expunged/1169137466/New Folder/1_The_Dave_Ramsey_Show/01 The Dave Ramsey Show - 11142008 1.mp3 Mar 31 18:14:07 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/1_The_Dave_Ramsey_Show/01 The Dave Ramsey Show - 11142008 1.mp3

It is possible that these files where deleted because of duplication.

What is happening? Why is it messing whith deleted files? Any advice??


http://wiki.amahi.org/index.php/Greyhol ... leshooting

1.

Code: Select all

uname -r; rpm -q samba hda-greyhole 2.6.35.11-83.fc14.i686 samba-3.5.8-74.fc14.i686 hda-greyhole-0.9.2-1.i386

2.

Code: Select all

yum -y install fpaste; fpaste /etc/samba/smb.conf; fpaste /etc/greyhole.conf Loaded plugins: fastestmirror, langpacks, presto, refresh-packagekit Adding en_US to language list Loading mirror speeds from cached hostfile * fedora: mirror01.th.ifl.net * rpmfusion-free: download1.rpmfusion.org * rpmfusion-free-updates: download1.rpmfusion.org * rpmfusion-nonfree: download1.rpmfusion.org * rpmfusion-nonfree-updates: download1.rpmfusion.org * updates: mirror01.th.ifl.net Setting up Install Process Package fpaste-0.3.5-1.fc14.noarch already installed and latest version Nothing to do Uploading (4.4K)... http://fpaste.org/kiew/ Uploading (1.4K)... http://fpaste.org/i850/

3.

Code: Select all

mount; fdisk -l; df -h; greyhole --stats /dev/mapper/VolGroup-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sdb1 on /boot type ext4 (rw) /dev/mapper/VolGroup-lv_home on /home type ext4 (rw) /dev/sda1 on /var/hda/files/drives/drive3 type ext4 (rw) fusectl on /sys/fs/fuse/connections type fusectl (rw) /dev/sdc1 on /var/hda/files/drives/drive4 type fuseblk (rw,noexec,nosuid,nodev,noatime,allow_other,blksize=4096,default_permissions) /dev/sdd1 on /var/hda/files/drives/drive7 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) //127.0.0.1/Books on /mnt/samba/Books type cifs (rw) //127.0.0.1/Docs on /mnt/samba/Docs type cifs (rw) //127.0.0.1/Drop on /mnt/samba/Drop type cifs (rw) //127.0.0.1/Movies on /mnt/samba/Movies type cifs (rw) //127.0.0.1/Music on /mnt/samba/Music type cifs (rw) //127.0.0.1/Pictures on /mnt/samba/Pictures type cifs (rw) //127.0.0.1/Software on /mnt/samba/Software type cifs (rw) gvfs-fuse-daemon on /home/Vertigo500/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=Vertigo500) Disk /dev/sda: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x43c88743 Device Boot Start End Blocks Id System /dev/sda1 2048 976773119 488385536 83 Linux Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xbe8be424 Device Boot Start End Blocks Id System /dev/sdb1 * 2048 1026047 512000 83 Linux /dev/sdb2 1026048 976773119 487873536 8e Linux LVM Disk /dev/sdc: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x8ecd2a9e Device Boot Start End Blocks Id System /dev/sdc1 2048 2930274303 1465136128 7 HPFS/NTFS Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x8ecd2a99 Device Boot Start End Blocks Id System /dev/sdd1 2048 2930276351 1465137152 83 Linux Disk /dev/dm-0: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 4227 MB, 4227858432 bytes 255 heads, 63 sectors/track, 514 cylinders, total 8257536 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/dm-2: 441.6 GB, 441643433984 bytes 255 heads, 63 sectors/track, 53693 cylinders, total 862584832 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-2 doesn't contain a valid partition table Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 50G 22G 26G 46% / tmpfs 1007M 240K 1006M 1% /dev/shm /dev/sdb1 485M 48M 412M 11% /boot /dev/mapper/VolGroup-lv_home 405G 44G 342G 12% /home /dev/sda1 459G 258G 178G 60% /var/hda/files/drives/drive3 /dev/sdc1 1.4T 1.1T 318G 78% /var/hda/files/drives/drive4 /dev/sdd1 1.4T 1.1T 282G 79% /var/hda/files/drives/drive7 //127.0.0.1/Books 3.2T 2.4T 777G 76% /mnt/samba/Books //127.0.0.1/Docs 3.2T 2.4T 777G 76% /mnt/samba/Docs //127.0.0.1/Drop 3.2T 2.4T 777G 76% /mnt/samba/Drop //127.0.0.1/Movies 3.2T 2.4T 777G 76% /mnt/samba/Movies //127.0.0.1/Music 3.2T 2.4T 777G 76% /mnt/samba/Music //127.0.0.1/Pictures 3.2T 2.4T 777G 76% /mnt/samba/Pictures //127.0.0.1/Software 3.2T 2.4T 777G 76% /mnt/samba/Software PHP Warning: date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'Europe/Berlin' for 'CEST/2.0/DST' instead in /usr/bin/greyhole on line 32 Greyhole Statistics =================== Storage Pool Total - Used = Free + Attic = Possible /var/hda/files/drives/drive3/gh: 458G - 257G = 178G + 0G = 178G /var/hda/files/drives/drive4/gh: 1397G - 1080G = 317G + 0G = 317G /var/hda/files/drives/drive7/gh: 1375G - 1024G = 281G + 0G = 281G ========================================== Total: 3231G - 2362G = 776G + 0G = 776G [root@localhost ~]#

4.

Code: Select all

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production +----+------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+------------------------------+--------------+---------------------+---------------------+ | 3 | /var/hda/files/drives/drive3 | 10 | 2011-03-17 15:26:12 | 2011-03-17 15:26:12 | | 5 | /var/hda/files/drives/drive4 | 10 | 2011-03-20 09:00:57 | 2011-03-20 09:00:57 | | 6 | /var/hda/files/drives/drive7 | 10 | 2011-03-20 10:15:49 | 2011-03-20 10:15:49 | +----+------------------------------+--------------+---------------------+---------------------+

5.

Code: Select all

ysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste Uploading (2.1K)... http://fpaste.org/O2e7/

6.

Code: Select all

greyhole --view-queue PHP Warning: date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'Europe/Berlin' for 'CEST/2.0/DST' instead in /usr/bin/greyhole on line 32 Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Books 0 0 0 Docs 0 2484 2 Drop 1352 1026 0 Movies 5 2 0 Music 0 0 0 Pictures 0 0 0 Software 1095 0 0 ======== Total 2452 + 3512 + 2 = 5966 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 228731

7. Tryed to debug random file, but sorry:

Code: Select all

greyhole --debug /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/29 linkan - better days (orginal_mix).mp3 -bash: syntax error near unexpected token `('

[Vertigo500]

Code: Select all

Write Delete Rename Books 0 0 0 Docs 0 2484 2 Drop 1352 1026 0 Movies 5 2 0 Music 0 0 0 Pictures 0 0 0 Software 1095 0 0 ======== Total 2452 + 3512 + 2 = 5966 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 585073

Its getting worse

[Vertigo500]

and suddenly 15 minutes after last post:

Code: Select all

Apr 01 06:51:43 6 daemon: Optimizing MySQL tables... Apr 01 06:51:43 6 daemon: Optimizing MySQL tables... Apr 01 06:51:44 6 daemon: Greyhole (version 0.9.2) daemon started. Apr 01 06:51:44 7 daemon: Loading graveyard backup directories... Apr 01 06:51:44 7 daemon: Found 2 directories in the settings table. Apr 01 06:52:06 7 read_smb_spool: Processing Samba spool...

and

Code: Select all

Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Books 0 0 0 Docs 0 2484 2 Drop 7076 5969 992 Movies 5 2 0 Music 0 0 0 Pictures 0 0 0 Software 1095 0 0 ======== Total 8176 + 8455 + 994 = 17625 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 560098

So this is Dropbox ? Is everything related to Dropbox, even the Spool, thats counting higher all the time?
But what about the trashfiles, thats taking 5 minutes to process each file?

[Vertigo500]

Code: Select all

Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename Books 0 0 0 Docs 0 2484 2 Drop 38258 33802 5933 Movies 5 2 0 Music 0 0 0 Pictures 0 0 0 Software 1095 0 0 ======== Total 39358 + 36288 + 5935 = 81581 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 760560

Its only growing.....

[Vertigo500]

Code: Select all

Write Delete Rename Books 0 0 0 Docs 0 2484 2 Drop 38258 33802 5933 Movies 5 2 0 Music 0 0 0 Pictures 0 0 0 Software 1095 0 0 ======== Total 39358 + 36288 + 5935 = 81581 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 1012267

Over 1 million, when will it stop?

Code: Select all

Apr 03 10:57:33 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/087 - Sean Kinston - Beautiful Girls (Robboz Rmx Edit).mp3 Apr 03 10:57:33 7 rename: Changing tombstone: /var/hda/files/drives/drive3/gh/Docs/.Trash-500/files/Music/New Folder/50 Best Trance Techno songs vol.2/08-Cascada - Truly Madly Deeply (Styles & Breeze Remix) - DoG.mp3 -> /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/08-Cascada - Truly Madly Deeply (Styles & Breeze Remix) - DoG.mp3 Apr 03 11:01:12 7 rename: Saving 1 tombstones for Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/08-Cascada - Truly Madly Deeply (Styles & Breeze Remix) - DoG.mp3 Apr 03 11:01:12 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/08-Cascada - Truly Madly Deeply (Styles & Breeze Remix) - DoG.mp3 Apr 03 11:01:12 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/08-Cascada - Truly Madly Deeply (Styles & Breeze Remix) - DoG.mp3 Apr 03 11:01:12 7 rename: Changing tombstone: /var/hda/files/drives/drive3/gh/Docs/.Trash-500/files/Music/New Folder/50 Best Trance Techno songs vol.2/05-stian_k_-_be_alive_(raaban_inc_rmx)-vanilla.mp3 -> /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-stian_k_-_be_alive_(raaban_inc_rmx)-vanilla.mp3 Apr 03 11:04:57 7 rename: Saving 1 tombstones for Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-stian_k_-_be_alive_(raaban_inc_rmx)-vanilla.mp3 Apr 03 11:04:57 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-stian_k_-_be_alive_(raaban_inc_rmx)-vanilla.mp3 Apr 03 11:04:57 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/05-stian_k_-_be_alive_(raaban_inc_rmx)-vanilla.mp3 Apr 03 11:04:57 7 rename: Changing tombstone: /var/hda/files/drives/drive3/gh/Docs/.Trash-500/files/Music/New Folder/50 Best Trance Techno songs vol.2/092 - Dreamsplash - Erica! (Original Edit).mp3 -> /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/092 - Dreamsplash - Erica! (Original Edit).mp3 Apr 03 11:08:58 7 rename: Saving 1 tombstones for Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/092 - Dreamsplash - Erica! (Original Edit).mp3 Apr 03 11:08:58 7 rename: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/092 - Dreamsplash - Erica! (Original Edit).mp3 Apr 03 11:08:58 7 rename: Saving tombstones (backup) in /var/hda/files/drives/drive7/gh/.gh_graveyard_backup/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/092 - Dreamsplash - Erica! (Original Edit).mp3 Apr 03 11:08:58 7 rename: Changing tombstone: /var/hda/files/drives/drive3/gh/Docs/.Trash-500/files/Music/New Folder/50 Best Trance Techno songs vol.2/027 - Banana Inc. - Norwegian Rebel (Prescription Vs Iz0r 2007 Radio Mix).mp3 -> /var/hda/files/drives/drive3/gh/Docs/.Trash-500/expunged/1169137466/New Folder/50 Best Trance Techno songs vol.2/027 - Banana Inc. - Norwegian Rebel (Prescription Vs Iz0r 2007 Radio Mix).mp3

Does anyone have an ida about what is going on here? Is it restoring deleted files? If so, why?

On of my next tasks is to find and delete duplicate photos, about 100Gb x 3 or 4 . Is this whats going to happen if i delete a lot of files from the "docs" share, and move some of them to another share?

Edit: It has finally begun decreasing. I would still like to know what is going on with the deleted files. Anyone know?