Deleting symlinks from Samba share does not delete file

[LaserlaserFaceMcgee]

I've recently run into an interesting problem. While deploying a new media center, I realized that some of my shows were misnamed, causing the TV scarper to fail. In order to combat this, I went though and renamed the offending shows via the samba share they are in, and moved the media files into these new folders. I then deleted these older, misnamed folders.

Now, whenever a fsck runs, it recreates these old, wrongly named folders, and often times deletes the files from the new, properly named folders. This is driving me absolutely nuts. The interesting thing is that when I "delete" these, and only these renamed files, they do not show up in the greyhole attic at all.This is the case for all files I delete. Its become apparent that deleting these files via the samba share they are in is just deleting the symlink, and not the actual file. All file movement/deleting is done via the share.

Example: re- name a tv show: blah and blah to blah & blah. Manually move and rename files. Delete original misnamed folder. Fsck runs a couple of hours later, returns the folder named blah and blah, and moves files back into it. This leaves the new folder, blah & blah. If the files were renamed, they are deleted from the new folder, and the older, wrongly named files are added back to the orginal.

Here is a copy/past from the greyhole log of one such instance:

ow working on task ID 392111: fsck_file TV/blah And blah/Season 1/01x01 - Episode1.avi
Jan 08 16:15:46 7 fsck_file: Found /var/hda/files/drives/drive2/gh/TV/blah And blah/Season 1/01x01 - Episode1.avi
Jan 08 16:15:46 7 fsck_file: Found /var/hda/files/drives/drive3/gh/TV/blah And blah/Season 1/01x01 - Episode1.avi
Jan 08 16:15:46 7 fsck_file: Loading tombstones for TV/blah And blah/Season 1/01x01 - Episode1.avi... Got 2 tombstones.
Jan 08 16:15:46 6 fsck_file: Symlink was missing. Creating new symlink.
Jan 08 16:15:46 7 fsck_file: Updating symlink at /var/hda/files/drives/drive1/TV/blah And blah/Season 1/01x01 - Episode1.avi to point to /var/hda/files/drives/drive2/gh/TV/blah And blah/Season 1/01x01 - Episode1.avi
Jan 08 16:15:46 7 fsck_file: Saving 2 tombstones for TV/blah And blah/Season 1/01x01 - Episode1.avi
Jan 08 16:15:46 7 fsck_file: Saving tombstones in /var/hda/files/drives/drive2/gh/.gh_graveyard/TV/blah And blah/Season 1/01x01 - Episode1.avi
Jan 08 16:15:46 7 fsck_file: Saving tombstones in /var/hda/files/drives/drive3/gh/.gh_graveyard/TV/blah And blah/Season 1/01x01 - Episode1.avi

uname:

2.6.35.14-106.fc14.x86_64

rpm -q samba hda-greyhole

samba-3.5.11-79.fc14.x86_64
hda-greyhole-0.9.2-1.x86_64

smb.conf paste:

# This file is automatically generated for WORKGROUP setup.
# Any manual changes MAY BE OVERWRITTEN
# Amahi configuration, generated on Sat Jan 07 19:54:16 -0800 2012
[global]
workgroup = WORKGROUP
server string = home.com
netbios name = hda
printing = cups
printcap name = cups
load printers = yes
cups options = raw
log file = /var/log/samba/%m.log
log level = 0
max log size = 150
socket options = TCP_NODELAY
preferred master = yes
os level = 60
time server = yes
unix extensions = no
wide links = yes
veto files = /*.nws/riched20.dll/*.{*}/
security = user
username map script = /usr/share/hda-platform/hda-usermap
large readwrite = yes
encrypt passwords = yes
dos charset = CP850
unix charset = UTF8
display charset = LOCALE
guest account = nobody
map to guest = Bad User
wins support = yes
printer admin = root, @ntadmin, administrator


[homes]
comment = Home Directories
valid users = %S
browseable = no
writable = yes
create mask = 0644
directory mask = 0755

[print$]
path = /var/lib/samba/drivers
read only = yes
force group = root
write list = @ntadmin root
force group = root
create mask = 0664
directory mask = 0775
guest ok = yes

[printers]
path = /var/spool/samba
writeable = yes
browseable = yes
printable = yes
public = yes

[Books]
comment = Books
path = /var/hda/files/drives/drive1/books
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Pictures]
comment = Pictures
path = /var/hda/files/drives/drive1/pictures
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Movies]
comment = Movies
path = /var/hda/files/drives/drive1/movies
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Music]
comment = Music
path = var/hda/files/drives/drive1/music
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Docs]
comment = Docs
path = /var/hda/files/drives/drive1/docs
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[TV]
comment = TV
path = /var/hda/files/drives/drive1/TV
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Backup]
comment = Backup
path = /var/hda/files/drives/drive1/Backup
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[sysadmin]
comment = sysadmin
path = /var/hda/files/drives/drive1/sysadmin
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Rifftrax]
comment = Rifftrax
path = /var/hda/files/drives/drive1/Rifftrax
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Games]
comment = Games
path = /var/hda/files/Games
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

[Greyhole Attic]
comment = Greyhole Attic
path = /var/hda/files/Greyhole Attic
writeable = no
browseable = yes
valid users = user, sam, admin, nobody
write list = user, sam, admin, nobody
guest ok = yes
create mask = 0775
force create mode = 0664
directory mask = 0775
force directory mode = 0775

dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole
Revise paste →

We are not responsible for any content on this site.
About this site. Coded a

greyhole.conf paste

# This file is automatically generated by Amahi on Sat Jan 07 19:54:16 -0800 2012
# Any manual changes WILL BE OVERWRITTEN!
# Set the defaults (carefully) in /var/hda/platform/html/config/greyhole.yml
db_engine = mysql
db_host = localhost
db_user = greyhole
db_pass = greyhole
db_name = greyhole
balance_modified_files = false
email_to = root
greyhole_log_file = /var/log/greyhole.log
log_level = DEBUG
log_memory_usage = false
dir_selection_algorithm = most_available_space
df_cache_time = 15
delete_moves_to_attic = true
# sticky_files and stick_into options are nested - see http://wiki.amahi.org/index.php/Greyhol ... ed_options
sticky_files = Music/
# add any group of other settings
# here in multiple lines
storage_pool_directory = /var/hda/files/drives/drive1/gh, min_free: 10gb
storage_pool_directory = /var/hda/files/drives/drive2/gh, min_free: 10gb
storage_pool_directory = /home/gh, min_free: 10gb
storage_pool_directory = /var/hda/files/drives/drive3/gh, min_free: 10gb
num_copies[Books] = 4
num_copies[Pictures] = 4
num_copies[Movies] = 2
num_copies[Music] = 2
num_copies[Docs] = 4
num_copies[TV] = 2
num_copies[Backup] = 1
num_copies[sysadmin] = 2
num_copies[Rifftrax] = 2
num_copies[Games] = 1
num_copies[Greyhole Attic] = 1

mount:
mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/VolGroup-lv_home on /home type ext4 (rw)
/dev/sdb1 on /var/hda/files/drives/drive1 type ext4 (rw)
/dev/sdd1 on /var/hda/files/drives/drive2 type ext4 (rw)
/dev/sdc1 on /var/hda/files/drives/drive3 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

fdisk -l

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
16 heads, 63 sectors/track, 3876021 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x90909090

Device Boot Start End Blocks Id System
/dev/sdb1 63 3907029167 1953514552+ 83 Linux

Disk /dev/dm-0: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 8355 MB, 8355053568 bytes
255 heads, 63 sectors/track, 1015 cylinders, total 16318464 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table

df -h

[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
50G 21G 26G 45% /
tmpfs 2.9G 88K 2.9G 1% /dev/shm
/dev/sda1 485M 49M 412M 11% /boot
/dev/mapper/VolGroup-lv_home
1.8T 1.4T 314G 82% /home
/dev/sdb1 1.8T 1.3T 483G 73% /var/hda/files/drives/drive1
/dev/sdd1 1.8T 1.2T 584G 67% /var/hda/files/drives/drive2
/dev/sdc1 1.8T 1.4T 318G 82% /var/hda/files/drives/drive3

greyhole --stats

Greyhole Statistics
===================

Storage Pool
Total - Used = Free + Attic = Possible
/var/hda/files/drives/drive1/gh: 1834G - 1258G = 483G + 0G = 483G
/var/hda/files/drives/drive2/gh: 1834G - 1157G = 583G + 0G = 583G
/home/gh: 1776G - 1373G = 313G + 0G = 313G
/var/hda/files/drives/drive3/gh: 1834G - 1424G = 317G + 0G = 317G
==========================================
Total: 7278G - 5212G = 1697G + 0G = 1697G

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production
+----+------------------------------+--------------+---------------------+---------------------+
| id | path | minimum_free | created_at | updated_at |
+----+------------------------------+--------------+---------------------+---------------------+
| 3 | /var/hda/files/drives/drive1 | 10 | 2011-08-20 19:22:59 | 2011-08-20 19:22:59 |
| 4 | /var/hda/files/drives/drive2 | 10 | 2011-08-20 20:05:07 | 2011-08-20 20:05:07 |
| 5 | /home | 10 | 2011-08-20 21:36:47 | 2011-08-20 21:36:47 |
| 6 | /var/hda/files/drives/drive3 | 10 | 2011-10-02 22:25:33 | 2011-10-02 22:25:33 |
+----+------------------------------+--------------+---------------------+---------------------+

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste

xargs: ls: No such file or directory
No text to send.

Greyhole Work Queue Statistics
==============================

This table gives you the number of pending operations queued for the Greyhole daemon, per share.

Write Delete Rename
Backup 0 0 0
Books 0 0 0
Docs 0 0 0
Games 0 0 0
Movies 0 0 0
Music 0 0 0
Pictures 0 0 0
Rifftrax 0 0 0
sysadmin 0 0 0
TV 0 0 0
========
Total 0 + 0 + 0 = 0

The following is the number of pending operations that the Greyhole daemon still needs to parse.
Until it does, the nature of those operations is unknown.
Spooled operations that have been parsed will be listed above and disappear from the count below.

Any help is appreciated.

[LaserlaserFaceMcgee]

Progress so far:

It turns out the Greyhole attic share was not actually working correctly, as restarting the samba service from inside the Amahi web interface was apparently not actually restarting the service on the server itself. I used VNC to login to the machine and reset it manually. This got the Greyhole attic share going, but was not the source of my issue.

Other attempts: Based on the greyhole log, I removed a source drive for one of the offending folders from my pool. I then --fsck in the hopes that without this drive I would be able to remove the offending links. No luck.

My next attempt is to access the drives directly thought the file system. I currently have four such drives, and my intend is to hand remove all trace of the folders from each one of them.

[LaserlaserFaceMcgee]

No dice. I disabled both the samba service and greyhole, and went into each individual drive,as listed in my storage pool. I then deleted each folder, some of which should only have been populated 2 times but were on all 4 drives. I restarted the services, --fsck, and sadly had the same result.

I have also followed the tut listed here to mount the shares locally on my amahi server:

http://wiki.amahi.org/index.php/Mount_Shares_Locally

While the script does run, and the shares are currently listed in the /mnt/samba path, there are no files in any of the shares.

At this point, I am likely to give up on Amahi. I love the platform, but if it is going to forever propagate any misnamed file I have, removing my capability to edit anything in the pool, it is not much use as a storage medium. Hopefully windows 8 does storage pooling comparably.

[bigfoot65]

Sorry to hear you are having issues. Maybe stop by the IRC channel amahi today and get some live help. Not sure if you have modified anything, but I am sure there is an solution.

How did you rename the files, in terminal or through a Windows client? That could make a difference.

[LaserlaserFaceMcgee]

I appreciate the help. I modified the files through a windows share. The renaming went well, and the files deleted as you would expect, at least on the surface. When checking the attic however, either via the greyhole attic share or using greyhole -a, none are shown. Interestingly, the attic in general is working, as I've attempted to toggled the number of copies for the shares in such a way as to force deletion of a whole iteration of said shares, which worked just as you would expect. The issue seems to be when I manually delete the files in a windows share.

I'll hunt down the IRC and pop in.

[LaserlaserFaceMcgee]

There has been an odd development here. I stepped away from this for a couple of days to take care of some other things. Well, I glance in my TV share and notice a couple of the zombie folders are actually still gone. This was great news, but baffling. My first assumption was that the --fsck was still my culprit, and for whatever reason the automated midnight run had stopped. Since I hadn't run it manually, that must be why the folders are still gone. Checking into the greyhole.log showed that I was wrong. The auto --fsck is running, and not resurrecting the samba links! At this point, I think they only come back when I manually run the --fsck, but after re-editting my library about 30 times, I'm heavily disinclined to test it.

So no actual solution, but the cause appears to be manually running the --fsck.

[ciscoh]

There has been an odd development here. I stepped away from this for a couple of days to take care of some other things. Well, I glance in my TV share and notice a couple of the zombie folders are actually still gone. This was great news, but baffling. My first assumption was that the --fsck was still my culprit, and for whatever reason the automated midnight run had stopped. Since I hadn't run it manually, that must be why the folders are still gone. Checking into the greyhole.log showed that I was wrong. The auto --fsck is running, and not resurrecting the samba links! At this point, I think they only come back when I manually run the --fsck, but after re-editting my library about 30 times, I'm heavily disinclined to test it.

So no actual solution, but the cause appears to be manually running the --fsck.

Thias actually explains some stranger behavior i noticed while running --fsck manually.

[bigfoot65]

Is this strange behavior with the Amahi installed version of Greyhole or the newer version from the Greyhole website? It would be interesting to figure out if its a bug that has been fixed or one that has not been reported.

On a side note, I hope everyone who updates Greyhole uses the Amahi version (hda-greyhole) vs the regular Greyhole version. The wiki states that this one must be used of there will be problems.

[ciscoh]

Is this strange behavior with the Amahi installed version of Greyhole or the newer version from the Greyhole website? It would be interesting to figure out if its a bug that has been fixed or one that has not been reported.

On a side note, I hope everyone who updates Greyhole uses the Amahi version (hda-greyhole) vs the regular Greyhole version. The wiki states that this one must be used of there will be problems.

The install process i used most recently......maybe a week ago to re-install on new hardware.

1. Install Fedora 14 per the instructions
2. Run yum to get all Fedora Updates
3. Run Amahi Installer

I am not sure if running the updates also installed Greyhole updates.

[bigfoot65]

No it does not. Amahi ships with a specific version. There is no automated Greyhole update at the moment built into Amahi. If you want the latest Greyhole, you have to do it manually.

The wiki provides guidance on how to update to the latest version of Greyhole via a repo or the RPM. That is why I believe we are seeing more issues lately is many are doing the upgrade on their own and sometimes things do not quite work as designed initially. It seems that a repeat install is needed of the RPM to keep things working smoothly.