SOLVED: Folders are not deleted from Greyhole pools in Amahi 10

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

SOLVED: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Sat Mar 17, 2018 8:31 am

Hi,

I just upgraded from Amahi 8 to Amahi 10 and have a problem with Greyhole. I made a complete Fedora/Amahi reinstall on a new disk and planned to rebuild the LZ and the GH Pools. I experienced a strange problem while carefully checking the setup. So I stepped back and used a new test user (testuser), temporary renamed the old pool directores on the two drives (gh -> gh1) and let Greyhole to create a new pool gh folder on each drives for testing. Thus now it is as if I would create a competely new Greyhole configuration. Still the problem is the same and is described below. On my Amahi 8 setup I did not have similar problem with Greyhole.

Hardware: HP Proliant MicroServer Gen8
System: Amahi10/Fedora25
User: testuser
Shared folder (LZ): /var/hda/files/testuser saved in 2 copies by Greyhole
Greyhole pools: /var/hda/files/drives/drive1 and /var/hda/files/drives/drive2

Additional version infos:

uname -a
Linux nas.home.hu 4.13.16-100.fc25.x86_64 #1 SMP Mon Nov 27 19:52:46 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/fedora-release
Fedora release 25 (Twenty Five)
rpm -q samba amahi-greyhole
samba-4.5.14-0.fc25.x86_64
amahi-greyhole-0.10.14-1.x86_64

Platform: 10.0.0-1 (as reported by Amahi Dashboard)
Core: 10.4.0-1 (as reported by Amahi Dashboard)


(UPDATE: more precise spool and log details can be found in my later posts)

Scenario 1:

Copy test1-folder with files test1a.txt and test1b.txt to Samba shared testuser directory via Windows 10 file Explorer. Copy is successful and also checked on the HDA terminal. In the landing zone it is absolutelly OK, in the drive pools an extra empty directory appears with a strange name.

Greyhole log:

Mar 17 15:04:08 INFO write: Now working on task ID 36: write testuser/test1-folder/test1a.txt
Mar 17 15:04:08 INFO write: File created: testuser/test1-folder/test1a.txt
Mar 17 15:04:08 INFO write: Now working on task ID 37: write testuser/test1-folder/test1b.txt
Mar 17 15:04:08 INFO write: File created: testuser/test1-folder/test1b.txt


In the pools (the two pools look identical):

[root@nas testuser]# pwd
/var/hda/files/drives/drive2/gh/testuser
[root@nas testuser]# ls -lRh
.:
total 8.0K
drwxrwxrwx 2 root root 4.0K Mar 17 15:04 ''$'\240\234\215''C(V'
drwxrwxr-x 2 testuser users 4.0K Mar 17 15:04 test1-folder

'./'$'\240\234\215''C(V':

./test1-folder:
total 8.0K
-rwxrw-r-- 1 testuser users 1.1K Mar 17 13:05 test1a.txt
-rwxrw-r-- 1 testuser users 1.1K Mar 17 13:05 test1b.txt
total 0


Scenario 2:

Delete the test1-folder with the two text files via Windows 10 file Explorer. The shared folder becomes empty and that is also reflected by checking in the terminal. However in the pools only the files are deleted and not the directory. Again the two pools look identical.

Greyhole log:

Mar 17 15:14:09 INFO unlink: Now working on task ID 38: unlink testuser/test1-folder/test1a.txt
Mar 17 15:14:09 INFO unlink: File deleted: /var/hda/files/testuser/test1-folder/test1a.txt
Mar 17 15:14:09 INFO unlink: Now working on task ID 39: unlink testuser/test1-folder/test1b.txt
Mar 17 15:14:09 INFO unlink: File deleted: /var/hda/files/testuser/test1-folder/test1b.txt
Mar 17 15:14:09 INFO rmdir: Now working on task ID 40: rmdir testuser/N�C(V
Mar 17 15:14:09 INFO rmdir: Directory deleted: /var/hda/files/testuser/N�C(V


In the pools (the two pools look identical):

[root@nas testuser]# ls -lRh
.:
total 8.0K
drwxrwxrwx 2 root root 4.0K Mar 17 15:04 ''$'\240\234\215''C(V'
drwxrwxr-x 2 testuser users 4.0K Mar 17 15:14 test1-folder

'./'$'\240\234\215''C(V':
total 0

./test1-folder:
total 0
total 0


During the tests I patiently waited for the automatic processes (every 10 minutes) and had not forced with FSCK. I later scheduled FSCK and when ran, it gave these log lines:

Mar 17 16:24:10 INFO fsck_file: Now working on task ID 52: fsck_file testuser/test2-folder/test2b.txt
Mar 17 16:24:10 WARN fsck_file: WARNING! No copies of this file are available in the Greyhole storage pool: "testuser/test2-folder/test2b.txt".
Mar 17 16:24:10 INFO fsck_file: Now working on task ID 53: fsck_file testuser/test2-folder/test2a.txt
Mar 17 16:24:10 WARN fsck_file: WARNING! No copies of this file are available in the Greyhole storage pool: "testuser/test2-folder/test2a.txt".


If I repeate the test with another folder name and folders, the result is the same. Another strangely named empty directory is created in the pools together with the correct directory with the files. After deleting only the files are deleted. The file's folder and the strange folder remains there. Thus more and more garbage directories remain in the LZ folders. They don't disappear after FSCK or trash deletion.

I made several tests. Disabled storage pools, recreated the user, the shared directory, restarted the HDA etc. All modification had been made via the Amahi Dashboard. The shared folder's name is identical of the user name and thus shadows the /home/<user> directory. However the situation is the same if I use a common share folder, i.e. with a directory name other than the user name.

What could be the problem?

Gabor
Last edited by ghrasko on Sun Mar 18, 2018 1:40 am, edited 2 times in total.

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Sat Mar 17, 2018 9:39 am

Additional information to the above problem. I reinstalled Greyhole UI several times. In each case when Greyhole starts the log file shows the following warning:

Mar 17 17:35:16 WARN initialize: PHP Warning [2]: Invalid argument supplied for foreach() in /usr/bin/greyhole on line 533; BT: greyhole[L3634] processCommandLine() => greyhole[L3400] getRunner() => greyhole[L3436] process_config() => greyhole[L6467] parse() => greyhole[L150] init() => greyhole[L423] getShares()

I have seen this issue in other forum posts with much earlier versions, but I don't know if it has any relation to my problem.

Gábor

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Sun Mar 18, 2018 12:14 am

I try to dig deeper. Now I even check the /var/spool/greyhole directory and the MySQL Greyhole database.

This is in the Greyhole database tasks records about moving in a folder with a text file in it:

5,write,testuser,testfolder/test1.txt,,yes,2018-03-18 07:16:28

and when deleting it. See that there is garbage in the 4th field ("full path"). In the database that field was emty, so this seems to be some variables with uninitialized values. This will correctly remove the file from the folder, but not the folder itself. That remains there forever.

6,unlink,testuser,testfolder/test1.txt,,yes,2018-03-18 07:26:29
7,rmdir,testuser,pBìU,,yes,2018-03-18 07:26:29


If I just create a new dirextory from Windors ("New directory") a garbage directory is created in the pool drives. In the Greyhole spool files I see thes types of items:

mkdir
testuser
▒J▒U


rmdir
testuser
`6▒U


So it seems that already the spoo files does not contain the folder names correctly.

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Sun Mar 18, 2018 1:17 am

Scenario 3:

I create an emty folder on my Windows 10 desktop named as test2folder. Then I drag (move) it to the testuser shared folder.

FIles generated in /var/spool/greyhole directory:

[root@nas greyhole]# ls -ls
total 36
4 -rw-rw-rw- 1 testuser users 19 Mar 18 08:57 1521359829398448
4 -rw-rw-rw- 1 testuser users 19 Mar 18 08:57 1521359829502594
4 -rw-rw-rw- 1 testuser users 18 Mar 18 08:57 1521359829522859
4 -rw-rw-rw- 1 testuser users 19 Mar 18 08:57 1521359829582545
4 -rw-rw-rw- 1 testuser users 23 Mar 18 08:57 1521359829596210
4 -rw-rw-rw- 1 testuser users 18 Mar 18 08:57 1521359829602795
4 -rw-rw-rw- 1 testuser users 18 Mar 18 08:57 1521359829636247
4 -rw-rw-rw- 1 testuser users 18 Mar 18 08:57 1521359829645450
4 -rw-rw-rw- 1 testuser users 19 Mar 18 08:57 1521359829722163
0 drwxrwxrwt 2 root root 40 Mar 18 07:56 mem


The content of the files are as follows. See the one with mkdir markd with red:

[root@nas greyhole]# cat 1*
close
testuser
38

close
testuser
38

close
testuser
9

close
testuser
35

mkdir
testuser
▒J▒U


close
testuser
9

close
testuser
9

close
testuser
9

close
testuser
35


Scenario 4:

In similar way I move in a folder test5folder but now it contains a text file test5.txt. The spool file items are correct:

open
testuser
test5folder/test5.txt
9
for writing
close
testuser
9


Scenario 5:

When deleting that folder (that contains the file), the file within the folder is deleted correctly, but not the folder itself:

close
testuser
37

close
testuser
37

close
testuser
37

close
testuser
9

close
testuser
9

unlink
testuser
test5folder/test5.txt

rmdir
testuser
`▒U


close
testuser
9


The corresponding Greyhole database task records:

id,action,share,full_path,additional_info,complete,event_date
12,write,testuser,test4folder/test4.txt,,yes,2018-03-18 09:06:30
13,unlink,testuser,test4folder/test4.txt,,yes,2018-03-18 09:16:31
14,rmdir,testuser, `ìU,,yes,2018-03-18 09:16:31

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Sun Mar 18, 2018 2:03 am

Relevant config files are as follows.

smb.conf


greyhole.conf
Attachments
greyhole.txt
(939 Bytes) Downloaded 16 times
smb.txt
(1.69 KiB) Downloaded 18 times
greyhole.txt
(939 Bytes) Downloaded 19 times
smb.txt
(1.69 KiB) Downloaded 16 times

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Sun Mar 18, 2018 4:34 am

I finally went to the Greyhole Github source and found some interesting things. I checked the Greyhole daemon C routines that write out the faulty spool files when triggered by Samba.

vfs_greyhole-samba-4.5.c

static int greyhole_mkdir(vfs_handle_struct *handle, const struct smb_filename *smb_fname, mode_t mode);
static int greyhole_rmdir(vfs_handle_struct *handle, const struct smb_filename *smb_fname);
static int greyhole_open(vfs_handle_struct *handle, struct smb_filename *fname, files_struct *fsp, int flags, mode_t mode);
static ssize_t greyhole_write(vfs_handle_struct *handle, files_struct *fsp, const void *data, size_t count);
static ssize_t greyhole_pwrite(vfs_handle_struct *handle, files_struct *fsp, const void *data, size_t count, off_t offset);
static ssize_t greyhole_recvfile(vfs_handle_struct *handle, int fromfd, files_struct *tofsp, off_t offset, size_t n);
static int greyhole_close(vfs_handle_struct *handle, files_struct *fsp);
static int greyhole_rename(vfs_handle_struct *handle, const struct smb_filename *oldname, const struct smb_filename *newname);
static int greyhole_unlink(vfs_handle_struct *handle, const struct smb_filename *path);


vfs_greyhole-samba-3.6.c

static int greyhole_connect(vfs_handle_struct *handle, const char *svc, const char *user);
static int greyhole_mkdir(vfs_handle_struct *handle, const char *path, mode_t mode);
static int greyhole_rmdir(vfs_handle_struct *handle, const char *path);
static int greyhole_open(vfs_handle_struct *handle, struct smb_filename *fname, files_struct *fsp, int flags, mode_t mode);
static ssize_t greyhole_write(vfs_handle_struct *handle, files_struct *fsp, const void *data, size_t count);
static ssize_t greyhole_pwrite(vfs_handle_struct *handle, files_struct *fsp, const void *data, size_t count, SMB_OFF_T offset);
static ssize_t greyhole_recvfile(vfs_handle_struct *handle, int fromfd, files_struct *tofsp, SMB_OFF_T offset, size_t n);
static int greyhole_aio_write(struct vfs_handle_struct *handle, struct files_struct *fsp, SMB_STRUCT_AIOCB *aiocb);
static int greyhole_close(vfs_handle_struct *handle, files_struct *fsp);
static int greyhole_rename(vfs_handle_struct *handle, const struct smb_filename *oldname, const struct smb_filename *newname);
static int greyhole_unlink(vfs_handle_struct *handle, const struct smb_filename *path);


Between the two versions greyhole_rmdir() and greyhole_mkdir() functions's 2nd parameter had been changed. The file read and write function were unchanged. The directory name retrieval thus was changed:

in 3.6: path
in 4.5: fname->basename

For the greyhole_write() and greyhole_read() functions the file name reference had not changed:

both in 3.6 and 4.5: fsp->fh->fd

This might be OK, but I see the problem exactly in these routines. As if fname->basename would point to an uninitialized storage area.

In the Samba project struct smb_filename is defned in vfs.h as

struct smb_filename {
char *base_name;
char *stream_name;
char *original_lcomp;
uint32_t flags;
SMB_STRUCT_STAT st;
};


I increased the Samba log level and it seems it is sending the information correclty. Most probably Greyhole is reading the directory information incorrectly as suggested by the latest Greyhole changes in the mkdir and rmdir functions. I can't imagine that this works the same way for everyone, so it must be some system specific issue, though I have no idea what it could be.

I should emphasize that this problem is not obvious for the normal user. Everything seems to work correctly if someone only checks the Samba shared directories (the Greyhole LZ). I had to check the Greyhole pool(s) to identify the problem, to realize that folders are not deleted and garbage folders are created. This could remain hidden for many users.

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Mon Mar 19, 2018 1:21 am

The following Greyhole bug could be related: https://github.com/gboudreau/Greyhole/issues/186. For any case I opened a new ticket there pointing to this issue: https://github.com/gboudreau/Greyhole/issues/191

I had not yet checked if the fix for that bug in included in my installation or not. I used the Amahi Dashboard to install Greyhole UI.

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Mon Mar 19, 2018 7:05 am

At the Greyhole ticketing platform I had been advised to implement the following patch:

[*]Bugfix: "VFS module for Samba 4.5 was creating randomly-named folders in shares root", but also descireibed as " Bugfix: Fix mkdir/rmdir issue on Samba 4.5": https://github.com/gboudreau/Greyhole/r ... ag/0.10.15

I am happy to test it, but I don't know how I should implement a patch (or install a specific package) within a standard Amahi implementation. I used to add packages through the Amahi Dashboard. Could someone provide me information on how to do this?

Gábor

User avatar
bigfoot65
Project Manager
Posts: 11104
Joined: Mon May 25, 2009 4:31 pm

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby bigfoot65 » Mon Mar 19, 2018 6:19 pm

We would appreciate if you would attach files with text versus creating such long posts.

See bug 2303 for progress on new version.
ßîgƒσστ65
Applications Manager

ghrasko
Posts: 21
Joined: Wed Nov 04, 2015 9:23 am

Re: Folders are not deleted from Greyhole pools in Amahi 10

Postby ghrasko » Mon Mar 19, 2018 11:15 pm

See bug 2303 for progress on new version.

Sorry for the long posts and thanks for the info. I appreciate for your work on this product and don't want to push you, just asking if you have any date in your mind for adding this fix to the standard package? You were not directly advising me to implement the patch manually, weren't you? But if yes, I would need some info as I would maintain a standard installation that could be managed using the Amahi Dashboard. I have never done this before, but perhaps could follow the instructions in the INSTALL file within the package, but I would prefer waiting for an official upgrade of the Amahi product.

What do you recommend?

Sincerely,
Gábor

Who is online

Users browsing this forum: No registered users and 1 guest