Greyhole not making copies

michaelcombs
Posts: 12
Joined: Sat Jun 26, 2010 1:11 pm

Greyhole not making copies

Postby michaelcombs » Sat Jun 26, 2010 1:22 pm

I recently added a new/second storage drive to my server and marked it for use in my storage pool. My primary drives is at /var/hda/files and my secondary drive is at /var/hda/drives/sdb2. I then added an additional share (Videos) and set 'Extra Copies' to max. It appears that greyhole placed the storage for the share on the second drive under /var/hda/drives/sdb2/gh/Videos. However, it does not appear to be making additional copies of my files. I would have expected a /var/hd/files/gh/videos as well. There appear to be no errors in the greyhole log and the entries regarding my new share appear as if it is configured to only make a single copy. I have run greyhole --fsck to see if it picks them up but no luck. I have tried switching the extra copies between 0 and max and the log shows 'simplify_tasks: Simplifying pending tasks.' and then a sleep: nothing to do. Any suggestions would be appreciated.

-Michael

User avatar
gboudreau
Posts: 606
Joined: Sat Jan 23, 2010 1:15 pm
Location: Montréal, Canada
Contact:

Re: Greyhole not making copies

Postby gboudreau » Sun Jun 27, 2010 5:51 am

When you change the number of copies, you need to launch fsck for GH to apply your new setting.
(Or wait for the nightly fsck to run automatically at midnight.)

So you have two drives selected in Shares > Storage Pool?
Make sure your Videos shares is configured to Max copies.
Then run

Code: Select all

greyhole --fsck
Then wait until fsck is done (tail -f /var/log/greyhole.log to see it work).

Report back the result of the following commands:

Code: Select all

cat /etc/greyhole.conf greyhole --debug Videos/some_existing_filename
Replace "some_existing_filename" by an existing file from your Videos shares...

Example:

Code: Select all

greyhole --debug Videos/Movies/9.mkv
- Guillaume Boudreau

michaelcombs
Posts: 12
Joined: Sat Jun 26, 2010 1:11 pm

Re: Greyhole not making copies

Postby michaelcombs » Sun Jun 27, 2010 11:37 am

Yes, I have 2 drives in the pool, a 1Tb (460Gb free) and 500Gb (240Gb free) and neither are near capacity. My videos folder is 64Gb.

When I created my Videos share it placed the dir on my second drive in /var/hda/files/drives/sdb2/gh/videos. It appears from the logs below that it recognizes there is a missing copy, but attempts to resolve the problem by making a second copy on the drive where the share is already located, leaving only 1 copy.

Thanks for the help. The logs are below....

greyhole.conf

# This file is automatically generated by Amahi on Sun Jun 27 00:56:12 -0500 2010
# Any manual changes WILL BE OVERWRITTEN!
# Set the defaults (carefully) in /var/hda/platform/html/config/greyhole.yml
db_engine = sqlite
db_path = /var/cache/greyhole.sqlite
balance_modified_files = false
email_to = root
samba_log_file = /var/log/messages
greyhole_log_file = /var/log/greyhole.log
log_level = DEBUG
log_memory_usage = false
dir_selection_algorithm = most_available_space
sticky_files = Music/
df_cache_time = 15
delete_moves_to_attic = true
# add any group of other settings
# here in multiple lines
storage_pool_directory = /var/hda/files/gh, min_free: 10gb
storage_pool_directory = /var/hda/files/drives/sdb2/gh, min_free: 10gb
num_copies[Pictures] = 999
num_copies[Movies] = 1
num_copies[Music] = 1
num_copies[Docs] = 2
num_copies[Videos] = 999

here is the debug text for MaccaConcert.m4v

Debugging file operations for file named "Videos/Working/MaccaConcert.m4v"

From DB
=======

From logs
=========
Jun 26 02:57:00 7 simplify_tasks: Removing 1 duplicate pending tasks for Videos/Working/Kids Dance Again.avi
Jun 26 02:57:00 7 simplify_tasks: Removing 1 duplicate pending tasks for Videos/Working/MaccaConcert.m4v -> 35
Jun 26 02:57:01 7 write: Now working on task ID 96487: write Videos/Working/vid5.09-07-31_00-21.00.avi
--
Jun 26 02:59:53 7 write: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 02:59:53 7 write: Now working on task ID 96491: write Videos/Working/MaccaConcert.m4v
Jun 26 02:59:53 6 write: File created: Videos/Working/MaccaConcert.m4v - 1.71GB
Jun 26 02:59:53 7 write: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 0 tombstones.
Jun 26 02:59:54 7 write: Drives with available space: /var/hda/files/drives/sdb2/gh (241GB avail)
Jun 26 02:59:54 7 write: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 02:59:54 7 write: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 02:59:54 7 write: Copying file to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 03:02:00 7 write: Creating symlink in share pointing to the above file copy.
Jun 26 03:02:00 7 write: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 03:02:00 7 write: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 03:02:01 7 simplify_tasks: Simplifying pending tasks.
Jun 26 15:04:56 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 15:04:56 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:04:56 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 15:04:56 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 15:08:00 7 fsck: Loading tombstones for Videos/Working/Kids Dance Again.avi... Got 1 tombstones.
Jun 26 15:08:00 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:08:00 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:08:00 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 15:08:16 7 fsck: Loading tombstones for Videos/Working/Kids Dance Again.avi... Got 1 tombstones.
Jun 26 15:08:16 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:08:16 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:08:16 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 15:09:04 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 15:09:04 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:09:04 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 15:09:04 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 22:10:57 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 22:10:57 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 22:10:57 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 22:10:57 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 22:11:10 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 22:11:10 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 22:11:10 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 22:11:10 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 27 00:42:21 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 27 00:42:21 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 27 00:42:21 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 27 00:42:21 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Entering /var/hda/files/videos/2003
Jun 27 13:24:41 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 27 13:24:41 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 27 13:24:41 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 27 13:24:41 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Entering /var/hda/files/videos/2003

From filesystem
===============
Landing Zone:
lrwxrwxrwx 1 root root 61 2010-06-27 13:24 /var/hda/files/videos/Working/MaccaConcert.m4v -> /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v

Graveyard:
-rw-r--r-- 1 root root 150 2010-06-27 13:24 /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
array (
0 =>
stdClass::__set_state(array(
'path' => '/var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v',
'is_linked' => true,
'state' => 'OK',
)),
)

File copies:
-rwxrw-r-- 1 michael users 1832767742 2010-06-26 03:02 /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v

User avatar
gboudreau
Posts: 606
Joined: Sat Jan 23, 2010 1:15 pm
Location: Montréal, Canada
Contact:

Re: Greyhole not making copies

Postby gboudreau » Sun Jun 27, 2010 11:57 am

Show me the result of the following commands:

Code: Select all

mount df -k cat /usr/bin/greyhole | fpaste
I'm not sure why it seems to ignore /var/hda/files as a valid partition in the storage pool...

If you're able to, try to edit this file: /usr/bin/greyhole

Find the line that looks like this:

Code: Select all

$config_text = file_get_contents($smb_config_file);
And just above it, add the following lines:

Code: Select all

gh_log(DEBUG, "DEBUG1: " . var_export($storage_pool_directories, TRUE)); gh_log(DEBUG, "DEBUG2: $df_command"); exec($df_command, $temp); gh_log(DEBUG, "DEBUG3: " . var_export($temp, TRUE)); gh_log(DEBUG, "DEBUG4: " . var_export(order_target_drives(1, TRUE, 'Videos', 'Working/MaccaConcert.m4v'), TRUE)); gh_log(DEBUG, "DEBUG5: " . var_export(get_free_space_in_storage_pool_dirs(), TRUE));
Then execute the following command:

Code: Select all

sudo service greyhole restart; sleep 2; grep -B 10 -A 10 DEBUG /var/log/greyhole.log
Show me what it showed.

If you're not able to edit that file manually, it's OK. I'll be able to give you a command to do it after you give me the result of the above 3 commands (mount, df, cat).

Cheers.
- Guillaume Boudreau

michaelcombs
Posts: 12
Joined: Sat Jun 26, 2010 1:11 pm

Re: Greyhole not making copies

Postby michaelcombs » Sun Jun 27, 2010 1:47 pm

Here it is....

mount

/dev/sdc2 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdc1 on /boot type ext4 (rw)
/dev/sda2 on /var/hda/files type ext4 (rw)
/dev/sdb2 on /var/hda/files/drives/sdb2 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

df -k

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdc2 10321208 3508876 6288044 36% /
tmpfs 1012648 88 1012560 1% /dev/shm
/dev/sdc1 198337 22806 165291 13% /boot
/dev/sda2 957305492 474858752 433818364 53% /var/hda/files
/dev/sdb2 476591984 214108228 238274204 48% /var/hda/files/drives/sdb2


cat /usr/bin/greyhole | fpaste

Uploading (163.3K)...
http://fpaste.org/3kn5/

I edited /usr/bin/greyhole ....then

sudo service greyhole restart; sleep 2; grep -B 10 -A 10 DEBUG /var/log/greyhole.log

Shutting down Greyhole: [ OK ]
Starting Greyhole ...


-Michael

User avatar
gboudreau
Posts: 606
Joined: Sat Jan 23, 2010 1:15 pm
Location: Montréal, Canada
Contact:

Re: Greyhole not making copies

Postby gboudreau » Sun Jun 27, 2010 2:25 pm

Code: Select all

curl -o /usr/bin/greyhole http://dl.dropbox.com/u/3022105/Amahi/greyhole.1 chmod +x /usr/bin/greyhole service greyhole restart
Wait a couple of seconds after the service restarted, then:

Code: Select all

tail -200 /var/log/greyhole.log | fpaste
- Guillaume Boudreau

michaelcombs
Posts: 12
Joined: Sat Jun 26, 2010 1:11 pm

Re: Greyhole not making copies

Postby michaelcombs » Sun Jun 27, 2010 5:46 pm


User avatar
gboudreau
Posts: 606
Joined: Sat Jan 23, 2010 1:15 pm
Location: Montréal, Canada
Contact:

Re: Greyhole not making copies

Postby gboudreau » Sun Jun 27, 2010 5:51 pm

Thanks. I think that's all I need.
There's probably a bug in the code that checks for free space in your partitions.
Issue filed in Greyhole bug tracker: http://code.google.com/p/greyhole/issues/detail?id=19
I should have a fix for it pretty soon.
Last edited by gboudreau on Sun Jun 27, 2010 6:27 pm, edited 1 time in total.
- Guillaume Boudreau

User avatar
gboudreau
Posts: 606
Joined: Sat Jan 23, 2010 1:15 pm
Location: Montréal, Canada
Contact:

Re: Greyhole not making copies

Postby gboudreau » Sun Jun 27, 2010 6:19 pm

Code: Select all

curl -o /usr/bin/greyhole http://dl.dropbox.com/u/3022105/Amahi/greyhole.1 chmod +x /usr/bin/greyhole service greyhole restart greyhole --fsck /var/hda/files/videos
Wait for fsck to complete:

Code: Select all

tail -f /var/log/greyhole.log
until it starts sleeping again; then Ctrl-C.

Code: Select all

greyhole --debug Videos/Working/MaccaConcert.m4v
That last command should now show you that you have two copies of the file.

If it worked, launch a fsck to fix all files:

Code: Select all

greyhole --fsck
- Guillaume Boudreau

michaelcombs
Posts: 12
Joined: Sat Jun 26, 2010 1:11 pm

Re: Greyhole not making copies

Postby michaelcombs » Sun Jun 27, 2010 8:04 pm

That did the trick. I actually followed the instructions for adding a second drive that are on the Amahi wiki ( http://wiki.amahi.org/index.php/Adding_ ... o_your_HDA ). That's why I had the nested mount.

Thanks again.

-Michael

Who is online

Users browsing this forum: No registered users and 14 guests