Amahi Forums

Posted: **Sat Jun 26, 2010 1:22 pm**

I recently added a new/second storage drive to my server and marked it for use in my storage pool. My primary drives is at /var/hda/files and my secondary drive is at /var/hda/drives/sdb2. I then added an additional share (Videos) and set 'Extra Copies' to max. It appears that greyhole placed the storage for the share on the second drive under /var/hda/drives/sdb2/gh/Videos. However, it does not appear to be making additional copies of my files. I would have expected a /var/hd/files/gh/videos as well. There appear to be no errors in the greyhole log and the entries regarding my new share appear as if it is configured to only make a single copy. I have run greyhole --fsck to see if it picks them up but no luck. I have tried switching the extra copies between 0 and max and the log shows 'simplify_tasks: Simplifying pending tasks.' and then a sleep: nothing to do. Any suggestions would be appreciated.

-Michael

Posted: **Sun Jun 27, 2010 5:51 am**

When you change the number of copies, you need to launch fsck for GH to apply your new setting.
(Or wait for the nightly fsck to run automatically at midnight.)

So you have two drives selected in Shares > Storage Pool?
Make sure your Videos shares is configured to Max copies.
Then run

Code: Select all

greyhole --fsck

Then wait until fsck is done (tail -f /var/log/greyhole.log to see it work).

Report back the result of the following commands:

Code: Select all

cat /etc/greyhole.conf
greyhole --debug Videos/some_existing_filename

Replace "some_existing_filename" by an existing file from your Videos shares...

Example:

Code: Select all

greyhole --debug Videos/Movies/9.mkv

Posted: **Sun Jun 27, 2010 11:37 am**

Yes, I have 2 drives in the pool, a 1Tb (460Gb free) and 500Gb (240Gb free) and neither are near capacity. My videos folder is 64Gb.

When I created my Videos share it placed the dir on my second drive in /var/hda/files/drives/sdb2/gh/videos. It appears from the logs below that it recognizes there is a missing copy, but attempts to resolve the problem by making a second copy on the drive where the share is already located, leaving only 1 copy.

Thanks for the help. The logs are below....

greyhole.conf

# This file is automatically generated by Amahi on Sun Jun 27 00:56:12 -0500 2010
# Any manual changes WILL BE OVERWRITTEN!
# Set the defaults (carefully) in /var/hda/platform/html/config/greyhole.yml
db_engine = sqlite
db_path = /var/cache/greyhole.sqlite
balance_modified_files = false
email_to = root
samba_log_file = /var/log/messages
greyhole_log_file = /var/log/greyhole.log
log_level = DEBUG
log_memory_usage = false
dir_selection_algorithm = most_available_space
sticky_files = Music/
df_cache_time = 15
delete_moves_to_attic = true
# add any group of other settings
# here in multiple lines
storage_pool_directory = /var/hda/files/gh, min_free: 10gb
storage_pool_directory = /var/hda/files/drives/sdb2/gh, min_free: 10gb
num_copies[Pictures] = 999
num_copies[Movies] = 1
num_copies[Music] = 1
num_copies[Docs] = 2
num_copies[Videos] = 999

here is the debug text for MaccaConcert.m4v

Debugging file operations for file named "Videos/Working/MaccaConcert.m4v"

From DB
=======

From logs
=========
Jun 26 02:57:00 7 simplify_tasks: Removing 1 duplicate pending tasks for Videos/Working/Kids Dance Again.avi
Jun 26 02:57:00 7 simplify_tasks: Removing 1 duplicate pending tasks for Videos/Working/MaccaConcert.m4v -> 35
Jun 26 02:57:01 7 write: Now working on task ID 96487: write Videos/Working/vid5.09-07-31_00-21.00.avi
--
Jun 26 02:59:53 7 write: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 02:59:53 7 write: Now working on task ID 96491: write Videos/Working/MaccaConcert.m4v
Jun 26 02:59:53 6 write: File created: Videos/Working/MaccaConcert.m4v - 1.71GB
Jun 26 02:59:53 7 write: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 0 tombstones.
Jun 26 02:59:54 7 write: Drives with available space: /var/hda/files/drives/sdb2/gh (241GB avail)
Jun 26 02:59:54 7 write: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 02:59:54 7 write: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 02:59:54 7 write: Copying file to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 03:02:00 7 write: Creating symlink in share pointing to the above file copy.
Jun 26 03:02:00 7 write: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 03:02:00 7 write: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 03:02:01 7 simplify_tasks: Simplifying pending tasks.
Jun 26 15:04:56 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 15:04:56 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:04:56 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 15:04:56 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 15:04:56 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 15:08:00 7 fsck: Loading tombstones for Videos/Working/Kids Dance Again.avi... Got 1 tombstones.
Jun 26 15:08:00 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:08:00 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:08:00 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 15:08:16 7 fsck: Loading tombstones for Videos/Working/Kids Dance Again.avi... Got 1 tombstones.
Jun 26 15:08:16 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:08:16 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:08:16 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 15:09:04 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 15:09:04 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 15:09:04 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 15:09:04 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 15:09:04 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 22:10:57 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 22:10:57 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 22:10:57 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 22:10:57 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 22:10:57 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 26 22:11:10 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 26 22:11:10 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 26 22:11:10 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 26 22:11:10 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 26 22:11:10 7 fsck: Entering /var/hda/files/videos/2003
--
Jun 27 00:42:21 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 27 00:42:21 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 27 00:42:21 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 27 00:42:21 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 27 00:42:21 7 fsck: Entering /var/hda/files/videos/2003
Jun 27 13:24:41 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/Kids Dance Again.avi
Jun 27 13:24:41 7 fsck: Found /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Loading tombstones for Videos/Working/MaccaConcert.m4v... Got 1 tombstones.
Jun 27 13:24:41 6 fsck: Missing file copies. Expected 2, got 1. Will create more copies using /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Drives with available space: /var/hda/files/drives/sdb2/gh (217GB avail)
Jun 27 13:24:41 7 fsck: Updating symlink at /var/hda/files/videos/Working/MaccaConcert.m4v to point to /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Saving 1 tombstones for Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Saving tombstones in /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
Jun 27 13:24:41 7 fsck: Entering /var/hda/files/videos/2003

From filesystem
===============
Landing Zone:
lrwxrwxrwx 1 root root 61 2010-06-27 13:24 /var/hda/files/videos/Working/MaccaConcert.m4v -> /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v

Graveyard:
-rw-r--r-- 1 root root 150 2010-06-27 13:24 /var/hda/files/drives/sdb2/gh/.gh_graveyard/Videos/Working/MaccaConcert.m4v
array (
0 =>
stdClass::__set_state(array(
'path' => '/var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v',
'is_linked' => true,
'state' => 'OK',
)),
)

File copies:
-rwxrw-r-- 1 michael users 1832767742 2010-06-26 03:02 /var/hda/files/drives/sdb2/gh/Videos/Working/MaccaConcert.m4v

Posted: **Sun Jun 27, 2010 11:57 am**

Show me the result of the following commands:

Code: Select all

mount
df -k
cat /usr/bin/greyhole | fpaste

I'm not sure why it seems to ignore /var/hda/files as a valid partition in the storage pool...

If you're able to, try to edit this file: /usr/bin/greyhole

Find the line that looks like this:

Code: Select all

$config_text = file_get_contents($smb_config_file);

And just above it, add the following lines:

Code: Select all

	gh_log(DEBUG, "DEBUG1: " . var_export($storage_pool_directories, TRUE));
	gh_log(DEBUG, "DEBUG2: $df_command");
	exec($df_command, $temp);
	gh_log(DEBUG, "DEBUG3: " . var_export($temp, TRUE));
	gh_log(DEBUG, "DEBUG4: " . var_export(order_target_drives(1, TRUE, 'Videos', 'Working/MaccaConcert.m4v'), TRUE));
	gh_log(DEBUG, "DEBUG5: " . var_export(get_free_space_in_storage_pool_dirs(), TRUE));

Then execute the following command:

Code: Select all

sudo service greyhole restart; sleep 2; grep -B 10 -A 10 DEBUG /var/log/greyhole.log

Show me what it showed.

If you're not able to edit that file manually, it's OK. I'll be able to give you a command to do it after you give me the result of the above 3 commands (mount, df, cat).

Cheers.

Posted: **Sun Jun 27, 2010 1:47 pm**

Here it is....

mount

/dev/sdc2 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdc1 on /boot type ext4 (rw)
/dev/sda2 on /var/hda/files type ext4 (rw)
/dev/sdb2 on /var/hda/files/drives/sdb2 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

df -k

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdc2 10321208 3508876 6288044 36% /
tmpfs 1012648 88 1012560 1% /dev/shm
/dev/sdc1 198337 22806 165291 13% /boot
/dev/sda2 957305492 474858752 433818364 53% /var/hda/files
/dev/sdb2 476591984 214108228 238274204 48% /var/hda/files/drives/sdb2

cat /usr/bin/greyhole | fpaste

Uploading (163.3K)...
http://fpaste.org/3kn5/

I edited /usr/bin/greyhole ....then

sudo service greyhole restart; sleep 2; grep -B 10 -A 10 DEBUG /var/log/greyhole.log

Shutting down Greyhole: [ OK ]
Starting Greyhole ...

-Michael

Posted: **Sun Jun 27, 2010 2:25 pm**

Code: Select all

curl -o /usr/bin/greyhole http://dl.dropbox.com/u/3022105/Amahi/greyhole.1
chmod +x /usr/bin/greyhole
service greyhole restart

Wait a couple of seconds after the service restarted, then:

Code: Select all

tail -200 /var/log/greyhole.log | fpaste

Posted: **Sun Jun 27, 2010 5:46 pm**

OK, done....

http://fpaste.org/mfM5/

Posted: **Sun Jun 27, 2010 5:51 pm**

Thanks. I think that's all I need.
There's probably a bug in the code that checks for free space in your partitions.
Issue filed in Greyhole bug tracker: http://code.google.com/p/greyhole/issues/detail?id=19
I should have a fix for it pretty soon.

Posted: **Sun Jun 27, 2010 6:19 pm**

Code: Select all

curl -o /usr/bin/greyhole http://dl.dropbox.com/u/3022105/Amahi/greyhole.1
chmod +x /usr/bin/greyhole
service greyhole restart
greyhole --fsck /var/hda/files/videos

Wait for fsck to complete:

Code: Select all

tail -f /var/log/greyhole.log

until it starts sleeping again; then Ctrl-C.

Code: Select all

greyhole --debug Videos/Working/MaccaConcert.m4v

That last command should now show you that you have two copies of the file.

If it worked, launch a fsck to fix all files:

Code: Select all

greyhole --fsck

Posted: **Sun Jun 27, 2010 8:04 pm**

That did the trick. I actually followed the instructions for adding a second drive that are on the Amahi wiki ( http://wiki.amahi.org/index.php/Adding_ ... o_your_HDA ). That's why I had the nested mount.

Thanks again.

-Michael

Amahi Forums

Greyhole not making copies

Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies

Re: Greyhole not making copies