Pool Using More Space Than "Copies" Should Allow

[luckifamily]

I have 5-1TB drives.

"Movies" is set to store 1 extra copy.

I have ~150GB worth of Movies

The Pool is showing each drive utilizing ~150GB space EACH.

What's going on here? I'm going to run out of space far too quickly.

[majide84]

Hi,

there is your problem.

"Movies is set to store 1 extra copy."

On each drive you store a extra copy. It is like a RAID1.

Here are more information. http://wiki.amahi.org/index.php/Storage_pooling

Regards

[luckifamily]

No extra copies means 1 instance of the file is stored (regardless of how many partitions used).

1 extra copy should mean (out of a pool of X partitions) that the file is stored on one partition and 1 extra copy is stored on a different partition. It is NOT like RAID1. Or at least that was my understanding of the features of greyhole.

Please, do not tell me I'm mistaken.

[lrevxl]

What else is on the drives? From my understanding it's just displaying the total used space, not just what Greyhole is taking up.

[luckifamily]

Nothing else (well, the last 5 drives anyway, disregard the 1st pictured as that's the OS which isn't included in the pool). Those drives are for the greyhole pool explicitly. I started copying my movies over and noticed it and stopped.

[lrevxl]

Yes, I understand that, but the problem is just showing the total used space doesn't really tell us anything. First thing to check, what is the x in 'num_copies[Movies]=x' in /etc/greyhole.conf

If that is only 2 as expected, then start going onto the drives see, do you see > 2 copies of the files spread across the partitions. (Be careful not to count tombstones -- these are not file copies!)

Also, though this does not account for the total space used in your case, remember that there is a 5% default reserved space when creating a new ext3/4 filesystem by default. So, 750 * .05 * 5 = 187.5gb. This has nothing to do with Greyhole, it's just a default system behavior, if you search the forum it's been asked plenty of times. (Search for 'greyhole maths'.)

Finally, have you been deleting / replacing files? You could have files sitting in the attics of those pool drives which would most certainly increase the space usage.

[luckifamily]

/etc/greyhole.conf does indeed show num_copies=2 for movies. I checked each drive and there doesn't seem to be any "extra"-extra copies. I'll add more and report back. Thanks for your suggestion.

I don't quite follow your 5% math though. .05x916GB=45.8GB=~870GB (useable space) per drive. 150GB averaged across 5 drives is ~30GB ea. 30 from 870=840GB. I'm about 90G short per drive. Right?

[lrevxl]

I don't quite follow your 5% math though.

You're right about the math, I only glanced quickly at your charts, it should indeed by .05 * 916.9 * 5 = ~230gb of reserved space. (Since your graph shows numbers like 748.7gb free + 168.2gb used = 916.9gb total it's pretty safe to assume that reserved space is being counted in the 'used' section I think.)

You've got 150gb of movies, and one extra copy so that should account for another 150gb, bringing the used total to 529.25gb (300 + 230). (I'm assuming your figure of 150gb means you've got 150gb of unique files, not 75gb, let me know if this is not the case.)

Your usage total based on the graphs is 844.15 (excluding the smaller partition, since it can't have been used as a destination yet).

You're left with 844.15 - 529.25 = 314.9gb unaccounted for. 314.9/5 = 62.98gb / drive unaccounted for. The question then becomes, what is using that space?

The next step, run 'greyhole --stats' and paste the output here, that will tell us how much space is being used by the attics. If that doesn't add up you can look at the actual space used up on the pool drives for Greyhole with something like 'du -sh /var/hda/files/drives/drive2/gh'

Find those two sets of data and we'll have a clearer idea of what's going on.

Also, you can verify the percentage of space reserved on your pool volumes with the following commands (again, default is 5%):

Where /dev/sdx1 is the drive in question

sudo tune2fs -l /dev/sdx1 | grep 'Reserved block count'
sudo tune2fs -l /dev/sdx1 | grep 'Block count'
(The above assumes you have sudo setup, if not you can become root and run the commands above without sudo.)

Divide the first by the second and you'll get a percentage, probably 0.05.

[luckifamily]

I wish I would have waited to try your other suggestions

As it is (and before you posted your last), I simply copied my data to my desktop, formatted the drives and began again. After copying the data back over, each drive is at ~60GB of data usage which falls in line with your's, mine and greyhole's math (I know I was off by ~30GB previously...mistakenly forgot to account for the duplicates).

I don't have a clue what was awry previously but at least now it's behaving as should be and I feel secure enough to begin transferring data en masse (particularly with some of the commands you have suggested). Many thanks.