Greyhole --stats = "error 28 from storage enginge" ??

[jbmia]

Hey guys... something screwy is going on with greyhole.. has been working fine the last few months.. just through another disk on it the other day and now I'm accessing it remotely via vpn and trying to check "greyhole --stats" and I'm getting a strange response:

Code: Select all

Can't describe tasks with query: DESCRIBE tasks - Error: Got error 28 from storage engine[

Any ideas?? Here's my troubleshooting points:

1. What version of Fedora, Samba & Greyhole are you running?

uname -r; rpm -q samba hda-greyhole

Code: Select all

2.6.35.13-92.fc14.x86_64 samba-3.5.8-76.fc14.x86_64 hda-greyhole-0.9.2-1.x86_64

2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):

yum -y install fpaste; fpaste /etc/samba/smb.conf; fpaste /etc/greyhole.conf

http://fpaste.org/Mav1/

http://fpaste.org/gOv9/

3. The result of the following commands:

mount; fdisk -l; df -h; greyhole --stats

Code: Select all

/dev/mapper/vg_9745-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sdi1 on /boot type ext4 (rw) /dev/mapper/vg_9745-lv_home on /home type ext4 (rw) /dev/sdi3 on /var/hda/files type ext4 (rw) /dev/sdb1 on /var/hda/files/drives/drive1 type ext4 (rw) /dev/sdc1 on /var/hda/files/drives/drive2 type ext4 (rw) /dev/sdd1 on /var/hda/files/drives/drive3 type ext4 (rw) /dev/sde1 on /var/hda/files/drives/drive4 type ext4 (rw) /dev/sdg1 on /var/hda/files/drives/drive5 type ext4 (rw) /dev/sdh1 on /var/hda/files/drives/drive6 type ext4 (rw) /dev/sdi4 on /var/media type ext4 (rw) /dev/sdf1 on /var/hda/files/drives/drive9 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) //127.0.0.1/Books on /mnt/samba/Books type cifs (rw) //127.0.0.1/Home_Movies on /mnt/samba/Home_Movies type cifs (rw) //127.0.0.1/Movies on /mnt/samba/Movies type cifs (rw) //127.0.0.1/Music on /mnt/samba/Music type cifs (rw) //127.0.0.1/Pictures on /mnt/samba/Pictures type cifs (rw) //127.0.0.1/Public on /mnt/samba/Public type cifs (rw) //127.0.0.1/Recipes on /mnt/samba/Recipes type cifs (rw) //127.0.0.1/Software on /mnt/samba/Software type cifs (rw) //127.0.0.1/TV on /mnt/samba/TV type cifs (rw) //127.0.0.1/Users on /mnt/samba/Users type cifs (rw) Disk /dev/sda: 250.1 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xd16ce398 Device Boot Start End Blocks Id System /dev/sda1 2048 488396799 244197376 83 Linux Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xc87ee887 Device Boot Start End Blocks Id System /dev/sdb1 2048 1953523711 976760832 83 Linux Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x4bafc7b3 Device Boot Start End Blocks Id System /dev/sdc1 2048 1953523711 976760832 83 Linux Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes 255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x18249b79 Device Boot Start End Blocks Id System /dev/sdd1 2048 2930276351 1465137152 83 Linux Disk /dev/sde: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x29cd4bd0 Device Boot Start End Blocks Id System /dev/sde1 2048 3907028991 1953513472 83 Linux Disk /dev/sdf: 2000.4 GB, 2000398934016 bytes 255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000bc6a4 Device Boot Start End Blocks Id System /dev/sdf1 2048 3907028991 1953513472 83 Linux Disk /dev/sdg: 300.1 GB, 300090728448 bytes 255 heads, 63 sectors/track, 36483 cylinders, total 586114704 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xd16ce393 Device Boot Start End Blocks Id System /dev/sdg1 2048 586113023 293055488 83 Linux Disk /dev/sdh: 1000.2 GB, 1000204886016 bytes 255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x5c2ff997 Device Boot Start End Blocks Id System /dev/sdh1 2048 1953523711 976760832 83 Linux Disk /dev/sdi: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000b3c51 Device Boot Start End Blocks Id System /dev/sdi1 * 2048 1026047 512000 83 Linux /dev/sdi2 1026048 78850047 38912000 8e Linux LVM /dev/sdi3 78850048 771973119 346561536 83 Linux /dev/sdi4 771973120 976773119 102400000 83 Linux Disk /dev/dm-0: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-0 doesn't contain a valid partition table Disk /dev/dm-1: 20.7 GB, 20736638976 bytes 255 heads, 63 sectors/track, 2521 cylinders, total 40501248 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-1 doesn't contain a valid partition table Disk /dev/dm-2: 10.5 GB, 10502537216 bytes 255 heads, 63 sectors/track, 1276 cylinders, total 20512768 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/dm-2 doesn't contain a valid partition table Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_9745-lv_root 20G 19G 0 100% / tmpfs 1.9G 0 1.9G 0% /dev/shm /dev/sdi1 485M 49M 411M 11% /boot /dev/mapper/vg_9745-lv_home 9.7G 545M 8.7G 6% /home /dev/sdi3 326G 59G 251G 19% /var/hda/files /dev/sdb1 917G 664G 254G 73% /var/hda/files/drives/drive1 /dev/sdc1 917G 660G 258G 72% /var/hda/files/drives/drive2 /dev/sdd1 1.4T 1.1T 255G 82% /var/hda/files/drives/drive3 /dev/sde1 1.8T 1.6T 258G 86% /var/hda/files/drives/drive4 /dev/sdg1 276G 260G 17G 95% /var/hda/files/drives/drive5 /dev/sdh1 917G 661G 256G 73% /var/hda/files/drives/drive6 /dev/sdi4 97G 91G 651M 100% /var/media /dev/sdf1 1.8T 1.4T 487G 74% /var/hda/files/drives/drive9 //127.0.0.1/Books 7.9T 6.2T 1.8T 78% /mnt/samba/Books //127.0.0.1/Home_Movies 7.9T 6.2T 1.8T 78% /mnt/samba/Home_Movies //127.0.0.1/Movies 7.9T 6.2T 1.8T 78% /mnt/samba/Movies //127.0.0.1/Music 7.9T 6.2T 1.8T 78% /mnt/samba/Music //127.0.0.1/Pictures 7.9T 6.2T 1.8T 78% /mnt/samba/Pictures //127.0.0.1/Public 7.9T 6.2T 1.8T 78% /mnt/samba/Public //127.0.0.1/Recipes 7.9T 6.2T 1.8T 78% /mnt/samba/Recipes //127.0.0.1/Software 7.9T 6.2T 1.8T 78% /mnt/samba/Software //127.0.0.1/TV 7.9T 6.2T 1.8T 78% /mnt/samba/TV //127.0.0.1/Users 7.9T 6.2T 1.8T 78% /mnt/samba/Users Can't describe tasks with query: DESCRIBE tasks - Error: Got error 28 from storage engine[root@9745 /]#

4. The list drives in your storage pool (per Amahi platform):

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production

Code: Select all

> Filesystem Size Used Avail Use% Mou > /dev/mapper/vg_9745-lv_root > 20G 19G > tmpfs 1.9G 0 1.9G 0% /dev/shm > /dev/sdi > /dev/mapper/vg_9745-lv_home > > /dev/sdi3 326G 59G 251G 19% /var/hda/files > /dev/sdb1 > /dev/sdc1 917G 660G 258G > /dev/sdd1 1.4T 1.1T 255G 82% /var/hda/files/drives/drive3 > /dev/sde1 1.8T 1.6T 258G 86% /var/hda/files > /dev/sdg1 276G 260G 17G 95% /var/hda/files/driv > /dev/sdh1 917G 661G 256G 73% /var/hda/files/dr > /dev/sdi4 97G 91G 651M 100% /v > /dev/sdf1 1.8T 1.4T 487G 74% /var/hda/files/drives/drive > //127.0.0.1/Books 7.9T 6.2T 1.8T > //127.0.0.1/Home_Movies > 7.9T 6 > //127.0.0.1/Movies 7.9T 6.2T 1.8T 78% /mnt/samba/Movies > //127.0.0.1/Mus > //127.0.0.1/Pictures 7.9T 6.2T 1.8T 78% /mnt/samba/Pictures > //127.0.0.1/Public 7.9T 6.2T 1.8T 78% /mnt/samba/Public > > //127.0.0.1/Software 7.9T 6.2T 1.8T 78% / > //127.0.0.1/TV 7.9T 6.2T 1.8T 78% /mnt/samba/TV > //127.0.0.1/U > Can't describe tasks with query: DESCR bash: Disk: command not found... [root@9745 /]# mysql -u root -phda -e "select * from disk_pool_partitions" hda_production +----+------------------------------+--------------+---------------------+---------------------+ | id | path | minimum_free | created_at | updated_at | +----+------------------------------+--------------+---------------------+---------------------+ | 2 | /var/hda/files/drives/drive1 | 10 | 2011-04-08 17:48:11 | 2011-04-08 17:48:11 | | 3 | /var/hda/files/drives/drive2 | 10 | 2011-04-08 17:48:12 | 2011-04-08 17:48:12 | | 4 | /var/hda/files/drives/drive3 | 10 | 2011-04-08 17:48:12 | 2011-04-08 17:48:12 | | 5 | /var/hda/files/drives/drive4 | 10 | 2011-04-08 17:48:13 | 2011-04-08 17:48:13 | | 6 | /var/hda/files/drives/drive5 | 10 | 2011-04-08 17:48:13 | 2011-04-08 17:48:13 | | 7 | /var/hda/files/drives/drive6 | 10 | 2011-04-08 17:48:15 | 2011-04-08 17:48:15 | | 8 | /var/hda/files/drives/drive9 | 10 | 2011-07-06 01:50:26 | 2011-07-06 01:50:26 | +----+------------------------------+--------------+---------------------+---------------------+

5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste

http://fpaste.org/GH7b/

6. The Greyhole work queue:

greyhole --view-queue

Code: Select all

Can't describe tasks with query: DESCRIBE tasks - Error: Got error 28 from storage engine

By the way, I also have this going on: http://forums.amahi.org/viewtopic.php?f=3&t=3119 at the same time...

Looking at this a little closer, it looks like I'm out of space on my / partition... Running du & df remotely and trying to trouble shoot... Hopefully I can solve this myself, but any pointers would be appreciated... /var/hda/files/ is on a separate partition, so that's not the issue I don't think... odd...

[jbmia]

So, I'm getting a bit closer on this one... the greyhole.log file is 7.1g in size... that seems really high to me... now adding 2Tb drive on to a 6Tb pool and running an fsck and a balance with a bunch of small files I imagine would gen a fair amount of log activity, so maybe this is normally.. but 7.1g is pretty big for a log file...

Anybody have any input on this? Anyway to set a size limit on the file...

Code: Select all

[root@9745 log]# du -sh *.log 4.0K amahi-app-installer.log 4.0K amahi-platform-migration.log 4.0K amahi-sync.log 44K anaconda.log 92K anaconda.program.log 192K anaconda.storage.log 140K anaconda.yum.log 8.0K boot.log 7.2G greyhole.log 2.9M hda-updates.log 12K mysqld.log 4.0K openvpn-status.log 4.0K pm-powersave.log 0 wpa_supplicant.log 88K Xorg.0.log 36K Xorg.9.log 76K yum.log

I put /var/hda/files and /home and /var/media on different partititions as I'd run out of space in the past, but didn't think I'd run into it with log files...

regards,

jbmia

[lrevxl]

Greyhole will roll its logs once a day, or rather logrotate will do this. Once the log gets to a certain size there can also be issues with Greyhole not being able to open / edit the log file.

A huge log file isn't too surprising given the type of load you're running, though. Think about it, if you're transferring 2tb worth of small files in your pool, you're going to have X number of lines in the log per file. Where X is probably > 6 (File opened for write, moving to volume, saving tombstones, etc) so if you're moving, say, 100k files, that's at least 600,000k lines in your text file. Add in a rebalancing call and you'll get plenty more lines in the log file.

What I can say is that re-balancing usually doesn't need to be done very often and once you're beyond the point where you're doing an initial load, the average use case does not create logs this large.

If you're still worried about running into big logs, there is a solution you can implement for yourself. As I said above, Greyhole log rotation is handled by logrotate. So there are two things you'd need to do here.

First, there is the logrotate config for greyhole which lives at /etc/logrotate.d/greyhole, you can see two things here, the log rotation is done daily and there is no size limit. We can leave the log rotation as daily, that's fine. (Log rotate only thinks in terms of days, weeks, months, etc -- not hours.) But we can add in a size limit such as 30M, see below for the modified config.

Code: Select all

/var/log/greyhole.log { daily size 30M missingok rotate 7 compress notifempty }

The above config will rotate whenever log rotate is run and the Greyhole log has exceeded 30mb or the log is older than a day old. The next problem is that most likely logrotate only runs once a day on your system. This is normal and under normal circumstances logrotate only lives under your cron.daily jobs. If you want it to run more often, you need to set that up. But more than likely you don't need logrotate to run globally. We can setup a new cron.hourly job to run.

Under /etc/cron.hourly create a file named 'greyhole' and inside of it you can put:

Code: Select all

#!/bin/sh # Hourly log rotate for Greyhole /usr/sbin/logrotate /etc/logrotate.d/greyhole

Once this is in place logrotate will be executed hourly specifically for greyhole.log and will rotate+compress if the log size is > 30mb.

[johnrbek]

Many thanks for the tips !!

I've implemented the changes and am just waiting to validate they're functioning correctly...

[johnrbek]

Again, many thanks for pointing me in the right direction Irevxl... If it wasn't for your feedback, I'd still be wandering around in clogged hard drive land...

So, with the volume of data I had going into greyhole.log, I ended up having to create a crontab file and set the greyhole logrotate script, provided for the hourly run in your post above, and set it up to run every 15 minutes...

My system drive is set to 20g and greyhole.log would fill up to 8.0g inside an hour so that when the hourly logrotate job ran, there was no room available for the compress step...

I then researched a bit and learned more about cron and crontab and the ability to customize jobs to run more granularly than weekly, daily, or hourly.

Seems to be running fine now... With disk space so cheap though, you have to wonder, for those of use with lots of data and drives, if a much larger system partition would make more sense. I know in my case, I'll never set up a 20g Amahi system partition again... Disk space is relatively cheap and I can help but think I'd be better off with 100g's of system partition and the ability to set that logrotate job back to hourly or even daily instead of in 15 min increments...

Of course, I'd already put my LZ and my /media/TV (in re SageTV recorded TV directory) into separate partitions.. just never foresaw an 8gb greyhole.log file ....

Hope this helps someone..

jbmia