Lost tasks_completed table - out of space?

[eappell]

I've got some big problems in gryehole that just happened in the past week. It kept reporting out of space on device, but I can't see where I'm out of space on ANY of my devices when I do a df -h. In the process of trying to fix this, and reading all sorts of posts on the topic, I dropped the tasks_completed table, which appears to have made things much worse. I've tried re-creating it using the sql, but I cannot get mysql to allow me to run it. It is still giving me out of space errors when I try to get stats or queue. I'm very confused, and about to throw in the towel. Hoping someone can help me figure out how to get this back to working... Here's the results of running the Greyhole Troubleshooting steps:

Greyhole troubleshooting:

2. $ pastebinit /etc/samba/smb.conf
http://paste.ubuntu.com/6954709/
2. pastebinit /etc/greyhole.conf
error: Unable to read from /etc/greyhole.conf​ - (will attach the file.)
2. $ mount (Attaching result of mount command.)

3. $ fdisk -l (no result)
3. $ df -h:

Code: Select all

$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 42G 27G 13G 68% / udev 3.9G 12K 3.9G 1% /dev tmpfs 1.6G 22M 1.6G 2% /run none 5.0M 0 5.0M 0% /run/lock none 3.9G 80K 3.9G 1% /run/shm /dev/sdb1 932G 623G 309G 67% /var/hda/drives/internal_1 /dev/sdd1 1.8T 1.5T 292G 84% /var/hda/drives/internal_2 /dev/sdc1 917G 587G 284G 68% /var/hda/drives/internal_3 /dev/sde1 2.7T 2.3T 325G 88% /var/hda/drives/SG3TB /dev/sda6 1.5T 1.1T 322G 77% /var/hda/drives/BigData /dev/sda4 327G 5.5G 305G 2% /home /dev/sdj1 917G 549G 322G 64% /var/hda/drives/external_5 /dev/sdk1 1.8T 1.4T 322G 82% /var/hda/drives/external_4 /dev/sdl1 1.4T 984G 322G 76% /var/hda/drives/external_3 /dev/sdm1 1.8T 1.4T 321G 82% /var/hda/drives/external_2 /dev/sdn1 917G 591G 281G 68% /var/hda/drives/external_1 //127.0.0.1/Audiobooks 15T 12T 3.1T 80% /mnt/samba/Audiobooks //127.0.0.1/Books 15T 12T 3.1T 80% /mnt/samba/Books //127.0.0.1/Docs 1.8T 1.6T 292G 85% /mnt/samba/Docs //127.0.0.1/Downloads 15T 12T 3.1T 80% /mnt/samba/Downloads //127.0.0.1/Greyhole Trash 15T 12T 3.1T 80% /mnt/samba/Greyhole Trash //127.0.0.1/Pictures 15T 12T 3.1T 80% /mnt/samba/Pictures //127.0.0.1/Software 15T 12T 3.1T 80% /mnt/samba/Software //127.0.0.1/TV 15T 12T 3.1T 80% /mnt/samba/TV //127.0.0.1/TimeMachine 327G 23G 305G 7% /mnt/samba/TimeMachine //127.0.0.1/Training 15T 12T 3.1T 80% /mnt/samba/Training //127.0.0.1/Videos 15T 12T 3.1T 80% /mnt/samba/Videos

3. $ greyhole --stats
Can't describe tasks with query: DESCRIBE tasks - Error: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql_64f_0.MYI' (Errcode: 28)

4. $ mysql -u root -phda -e "select * from disk_pool_partitions" hda_production
ERROR 1049 (42000): Unknown database 'hda_production'
Could not open file /var/cache/apt/archives/partial/greyhole_0.9.44-1_amd64.deb - open (28: No space left on device) [IP: 108.162.197.125 80]

5. $ mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste
fpaste: command not found
xargs: ls: terminated by signal 13

6. $ greyhole -q
Can't describe tasks with query: DESCRIBE tasks - Error: SQLSTATE[HY000]: General error: 1 Can't create/write to file '/tmp/#sql_64f_0.MYI' (Errcode: 28)

6. $ tail /var/log/greyhole.log
tail: cannot open `/var/log/greyhole.log' for reading: No such file or directory
$ ls /var/log/g* -lh
-rw-rw-rw- 1 root root 3.0K Feb 17 17:10 /var/log/greyhole.log
-rw-r--r-- 1 root root 79M Feb 6 06:45 /var/log/greyhole.tar.gz

(Clearly the file is there, but cannot be read)

None of the sql stuff has worked for me. I get access denied errors, and I don't know what else to do beyond this point...

Thanks for your help!
Eddie

[bigfoot65]

Might need to execute tasks as privileged user, i.e. sudo.

As for dropping the completed_tasks table, where did you find that guidance? I could see maybe deleting rows, but not dropping the table as that would break Greyhole as you found out.

To put it back, you should ne able to find the create table syntax in the following file:
/usr/share/greyhole/schema-mysql.sql\

You might find it easier to install a MySQL front end and execute the commands. We have apps in the Amahi store, i.e. phpMyAdmin, Adminer, Chive, etc. I believe they are all free as well.

Once you get that all sorted out, try restarting Greyhole and see what the logs reflect. Need to tackle one problem at a time here.

[diggityDawg]

Thanks for helping out - I know this is a bit of a mess at this point... When I try to run the sql to create that table in mysql I get an error indicating I'm out of space:

Code: Select all

mysql> source /home/homeserver/tasks_completed.sql ERROR 1005 (HY000): Can't create table 'tasks_completed' (errno: 28)

Here's the sql in that file:

Code: Select all

CREATE TABLE `tasks_completed` ( `id` bigint(20) unsigned NOT NULL, `action` varchar(10) NOT NULL, `share` tinytext NOT NULL, `full_path` tinytext, `additional_info` tinytext, `complete` enum('yes','no') NOT NULL, `event_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ) ENGINE=MyISAM DEFAULT CHARSET=latin1

What next? I think if I could figure out what device or partition is full and clean that up things might start working again. But everything I have tried doesn't show any full partitions...

[bigfoot65]

Need to ensure you are in the correct database.

Code: Select all

mysql -uroot -phda use greyhole; CREATE TABLE `tasks_completed` ( `id` bigint(20) unsigned NOT NULL, `action` varchar(10) NOT NULL, `share` tinytext NOT NULL, `full_path` tinytext, `additional_info` tinytext, `complete` enum('yes','no') NOT NULL, `event_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ) ENGINE=MyISAM DEFAULT CHARSET=latin1;

If that still does not work, will have to figure out something else. If you do not see any partition full, then seems odd that you receive that message.

[diggityDawg]

I agree!! There may be some commands I'm unaware of that will show hidden partitions?? I don't know, just trying to think of whatever else I can try to resolve this... I tried the commands as you suggested and got the same error (errno: 28):

Code: Select all

# mysql -uroot -phda (cutting out welcome text...) mysql> use greyhole Database changed mysql> CREATE TABLE `tasks_completed` ( -> `id` bigint(20) unsigned NOT NULL, -> `action` varchar(10) NOT NULL, -> `share` tinytext NOT NULL, -> `full_path` tinytext, -> `additional_info` tinytext, -> `complete` enum('yes','no') NOT NULL, -> `event_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP -> ) ENGINE=MyISAM DEFAULT CHARSET=latin1; ERROR 1005 (HY000): Can't create table 'tasks_completed' (errno: 28) mysql>

If this helps at all, here's an exhaustive df command response:

Code: Select all

# df -ahT Filesystem Type Size Used Avail Use% Mounted on /dev/sda2 ext4 42G 27G 13G 68% / proc proc 0 0 0 - /proc sysfs sysfs 0 0 0 - /sys none fusectl 0 0 0 - /sys/fs/fuse/connections none debugfs 0 0 0 - /sys/kernel/debug none securityfs 0 0 0 - /sys/kernel/security udev devtmpfs 3.9G 12K 3.9G 1% /dev devpts devpts 0 0 0 - /dev/pts tmpfs tmpfs 1.6G 24M 1.6G 2% /run none tmpfs 5.0M 0 5.0M 0% /run/lock none tmpfs 3.9G 80K 3.9G 1% /run/shm /dev/sdb1 fuseblk 932G 623G 309G 67% /var/hda/drives/internal_1 /dev/sdd1 ext4 1.8T 1.5T 292G 84% /var/hda/drives/internal_2 /dev/sdc1 ext4 917G 587G 284G 68% /var/hda/drives/internal_3 /dev/sde1 ext4 2.7T 2.3T 325G 88% /var/hda/drives/SG3TB /dev/sda6 ext4 1.5T 1.1T 322G 77% /var/hda/drives/BigData /dev/sda4 ext4 327G 5.5G 305G 2% /home /dev/sdj1 ext4 917G 549G 322G 64% /var/hda/drives/external_5 /dev/sdk1 ext4 1.8T 1.4T 322G 82% /var/hda/drives/external_4 /dev/sdl1 ext4 1.4T 984G 322G 76% /var/hda/drives/external_3 /dev/sdm1 ext4 1.8T 1.4T 321G 82% /var/hda/drives/external_2 /dev/sdn1 ext4 917G 591G 281G 68% /var/hda/drives/external_1 binfmt_misc binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc //127.0.0.1/Audiobooks cifs 15T 12T 3.1T 80% /mnt/samba/Audiobooks //127.0.0.1/Books cifs 15T 12T 3.1T 80% /mnt/samba/Books //127.0.0.1/Docs cifs 1.8T 1.6T 292G 85% /mnt/samba/Docs //127.0.0.1/Downloads cifs 15T 12T 3.1T 80% /mnt/samba/Downloads //127.0.0.1/Greyhole Trash cifs 15T 12T 3.1T 80% /mnt/samba/Greyhole Trash //127.0.0.1/Pictures cifs 15T 12T 3.1T 80% /mnt/samba/Pictures //127.0.0.1/Software cifs 15T 12T 3.1T 80% /mnt/samba/Software //127.0.0.1/TV cifs 15T 12T 3.1T 80% /mnt/samba/TV //127.0.0.1/TimeMachine cifs 327G 23G 305G 7% /mnt/samba/TimeMachine //127.0.0.1/Training cifs 15T 12T 3.1T 80% /mnt/samba/Training //127.0.0.1/Videos cifs 15T 12T 3.1T 80% /mnt/samba/Videos

Again, no empty partitions, although there are some with 0 bytes...?

Is that helpful at all?

[bigfoot65]

Has the box been rebooted since all this started? Not sure what to suggest next. Will have to think about this one for a bit.

[diggityDawg]

Yes, it has been rebooted several times... Thanks again for your help!

[bigfoot65]

See if this article helps.
http://www.cyberciti.biz/faq/mysql-erro ... -solution/

[eappell]

I went through the entire article, and did everything it requested. I then went back into mysql as root and ran the create command and it worked, as far as I can tell the table was created. Then I quit mysql and restarted greyhole. But when I try to get stats I get the same error about no space in /tmp.

I tried to create a new, empty file in /tmp, and I got an error in nano: Error writing testfile: No space left on device - so there's definitely something up with /tmp. It's not a device or share. It looks just like a directory, but in iTerm it's the only directory in the root that has a green background. Not sure what that means...

I've tried removing the directory so I could recreate it, but I can't remove it no matter what I try. And the only file in it is greyhole_df_error.log, and it's 0 bytes. As soon as I delete that file, it gets recreated, and greyhole is NOT running. Very weird... I don't know what else to do. I believe something is reporting that /tmp is full, and I have no way of figuring out what is causing that, and no way to fix it.

[eappell]

I've finally made some headway. I found this article, so I did a df -i, and sure enough my /dev/sda2 is full:

Code: Select all

# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 2779840 2779840 0 100% / udev 1009233 709 1008524 1% /dev tmpfs 1011569 751 1010818 1% /run none 1011569 4 1011565 1% /run/lock none 1011569 5 1011564 1% /run/shm /dev/sdb1 323849288 90981 323758307 1% /var/hda/drives/internal_1 /dev/sdd1 122101760 1075582 121026178 1% /var/hda/drives/internal_2 /dev/sdc1 61054976 677528 60377448 2% /var/hda/drives/internal_3 /dev/sdi1 61054976 216635 60838341 1% /var/hda/drives/external_1 /dev/sdg1 91578368 73322 91505046 1% /var/hda/drives/external_3 /dev/sdf1 122101760 64034 122037726 1% /var/hda/drives/external_4 /dev/sdh1 122101760 142808 121958952 1% /var/hda/drives/external_2 /dev/sdj1 183148544 74801 183073743 1% /var/hda/drives/SG3TB /dev/sda6 97427456 60686 97366770 1% /var/hda/drives/BigData /dev/sde1 61054976 98484 60956492 1% /var/hda/drives/external_5 /dev/sda4 21766144 250327 21515817 2% /home

So now I just have to figure out what is taking up that space and how to remove it... If anyone has any advice, let me know. I'll try to follow the steps outlined in the article and see if I can free up the space.