SOLVED: Server has been constantly running for 2 days

[tomtheappraiser]

I've noticed my server running constantly now for 2 days. I satrted monitoring the Greyhole Logmon and noticed it is going through every file on my server and trying to write it, but failing. there are 2 sets of warnings I get. The following is the exampls of each for the same file:

Feb 19 07:06:40 WARN write: PHP Warning [2]: file_put_contents(): Only 0 of 386 bytes written, possibly out of free disk space in /usr/bin/greyhole on line 4842; BT: greyhole[L3585] run() => greyhole[L3558] execute_next_task() => greyhole[L3871] gh_write(Docs,/win 8.dotx,242531) => greyhole[L4405] gh_write_process_metafiles(3,array (),Docs,/win 8.dotx,/var/hda/files/docs/win 8.dotx,12033,242531) => greyhole[L4445] create_copies_from_metafiles(array ( '/home/gh/Docs/win 8.dotx' => stdClass::__set_state(array( 'path' => '/home/gh/Docs/win 8.dotx', 'is_linked' => false, 'state' => 'Gone', )), '/var/hda/files/drives/WD2TBsdb/gh/Docs/win 8.dotx' => stdClass::__set_state(array( 'path' => '/var/hda/files/drives/WD2TBsdb/gh/Docs/win 8.dotx', 'is_linked' => false, 'state' => 'Pending', )), '/var/hda/files/drives/WD3TB/gh/Docs/win 8.dotx' => stdClass::__set_state(array( 'path' => '/var/hda/files/drives/WD3TB/gh/Docs/win 8.dotx', 'is_linked' => false, 'state' => 'Pending', )),),Docs,/win 8.dotx,/var/hda/files/docs/win 8.dotx) => greyhole[L4488] save_metafiles(Docs,/,win 8.dotx,array ( '/home/gh/Docs/win 8.dotx' => stdClass::__set_state(array( 'path' => '/home/gh/Docs/win 8.dotx', 'is_linked' => false, 'state' => 'Gone', )), '/var/hda/files/drives/WD2TBsdb/gh/Docs/win 8.dotx' => stdClass::__set_state(array( 'path' => '/var/hda/files/drives/WD2TBsdb/gh/Docs/win 8.dotx', 'is_linked' => false, 'state' => 'Pending', )), '/var/hda/files/drives/WD3TB/gh/Docs/win 8.dotx' => stdClass::__set_state(array( 'path' => '/var/hda/files/drives/WD3TB/gh/Docs/win 8.dotx', 'is_linked' => false, 'state' => 'Pending', )),)) => greyhole[L4842] file_put_contents(/home/gh/.gh_metastore/Docs//win 8.dotx,a:3:{i:0;O:8:"stdClass":3:{s:4:"path";s:24:"/home/gh/Docs/win 8.dotx";s:9:"is_linked";b:0;s:5:"state";s:4:"Gone";}i:1;O:8:"stdClass":3:{s:4:"path";s:49:"/var/hda/files/drives/WD2TBsdb/gh/Docs/win 8.dotx";s:9:"is_linked";b:0;s:5:"state";s:7:"Pending";}i:2;O:8:"stdClass":3:{s:4:"path";s:46:"/var/hda/files/drives/WD3TB/gh/Docs/win 8.dotx";s:9:"is_linked";b:0;s:5:"state";s:7:"Pending";}})
Feb 19 07:06:40 WARN write: Failed file copy (cont). Will try to re-process this write task, since the source file seems intact.
Feb 19 07:06:40 INFO write: Now working on task ID 0: write Docs/win 8.dotx
Feb 19 07:06:40 INFO write: File created: Docs//win 8.dotx

and then later:

Feb 19 07:06:42 WARN write: Failed file copy. Will mark this metadata file 'Gone'.
Feb 19 07:06:42 ERROR write: Failed file copy (cont). We already retried this task. Aborting.
Feb 19 07:06:42 INFO write: Now working on task ID 238142: write Docs/Win 8.dotx
Feb 19 07:06:42 INFO write: File created: Docs//Win 8.dotx
Feb 19 07:06:42 WARN write: Failed file copy. Will mark this metadata file 'Gone'.
Feb 19 07:06:42 WARN write: Failed file copy (cont). Will try to re-process this write task, since the source file seems intact.
Feb 19 07:06:42 INFO write: Now working on task ID 0: write Docs/Win 8.dotx
Feb 19 07:06:42 INFO write: File created: Docs//Win 8.dotx
Feb 19 07:06:42 WARN write: Failed file copy. Will mark this metadata file 'Gone'.
Feb 19 07:06:42 ERROR write: Failed file copy (cont). We already retried this task. Aborting.

I'm using Amahi 23. I haven't made any hardware or software changes recently. All of my drives have room except /home, which is full.

[bigfoot65]

Not sure what the cause could be.

Might check that you have plenty of space left on the drives, to include the Landing Zone. Are the permissions correct?

Also might find some help in the Greyhole Troubleshooting page in the #Amahi wiki.

Check Greyhole stats and the .conf file. Are you using the latest version which is 0.10.0?

[tomtheappraiser]

OK...wall of text coming. I followed the Greyhole troubleshooter, but the end result didn't seem to be the problems I was having. Here is what I found:

4.7.10-100.fc23.x86_64
samba-4.3.11-1.fc23.x86_64
amahi-greyhole-0.10.0-1.x86_64



[root@localhost ~]# fpaste /etc/samba/smb.conf
Uploading (5.0KiB)...
https://da.gd/cdTEn -> https://paste.fedoraproject.org/paste/V ... ivL9gydE=/
[root@localhost ~]# fpaste /etc/greyhole.conf
Uploading (1.5KiB)...
https://da.gd/B4D4 -> https://paste.fedoraproject.org/paste/2 ... ivL9gydE=/
[root@localhost ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=3811308k,nr_inodes=952827,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/sda3 on / type xfs (rw,relatime,attr2,inode64,noquota)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13713)
tmpfs on /tmp type tmpfs (rw)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
/dev/sdb1 on /var/hda/files/drives/WD2TBsdb type ext4 (rw,relatime,data=ordered)
/dev/sdd1 on /var/hda/files/drives/CrashplanDisk type ext4 (rw,relatime,data=ordered)
/dev/sdc1 on /var/hda/files/drives/WD3TB type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot type ext4 (rw,relatime,stripe=4,data=ordered)
/dev/sda5 on /home type xfs (rw,relatime,attr2,inode64,noquota)
none on /var/spool/greyhole/mem type tmpfs (rw,relatime,size=4096k)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=764072k,mode=700,uid=1000,gid=100)
[root@localhost ~]# fdisk -1
fdisk: invalid option -- '1'

Usage:
fdisk [options] <disk> change partition table
fdisk [options] -l [<disk>] list partition table(s)

Display or manipulate a disk partition table.

Options:
-b, --sector-size <size> physical and logical sector size
-B, --protect-boot don't erase bootbits when creating a new label
-c, --compatibility[=<mode>] mode is 'dos' or 'nondos' (default)
-L, --color[=<when>] colorize output (auto, always or never)
colors are enabled by default
-l, --list display partitions end exit
-o, --output <list> output columns
-t, --type <type> recognize specified partition table type only
-u, --units[=<unit>] display units: 'cylinders' or 'sectors' (default)
-s, --getsz display device size in 512-byte sectors [DEPRECATED]
--bytes print SIZE in bytes rather than in human readable format
-w, --wipe <mode> wipe signatures (auto, always or never)

-C, --cylinders <number> specify the number of cylinders
-H, --heads <number> specify the number of heads
-S, --sectors <number> specify the number of sectors per track

-h, --help display this help and exit
-V, --version output version information and exit

Available columns (for -o):
gpt: Device Start End Sectors Size Type Type-UUID Attrs Name UUID
dos: Device Start End Sectors Cylinders Size Type Id Attrs Boot End-C/H/S
Start-C/H/S
bsd: Slice Start End Sectors Cylinders Size Type Bsize Cpg Fsize
sgi: Device Start End Sectors Cylinders Size Type Id Attrs
sun: Device Start End Sectors Cylinders Size Type Id Flags

For more details see fdisk(8).
[root@localhost ~]# fdisk -l
Disk /dev/sda: 298.1 GiB, 320072933376 bytes, 625142448 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x7579c167

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 1026047 1024000 500M 83 Linux
/dev/sda2 1026048 16494591 15468544 7.4G 82 Linux swap / Solaris
/dev/sda3 16494592 121352191 104857600 50G 83 Linux
/dev/sda4 121352192 625141759 503789568 240.2G 5 Extended
/dev/sda5 121354240 625141759 503787520 240.2G 83 Linux


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x968de395

Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 3907028991 3907026944 1.8T 83 Linux


Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 77C2F59D-B65C-4C38-A052-617662EC3114

Device Start End Sectors Size Type
/dev/sdc1 2048 5860532223 5860530176 2.7T Linux filesystem


Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xccfb05e2

Device Boot Start End Sectors Size Id Type
/dev/sdd1 2048 1953523711 1953521664 931.5G 83 Linux
[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.7G 0 3.7G 0% /dev
tmpfs 3.7G 4.0K 3.7G 1% /dev/shm
tmpfs 3.7G 792K 3.7G 1% /run
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/sda3 50G 27G 24G 54% /
tmpfs 3.7G 64K 3.7G 1% /tmp
/dev/sdb1 1.8T 1.2T 534G 70% /var/hda/files/drives/WD2TBsdb
/dev/sdd1 917G 72M 871G 1% /var/hda/files/drives/CrashplanDisk
/dev/sdc1 2.7T 1.4T 1.3T 52% /var/hda/files/drives/WD3TB
/dev/sda1 477M 106M 342M 24% /boot
/dev/sda5 241G 241G 20K 100% /home
none 4.0M 0 4.0M 0% /var/spool/greyhole/mem
tmpfs 747M 0 747M 0% /run/user/1000
[root@localhost ~]# greyhole --stats

Greyhole Statistics
===================

Storage Pool
Total - Used = Free + Trash = Possible
/var/hda/files/drives/WD2TBsdb/gh: 1834G - 1207G = 534G + 80G = 614G
/home/gh: 240G - 240G = 0G + 0G = 0G
/var/hda/files/drives/WD3TB/gh: 2751G - 1346G = 1265G + 219G = 1484G
==========================================
Total: 4824G - 2793G = 1799G + 300G = 2099G

[root@localhost ~]# mysql -u root -phda -e "select * from disk_pool_partitions" hda_production +----+--------------------------------+--------------+---------------------+---------------------+
| id | path | minimum_free | created_at | updated_at |
+----+--------------------------------+--------------+---------------------+---------------------+
| 2 | /var/hda/files/drives/WD2TBsdb | 10 | 2016-10-14 23:09:05 | 2016-10-14 23:09:05 |
| 3 | /home | 10 | 2016-10-14 23:09:07 | 2016-10-14 23:09:07 |
| 4 | /var/hda/files/drives/WD3TB | 10 | 2016-10-15 17:13:34 | 2016-10-15 17:13:34 |


[root@localhost ~]# mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions"
ERROR 1046 (3D000) at line 1: No database selected


[root@localhost ~]# hda_production | grep -v 'concat(' | xargs ls -la | pastebinit
-bash: pastebinit: command not found
-bash: hda_production: command not found
xargs: ls: terminated by signal 13

[root@localhost ~]# greyhole --view-queue

Greyhole Work Queue Statistics
==============================

This table gives you the number of pending operations queued for the Greyhole daemon, per share.

Write Delete Rename Check
Appraisal 1,143 0 0 0
Docs 50 0 0 2
Movies 2,256 4 2 368
Music 64 0 0 0
Pathfinder 32 7 5 0
Pictures 0 0 0 0
Software 3,452 0 0 0
TV 4 0 0 0
Videos 0 0 0 0
==============================================
Total 7,001 11 7 370
Write Delete Rename Check

The following is the number of pending operations that the Greyhole daemon still needs to parse.
Until it does, the nature of those operations is unknown.
Spooled operations that have been parsed will be listed above and disappear from the count below.

Spooled 0

[bigfoot65]

OK...wall of text coming.

While we appreciate the details, in future please attach as a file vs posting the long list of text. This makes it easier for all to read.

I had a similar problem since my last reply, but not sure the root cause. What I had to do was empty the Greyhole queue and restart operations. The task table in the greyhole database holds all pending tasks. This can be done using phpmyadmin App, adminer App, or manually as noted in the Greyhole Troubleshooting, Greyhole Work Queue wiki page.

Once I did that, everything went back to normal. I think this may have resulted from the recent update from Greyhole version 0.9.67 to 0.10.0. BTW, there is a new version, 0.10.2 that has been released in the repo. To get it, you may need to do:

Code: Select all

sudo dnf clean all sudo dnf update

[tomtheappraiser]

I tried doing the Troubleshooter, and it didn't work. The next option was to wipe out the Landing zone, but I don't understand exactly which shares are where and which are supposed to be moved.
I got both the phpMyAdmin app and Adminer, but I can't find anything that does that.
Would you be able to walk me through it?

Also, sorry about the WOT. For future reference what type of file should I post? Notebook?


Thanks.

[bigfoot65]

Might be easier to follow the guidance to do it manually as documented in the last link I listed. The steps are pretty clear.

As for file type, notepad is fine.

BTW the path for each share is listed in the dashboard in the shares tab. Expand the share and you should see it. See Shares extra parameters wiki page for a screen shot.

[tomtheappraiser]

OK,

My problem is I don't understand what I'm doing at all. From what I'm reading and what your saying, I am supposed to move my shares to another drive? I don't have a big enough drive to move all of the data in my shares.

This is the first line:
cp -rv /var/hda/files/shareName /mnt/timemachine/landingzonebackup/shareName

Do I have to do this for each share individually? I have a Crashplan 1TB disk that I haven't used yet that is not part of Greyhole. I could move some it there. What would the path be?
I thought I was just clearing the Greyhole LZ (which is on /).

I'm so confused.

Here is a list of my shares and their paths

• Appraisal \\hda\Appraisal
  Crashplan \\hda\Crashplan
  Docs \\hda\Docs
  Movies \\hda\Movies
  Music \\hda\Music
  Pathfinder \\hda\Pathfinder
  Pictures \\hda\Pictures
  Software \\hda\Software
  TV \\hda\TV
  Videos \\hda\Videos

with /var/hda/files preceding each share name.

Here is a list of all of my mounted disks and their contents:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 298.1G 0 disk
├─sda1 8:1 0 500M 0 part /boot
├─sda2 8:2 0 7.4G 0 part [SWAP]
├─sda3 8:3 0 50G 0 part /
├─sda4 8:4 0 1K 0 part
└─sda5 8:5 0 240.2G 0 part /home
sdb 8:16 0 1.8T 0 disk
└─sdb1 8:17 0 1.8T 0 part /var/hda/files/drives/WD2TBsdb
sdc 8:32 0 2.7T 0 disk
└─sdc1 8:33 0 2.7T 0 part /var/hda/files/drives/WD3TB
sdd 8:48 0 931.5G 0 disk
└─sdd1 8:49 0 931.5G 0 part /var/hda/files/drives/CrashplanDisk

My LZ is on sda3. The disk NOT in the Greyhole is sdd1

So what do I substitute in that command line I posted above?

[bigfoot65]

No no no.

Just do the MySQL portion for stuck work queue. That's what the link points to for me. Don't do the Queue Still Stuck part.

That should clear the database pending Greyhole tasks.

The restart Greyhole and monitor the log.

[tomtheappraiser]

I feel like an idiot...but which specific part is the MySql section? The only thing I see is talking about shares.

[bigfoot65]

No worries.

It's the area titled "Greyhole Work Queue" and sows commands for MySQL.