Clear stuck queue?/Remove all tombstones and re-fsck

[lrevxl]

As root you can run `blkid -L` and this will spit out the drive, filesystem type, mount point, and UUID, that should give tell you which drive has the LZ partition.

[AndyNJ]

Well, according to this, sdb is not the landing zone disk (I didn't think it was, but wasn't sure). My LZ is on sda.

Should I do a --going on sdb or will that cause me more problems since my database isn't currently in an accurate state?

[lrevxl]

The database doesn't maintain state information, really. That all resides in configs and graveyards. You could try to do a --going on sdb, but if the drive is already in a state where the system can't read from it (I'm assuming that's the case -- you'd know better than I, having access to the system), I don't think it's going to do you a whole lot of good. You could cut sdb from the greyhole config and restart greyhole, that would get you past the point where it's failing on I/O, but any files that exist only on that drive would not be accessible -- granted they're not accessible now.

[radioz]

Post the output of 'df -h'. That should show the drives/partitions.
Also post the output of: 'sudo fdisk -l'

[AndyNJ]

I think I'd at least like to give greyhole a shot at trying to get what it can off of the drive. I have all shares set to keep 2 copies, but a couple of them weren't set to keep an extra copy until recently, so I'm not entirely sure that everything got duplicated given this mess.

Is there any way to find out what files you're missing? Like say a drive just died entirely and you couldn't access it at all, can greyhole give you a list of files for which it doesn't have (or can't find) any copies of?

[lrevxl]

As radioz said, post the output of `df -h` and that'll give us an idea of what/where exactly /dev/sdb is mounted to (I'm assuming it's a pool drive -- but we should verify this). If it is a pool drive, can you actually cd to the mounted path and look at it? If you can't read from the drive Greyhole isn't going to be able to get anything off of it either.

You can get a list of files with 0 copies by running an fsck with the --email-report option, it'll provide, among other things, that list. If you haven't changed the defaults, this email report will get sent to root on the local system, so as root you can just type `mail` and look for the most recent email from Greyhole.

[AndyNJ]

Now that I'm home, I was able to get a better look at everything.

That drive is definitely failing (SMART is throwing up all kinds of red flags). It's a 2TB WD Green that's pretty new (a few months). Must be defective. I can somewhat get into it to read the files, but it's slow.

I kicked off the --going command to at least try it and see what happens. In theory, I wouldn't be losing any data because everything is duped to two drives. However, that's assuming that greyhole was actually able to make that happen before the drive starting causing problems. I didn't set the duplication on all shares until a week or two ago (the most important stuff was already). With the drive being almost entirely full, this could take a little while.

Thanks for all the info, guys. It helps a lot.