SOLVED: Pool Drive Failing - What are my next steps?

[ksburgess]

One of the HDD in my 2-drive pool is failing and my HDA will not boot properly after the SMART drive failure warning is displayed upon boot. It will boot into emergency mode.

Only one my shares (business files) is mirrored on both disk drives. I have currently mounted an external HDD and am attempting to copy files from each pool drive (I'm not sure which one is the failing drive, but I can browse the file structure in the gh directory on both). I also have one share that is not on a pool drive, it's on the main HDD (where Amahi is installed) and I'm copying that one just in case.

I cannot exactly copy the file structure of the pool drives because my external HDD isn't big enough, but I will copy all shares I can.

Once this happens, what are my next steps? Even with the failing drive unplugged (physically we know which drive is failing but I don't know if it's Pool Drive One or Two) my HDA will not boot.

Is the HDA completely hosed and I need to start over Or can I bring in a new pool drive (or likely will replace both pool drives)?

[bigfoot65]

Is the HDA completely hosed and I need to start over Or can I bring in a new pool drive (or likely will replace both pool drives)?

No the HDA should be fine. You can remove bad drives and replace them. Greyhole makes things even easier since there are copies on multiple drives.

Will it still boot with the bad drive connected? If so, determine which drive is bad and comment out that entry in /etc/fstab.

The moving data out of the storage pool wiki guidance should be helpful.

[ksburgess]

It will only boot in emergency mode, so I can't access the HDA interface. I know that it has minimal services running in this mode, so I'm not sure if I'll be able to tell greyhole to remove the bad drive

It's still copying files to the external HDD right now but it's on the last share so I can try and work with it shortly. I'm frustrated that the one bad pool drive has made it so it will only boot in emergency mode - the OS drive is just fine

[bigfoot65]

I'm frustrated that the one bad pool drive has made it so it will only boot in emergency mode

This is standard Linux behavior and nothing to do with Amahi or Greyhole. Anytime a drive fails, it will boot to emergency mode.

You could boot the HDA using a Live CD, such as Ubuntu and the edit the /etc/fstab to remove the bad drive. Once booted, you can mount your HDA OS drive and make the change. I had to do this once myself.

From emergency mode, you have extremely limited command line capability. Greyhole is not one of them.

EDIT: This might help: edit /etc/fstab system file in linux maintenance mode

[ksburgess]

Thanks bigfoot - I'm not frustrated with amahi or greyhole really, just frustrated mostly that the drive decided to fail today, right after I finished work and moved files to the HDA, before those files had a chance to backup to the cloud, etc... just bad timing, but then, when is a HDD failure ever good timing.

I did try a LiveUSB boot but that wasn't working - I think it was my flash drive as it's a few years old. So downloading an Ubuntu ISO and will make a new flash drive (very, very slow internet).

My last share is still copying. Thanks for the links. About to turn in for the night but will look at it tomorrow - maybe with better luck on a new day to get it squared away and pick a new drive to order

[ksburgess]

Okay this morning I feel a bit stupid but very relieved.

I created the Ubuntu LiveUSB, and restarted to boot into that. Apparently my BIOS had gotten changed back from booting from USB first, so it went into emergency mode. I decided to scour the log file to try and find which disk was failing and noticed a line I didn't see yesterday - failed dependency on an external HDD I'd mounted with Disk Wizard (so that Crashplan could do backups of the business docs share to that continuously)... I'd unplugged it yesterday morning when we were vacuuming around the HDA / router etc. and forgotten to plug it back in!!!

So again, I feel stupid, but upon plugging that in, my HDA now boots just fine. Still getting the SMART failure notice from the BIOS, but this gives me time to order a new drive and then follow the instructions in your link to swap it out. Thanks Bigfoot

And it did allow me to see that SMART failure warning, which I don't normally see since we run the HDA headless. Will have to set a reminder to check SMART drive status periodically.

Could I suggest that we add a line to the wiki page for the drive wizard: https://wiki.amahi.org/index.php/New_Disk_Wizard that if you remove an external HDD that has been mapped in with the wizard, a dependency will fail on your HDA? It would have been good to read that when I referenced that page before adding the external HDD in a few weeks ago!

[bigfoot65]

And it did allow me to see that SMART failure warning, which I don't normally see since we run the HDA headless. Will have to set a reminder to check SMART drive status periodically.

You might be interested in the disk error alert emails tutorial.

Could I suggest that we add a line to the wiki page for the drive wizard: New Disk Wizard

Done. There is now a cautionary message in the Introduction paragraph of the page.

Glad you got it resolved. Will mark this one as solved.