Server locked up, primary drive full.

nstallbaumer
Posts: 5
Joined: Thu Oct 06, 2011 10:03 pm

Server locked up, primary drive full.

Postby nstallbaumer » Wed Nov 23, 2011 9:41 am

Running Amahi on Fedora 12, with a 500 GB internal drive, and a 5.5TB Rosewill enclosed attached via eSata and mounted at /var/hda/files/disks/drive1. Both drives are part of Greyhole and all of my shares are configured to use the pool.

I am running Sabnzbd with Couch Potato and Sick Beard and this morning was checking into why some downloads weren't there and saw that Sabnzbd was reporting that there was 0.0GB available and the download was froze.

I switch over to Amahi web util and check my drives. The internal drive is reporting that there is 20GB available while the 5.5TB is reporting that 4.7TB is available.

At this point with the exception of my stalled download all other services (VPN, SMB, etc) are working without issue.

I ssh'ed into my box as root (I'm at work right now) and initiated a greyhole --fsck at which point it stalled out, and all services became inaccessible and disconnected.

I'm at a loss for what I need to do to get everything back up and running. Thanks.

User avatar
radioz
Posts: 406
Joined: Tue Nov 24, 2009 1:07 pm

Re: Server locked up, primary drive full.

Postby radioz » Wed Nov 23, 2011 6:41 pm

Open up a terminal on your hda and type: df -h (this will display all your partitions and the amount of space on them)

Copy the output and post it here. Perhaps we can help.

nstallbaumer
Posts: 5
Joined: Thu Oct 06, 2011 10:03 pm

Re: Server locked up, primary drive full.

Postby nstallbaumer » Thu Dec 22, 2011 9:32 pm

So I am apparently back in the situation.

Last time this happened, by the time that I returned home and began to look as to what was going on, the fsck had finished, everything was balanced out and after a reboot everything came back up no issue. Everything has been working fine, have downloaded lots of file and haven't had any issues, until today, I am once again showing 0 GB free on my main drive and nothings working again.

I am currently running greyhole --balance and will follow up with a greyhole --fsck reboot and see what happens, but I would like to find a permanent fix so I'm not doing this once a month.

As requested below is the output when I run df -h


Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_amahi-lv_root
457G 434G 0 100% /
tmpfs 502M 0 502M 0% /dev/shm
/dev/sda1 194M 22M 163M 12% /boot
/dev/sdb 5.5T 1.3T 4.0T 25% /var/hda/files/drives/sdb


Thanks

User avatar
moredruid
Expert
Posts: 791
Joined: Tue Jan 20, 2009 1:33 am
Location: Netherlands
Contact:

Re: Server locked up, primary drive full.

Postby moredruid » Fri Dec 23, 2011 3:24 am

login with ssh (as root or become root)

Code: Select all

cd / du -kh --max-depth=1 .
you'll probably see /var being very large

Code: Select all

cd /var du -kh --max-depth=1 .
you'll probably notice /log being very large
you can remove all log files (or just 1 big log file) in /var/log and run "logrotate -f /etc/logrotate.conf" to get everything back in order.
echo '16i[q]sa[ln0=aln100%Pln100/snlbx]sbA0D2173656C7572206968616D41snlbxq' | dc
Galileo - HP Proliant ML110 G6 quad core Xeon 2.4GHz, 4GB RAM, 2x750GB RAID1 + 2x1TB RAID1 HDD

nstallbaumer
Posts: 5
Joined: Thu Oct 06, 2011 10:03 pm

Re: Server locked up, primary drive full.

Postby nstallbaumer » Sat Dec 31, 2011 9:52 am

Greyhole is what is taking up a majority of my 500gb drive.

I removed that drive from the greyhole pool and ran --fsck and --balance, but the files that were stored on that partition were no longer accessible.

How do I remove the 500gb drive leaving the 5.5tb raid array and move the files to it? Because that would obviously solve my drive space issues on the 500gb

User avatar
bigfoot65
Project Manager
Posts: 11924
Joined: Mon May 25, 2009 4:31 pm

Re: Server locked up, primary drive full.

Postby bigfoot65 » Sat Dec 31, 2011 9:59 am

It is not recommended to use Raid and Greyhole. If you removed the drive from the pool, the files are still on the drive in the directory. You will have to remove that directory manually.

Since you have removed it from the pool, it will not longer be used. If there were files on that drive that were not anywhere else, you might wanna manually copy them to a different drive.

The best way to remove a drive from the pool, is to run the command as root user:

Code: Select all

greyhole --going /var/hda/files/drives/drive1
That will ensure any files that exist only on that drive are moved to another one. Then after that, do the following:

Code: Select all

greyhole --gone /var/hda/files/drives/drive1
Then you can remove it from the pool. That is how I do it.

This is a topic best addressed in the Greyhole topic area for future. There are advanced users who watch that area and can provide assistance.
ßîgƒσστ65
Applications Manager

My HDA: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz on MSI board, 16GB RAM, 1TBx1+2TBx2+4TBx2

rcrh
Posts: 50
Joined: Fri Sep 02, 2011 1:40 pm

Re: Server locked up, primary drive full.

Postby rcrh » Fri Mar 02, 2012 3:50 pm

moredruid, I'm trying to follow your approach and am not having much luck.

I've run "logrotate -f /etc/logrotate.conf" and here is the result of the two du commands:

[root@localhost ~]# cd /
[root@localhost /]# logrotate -f /etc/logrotate.conf
[root@localhost /]# du -kh --max-depth=1
25M ./root
34M ./etc
18M ./boot
16K ./lost+found
8.0K ./1. Waiting to Encode
4.0K ./srv
1.5T ./home
14M ./sbin
4.0K ./cgroup
11T ./var
du: cannot access `./proc/11077/task/11077/fd/4': No such file or directory
du: cannot access `./proc/11077/task/11077/fdinfo/4': No such file or directory
du: cannot access `./proc/11077/fd/4': No such file or directory
du: cannot access `./proc/11077/fdinfo/4': No such file or directory
du: cannot access `./proc/27167': No such file or directory
du: cannot access `./proc/27185': No such file or directory
0 ./proc
0 ./sys
8.0K ./opt
8.4M ./bin
122M ./lib
4.0K ./selinux
4.0K ./mnt
4.0K ./media
104K ./tmp
1.8G ./usr
3.3M ./.gem
612K ./dev
12T .
[root@localhost /]# cd /var
[root@localhost var]# du -kh --max-depth=1
12K ./yp
1.2M ./www
252K ./run
11T ./hda
4.0K ./account
20K ./lock
402M ./spool
4.0K ./preserve
4.0K ./opt
4.0K ./local
353M ./log
4.0K ./gdm
20K ./db
330M ./lib
4.0K ./games
456M ./cache
4.0K ./nis
4.0K ./tmp
8.0K ./empty
4.0K ./report
220K ./named
11T .
[root@localhost var]#

And, here's my available space after this:
[root@localhost var]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
50G 48G 0 100% /
tmpfs 1.8G 0 1.8G 0% /dev/shm
/dev/sda1 485M 34M 426M 8% /boot
/dev/mapper/VolGroup-lv_home
1.8T 1.5T 170G 90% /home
/dev/sdb1 1.8T 1.6T 171G 91% /var/hda/drives/drive1
/dev/sdc1 1.8T 1.6T 171G 91% /var/hda/drives/drive3
/dev/sdg3 1.8T 1.6T 172G 91% /var/hda/drives/drive4
/dev/sdd1 1.8T 1.5T 282G 84% /var/hda/drives/drive5
/dev/sde1 1.8T 1.5T 282G 84% /var/hda/drives/drive6
/dev/sdi1 917G 700G 171G 81% /var/hda/drives/drive7
/dev/sdf1 917G 589G 282G 68% /var/hda/drives/drive8
/dev/sdj1 917G 700G 171G 81% /var/hda/drives/drive9
/dev/sdh1 917G 701G 170G 81% /var/hda/drives/drive10
[root@localhost var]#

Any other suggestions on how I can either make space available on my root partition or how I can increase it's size without creating more issues?

Until I fix this available space issue I don't think I can fix the "ruby on rails"/greyhole database errors I'm getting.

Thanks.

User avatar
radioz
Posts: 406
Joined: Tue Nov 24, 2009 1:07 pm

Re: Server locked up, primary drive full.

Postby radioz » Fri Mar 02, 2012 7:06 pm

Are you running Greyhole in addition to using the logical-volume manager?

Post the output of this command:
sudo lvs

This will show all of the logical volumes you have.

I'm having a hard time understanding where all the / space went.

rcrh
Posts: 50
Joined: Fri Sep 02, 2011 1:40 pm

Re: Server locked up, primary drive full.

Postby rcrh » Fri Mar 02, 2012 8:18 pm

If I am running LVM it isn't on purpose. I'm running what I thought was a standard install from the install CD.

Here is what I get from sudo LVM


[root@localhost /]# sudo lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
lv_home VolGroup -wi-ao 1.76t
lv_root VolGroup -wi-ao 50.00g
lv_swap VolGroup -wi-ao 5.19g

You'll notice that it isn't quite consistent with the first listings I had above as I've found some files that appear to be on my landing zone and not moved to the correct drives. I've been removing those files and have freed up a bit of space.

Any further help you can provide would be great.

Richard

User avatar
radioz
Posts: 406
Joined: Tue Nov 24, 2009 1:07 pm

Re: Server locked up, primary drive full.

Postby radioz » Fri Mar 02, 2012 9:50 pm

I'm not a greyhole expert, but it sounds like you must have your landing zone on your OS drive.

Search around the forums, there might be a way to move your landing zone to a different drive.
If your LZ fills up, there is no room left for the OS which can lead to problems.

I don't run greyhole myself, so I won't be much help to you.

Good luck.

Who is online

Users browsing this forum: No registered users and 10 guests