Page 1 of 1

Root directory full, no longer boots

Posted: Mon Mar 15, 2010 8:10 pm
by Coolbeans
I'm still new at this so I apologize in advance for things I may have looked over, but I'm at my wits end.

Yesterday I got a message that my /root directory was full. No big deal I think, maybe some temp files or something are hogging space? Rebooted (first mistake) and it will no longer finish booting, just sits at the login screen teasing me.

tried the recovery option via the install DVD, deleted the /home directories (was intending on removing them anyways - maybe second mistake?) to free up some space to let it boot up. No dice. Slightly faster but not enough to log in.

I can't figure out where to go or what to remove. Sadly I had though the best course of action was to put the main OS on a CF card (8gb) and the rest of the shares on separate drives for funzies.

I tried using a Mint live CD but it can't even mount the /root drive, ext4 not supported or some jazz like that.

I'm thinking I may have messed it all up and a re-install is the only way to fix things. If anyone has any ideas though I am all ears :)

Cheers,

Andrew

Re: Root directory full, no longer boots

Posted: Tue Mar 16, 2010 2:30 am
by moredruid
You could try booting in rescue mode (I had this same issue a week or so ago).

When you see the Grub boot screen press a key to enter the edit screen (don't type the quotes in the instructions btw).
go to the line that starts with "kernel"
press "e" to edit
remove the "splash" comment if it exists
add the following on that line: "init=/bin/bash"
press enter
press "b" to boot

You should now be able to boot into a bash shell environment. You don't need the root password.
Now your / is read only, mount it read-write:

Code: Select all

root@host# mount -o remount,rw /
Fix what needs to be fixing.
if /var is not a separate mount point you could check the log files there (/var/log).
if you see some very large log files either move them to another partition or delete them. if you see for instance a messages.1, messages.2 file you can delete those (those are older logs).
on my system it was hda-updates.log that was hogging a lot of space (2G+) because it wasn't properly logrotated (fixed that too :)).
you can empty it by doing the following:

Code: Select all

root@host# echo "" > /var/log/hda-updates.log
or by simply removing the file and recreating it:

Code: Select all

root@host# rm -rf /var/log/hda-updates.log && touch /var/log/hda-updates.log
check if your system now has some free space on / with "df -h"
if it has, reboot & check if everything comes up as it should.
check your /etc/logrotate.conf; if it has the "compress" option hashed out you can enable it by removing the leading # - this will make sure your old logfiles are compressed. compression on log files is usually quite useful since logfiles are easy to compress (51K file > 12K file compressed).

Re: Root directory full, no longer boots

Posted: Tue Mar 16, 2010 4:07 am
by lou1z
i'd certainly look at getting the log files off the cf card. although modern cf cards last longer than they used to (particually industrial cf), they still have limited write life.

Re: Root directory full, no longer boots

Posted: Tue Mar 16, 2010 10:50 am
by Coolbeans
Thanks for the replies, I really appreciate the help.

Sadly I'm at work so I cannot try the fix posted by moredruid, I have a question about my screw-up (deleting the /home folder). Is it possible to just bypass needing it? I let it to try logging in overnight and it gave some crazy 'sanity' check error or some jazz. Likely if it is exceptionally difficult to fix I'll follow lou1z's advice and nuke the install and toss it onto a different proper HDD.

Re: Root directory full, no longer boots

Posted: Wed Mar 17, 2010 12:21 am
by moredruid
well, you do need the /home mount for all your user accounts (except for user root who lives in /root/).

check if you can fix it, and even if you can fix it I'd recommend the reinstall after fixing it... you'll learn much in the process.