SOLVED - HELP! Disk showing full, nothing works

[t0bbe]

I've used Amahi for a few weeks now, most things work nicely. I should also say I'm very new to Linux and prefer a gui whenever available.

The problem I have now is Amahi is not working anymore, I can't access http://hda, I get the message:

Code: Select all

The application spawner server exited unexpectedly: No space left on device

I can log in to the hda using ssh, but if I try to start a vncserver I also get an error message about disk space.
I got this problem after running Crashplan to backup the Amahi installation to another computer on the network.
The backup completed ok, but then this problem started.

I ran

Code: Select all

df

and got the printout:

Code: Select all

Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_amahi-lv_root 3434980 3434680 0 100% / tmpfs 1025908 0 1025908 0% /dev/shm /dev/sdc1 198337 22084 166013 12% /boot /dev/mapper/storage-storage 3845710856 1060562804 2589797116 30% /var/hda/files

And it looks like my root partition is completely filled up.
How can this be?
I am not very comfortable using the cli and don't really know where to start looking for the files that must be filling up my root drive.

I use a 8GB CF card as my system disk, and it has never even been close to full before.


Any help really appreciated!!


EDIT:
I've done some searching, using the "du" command, and the only folder that seems too big is the "usr" folder that is 2,5GB.
Is this normal or could my problem be in there?

[moredruid]

ssh into your hda as root
give the following command:

Code: Select all

root@host# cd / root@host# du -kh --max-depth=1 .

this will give you a quick overview of what directories are using a lot of space.
probably this will be in /var
if you go to /var, you can run the du command again for further narrowing down.
I'd go to /var/log and run the following command:

Code: Select all

root@host# ls -lh

and check for large logfiles.
if you've found one or more large logfiles you can either move them to another filesystem/disk or you can wipe them completely if you don't want to keep 'em. be sure to recreate the file if you've removed it.
you can also empty the files with the following command (put the file you want to clear in place of the "file" descriptor):

Code: Select all

root@host# echo "" > file

good luck and tell us if it worked out!

[t0bbe]

Thank you for the quick reply!

I did what you said and it gives me:

Code: Select all

root@host# du -kh --max-depth=1 1012G ./var 17M ./boot 1.3M ./tmp 4.0K ./opt du: cannot access `./proc/2751/task/2751/fd/4': No such file or directory du: cannot access `./proc/2751/task/2751/fdinfo/4': No such file or directory du: cannot access `./proc/2751/fd/4': No such file or directory du: cannot access `./proc/2751/fdinfo/4': No such file or directory du: cannot access `./proc/2796': No such file or directory 0 ./proc 1.9M ./.gem 4.0K ./media 0 ./sys 105M ./home 16K ./lost+found 220K ./dev 6.6M ./bin 14M ./sbin 2.5G ./usr 8.0K ./.smolt 95M ./lib 4.0K ./mnt 30M ./etc 1.4M ./root 4.0K ./srv 4.0K ./selinux 12K ./.dbus 1015G .

I have my data drives mounted under /var/hda/files/ som I guess thats why that folder is 1015GB.
My /var/log/ folder is only 32MB, so that should not be the problem.
My /usr/ folder is 2,5GB, is this normal?
Also are the "cannot access ./proc" messages normal?


Running your command under /var/ gives:

[root@Amahi var]# du -kh --max-depth=1
346M ./cache
4.0K ./tmp
208K ./spool
4.0K ./nis
4.0K ./opt
4.0K ./games
16K ./db
4.0K ./gdm
1.2M ./www
4.0K ./preserve
12K ./yp
32M ./log
160K ./named
360K ./run
4.0K ./local
126M ./lib
1012G ./hda
8.0K ./empty
4.0K ./account
16K ./lock
1012G .

Thank you very much for your help so far!

[gboudreau]

The du command you should use is:

Code: Select all

du -xh --max-depth=1 /

Then work from there.
-x means just look for files on the current partition, not other mounted partitions it finds along the way.


The df command you showed on your first post seems to indicate only 3GB on your system drive, not 8GB...
I don't know much about LVM, but I'm pretty sure that number there (3434980 1K-blocks) should be near 8,000,000.

Maybe the rest of your 8GB card is taken by a swap partition?

Code: Select all

swapon -s

The above command should list the swap usage.

Maybe you'd want to move the swap file to your secondary HDD, and resize your / partition to 8GB.
I doubt 3GB is enough for Amahi.

[t0bbe]

Thanks very much guys for your help, everything is working again now.

I assumed 8GB would be enough to install Amahi, but gboudreau, you were right, my swap partition was over 4GB leaving only just over 3GB for Amahi.
I had left everything at "default" during installation.

With your help I found a few large log files that I deleted to clear just enough space to get vncserver going.
I then tried to run gparted to change the size of my partitions, but since the installer by default had installed some sort of LVM partitions, I couldn't change that from gparted.
Under administration/logical volume management I could see my swap partition but not change the size.
I did

Code: Select all

swapoff

then deleted the swap partition, increased the size of my root partition then tried to create a new swap partition.
But here is where I ran into problems, under filesystem, I can't select "swap" as it said for the old swap partition. I created a partition called swap with ext4, but I guess that this is not working and I'm actually running without swap partition at the moment.

Anyway, everything else is working, and I'm quite happy to let everything be the way it is, I have a feeling it could be rather difficult to create a new swap partition under LVM.

Thanks again!

[moredruid]

you can't use swap on ext4 as a partition, since swap has it's own filesystem structure. you can "format" the partition by setting the partition type to swap. If you use the proper values in /etc/fstab your swap will then be mounted on boot or with "mount -a". If you have enough memory (say > 1GB) you shouldn't need swap unless you really hammer your server. If you want you can also create a swap file on a filesystem. This doesn't use a full partition as swap but creates a file (size must be specified in your command) that the system will use as swap space. Note that the performance of such a swap file will be quite slow.

[gboudreau]

I'm not sure I'd recommend running any server without some swap space, like moredruid suggests.
If your not keen on swap usage, you can just adjust some kernel parameters (swapiness) to ensure your system won't swap until it's really out of physical memory.

You should Google something like "do i need swap on Linux" or something similar, and read some more about that. I'm pretty sure many kernel developers do recommend swap, for both protection against out of memory errors, and for performance.
Just make sure the articles you read are recent; the facts about how linux use swap can change as the linux kernel evolves, so what was factually true in 2004 might be completely false now!

[moredruid]

ah yes, some swap is indeed recommended (though you might never need it). The "golden rule" of swap=mem*1,5 is not valid anymore if you're going over 8GB of swap unless your application really needs it (e.g. SAP and Oracle do) but then you'd probably be running a dual quad core with 32GB RAM or more. RedHat recommends 8GB max for normal usage.

A lot of this is tunable (of course) and can be setup to your likings.