root partition full after deleting large amount of data

[mudvane]

So I have had everything working just fine for some time. Just last night I deleted about 190GB of data off the server all of which lived on the storage pool. This morning I am unable to access any shares and I get this when trying to access \hda
Ruby on Rails application could not be started
There appears to be a database problem.
Your config/database.yml may not be written correctly. Please check it and fix any errors.
Your database schema may be out of date or nonexistant. Please run rake db:migrate to ensure that the database schema is up-to-date.
The database server may not be running. Please check whether it's running, and start it if it isn't.
Error message:
Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) (Mysql::Error)
Exception class:
Mysql::Error
Application root:
/var/hda/platform/html

After digging around I find that my root partition is filled completely, although I am not sure why. I have a 20GB / drive and /var/hda/files is mounted to a 2TB drive with several others mounted in /var/hda/files/drives/drive# which are all part of the storage pool.

The SQL server is running
when I run du -kh --max-depth=1 this is my output

Code: Select all

[root@localhost /]# du -kh --max-depth=1 36K ./mnt 4.0K ./.pulse 34M ./etc 492K ./dev 4.0K ./srv 12K ./.dbus 14M ./sbin 6.3M ./.gem 4.0K ./opt 4.0K ./media 0 ./sys 35M ./boot 87M ./root 42M ./home 4.0K ./selinux 16K ./lost+found 4.1T ./var 171M ./lib 3.1G ./usr 7.2M ./bin du: cannot access `./proc/13908': No such file or directory du: cannot access `./proc/24617/task/24617/fd/4': No such file or directory du: cannot access `./proc/24617/task/24617/fdinfo/4': No such file or directory du: cannot access `./proc/24617/fd/4': No such file or directory du: cannot access `./proc/24617/fdinfo/4': No such file or directory 0 ./proc 25M ./lib64 39M ./tmp 4.1T . [root@localhost /]#

so the concern is not being able to see the /proc directory

I need some help on how to fix this and get things back to some normal, Also should I have a larger root drive so this does not happen in the future? Ideas as to what caused this to happen?


update:
So I noticed something that being the funionfs mounted in /var/hda/apps/ushare/files is rather large. the output of just df at the root looks like this:

Code: Select all

[root@localhost /]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdc1 20642428 20602116 0 100% / tmpfs 1995956 100 1995856 1% /dev/shm /dev/sdc3 61927420 226696 58554996 1% /home /dev/sda1 1922859824 1532145088 293039064 84% /var/hda/files /dev/sdb1 1442146364 1229458652 139430856 90% /var/hda/files/drives/drive1 /dev/sdd1 961432904 773348168 139246696 85% /var/hda/files/drives/drive2 /dev/sdc2 590378640 459136512 101252548 82% /var/hda/files/drives/drive3 /dev/sde1 480720592 317953820 138347496 70% /var/hda/files/drives/drive4 //127.0.0.1/Backups 5397538324 4586221664 811316660 85% /mnt/samba/Backups //127.0.0.1/CrashplanData 480720592 342373096 138347496 72% /mnt/samba/CrashplanData //127.0.0.1/Docs 5397538324 4586221664 811316660 85% /mnt/samba/Docs //127.0.0.1/Movies 5397538324 4586221664 811316660 85% /mnt/samba/Movies //127.0.0.1/Music 5397538324 4586221664 811316660 85% /mnt/samba/Music //127.0.0.1/Pictures 5397538324 4586221664 811316660 85% /mnt/samba/Pictures //127.0.0.1/Public 5397538324 4586221664 811316660 85% /mnt/samba/Public funionfs 1922859824 1532145088 293039064 84% /var/hda/apps/ushare/files [root@localhost /]#

That seems rather large, is there a way to reconfigure that? Could this be my problem?

[moredruid]

check the /var/log directory, probably there's a lot of info there.