O.K. I filled up my Landing Zone...DOH!

[rjacobsen]

Greyhole was running fine, I'd transfer files to the storage shares and everything was running fine.
The root drive was 101GB, yeah these days that was rather small, I transfered about 70G and thought I'd be fine. Now I'm getting the classic
"Ruby on Rails application could not be started"
and the message
"No Space left on device"

Here's a log

root@amahi:~# tail /var/log/greyhole.log
Aug 21 13:53:25 4 daemon: PHP Warning [2]: mysql_connect(): Can't connect to loc al MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/bin/gre yhole on line 427; BT: greyhole[L1935] db_connect() => greyhole[L427] mysql_conn ect(localhost,greyhole,greyhole)
Aug 21 13:53:25 2 daemon: Can't connect to mysql database.
Aug 21 13:55:25 4 daemon: PHP Warning [2]: mysql_connect(): Can't connect to loc al MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/bin/gre yhole on line 427; BT: greyhole[L1935] db_connect() => greyhole[L427] mysql_conn ect(localhost,greyhole,greyhole)
Aug 21 13:55:25 2 daemon: Can't connect to mysql database.
Aug 21 13:57:25 4 daemon: PHP Warning [2]: mysql_connect(): Can't connect to loc al MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/bin/gre yhole on line 427; BT: greyhole[L1935] db_connect() => greyhole[L427] mysql_conn ect(localhost,greyhole,greyhole)
Aug 21 13:57:25 2 daemon: Can't connect to mysql database.
Aug 21 13:58:58 4 stats: PHP Warning [2]: mysql_connect(): Can't connect to loca l MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/bin/grey hole on line 427; BT: greyhole[L1935] db_connect() => greyhole[L427] mysql_conne ct(localhost,greyhole,greyhole)
Aug 21 13:58:58 2 stats: Can't connect to mysql database.
Aug 21 13:59:25 4 daemon: PHP Warning [2]: mysql_connect(): Can't connect to loc al MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in /usr/bin/gre yhole on line 427; BT: greyhole[L1935] db_connect() => greyhole[L427] mysql_conn ect(localhost,greyhole,greyhole)
Aug 21 13:59:25 2 daemon: Can't connect to mysql database.

I did not find a guide on how to clear this, could someone reply with a suggested plan to get drive cleaned up enough and how to restart greyhole?

I'm running the latest ubuntu 12.04? on a headless server, I have root access through putty and WinSCP.

Amahi DNS runs fine, even the shares work, just when going to \\hda reports ruby on rails could not be started

I probably will not be the last one to fill up the last drive, if there is already a post to deal with this please reply with a link.

[rjacobsen]

FYI on the \\hda page there's a backtrace:

Backtrace:
# File Line Location
0 /usr/lib/ruby/1.8/phusion_passenger/utils.rb 219 in `write'
1 /usr/lib/ruby/1.8/phusion_passenger/utils.rb 219 in `send'
2 /usr/lib/ruby/1.8/phusion_passenger/utils.rb 219 in `method_missing'
3 /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb 62 in `warn'
4 /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb 62 in `add_spec'
5 /usr/lib/ruby/vendor_ruby/1.8/rubygems/source_index.rb 127 in `_deprecated_load_gems_in'
6 /usr/lib/ruby/vendor_ruby/1.8/rubygems/source_index.rb 123 in `each'
7 /usr/lib/ruby/vendor_ruby/1.8/rubygems/source_index.rb 123 in `_deprecated_load_gems_in'
8 /usr/lib/ruby/vendor_ruby/1.8/rubygems/source_index.rb 120 in `reverse_each'
9 /usr/lib/ruby/vendor_ruby/1.8/rubygems/source_index.rb 120 in `_deprecated_load_gems_in'
10 /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb 63 in `send'
11 /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb 63 in `load_gems_in'
12 /usr/lib/ruby/vendor_ruby/1.8/rubygems/source_index.rb 322 in `_deprecated_refresh!'
13 /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb 63 in `send'
14 /usr/lib/ruby/vendor_ruby/1.8/rubygems/deprecate.rb 63 in `refresh!'
15 /var/lib/gems/1.8/gems/rails-2.3.4/lib/rails/vendor_gem_source_index.rb 34 in `refresh!'
16 /var/lib/gems/1.8/gems/rails-2.3.4/lib/rails/vendor_gem_source_index.rb 29 in `initialize'
17 /var/lib/gems/1.8/gems/rails-2.3.4/lib/rails/gem_dependency.rb 21 in `new'
18 /var/lib/gems/1.8/gems/rails-2.3.4/lib/rails/gem_dependency.rb 21 in `add_frozen_gem_path'
19 /var/lib/gems/1.8/gems/rails-2.3.4/lib/initializer.rb 298 in `add_gem_load_paths'
20 /var/lib/gems/1.8/gems/rails-2.3.4/lib/initializer.rb 132 in `process'
21 /var/lib/gems/1.8/gems/rails-2.3.4/lib/initializer.rb 113 in `send'
22 /var/lib/gems/1.8/gems/rails-2.3.4/lib/initializer.rb 113 in `run'
23 /var/hda/platform/html/config/environment.rb 9
24 /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb 36 in `gem_original_require'
25 /usr/lib/ruby/vendor_ruby/1.8/rubygems/custom_require.rb 36 in `require'
26 /usr/lib/ruby/1.8/phusion_passenger/railz/application_spawner.rb 303 in `preload_application'
27 /usr/lib/ruby/1.8/phusion_passenger/railz/application_spawner.rb 252 in `initialize_server'
28 /usr/lib/ruby/1.8/phusion_passenger/utils.rb 255 in `report_app_init_status'
29 /usr/lib/ruby/1.8/phusion_passenger/railz/application_spawner.rb 237 in `initialize_server'
30 /usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb 194 in `start_synchronously'
31 /usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb 163 in `start'
32 /usr/lib/ruby/1.8/phusion_passenger/railz/application_spawner.rb 213 in `start'
33 /usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb 262 in `spawn_rails_application'
34 /usr/lib/ruby/1.8/phusion_passenger/abstract_server_collection.rb 126 in `lookup_or_add'
35 /usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb 256 in `spawn_rails_application'
36 /usr/lib/ruby/1.8/phusion_passenger/abstract_server_collection.rb 80 in `synchronize'
37 /usr/lib/ruby/1.8/phusion_passenger/abstract_server_collection.rb 79 in `synchronize'
38 /usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb 255 in `spawn_rails_application'
39 /usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb 154 in `spawn_application'
40 /usr/lib/ruby/1.8/phusion_passenger/spawn_manager.rb 287 in `handle_spawn_application'
41 /usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb 352 in `__send__'
42 /usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb 352 in `main_loop'
43 /usr/lib/ruby/1.8/phusion_passenger/abstract_server.rb 196 in `start_synchronously'
44 /usr/lib/phusion_passenger/passenger-spawn-server 61

It probably means "you filled up the drive" but I'm no expert here

[bigfoot65]

You did not include your root partition in the storage pool I hope.

The only fix is to boot the HDA via a Live CD and move some files to another drive or remove them or some other files that might not be needed. I used a 40GB drive for the longest time for my OS and never had an issue. You might consider moving smaller chunks of data or move your landing zone to another drive.

http://wiki.amahi.org/index.php/Moving_landing_zone

There is no magic link to fix this problem, it's something that we have yet to create guidance for in the wiki. If you can help, it would be greatly appreciated.

[rjacobsen]

The root was on a separate disk, actually a raid, not included in the greyhole, with 3 other disks for greyhole storage.
I believe you would be correct if the root was part of greyhole.

I was able to find ways to clean the root drive to find some space

http://ubuntuforums.org/showthread.php?t=140920
basically running

Code: Select all

sudo apt-get autoclean

and

Code: Select all

sudo deborphan | xargs sudo apt-get -y remove --purge

cleared enough room for Amahi to run

I'm having other issues, mounting a amahi-swap_1 which may have been self inflicted while trying to resolve the initial problem

[bigfoot65]

You can also try:

Code: Select all

sudo apt-get autoremove

which removes packages no longer needed.

Not sure of the need for RAID when using Greyhole. Does that seem like duplicate effort and more likely to fail?

[rjacobsen]

Only the root drive is a raid 5, the intent was fast system disk, I inherited 3 15KRPM 33GB SCSI SDI drives, small but fast, they where going to be disposed of.

Greyhole 3 1.8TB drives for storage.

[bigfoot65]

I see. Does it make that much of a difference? I have not experience with RAID, so not sure of the benefits.

[rjacobsen]

Well I'm not sure, I did enable jumbo packets on Amahi, it's best to just google "jumbo packets ubuntu" to find out how, it's basically changing the MTU 9000 in a config file.

Now when I move video teracopy reports transfer rates 100+MB/Sec yes that's a big B as in Bytes, to Amahi over a gigabit connection.
Most standalone spinning disk hard drives can do 35-40MB on a good day.

All this spinning disk raid is getting to be antique hardware with the new SDD's easily pushing 200MB/sec though :/

[rjacobsen]

DOH!

This is not a greyhole issue, it's an operator issue

When making a share with the intent of using greyhole's storage pooling It's important to put a check in the box for "Uses pool" (and of course how many copies)

Making a share without pooling then moving in files larger then the root drive will fill up you drive...DOH!

nevermind

[rjacobsen]

Well I did it again, when moving large amounts of video into the greyhole, the landing zone drive, in my case was 85GB can get filled up.

DNS and the file system work, it's just Amahi's home page fails with not enough space, and greyhole -f does NOT work.

BUT this time I stuck a large file, about 5GB on landing zone drive, just in case.

Deleting the large file, running

Code: Select all

greyhole -f

got me back up, once greyhole -f ran the landing drive moved the files to greyhole drives.

So lesson learned leave a large file at least 5% of your drive on the landing drive you can easily delete to get your system back.