PHP Fatal error (over php memory limit)

[lordpookdai]

My greyhole service seems to be halting with this error:

"PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8192 bytes) in /usr/bin/greyhole on line 2211"

After doing a bit of Googling it looks like I stumbled upon a bug that occurs when a rename is done high in the folder hierarchy with a ton of files multiple folders below (I'm sure I did a terrible job of explaining that, the link to the issue tracker is below):
http://code.google.com/p/greyhole/issues/detail?id=68

The specific change that I made was changing /Users/Steve/Downloads/Programs/.... to /Users/steve/Downloads/Programs/....

My /var/log/greyhole.log seems to have stopped in the middle of a tombstone process and refuses to continue.

It looks like my best bet is to change the php memory limit but I'm not sure how to go about doing that...I'll provide all of my log info below. Thank you so much for any help!!!

uname -r; rpm -q samba hda-greyhole

2.6.35.6-45.fc14.i686
samba-3.5.5-68.fc14.1.i686
hda-greyhole-0.9.2-1.i386

yum -y install fpaste; fpaste /etc/samba/smb.conf; fpaste /etc/greyhole.conf

Uploading (3.4K)...
http://fpaste.org/gwD3/
Uploading (1.4K)...
http://fpaste.org/WcZy/

mount; fdisk -l; df -h; greyhole --stats

/dev/mapper/vg_snabsvr-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg_snabsvr-lv_home on /home type ext4 (rw)
/dev/sda3 on /var/hda/files type ext4 (rw)
/dev/sdb1 on /var/hda/files/drives/drive2 type ext4 (rw)
/dev/sdc1 on /var/hda/files/drives/drive3 type ext4 (rw)
/dev/sdd1 on /var/hda/files/drives/drive4 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
//127.0.0.1/backup on /mnt/samba/backup type cifs (rw)
//127.0.0.1/music on /mnt/samba/music type cifs (rw)
//127.0.0.1/pictures on /mnt/samba/pictures type cifs (rw)
//127.0.0.1/Pictures on /mnt/samba/Pictures type cifs (rw)
//127.0.0.1/Users on /mnt/samba/Users type cifs (rw)
//127.0.0.1/videos on /mnt/samba/videos type cifs (rw)

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x181807e1

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 70658047 34816000 8e Linux LVM
/dev/sda3 70658048 1953523711 941432832 83 Linux

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x48933a6a

Device Boot Start End Blocks Id System
/dev/sdb1 2048 1953523711 976760832 83 Linux

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd53b391a

Device Boot Start End Blocks Id System
/dev/sdc1 2048 1953523711 976760832 83 Linux

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x489335db

Device Boot Start End Blocks Id System
/dev/sdd1 2048 1953523711 976760832 83 Linux

Disk /dev/dm-0: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 20.8 GB, 20837302272 bytes
255 heads, 63 sectors/track, 2533 cylinders, total 40697856 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/sde: 256 MB, 256901120 bytes
255 heads, 63 sectors/track, 31 cylinders, total 501760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x4f1e04e0

Device Boot Start End Blocks Id System
/dev/sde1 * 63 481949 240943+ 7 HPFS/NTFS

Disk /dev/dm-2: 10.5 GB, 10502537216 bytes
255 heads, 63 sectors/track, 1276 cylinders, total 20512768 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-2 doesn't contain a valid partition table
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_snabsvr-lv_root
20G 6.5G 12G 36% /
tmpfs 1002M 76K 1002M 1% /dev/shm
/dev/sda1 485M 29M 431M 7% /boot
/dev/mapper/vg_snabsvr-lv_home
9.7G 195M 9.0G 3% /home
/dev/sda3 884G 620G 220G 74% /var/hda/files
/dev/sdb1 917G 639G 232G 74% /var/hda/files/drives/drive2
/dev/sdc1 917G 638G 233G 74% /var/hda/files/drives/drive3
/dev/sdd1 917G 633G 238G 73% /var/hda/files/drives/drive4
//127.0.0.1/backup 3.6T 2.7T 922G 75% /mnt/samba/backup
//127.0.0.1/music 3.6T 2.7T 922G 75% /mnt/samba/music
//127.0.0.1/pictures 3.6T 2.7T 922G 75% /mnt/samba/pictures
//127.0.0.1/Pictures 3.6T 2.7T 922G 75% /mnt/samba/Pictures
//127.0.0.1/Users 3.6T 2.7T 922G 75% /mnt/samba/Users
//127.0.0.1/videos 3.6T 2.7T 922G 75% /mnt/samba/videos
PHP Warning: date_default_timezone_get(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/New_York' for 'EDT/-4.0/DST' instead in /usr/bin/greyhole on line 32

Greyhole Statistics
===================

Storage Pool
Total - Used = Free + Attic = Possible
/var/hda/files/gh: 884G - 619G = 220G + 0G = 220G
/var/hda/files/drives/drive2/gh: 917G - 639G = 231G + 0G = 231G
/var/hda/files/drives/drive3/gh: 917G - 638G = 232G + 0G = 232G
/var/hda/files/drives/drive4/gh: 917G - 633G = 238G + 0G = 238G
==========================================
Total: 3634G - 2529G = 921G + 0G = 921G

mysql -u root -phda -e "select * from disk_pool_partitions" hda_production

+----+------------------------------+--------------+---------------------+---------------------+
| id | path | minimum_free | created_at | updated_at |
+----+------------------------------+--------------+---------------------+---------------------+
| 1 | /var/hda/files | 10 | 2011-03-05 16:10:59 | 2011-03-05 16:10:59 |
| 2 | /var/hda/files/drives/drive2 | 10 | 2011-03-06 16:06:37 | 2011-03-06 16:06:37 |
| 3 | /var/hda/files/drives/drive3 | 10 | 2011-03-08 02:52:41 | 2011-03-08 02:52:41 |
| 4 | /var/hda/files/drives/drive4 | 10 | 2011-03-09 03:15:49 | 2011-03-09 03:15:49 |
+----+------------------------------+--------------+---------------------+---------------------+

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste

http://fpaste.org/Ay7r/

[lrevxl]

Upping your memory limit will help this, but it depends on how many files you're loading data for. Your php globally right now has a 128mb limit. As of version 0.9.4 (I think) I changed how greyhole loads meta data, it tries to this in chunks now so you're loading (hopefully) much smaller blocks of data at once. The latest version has this feature. (0.9.6) It also introduces a setting in the greyhole.conf file called memory_limit which will allow you to give greyhole it's own memory limit without touching your system's global php.ini limit.

So basically the quick way is to up your memory limit (there's no telling how much memory you'll need though -- so it might take a couple tries to get it right).

You might be able to avoid the issue entirely by upgrading to greyhole 0.9.6. (The new version will also quit before it hits memory exhaustion and give you an error in the greyhole.log file -- unlike now where it throws the php error where ever php defaults to.)

[lordpookdai]

Thanks so much for the response. I've actually modified the way those files are stored so that this wont happen again in the future - so hopefully upping the memory limit can resolve the issue this time (and then it won't happen again, at least for me).

How would I go about modifying the memory limit? I'm not well versed on php.

Thank you!

[lrevxl]

For your version of greyhole, you just set the global php limit, inside /etc/php.ini you should see a memory_limit property, it's set at 128M, you'll have to up that to something higher (how much higher, I don't know).

In 0.9.4+ you can just set memory_limit in /etc/greyhole.conf

[lordpookdai]

I did that, and I don't seem to be getting any errors anywhere now, however the greyhole.log file has not changed. The greyhole --view-queue gives:

Code: Select all

Greyhole Work Queue Statistics ============================== This table gives you the number of pending operations queued for the Greyhole daemon, per share. Write Delete Rename backup 30 0 0 music 0 0 0 pictures 0 0 0 Users 44 281 0 videos 0 0 0 ======== Total 74 + 281 + 0 = 355 The following is the number of pending operations that the Greyhole daemon still needs to parse. Until it does, the nature of those operations is unknown. Spooled operations that have been parsed will be listed above and disappear from the count below. Spooled 0

Which is about 98k fewer operations then before...but nothing i'm doing seems to cause greyhole to have any action whatsoever (tried a fsck, tried starting the daemon with -S). The last logged daemon action (expected from the greyhole.log file) was the rename that it halted on almost 7 hours ago. Any suggestions?

Thank you!

[lordpookdai]

I created a new greyhole.log file (as well as updated to 0.9.6) and it seems to be fsck'ing its heart out right now.

Thank you very much for your quick and helpful response!!!

[lrevxl]

I created a new greyhole.log file (as well as updated to 0.9.6) and it seems to be fsck'ing its heart out right now.

Thank you very much for your quick and helpful response!!!

I'm glad things are working. The log thing could be due to the fact that php has issues opening handles to files >2gb. And if greyhole gets really busy... Well let's just say I've seen a few 2+gb logs.