52 lines
1.9 KiB
ReStructuredText
52 lines
1.9 KiB
ReStructuredText
Webmaster tools alerted issue turned out Pylon session files flooded inodes
|
|
###########################################################################
|
|
:date: 2011-10-29 15:30
|
|
:author: Russell Ballestrini
|
|
:tags: DevOps, Opinion
|
|
:slug: webmaster-tools-alerted-issue-turned-out-pylon-session-files-flooded-inodes
|
|
:status: published
|
|
:summary:
|
|
|
|
**This graph could happen to you if you ever forget to configure munin
|
|
email alerting:**
|
|
|
|
|image0|
|
|
|
|
It only took approximately 1 hour to diagnoses and resolve this issue
|
|
however most of my web applications hosted on this server were down for
|
|
about 11 hours. I was lucky that this outage fell on a weekend otherwise
|
|
I would not have known about the problem till around 6:30pm!
|
|
|
|
|image1|
|
|
|
|
**Diagnosis:**
|
|
|
|
Two of my pylons apps had session files that slowly ran away on me. The
|
|
session files don't consume much capacity however the shear quantity of
|
|
them caused my inode usage to hit 100%.
|
|
|
|
Had I properly configured Munin's email alerting this issue would have
|
|
been identified well before it was a problem.
|
|
|
|
Want to know what alerted me to the problem? G Webmaster's tools claimed
|
|
it could not read my robots.txt on a few of my sites... After
|
|
investigating I learned the site was down. Checking the Apache error
|
|
logs pointed me to disk space issues. ``df -ha`` reported everything was
|
|
fine, however ``df -hi`` reported 100% inode usage! At this point I
|
|
started looking to cache and log locations to find lots of files, which
|
|
lead me to my pylons web applications data/sessions directories.
|
|
|
|
**Resolution:**
|
|
|
|
Delete the session cache tree directories and allow the applications to
|
|
rebuild them.
|
|
|
|
Todo: move /www off the root disk partition. This issue could have been
|
|
much worse if I was unable to boot or login to remedy. Moving /www off
|
|
root should prevent the web server from effecting the systems ability to
|
|
boot.
|
|
|
|
.. |image0| image:: /uploads/2011/10/df_inode-year.png
|
|
.. |image1| image:: /uploads/2011/10/df_inode-day.png
|
|
|
|
.. contents::
|