On remote logging with syslog

We’ve been doing some interesting things at work, as always, with yet more people and Linux boxes.  And of the side effects of mixing people, Linux boxes, and several locations is this need for some sort of centralized logging.  Luckily we have either syslog-ng or rsyslog daemons installed on each machine, so the only two issues seemed to be reconfiguration of syslog services for remote logging and setup of some log reading/searching tool for everyone to enjoy.

As for log reading and searching, there seems to be no end of tools.  We picked php-syslog-ng, which has web interface, MySQL back-end, access control, and more.  There were a few minor issues during setup and configuration, but overall it seemed to be OK.  I also patched the source code a bit in a few places, just to make it work nicer with our setup and our needs  (both numerical and symbolic priorities, preference for include masks over excludes, and full functionality with disabled caching).  In case you are interested, here is a patch against php-syslog-ng 2.9.8f tarball.

Once everything was up and running and we started looking through logs from all our hosts in the same place, there was one thing that surprised me a lot.  Either I don’t understand the syslog facilities and priorites fully (and I don’t claim that I do), or there is just too many software authors who don’t care much.  Most of our logs are coming in at priority critical.  Even if there isn’t much critical about them.  Emergency is also used way too much.  And there is hardly anything at debug or info or notice levels.  (RT, SpamAssassin, and many other applications seem to be using critical as their default log level).  Luckily, that  almost always is trivial to fix using either the configuration files or applications’ source code directly.