We spent a couple of weeks now at work, setting up our Nagios installation. This is one of those things that one can never finish. Monitoring is like security – no matter how good it is, it can always be better.
One thing that I particularly enjoyed figuring out was how to monitor dynamic stuff. Hosts and services are the easy part – they are always there and should be there. If they aren’t – let me know. But how to monitor dynamic values, that change based on the time of day or day of week? How to configure the monitoring so that we don’t need to update the limits every other week?
For example, consider the number of the user registrations through a web form. If we are to measure this number over any sensible (monitoring-wise) period of time, such as one hour, we’d see that it fluctuates a lot during the day. Furthermore, this number fluctuates differently depending on the day of the week. We have three separate pick hours during the day, and we have a great decrease in numbers over the weekend. Plus the amount of user registrations is linked to all the advertising that company does, so this week might different from the last week and from the next week. How can we measure it so that a notiification is sent when the number is abnormal? What is abnormal?
The solution turned out to be much simpler than I originally thought it would. It is sufficient to get a few samples of the data in same hour last week and a week before. If current value is more than twice the maximum or less than twice the minimum from the sample data, then we should be notified. This, in fact, works pretty well. The only time when we get a lot of false positives is when the values in the sample data are small. With values under 10, it’s very easy to jump over or under the limit. When the sample values are higher, there is more space between the boundaries and the system works as expected.
We’ll get some more sample data now and we’ll be adjusting the formulat accordingly. But as I said, even as it is, it’s pretty good.