{"id":27618,"date":"2017-05-25T19:55:00","date_gmt":"2017-05-25T17:55:00","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=27618"},"modified":"2017-05-25T19:55:00","modified_gmt":"2017-05-25T17:55:00","slug":"building-the-right-alerting-system","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2017\/05\/25\/building-the-right-alerting-system\/","title":{"rendered":"Building the Right Alerting System"},"content":{"rendered":"<!-- google_ad_section_start -->\n<p>Here&#8217;s something I wanted to get into for a while now, but haven&#8217;t had the time yet &#8211; <a href=\"https:\/\/thoughts.t37.net\/whats-not-actionable-business-critical-shouldn-t-ring-building-the-right-alerting-system-e8f4b085a2cb\">switching the monitoring \/ alerting system from server-oriented to business-oriented<\/a>. \u00a0The gist of the story is:<\/p>\n<blockquote><p>If it&#8217;s not actionable and business critical, then it shouldn&#8217;t ring.<\/p><\/blockquote>\n<p>The article has some statistics and summaries as well. \u00a0The reasoning behind the switch is obvious, but it&#8217;s good to have it formulated:<\/p>\n<blockquote><p>After a few months, I can tell reducing our alerting rate should have been a top priority before things got out of hands, for a few reasons.<\/p>\n<ul>\n<li>Constant alerts prevented the team to focus on what was important. Being interrupted even for things that can wait for a few hours lowers our productivity when we work on things that can&#8217;t wait.<\/li>\n<li>Being awaken every night, several times a night exhausts a team and make people less productive at day, and more prone to do errors.<\/li>\n<li>Too many off hours interventions cost the company a lot of money that could be invested in hardening the infrastructure or hiring someone else instead.<\/li>\n<\/ul>\n<\/blockquote>\n<!-- google_ad_section_end -->\n","protected":false},"excerpt":{"rendered":"<!-- google_ad_section_start -->\n<p>Here&#8217;s something I wanted to get into for a while now, but haven&#8217;t had the time yet &#8211; switching the monitoring \/ alerting system from server-oriented to business-oriented. \u00a0The gist of the story is: If it&#8217;s not actionable and business critical, then it shouldn&#8217;t ring. The article has some statistics and summaries as well. \u00a0The &hellip; <a href=\"https:\/\/mamchenkov.net\/wordpress\/2017\/05\/25\/building-the-right-alerting-system\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Building the Right Alerting System<\/span><\/a><\/p>\n<!-- google_ad_section_end -->\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Building the Right Alerting System #SysAdmin #DevOps #monitoring #productivity","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[1,133,62,1334],"tags":[127,1079,1330,2289],"keyring_services":[],"class_list":["post-27618","post","type-post","status-publish","format-standard","hentry","category-general","category-sysadmin","category-technology","category-web-work","tag-monitoring","tag-productivity","tag-web-development","tag-web-hosting"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":16722,"url":"https:\/\/mamchenkov.net\/wordpress\/2012\/09\/19\/zabbix-the-enterprise-class-monitoring-solution\/","url_meta":{"origin":27618,"position":0},"title":"Zabbix &#8211; The Enterprise-class Monitoring Solution","author":"Leonid Mamchenkov","date":"September 19, 2012","format":"link","excerpt":"Zabbix - The Enterprise-class Monitoring Solution Zabbix is the ultimate open source availability and performance monitoring solution. Zabbix offers advanced monitoring, alerting, and visualization features today which are missing in other monitoring systems, even some of the best commercial ones. Below is a short list of features available in Zabbix:\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":18182,"url":"https:\/\/mamchenkov.net\/wordpress\/2013\/05\/20\/quinico-web-insights\/","url_meta":{"origin":27618,"position":1},"title":"Quinico web insights","author":"Leonid Mamchenkov","date":"May 20, 2013","format":"link","excerpt":"Quinico web insights Quinico is an open source web application designed to help you easily improve your website's performance, reduce errors and optimize for search engines (SEO). \u00a0Quinico can constantly monitor your websites and alert you when there is a problem that requires attention. \u00a0Using Quinico, you can automate the\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":27131,"url":"https:\/\/mamchenkov.net\/wordpress\/2016\/12\/15\/what-does-operations-do\/","url_meta":{"origin":27618,"position":2},"title":"What does Operations *do*?","author":"Leonid Mamchenkov","date":"December 15, 2016","format":false,"excerpt":"SysAdvent runs a blog post about Operations (in IT sense of a word), explaining what the department (hopefully it's a department and not a single guy who doesn't remember the meaning of the word \"sleep\") does, and how the job complexity silently escalates with time. An operations team, almost by\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":12086,"url":"https:\/\/mamchenkov.net\/wordpress\/2010\/01\/06\/day-in-brief\/","url_meta":{"origin":27618,"position":3},"title":"Day in brief","author":"Leonid Mamchenkov","date":"January 6, 2010","format":false,"excerpt":"Bookmarked: Productivity Analysis of Object-Oriented Software (PDF) http:\/\/icio.us\/kxccpk # Bookmarked: Pitfalls of Object Oriented Programming (PDF) http:\/\/icio.us\/yaac34 #","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":42632,"url":"https:\/\/mamchenkov.net\/wordpress\/2019\/05\/20\/faast-js-serverless-batch-computing-made-simple\/","url_meta":{"origin":27618,"position":4},"title":"faast.js &#8211; serverless batch computing made simple","author":"Leonid Mamchenkov","date":"May 20, 2019","format":false,"excerpt":"faast.js is a new framework that makes writing serverless functions super easy. Read more about it in this introductory blog post: Faast.js started as a side project to solve the problem of large scale software testing. Serverless functions seemed like a good fit because they could scale up to perform\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/05\/faast-architecture-aws.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/05\/faast-architecture-aws.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/05\/faast-architecture-aws.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/05\/faast-architecture-aws.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":18180,"url":"https:\/\/mamchenkov.net\/wordpress\/2013\/05\/20\/mine-seeking-dolphins-make-historic-discovery\/","url_meta":{"origin":27618,"position":5},"title":"Mine-seeking dolphins make historic discovery","author":"Leonid Mamchenkov","date":"May 20, 2013","format":false,"excerpt":"Slashdot shares a story of mine-seeking dolphins making a historic discovery - a 11-foot long torpedo from 19th century. \u00a0Apparently, there were only 50 of this ever made, and this is the second one being discovered. When first created, the Howell torpedo had major implications for helping the United States\u2026","rel":"","context":"In &quot;All&quot;","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"howell torpedo","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2013\/05\/howell-torpedo-500x331.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/27618","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/comments?post=27618"}],"version-history":[{"count":0,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/27618\/revisions"}],"wp:attachment":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/media?parent=27618"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/categories?post=27618"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/tags?post=27618"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/keyring_services?post=27618"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}