{"id":26926,"date":"2016-11-21T09:28:43","date_gmt":"2016-11-21T07:28:43","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=26926"},"modified":"2016-11-21T09:28:43","modified_gmt":"2016-11-21T07:28:43","slug":"monitoring-the-monitoring-keeping-zabbix-server-service-up","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2016\/11\/21\/monitoring-the-monitoring-keeping-zabbix-server-service-up\/","title":{"rendered":"Monitoring the monitoring : keeping Zabbix server service up"},"content":{"rendered":"\n

After our recent MySQL migrations<\/a>, I started getting a weird issue – Zabbix<\/a> server process was crashing periodically (several times a day).<\/p>\n

\r\n8395:20161109:175408.023 [Z3005] query failed: [2013] Lost connection to MySQL server during query [begin;]\r\n8395:20161109:175408.024 [Z3001] connection to database 'zabbix_database_name_here' failed: [2003] Can't connect to MySQL server on 'zabbix_database_host_here' (111)\r\n8395:20161109:175408.024 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. Crashing ...\r\n<\/pre>\n
Digging around for a bit, it seems like a widely reported issue, related Zabbix server using the same database connection as one of its agents is monitoring (here is an example bug report<\/a>).<\/p>\n
Not having enough time to troubleshoot and fix it properly, I decided for the time being to use another monitoring tool – monit<\/a> – to keep an eye on the Zabbix server process and restart it, if it’s down. \u00a0After “yum install monit<\/em>“, the following was dropped into \/etc\/monit.d\/zabbix<\/em>:<\/p>\n
\r\ncheck process zabbix_server with pidfile \/var\/run\/zabbix\/zabbix_server.pid\r\n    start program = "\/sbin\/service zabbix-server start" with timeout 60 seconds\r\n    stop program = "\/sbin\/service zabbix-server stop"\r\n<\/pre>\n
Start the monit service, make sure it also starts at boot, and watch it in action via the \/var\/log\/monit<\/em>:<\/p>\n
\r\n[UTC Nov 20 20:49:18] error    : 'zabbix_server' process is not running\r\n[UTC Nov 20 20:49:18] info     : 'zabbix_server' trying to restart\r\n[UTC Nov 20 20:49:18] info     : 'zabbix_server' start: \/sbin\/service\r\n[UTC Nov 20 20:50:19] info     : 'zabbix_server' process is running with pid 28941\r\n<\/pre>\n
The chances of both systems failing at once are slim, so I think this will buy me some time.<\/p>\n\n","protected":false},"excerpt":{"rendered":"\n
After our recent MySQL migrations, I started getting a weird issue – Zabbix server process was crashing periodically (several times a day). 8395:20161109:175408.023 [Z3005] query failed: [2013] Lost connection to MySQL server during query [begin;] 8395:20161109:175408.024 [Z3001] connection to database ‘zabbix_database_name_here’ failed: [2003] Can’t connect to MySQL server on ‘zabbix_database_host_here’ (111) 8395:20161109:175408.024 Got signal [signal:11(SIGSEGV),reason:1,refaddr:(nil)]. … Continue reading Monitoring the monitoring : keeping Zabbix server service up<\/span><\/a><\/p>\n\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Monitoring the monitoring : keeping Zabbix server service up #Linux #Zabbix #monitoring #SysAdmin","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[1,6,133,62],"tags":[127,3216],"keyring_services":[],"class_list":["post-26926","post","type-post","status-publish","format-standard","hentry","category-general","category-linux","category-sysadmin","category-technology","tag-monitoring","tag-zabbix"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":42551,"url":"https:\/\/mamchenkov.net\/wordpress\/2019\/04\/18\/zabbix-in-the-munich-subway\/","url_meta":{"origin":26926,"position":0},"title":"Zabbix in the Munich subway","author":"Leonid Mamchenkov","date":"April 18, 2019","format":false,"excerpt":"Zabbix blog shares a very inspiring post \"Zabbix in the subway. Munich Transport Corporation Case Study\", which shows how Zabbix is used for the monitoring of the trains and trams in Munich. Here are some implementation details to get you started: Currently, there are 3796 devices monitored by the Zabbix\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/Zabbix-Munich-Trains.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/Zabbix-Munich-Trains.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/Zabbix-Munich-Trains.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/Zabbix-Munich-Trains.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":16722,"url":"https:\/\/mamchenkov.net\/wordpress\/2012\/09\/19\/zabbix-the-enterprise-class-monitoring-solution\/","url_meta":{"origin":26926,"position":1},"title":"Zabbix – The Enterprise-class Monitoring Solution","author":"Leonid Mamchenkov","date":"September 19, 2012","format":"link","excerpt":"Zabbix - The Enterprise-class Monitoring Solution Zabbix is the ultimate open source availability and performance monitoring solution. Zabbix offers advanced monitoring, alerting, and visualization features today which are missing in other monitoring systems, even some of the best commercial ones. Below is a short list of features available in Zabbix:\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":26692,"url":"https:\/\/mamchenkov.net\/wordpress\/2016\/09\/16\/zabbix-3-2-0\/","url_meta":{"origin":26926,"position":2},"title":"Zabbix 3.2.0","author":"Leonid Mamchenkov","date":"September 16, 2016","format":false,"excerpt":"The next major release version of Zabbix is now available! Zabbix 3.2 brings both new features and many UI and performance improvements along with it. Go here for the details.","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"Zabbix 3.2.0","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2016\/09\/zabbix_3_2_0_released-300x300.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":18528,"url":"https:\/\/mamchenkov.net\/wordpress\/2013\/07\/30\/looking-into-pros-and-cons-of-nagios-vs-zabbix-vs\/","url_meta":{"origin":26926,"position":3},"title":"Looking into pros and cons of Nagios vs. Zabbix vs…","author":"Leonid Mamchenkov","date":"July 30, 2013","format":"status","excerpt":"Looking into pros and cons of Nagios vs. Zabbix vs. Zenoss vs. whatever else. \u00a0If you have something to say, say it now or forever hold your silence.","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":27709,"url":"https:\/\/mamchenkov.net\/wordpress\/2017\/06\/20\/zabbix-no-more-flapping-define-triggers-the-smart-way\/","url_meta":{"origin":26926,"position":4},"title":"Zabbix : No more flapping. Define triggers the smart way.","author":"Leonid Mamchenkov","date":"June 20, 2017","format":false,"excerpt":"\"No more flapping. Define triggers the smart way.\" is a very useful article from the Zabbix Weblog on how to setup sensible, flapping-aware triggers in Zabbix. I'm sure every single person on this planet has a limit to how many up and down notifications he can receive ...","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":41537,"url":"https:\/\/mamchenkov.net\/wordpress\/2019\/04\/02\/zabbix-4-2-is-out\/","url_meta":{"origin":26926,"position":5},"title":"Zabbix 4.2 is out!","author":"Leonid Mamchenkov","date":"April 2, 2019","format":false,"excerpt":"Zabbix 4.2 has been released and it brings an impressive array of new features and improvements. Some of these are: Built-in support of Prometheus data collectionEfficient high-frequency monitoringValidation of collected data and error handlingPreprocessing data with JavaScriptTest preprocessing rules from UITest media type from Web UISupport of TimescaleDBSimplified tag managementMore\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/zabbix.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/zabbix.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/zabbix.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/zabbix.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/zabbix.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2019\/04\/zabbix.png?resize=1400%2C800&ssl=1 4x"},"classes":[]}],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/26926","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/comments?post=26926"}],"version-history":[{"count":0,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/26926\/revisions"}],"wp:attachment":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/media?parent=26926"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/categories?post=26926"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/tags?post=26926"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/keyring_services?post=26926"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}