{"id":27286,"date":"2017-02-01T18:27:46","date_gmt":"2017-02-01T16:27:46","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=27286"},"modified":"2017-02-11T22:26:29","modified_gmt":"2017-02-11T20:26:29","slug":"gitlab-horror-story-backup-restore-failure","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2017\/02\/01\/gitlab-horror-story-backup-restore-failure\/","title":{"rendered":"GitLab horror story : backup \/ restore failure"},"content":{"rendered":"\n

As I am reading this story – GitLab.com melts down after wrong directory deleted, backups fail<\/a>\u00a0and these details<\/a> – every single hair I have, moves … I don’t (and didn’t) have any data on GitLab<\/a>, so I haven’t lost anything. \u00a0But as somebody who worked as a system administrator (and backup administrator) for years, I can imagine the physical and psychological state of the team all too well.<\/p>\n

Sure, things could have been done better. \u00a0But it’s easier said than done. \u00a0Modern technology is very complex. \u00a0And it changes fast. \u00a0And businesses want to move fast too. \u00a0And the proper resources (time, money, people) are not always allocated for mission critical tasks. \u00a0One thing is for sure, the responsibility lies on a whole bunch of people for a whole bunch of decisions. \u00a0But the hardest job is right now upon the tech people to bring back whatever they can. \u00a0There’s no sleep. \u00a0Probably no food. \u00a0No fun. \u00a0And a tremendous pressure all around.<\/p>\n

I wish the guys and gals at GitLab a super good luck. \u00a0Hopefully they will find a snapshot to restore from and this whole thing will calm down and sort itself out. \u00a0Stay strong!<\/p>\n

And I guess I’ll be doing test restores all night today, making sure that all my things are covered…<\/p>\n

Update: you can now read the full post-mortem<\/a> as well.<\/p>\n\n","protected":false},"excerpt":{"rendered":"\n

As I am reading this story – GitLab.com melts down after wrong directory deleted, backups fail\u00a0and these details – every single hair I have, moves … I don’t (and didn’t) have any data on GitLab, so I haven’t lost anything. \u00a0But as somebody who worked as a system administrator (and backup administrator) for years, I … Continue reading GitLab horror story : backup \/ restore failure<\/span><\/a><\/p>\n\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"GitLab horror story : backup \/ restore failure #GitLab #backup #disaster","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false,"_links_to":"","_links_to_target":""},"categories":[1,133,62,1334],"tags":[251,2366,2264,3490,1588],"keyring_services":[],"class_list":["post-27286","post","type-post","status-publish","format-standard","hentry","category-general","category-sysadmin","category-technology","category-web-work","tag-backup","tag-cloud-computing","tag-disaster-recovery","tag-gitlab","tag-version-control"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t