As I am reading this story – GitLab.com melts down after wrong directory deleted, backups fail and these details – every single hair I have, moves … I don’t (and didn’t) have any data on GitLab, so I haven’t lost anything.  But as somebody who worked as a system administrator (and backup administrator) for years, I can imagine the physical and psychological state of the team all too well.
Sure, things could have been done better. Â But it’s easier said than done. Â Modern technology is very complex. Â And it changes fast. Â And businesses want to move fast too. Â And the proper resources (time, money, people) are not always allocated for mission critical tasks. Â One thing is for sure, the responsibility lies on a whole bunch of people for a whole bunch of decisions. Â But the hardest job is right now upon the tech people to bring back whatever they can. Â There’s no sleep. Â Probably no food. Â No fun. Â And a tremendous pressure all around.
I wish the guys and gals at GitLab a super good luck. Â Hopefully they will find a snapshot to restore from and this whole thing will calm down and sort itself out. Â Stay strong!
And I guess I’ll be doing test restores all night today, making sure that all my things are covered…
Update: you can now read the full post-mortem as well.