GitLab horror story : backup / restore failure

As I am reading this story – GitLab.com melts down after wrong directory deleted, backups fail and these details – every single hair I have, moves … I don’t (and didn’t) have any data on GitLab, so I haven’t lost anything.  But as somebody who worked as a system administrator (and backup administrator) for years, I can imagine the physical and psychological state of the team all too well.

Sure, things could have been done better.  But it’s easier said than done.  Modern technology is very complex.  And it changes fast.  And businesses want to move fast too.  And the proper resources (time, money, people) are not always allocated for mission critical tasks.  One thing is for sure, the responsibility lies on a whole bunch of people for a whole bunch of decisions.  But the hardest job is right now upon the tech people to bring back whatever they can.  There’s no sleep.  Probably no food.  No fun.  And a tremendous pressure all around.

I wish the guys and gals at GitLab a super good luck.  Hopefully they will find a snapshot to restore from and this whole thing will calm down and sort itself out.  Stay strong!

And I guess I’ll be doing test restores all night today, making sure that all my things are covered…

Power cut

Most of Cyprus experienced a major power cut today.  Cyprus Mail reports that it was due to a malfunction at Dhekelia power station.

A malfunction at the Dhekelia power station in the early morning hours caused a major power failure, which affected most of the island.

The Electricity company (EAC) said a unit at the Dhekelia station failed at 4.45 am triggering the collapse of the whole system. At the time, Dhekelia was supplying the whole island with power.

It is unclear what caused the initial failure.

Since last year’s explosion and power cuts, a lot of people, including yours truly, have been wondering if any changes were made to the management of the infrastructure.  Last year made it clear that there was no disaster recovery plan of any kind.  And while 9 month is not a lot of time, given the complexity of the problem, it looks like nothing changed.  We are still at the stage where a single mistake or malfunction can throw the country into darkness.  With summer approaching and electricity demand going up, I think we’ll see more of these cuts and blackouts.