The long story short : I lost my blog, as well as a few other web sites.
Here goes the longer version. I have been moving a whole bunch of web sites from my reseller hosting account at EuroVPS to a brand new VPS account at VAServ. Site by site, blog by blog, database by database. To keep things simple, once I made sure that the site was moved properly, I deleted the copy from the old hosting (after a week or so).
When I was almost done with the move and there were just a few more left, something really bad happened a VAServ. All company’s servers got compromised. The attackers gained control over thousands of VPS accounts across hundreds of servers, and then they deleted whatever they could. The effect of this was so devastating that it was extensively covered in the news.
According the VAServ, hackers utilized a security hole in the HyperVM software, which was written by LXLabs. Apparently, HyperVM is known for its poor security, but things never went wrong at this scale. (In other news, LXLabs founder was found dead in a suspected suicide a day or so later. And the rumour has it that the break-in had nothing to do with HyperVM, but was VAServ negligence)
Now for the most interesting part of the story – the lost data. How did that happen? OK, the company got hacked and all data was deleted. But what about the backups? It turned out, there were no tape backups. The only backups VAServ had were on the network storage. And, of course, that data got deleted by the attackers. Imagine that. Web sites, databases, emails, DNS records. Everything is gone. Well, not everything – they managed to recover some servers, but not all by far.
My sites were on one of those servers which experienced 100% data loss, and which had no backup. That was when I contacted EurVPS support and asked them to restore my recently deleted sites from tapes. After all, it’s better to lose a few weeks of work, rather than a few years. Guess what? It turned out, EuroVPS has no backups either. When I insisted, saying that backups are a part of my hosting plan, they corrected themselves and said that they have backups, but, as advertised on the site – weekly only.
Let me ask you a simple question. How do you understand the phrase “weekly backups on tape”? My understanding was that there’s a scheduled backup task (every weekend or so), which dumps data on tapes, and those tapes are moved out of the building somewhere. Eventually, of course, they are rotated (monthly, or annually, or so). But there is a certain period which you can go back to and restore from those weekly tapes.
It so happened, my understanding was wrong. Weekly tape backup means one backup within a week on tape. That is, there is no way to go more than one week back using tape backups. I was shocked a bit, but there was still a chance to get something. I clearly remember that I deleted two sites five days ago. I asked EuroVPS support to restore at least those. To which they replied that those two sites aren’t on the backups either.
What? How? Err… I know, of course, that the loss of data is my fault as much as theirs. I should have done my own backups, downloading them to my own machine. And I’m deeply sorry for not doing so. But on the other hand, having paid for hosting, I do expect uninterrupted power, redundant network connection, and properly organized backups. If that’s not how commercial hosting is different from home servers, than I don’t know how.
Currently, I am setting up a new VPS host, reconfiguring domains for the new IP, installing a bunch of WordPress blogs, and issuing a whole lot of apologies. Those things that can be recovered, will be recovered. Those things that were important and were lost, will be started a new. And those things that were not important and were lost, will remain lost.
Let this be yet another painful lesson on the importance of backups.
12 thoughts on “What happened?”
I’m very sorry to hear that. :( Very few people realize importance of off-site tape backups nowadays. They think that just storing stuff on the NAS is good enough.
But, I think I have good news for you. :) I still have all your articles in my Google Reader. Contact me and we’ll figure how to fetch them.
yeah, I am thinking of fetching stuff from the RSS feed caches, but that will take some time. I need to get the hosting and the essentials up first.
Thanks for the support :)
I’m sorry too
the horror! i’m gonna backup my photos on dvd’s all day today…
woooah… I had an idea from your twitter messages but thought it was just some one of your sites – just got the full impact of it now. And to think I felt sore when my Kannada language posts were corrupted after an upgrade – I just can’t believe the scale of losing data here – it sucks big time.
Glad to see you going in full swing again – looking forward to your brand new posts! And yeah lesson learnt, Im going to download my backup offline from now on every now and then thanks!
How do you understand the phrase “weekly backups on tape”? My understanding was that there’s a scheduled backup task (every weekend or so), which dumps data on tapes, and those tapes are moved out of the building somewhere.
– Exactly my understanding as well. Its so crazily incredible what they actually meant to say – it sounds like a cheap advertising gimmick for the hosting service actually! :)
Yeah, Sanjay, that’s pretty devastating. But I decided to adopt the “the king is dead! long live the king!” approach this time. :)
It’s very sad to lose a blog with such a huge amount of quality posts due to the incompetence of hosting providers. I hope you can recover as much as you can, and good luck with your new blog.
===Let this be yet another painful lesson on the importance of backups===
A lesson for ALL of us.
Glad to step in your new pile of posts :) and thank you for telling this terrible yet very important story in details so that we can learn from your experience.