Month: June 2009

Fedora 11

Fedora 11 – code named Leonidas – was released just a few days ago. Having access to an excellent Internet connection in the office, I immediately downloaded it and upgraded. It didn’t go as smooth as I wish it did, and I still don’t have everything working properly, but I’m glad that I did upgrade.

Here are the issues that I came across:

I was upgrading using the preupgrade utility. It downloaded the packages nicely and created a “Upgrade …” option in the grub boot loader. Once I rebooted, the upgraded process started and everything was looking good. However, when the last package was installed, and a popup came up saying “Finalizing install process. Please wait, it can take a while”, the upgrade actually hang. The progressbar was going back and forward, but nothing was happening. I waited for about 40 minutes or so and decided to reboot the machine. Upon the reboot, it seemed like the upgrade process actually did everything it had to do. So that was a minor issue.
My dual monitor setup broke. I am using a Lenovo T61 laptop with an external Samsung SyncMaster 2053bw monitor. Laptop’s resolution remained at 1280×1024, but Samsung monitor went down to 1024×768 and it seems there is no way to push it up. This is probably due to new kernel and xorg, and I guess has something to do with kernel mode setting. I tried those few tips that I could find, but nothing worked. I still have the problem, so if you by any chance can suggest anything – I am all ears.
Once I got to my desktop after the upgrade, Firefox refused to start. It was crashing with a whole bunch of debug output, but nothing that made any sense. I had to spend a day in Opera, which turned out to be a nice browser for as long as you don’t need your extensions. Gladly, the Firefox problem as resolved the next day with the help of some folks in #fedora IRC channel. The issue was a plugin conflict (not addon, but plugin). Once I removed conflicting plugins and restarted Firefox – it worked automagically.
Flash was broken in the browser, but once I removed all plugins and re-installed the ones that I needed, the problem was solved.

That was pretty much all in the troubles department. Now for the good stuff.

My filesystem wasn’t upgraded to ext4. This is probably a bug or something, but I’m glad it wasn’t. I have a single / partition with /boot, /home, /var, and everything else on this same partition. Probably Fedora decided that it’s too precious for the upgrade.
Booting is faster now. I can feel it. And being on the laptop, I do lots of booting (between home and office, due to very different setups).
KDE 4 is a pleasantly usable stage. After using Gnome for the last year, I decided to give KDE 4 another chance, and it happily took it. It’s a fast, slick, efficient desktop. And I enjoy using it.
Firefox 3.5 beta 4 is way faster than whatever I had before. Some addons are still not supporting this version, but the speed with which it starts up and renders pages provides for some balance.

So, apart from having just one screen now, I am enjoying the ride. It was worth waiting for, downloading, and upgrading. And the screen … the screen will come.

What will happen

The data loss was horrible, but life still goes on. Since I already told you what happened, I’d like to share my thoughts on what will happen.

Firstly, as you can see, here goes a new blog – from scratch, with no archives or pretty looks. Just yet. As with everything else, there is some good in everything bad. And the data loss is not an exception. Having archives for 10+ years is cool. But it also requires a tremendous amount of work in maintenance – moving things around, correcting them, making sure everything is at least visible in yet another new theme – that takes a lot of effort. With all those archives gone now, I don’t have to restructure or re-organize. I can start a new. And I will.

Secondly, I will move movie reviews into a separate blog. I tried it already before and then merge the reviews back in. But now I think the time came to try it once again. I’ve been trying quite a few ideas for my reviews lately, and I like where it goes. For example, using directors, actors, and genres as tags in separate taxonomies finally made it easier for me to find movies by actor or director combination. There is a lot that can be expanded and tried out there. So, stay tuned.

Thirdly, I want to extend the networking part of mamchenkov.net a bit. Back when I bought the domain, I was the only Mamchenkov using it. Now there is a small crowd, with my brother, my mother (blog lost together with mine, but will be restarted), and a few posts that I’ve been doing on behalf of my kid (two very outdated blogs, one of which was lost too). Plus there is a whole range of RSS feeds from Flickr, Twitter, Delicious, and such, that should be grouped together somehow. I am trying out the Planet software for that and so far it looks good.

Fourthly, with all those archives gone and blogs destroyed, there will be a huge black void, which, I am sure, will inspire more blogging. That is of course just my wild guess.

Now that my mind recovers from the loss (not without the help of friends and alcohol), I am having more and more ideas on what I want to do and where I want to go with this. I hope I’ll have the will and strength to pull it over.

What happened?

The long story short : I lost my blog, as well as a few other web sites.

Here goes the longer version. I have been moving a whole bunch of web sites from my reseller hosting account at EuroVPS to a brand new VPS account at VAServ. Site by site, blog by blog, database by database. To keep things simple, once I made sure that the site was moved properly, I deleted the copy from the old hosting (after a week or so).

When I was almost done with the move and there were just a few more left, something really bad happened a VAServ. All company’s servers got compromised. The attackers gained control over thousands of VPS accounts across hundreds of servers, and then they deleted whatever they could. The effect of this was so devastating that it was extensively covered in the news.

According the VAServ, hackers utilized a security hole in the HyperVM software, which was written by LXLabs. Apparently, HyperVM is known for its poor security, but things never went wrong at this scale. (In other news, LXLabs founder was found dead in a suspected suicide a day or so later. And the rumour has it that the break-in had nothing to do with HyperVM, but was VAServ negligence)

Now for the most interesting part of the story – the lost data. How did that happen? OK, the company got hacked and all data was deleted. But what about the backups? It turned out, there were no tape backups. The only backups VAServ had were on the network storage. And, of course, that data got deleted by the attackers. Imagine that. Web sites, databases, emails, DNS records. Everything is gone. Well, not everything – they managed to recover some servers, but not all by far.

My sites were on one of those servers which experienced 100% data loss, and which had no backup. That was when I contacted EurVPS support and asked them to restore my recently deleted sites from tapes. After all, it’s better to lose a few weeks of work, rather than a few years. Guess what? It turned out, EuroVPS has no backups either. When I insisted, saying that backups are a part of my hosting plan, they corrected themselves and said that they have backups, but, as advertised on the site – weekly only.

Let me ask you a simple question. How do you understand the phrase “weekly backups on tape”? My understanding was that there’s a scheduled backup task (every weekend or so), which dumps data on tapes, and those tapes are moved out of the building somewhere. Eventually, of course, they are rotated (monthly, or annually, or so). But there is a certain period which you can go back to and restore from those weekly tapes.

It so happened, my understanding was wrong. Weekly tape backup means one backup within a week on tape. That is, there is no way to go more than one week back using tape backups. I was shocked a bit, but there was still a chance to get something. I clearly remember that I deleted two sites five days ago. I asked EuroVPS support to restore at least those. To which they replied that those two sites aren’t on the backups either.

What? How? Err… I know, of course, that the loss of data is my fault as much as theirs. I should have done my own backups, downloading them to my own machine. And I’m deeply sorry for not doing so. But on the other hand, having paid for hosting, I do expect uninterrupted power, redundant network connection, and properly organized backups. If that’s not how commercial hosting is different from home servers, than I don’t know how.

Currently, I am setting up a new VPS host, reconfiguring domains for the new IP, installing a bunch of WordPress blogs, and issuing a whole lot of apologies. Those things that can be recovered, will be recovered. Those things that were important and were lost, will be started a new. And those things that were not important and were lost, will remain lost.

Let this be yet another painful lesson on the importance of backups.