What happened?

The long story short : I lost my blog, as well as a few other web sites.

Here goes the longer version.  I have been moving a whole bunch of web sites from my reseller hosting account at EuroVPS to a brand new VPS account at VAServ.  Site by site, blog by blog, database by database.  To keep things simple, once I made sure that the site was moved properly, I deleted the copy from the old hosting (after a week or so).

When I was almost done with the move and there were just a few more left, something really bad happened a VAServ.  All company’s servers got compromised.  The attackers gained control over thousands of VPS accounts across hundreds of servers, and then they deleted whatever they could.  The effect of this was so devastating that it was extensively covered in the news.

According the VAServ, hackers utilized a security hole in the HyperVM software, which was written by LXLabs.  Apparently, HyperVM is known for its poor security, but things never went wrong at this scale. (In other news, LXLabs founder was found dead in a suspected suicide a day or so later.  And the rumour has it that the break-in had nothing to do with HyperVM, but was VAServ negligence)

Now for the most interesting part of the story – the lost data.  How did that happen?  OK, the company got hacked and all data was deleted.  But what about the backups?  It turned out, there were no tape backups.  The only backups VAServ had were on the network storage.  And, of course, that data got deleted by the attackers.  Imagine that.  Web sites, databases, emails, DNS records.  Everything is gone.  Well, not everything – they managed to recover some servers, but not all by far.

My sites were on one of those servers which experienced 100% data loss, and which had no backup.  That was when I contacted EurVPS support and asked them to restore my recently deleted sites from tapes.  After all, it’s better to lose a few weeks of work, rather than a few years.  Guess what?  It turned out, EuroVPS has no backups either.  When I insisted, saying that backups are a part of my hosting plan, they corrected themselves and said that they have backups, but, as advertised on the site – weekly only.

Screenshot
Screenshot

Let me ask you a simple question.  How do you understand the phrase “weekly backups on tape”?  My understanding was that there’s a scheduled backup task (every weekend or  so), which dumps data on tapes, and those tapes are moved out of the building somewhere.  Eventually, of course, they are rotated (monthly, or annually, or so).  But there is a certain period which you can go back to and restore from those weekly tapes.

It so happened, my understanding was wrong.  Weekly tape backup means one backup within a week on tape.  That is, there is no way to go more than one week back using tape backups.  I was shocked a bit, but there was still a chance to get something.  I clearly remember that I deleted two sites five days ago.  I asked EuroVPS support to restore at least those.  To which they replied that those two sites aren’t on the backups either.

What?  How? Err…  I know, of course, that the loss of data is my fault as much as theirs. I should have done my own backups, downloading them to my own machine.  And I’m deeply sorry for not doing so.  But on the other hand, having paid for hosting, I do expect uninterrupted power, redundant network connection, and properly organized backups.  If that’s not how commercial hosting is different from home servers, than I don’t know how.

Currently, I am setting up a new VPS host, reconfiguring domains for the new IP, installing a bunch of WordPress blogs, and issuing a whole lot of apologies.  Those things that can be recovered, will be recovered.  Those things that were important and were lost, will be started a new.  And those things that were not important and were lost, will remain lost.

Let this be yet another painful lesson on the importance of backups.

Hosting downtime

None of the sites hosted on my sever were accessible for most of yesterday.  That was caused by some emergency maintenace done by the hosting company.  They didn’t warn me before, so I weren’t aware of it coming and for how long it would last.

This is the third downtime for this month.  Needless to say, I am not satisfied with the service no more.  Firstly, the downtimes are too frequent and too lengthy.  Secondly, total absense of notificatios – either before the downtime or after.  No explanations.  Nothing.

I’ve been with this hosting company for more than two years now and it was OK most of the time.  But now, once again, I am thinking about moving somewhere else.   Suggestions?

Web technology behind Cyprus presidential elections

Cyprus is preparing for the presidential elections, which will take place this coming Sunday – February 17th, 2008 – and then another Sunday after that – February 24th, 2008. Unfortunately, most of the information about the elections is in Greek, so there isn’t much point in linking to it or quoting it.

Anyway, I came across this post in Linkbox blog, which links to web sites of some candidates, as well as the main web site of the elections.  Being a curious web worker, I wanted to see which tools these web sites use, and how well they use them.  Here are my findings.

Continue reading “Web technology behind Cyprus presidential elections”

Cyprus web hosting? Really? Think again …

I’m far from honest myself, but at least I don’t charge for my lies. Plenty of people do, however. One of the commonly told lies that I come across rather often these days is the one of Cyprus web hosting. Some companies are telling lies out loud, others are just being overly silent about the truth. But the fact of the matter is that Cyprus web hosting is almost non-existing. There are only just a few companies providing web hosting in Cyprus (PrimeTel and CytaNet, for example), and in most of the cases, their offerings don’t make sense financially. That’s one of the biggest reasons for most of Cyprus web sites being hosted outside of Cyprus, in countries like USA, UK, Germany, and Netherlands.

Still, most of the web design and development companies in Cyprus offer “Cyprus web hosting”. Here is just one example, that I found (ironically, I found it through AdSense block on my own blog) – CyprusWebSpace.com . Here is a partial snapshot of their front page.

CyprusWebSpace.com

If you are using Firefox browser with Flagfox plugin, than you will immediately notice the United States of America flag in the bottom right corner of your browser window. If you don’t have the plugin or don’t trust the information that it provides, or use a totally different browser, here is how you can arrive to the same results:

  1. Go to DomainTools.com
  2. Type cypruswebspace.com in the Whois Lookup form at the top of the page and press the Search button
  3. Scroll down the resulting page to where it says “Server Data”
  4. Find the line with the American flag and a description “Texas – Dallas – Theplanet.com Internet Services Inc”.
  5. Bonus point : look at the map, provided by GeoTool. The area you see is pretty far from Cyprus.

Now, there is always this argument, that the web hosting company runs its own web site on a different server from the one that it uses for its customers. It is a theoretically valid point, but the one I haven’t seen in practice just yet. Most of the web companies offering web hosting in Cyprus only have a single server (usually it is a shared or dedicated server, or a VPS account), which they use for all of their hosting needs.

There is also another argument to this point – what exactly is a Cyprus web hosting? Is it hosting on the web server which physically resides in Cyprus, or is hosting on a web server, which is assigned one of the Cyprus IP addresses? And it is, in fact, an interesting point. There is significant difference between the two. If the server is physically located in Cyprus, then chances are, that it will have a lower latency, meaning that web sites on such a server will appear to respond much faster to visitors who are also in Cyprus. This is what matters for many site owners. On the other hand, Cyprus IP address of the web server might be more important to other site owners, since some search engines use this information in scoring their search results (that is considering, for example, that web sites residing on servers with Cyprus IP addresses are more relevant to Cyprus related queries than web sites residing on other web servers).

Technically, it is quite possible for a web server to be physically located in one country, while carrying the IP address, which belongs to another country. But situations like this, aren’t so common, and that is specifically the reason why search engines (and other tools) utilize the information of IP address location.

So, back to the issue of Cyprus web hosting. How bad is it? Let’s see. Examine the first 10 or so results for each of these queries at Google, using the method above:

Impressed? I am. Results:

  • Companies with web sites physically located in Cyprus: 0 (zero).
  • Companies with web sites hosted on a server with Cyprus IP address: 1 (one).

How do I know about this one company with Cyprus IP address that it doesn’t have the server physically located in Cyprus? Because I am an employee of that company (disclaimer and blah blah blah). And even if I wasn’t, finding this information is rather trivial with any traceroute tool (here is one, for example).

So, keep this stuff in mind before you pay for you Cyprus web hosting next time. Maybe this Cyprus offering is not so Cyprus after all. But then again, maybe you just don’t care…

Downtime, with database error

Apologies to those of you who noticed a brief downtime last night.  My hosting company was moving the database from one IP address to another, and they have warned me about it a few times early on the way.  All I had was to update configuration of the blog, but somehow that slipped through my memory.  Once I noticed the problem, it was trivial to fix.  Everything is back and should be working just fine now.

This blog wasn’t the only one affected – I’m hosting quite a few  web sites, and most of them had the same problem.  As far as I can see, all of them are up now.  Let me know if you notice that something is still broken.

Managed dedicated hosting anyone?

If anyone of you guys knows of any good hosting company that offers managed dedicated servers, now is an excellent time to let me know via comments or the contact form. So far the best I’ve found is XLHost.com . I am also talking with Rackspace.com, but something tells me that they will be a bit too expensive – not that I am jumping to conclusions here though.

I’ll need two servers to start with. I might grow up to anywhere from 6 to 20 in the next 6-8 month. Servers should have fast processors (3.0 GHz is ideal). Better even if they will be duel CPUed. 2 GBytes of RAM should fit me fine. I am not yet sure about the storage. I know that it has to be SCSI and that there should be at least 40-60 GBytes of it. Maybe more. I’ll have better numbers later. I will also need a lot of bandwidth. Both incoming and outgoing. 20 GBytes per month is the red line minimum. 200 GBytes per month is something I feel more comfortable with. 2 TBytes will make me smile one extra time.

Software-wise, I’ll need a Linux-only setup. Fedora Core 4 is preferrable, but anything Red Hat labeled should do just fine. I’ll need MySQL 4 or above, perl 5.8 with A LOT of CPAN modules that I’ll need to install myself, python 2.4, and a Subversion client.

What do I want from the hosting company? Well, I want my servers to be available 24×7. That’s the main requirement. Then, I’ll need their help with backup configuration. I’ll have a large MySQL database to backup and a lot of small files (think mail spool and proxy cache scale). Also, I would expect them to manage security updates and fixes for all the servers – I’m really out of time to keep up with that right now.

An additional strong wishlist item would be a LAN interconnecting all my servers. I’ll have a lot of traffic between the servers and I don’t see any reason why I should pass it via a outside network, where it is slow and expensive.

That’s about it.

Oh, the budget line? Let’s say anything within $300 USD.

Are you still with me?