{"id":24614,"date":"2015-08-20T15:00:44","date_gmt":"2015-08-20T13:00:44","guid":{"rendered":"https:\/\/mamchenkov.net\/wordpress\/?p=24614"},"modified":"2015-08-20T15:03:21","modified_gmt":"2015-08-20T13:03:21","slug":"backup-your-data-unless-you-are-google","status":"publish","type":"post","link":"https:\/\/mamchenkov.net\/wordpress\/2015\/08\/20\/backup-your-data-unless-you-are-google\/","title":{"rendered":"Backup your data! Unless you are Google"},"content":{"rendered":"\n

BBC reports<\/a> that one of the Google data centers experienced a data loss, after a nearby power power facility was struck by lightnings four times in a row. \u00a0Only about 0.000001% of total disk space was permanently affected, it is said.<\/p>\n

A thing called “backup” immediately comes to mind. \u00a0This was something I had to deal with in pretty much every company I worked for as a sysadmin. \u00a0Backup your data or lose it, right?<\/p>\n

Well, maybe. \u00a0For most of those companies a dedicated storage or a couple of tape drives could easily solve the problem. \u00a0But Google is often special in one way or the other.<\/p>\n

A quick Google search (hehe, yup) for how much data Google stores, brings up this article<\/a> from last year, linking to this estimation approach<\/a> – there are no officially published numbers, so the estimate is all we can do: 10-15 exabytes. \u00a0(10-15 exabytes, Carl!<\/a>) \u00a0And that’s from the last year.<\/p>\n

Using this method, they determined that Google holds somewhere around 10-15 exabytes<\/a> of data. If you are in the majority of the population that doesn\u2019t know what an exabyte is, no worries. An exabyte equals 1 million terabytes, a figure that may be a bit easier to relate to.<\/p><\/blockquote>\n

Holy Molly, that’s a lot of data! \u00a0To back this up, you’ll need at least double the storage. \u00a0And some really lightning-fast (pan intended) technology. \u00a0Just to give you an idea, some of the fastest tape drives have a throughput of about 1 TB \/ hour and a native capacity of about 10 TB (have a look here<\/a>, for example). \u00a0The backup process will take about … forever to complete.<\/p>\n

So if tapes are out, then we are backing up onto another storage. \u00a0Having the storage in the same data center sort of defeats the purpose (see above regarding “lightning”). \u00a0Having a storage in another data center (or centers) means you’ll need some super fast networks.<\/p>\n

You could probably do quite a bit of optimization with incremental and differential backups<\/a>, but you’d still need quite a substantial infrastructure.<\/p>\n

Simpler, I guess, just spread your data across many data centers with several copies all over the place, and hope for the best.<\/p>\n

But that’s for Google. \u00a0For the rest of us, backup is still an option. \u00a0(Read some of these horror stories<\/a> if you are not convinced yet.)<\/p>\n

And since we are on the subject of backups, let me ask you this: how are you doing backups? \u00a0Are you still with tapes, or local NAS, or, maybe, something cloud-based? \u00a0Which software do you use? What’s your strategy?<\/p>\n

For me, dealing with mostly small setups, Amazon S3 with HashBackup<\/a> is sufficient enough. \u00a0I don’t even need to rotate the backups anymore. Just do a full daily.<\/p>\n\n","protected":false},"excerpt":{"rendered":"\n

BBC reports that one of the Google data centers experienced a data loss, after a nearby power power facility was struck by lightnings four times in a row. \u00a0Only about 0.000001% of total disk space was permanently affected, it is said. A thing called “backup” immediately comes to mind. \u00a0This was something I had to … Continue reading Backup your data! Unless you are Google<\/span><\/a><\/p>\n\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Backup your data! Unless you are Google #Google #sysadmin #backup","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_links_to":"","_links_to_target":""},"categories":[1,133,62],"tags":[251,2068,2289],"keyring_services":[],"class_list":["post-24614","post","type-post","status-publish","format-standard","hentry","category-general","category-sysadmin","category-technology","tag-backup","tag-google","tag-web-hosting"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":18919,"url":"https:\/\/mamchenkov.net\/wordpress\/2013\/12\/05\/download-your-gmail-and-google-calendar-data-soon-or-now\/","url_meta":{"origin":24614,"position":0},"title":"Download your Gmail and Google Calendar data … soon or now","author":"Leonid Mamchenkov","date":"December 5, 2013","format":false,"excerpt":"I am a well known Google fan. \u00a0But even those who call it an Evil Corporation and a Global Spy, can't argue with the awesomeness of these news: Starting today we're rolling out the ability to export a copy of your Gmail and Google Calendar data, making it easy to\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"gmail data export","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2013\/12\/gmail_data_export-500x353.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":12062,"url":"https:\/\/mamchenkov.net\/wordpress\/2009\/12\/15\/disaster-recovery-plan\/","url_meta":{"origin":24614,"position":1},"title":"Disaster recovery plan","author":"Leonid Mamchenkov","date":"December 15, 2009","format":false,"excerpt":"While reading yet another sad story of the data loss, I came across a really awesome picture. \u00a0Whether you like it or not, it perfectly describes the disaster recovery plan and implementation in way too many companies.","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2009\/12\/dilbert-our-disaster-recovery-plan.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":27286,"url":"https:\/\/mamchenkov.net\/wordpress\/2017\/02\/01\/gitlab-horror-story-backup-restore-failure\/","url_meta":{"origin":24614,"position":2},"title":"GitLab horror story : backup \/ restore failure","author":"Leonid Mamchenkov","date":"February 1, 2017","format":false,"excerpt":"As I am reading this story - GitLab.com melts down after wrong directory deleted, backups fail\u00a0and these details - every single hair I have, moves ... I don't (and didn't) have any data on GitLab, so I haven't lost anything. \u00a0But as somebody who worked as a system administrator (and\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":22912,"url":"https:\/\/mamchenkov.net\/wordpress\/2014\/11\/18\/bup-very-efficient-backup-system-based-on-the-git-packfile-format\/","url_meta":{"origin":24614,"position":3},"title":"bup – very efficient backup system based on the git packfile format","author":"Leonid Mamchenkov","date":"November 18, 2014","format":"link","excerpt":"bup - very efficient backup system based on the git packfile format","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":18677,"url":"https:\/\/mamchenkov.net\/wordpress\/2013\/10\/15\/helium-app-sync-and-backup-for-android\/","url_meta":{"origin":24614,"position":4},"title":"Helium – App Sync and Backup for Android","author":"Leonid Mamchenkov","date":"October 15, 2013","format":"link","excerpt":"Helium - App Sync and Backup for Android","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":9997,"url":"https:\/\/mamchenkov.net\/wordpress\/2006\/03\/22\/daily-del-icio-us-bookmarks\/","url_meta":{"origin":24614,"position":5},"title":"Daily del.icio.us bookmarks","author":"Leonid Mamchenkov","date":"March 22, 2006","format":false,"excerpt":"Shared bookmarks for del.icio.us user tvset on 2006-03-21 mythago - Using Subversion for Backup Tagged as: article automation backup development documentation howto life linux personal productivity subversion sysadmin technology tips tools unix Bittorrent for torrent.fedoraproject.org -- Bittorrent for Fedora project. Tagged as: download fedora software torrent ONLamp.com -- Keeping Your\u2026","rel":"","context":"In "All"","block_context":{"text":"All","link":"https:\/\/mamchenkov.net\/wordpress\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":true,"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/24614","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/comments?post=24614"}],"version-history":[{"count":0,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/posts\/24614\/revisions"}],"wp:attachment":[{"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/media?parent=24614"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/categories?post=24614"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/tags?post=24614"},{"taxonomy":"keyring_services","embeddable":true,"href":"https:\/\/mamchenkov.net\/wordpress\/wp-json\/wp\/v2\/keyring_services?post=24614"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}