Google Infrastructure Security Design Overview

If you ever wanted to know what Google does to maintain its high level of security, here’s your chance. Google Infrastructure Security Design Overview provides quite a bit of information on the subject.

This document gives an overview of how security is designed into Google’s technical infrastructure. This global scale infrastructure is designed to provide security through the entire information processing lifecycle at Google. This infrastructure provides secure deployment of services, secure storage of data with end user privacy safeguards, secure communications between services, secure and private communication with customers over the internet, and safe operation by administrators.

How Linux got to be Linux: Test driving 1993-2003 distros

Here’s a trip down the memory lane – “How Linux got to be Linux: Test driving 1993-2003 distros“.  The article looks at some of the early Linux distributions, remember what was already in and what came later.  Complete with screenshots.

I don’t remember for sure which versions of which distributions I used in the early days, but Slackware, Suse, RedHat and Mandrake were definitely among those.  Slackware was probably my first one, when I found the floppies in the only book on Linux at my college library.  Then, somehow, I found RedHat (I think 5.1 or so) in one of the local computer shops.  Later I tried Mandrake and Suse, cause those were laying around at work.  But RedHat stuck with me ever since.  I think I’ve used pretty much every version, including the move to Fedora, CentOS, and even the Red Hat Enterprise Linux, which we had the licenses for at some of my early work places.

Fun times!

dotfiles – your unofficial guide to dotfiles on GitHub

Warning: you will lose a lot of sleep if you follow the link below. :)

No matter how well you know Vim, bash, git, and a whole slew of other command line tools, I promise you, you’ll find something new, something you had no idea existed, something that will help you save hours and hours of your life by shaving off a few seconds here and there on the tasks you perform on a daily basis, in the repositories link to from this site.

I think I’ve spent most of my Sunday there and my dotfiles are so different now that I’m not sure I should commit and push them all in one go.  I think I might need to get used to the changes first.

Some of the things that I’ve found for myself:

  • PHP Integration environment for Vim (spf13/PIV).
  • myrepos – provides a mr command, which is a tool to manage all your version control repositories.
  • bash-it – a community Bash framework.
  • Awesome dotfiles – a curated list of dotfiles resources.

… and a whole lot of snippets, tips, and tricks.

P.S.: Make sure you don’t spend too much time on these things though :)

Amazon Linux AMI : Let’s Encrypt : ImportError: No module named interface

Let’s Encrypt has only experimental support for the Amazon Linux AMI, so it’s kind of expected to have issues once in a while.   Here’s one I came across today:

# /opt/letsencrypt/certbot-auto renew
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Traceback (most recent call last):
File "/root/.local/share/letsencrypt/bin/letsencrypt", line 7, in <module>
from certbot.main import main
File "/root/.local/share/letsencrypt/local/lib/python2.7/dist-packages/certbot/", line 12, in <module>
import zope.component
File "/root/.local/share/letsencrypt/local/lib/python2.7/dist-packages/zope/component/", line 16, in <module>
from zope.interface import Interface
ImportError: No module named interface

My first though was to install the system updates. It looks like something is off in the Python-land. But even after the “yum update” was done, the issue was still there. A quick Google search later, thanks to the this GitHub issue and this comment, the solution is the following:

pip install pip --upgrade
pip install virtualenv --upgrade
virtualenv -p /usr/bin/python27 venv27

Running the renewal of the certificates works as expected after this.

P.S.: I wish we had fewer package and dependency managers in the world…

CentOS 7.3 released

CentOS 7.3 was released rather quietly a couple of days ago.  Or maybe it wasn’t quietly, but I still somehow missed it.  Here is a list of major changes:

  • Since release 1503 (abrt>= 2.1.11-19.el7.centos.0.1) CentOS-7 can report bugs directly to
  • Various new packages include among others: python-gssapi, python-netifaces, mod_auth_openidc, pidgin and Qt5.
  • Support for the 7th-generation Core i3, i5, and i7 Intel processors and I2C on 6th-generation Core Processors has been added.
  • Various packages have been rebased. Some of those are samba, squid, systemd, krb5, gcc-libraries, binutils, gfs-utils, libreoffice, GIMP,SELinux, firewalld, libreswan, tomcat and open-vm-tools.
  • SHA2 is now supported by OpenLDAP.
  • ECC-support has been added to OPenJDK-8, PerlNet:SSLeay and PerlIO::Socket::SSL.
  • Bluetooth LE is now supported.
  • virt-p2v is now fully supported. virt-v2v and virt-p2v add support for the latest windows releases.
  • Lots of updated storage, network and graphics drivers.
  • Technology Preview: Among others support of Btrfs, OverlayFS, CephFS, DNSSEC, kpatch, the Cisco VIC and usNIC kernel driver, nested virtualization with KVM and multi-threaded xz compression with rpm-builds.

More information is here.

Also, make sure you read the Known Issues section, as it might surprise you:

  • SElinux received major changes in this release, which might break certain functionality on your system. You might want to take a look at this bugzilla entry for further information.
  • The initramfs files are now significantly bigger than in CentOS-7 (1503). You may want to consider lowering installonly_limit in /etc/yum.conf to reduce the number of installed kernels if your /boot partition is smaller than 400MB. New installations should consider using 1GB as the size of the /boot partition.
  • The newer version of openssh in this release does not exit on the first match in the .ssh/config file as the older version did. This means if you have multiple host sections that match in your config for a given host, ALL will be applied. As an example, if you have a “” entry and a “*” entry, it will apply BOTH sets of instructions to “” but only the “*” section for “”.
  • Many people have complained that Ethernet interfaces are not started with the new default NetworkManager tool/have to be explicitly enabled during installation. See CentOS-7 FAQ#2.
  • At least 1024 MB RAM is required to install and use CentOS-7 (1611). When using the Live ISOs for install, 1024 MB RAM produces very slow results and even some install failures. At least 1344 MB RAM is recommend for LiveGNOME or LiveKDE installs.
  • If your screen resolution is 800×600 or lower, parts of the images shown at the bottom during install are clipped.
  • VMware Workstation/VMware ESXi allow to install two different virtual SCSI adapters: BusLogic and LsiLogic. However the default kernel from CentOS-7 does not include the corresponding driver for any of them thus resulting in an unbootable system if you install on a SCSI disk using the defaults for CentOS Linux. If you select ‘Red Hat Enterprise Linux’ as OS, the paravirtualized SCSI adapter is used, which works.
  • Commonly used utilities such as ifconfig/netstat have been marked as deprecated for some considerable time and the ‘net-tools’ package is no longer part of the @core group so will not be installed by default. Use nmcli c up ifname <interfacename> to get your network up and running and use yum to install the package if you really need it. Kickstart users can pull in the net-tools package as part of the install.
  • The AlpsPS/2 ‘ALPS DualPoint TouchPad’ edge scrolling does not work by default on CentOS-7. See bug 7403 for the command to make this feature work.
    After the update, some NICs may change their name from something like enoxxxxxxxx to something like ensxxx. This is due to the updated systemd package.
  • The 4 STIG Security Profiles in the anaconda installer produce a broken sshd_config that must be edited before sshd will start (BZ 1401069)

What does Operations *do*?

SysAdvent runs a blog post about Operations (in IT sense of a word), explaining what the department (hopefully it’s a department and not a single guy who doesn’t remember the meaning of the word “sleep”) does, and how the job complexity silently escalates with time.

An operations team, almost by definition, focuses on the steady-state “run” of whatever that team is responsible for … well, operating. This could be an electrical plant, an HR department, or an IT operations team; anything which needs to function day-in and day-out, reliably and without fuss. Those things the rest of us assume “just work”.

But just because we don’t notice doesn’t mean there isn’t anything happening. Much like an iceberg, where 90% of its mass sits below the waterline, nobody outside the team is aware of what the team actually does in order to make sure everything “just works”.

Anybody who’s been involved in Operations, at some point goes through this bit:

We can see how Oscar’s responsibilities grew over just two years. At first, it was just 6–8 laptops, office wifi, and a third-party office solution. Then it’s a cobbled-together server. Then development environments. Within a year, it’s 20 laptops, two application environments in the cloud, monitoring, alerting, and backups. After another 12 months, it’s a dozen third-party services, 40 laptops, 2 team members, offboarding processes, and monthly security audits.

Over beers one evening, Oscar makes the comment to a teammate that even he didn’t understand just how profoundly the company depended on the operations team; how much impact his work had on everyone else’s ability to do their jobs.

This blog post actually goes well with the one I am planning to write shortly about software features being like babies.  Stay tuned.

100 Favorite Programming, Computer and Science Books

Peteris Krumins, of the Browserling fame, has a series of blog posts on his top favorite programming, computer and science books.  It’s an excellent selection of titles, from which I’ve read only a fraction.  Good timing for the Christmas shopping too.  Here are the blog posts in the series so far (5 books per post):

Even with the 30 books mentioned so far, there are new things to read and learn.  I wonder how many of the notes to self I’ll have by the time the whole 100 are listed.

Amazon RDS and Amazon Virtual Private Cloud (VPC)

Yesterday I helped a friend to figure out why he couldn’t connect to his Amazon RDS database inside the Amazon VPC (Virtual Private Cloud).  It was the second time someone asked me to help with the Amazon Web Services (AWS), and it was the first time I was actually helpful.  Yey!

While I do use quite a few of the Amazon Web Services, I don’t have any experience with the Amazon RDS yet, as I’m managing my own MySQL instances.  It was interesting to get my toes wet in the troubleshooting.

Here are a few things I’ve learned in the process.

Lesson #1: Amazon supports two different ways of accessing the RDS service.  Make sure you know which one you are using and act accordingly.


If you run an Amazon RDS instance in the VPC, you’ll have to setup your networking and security access properly.  This page – Connecting to a DB Instance Running the MySQL Database Engine – will only be useful once everything else is taken care of.  It’s not your first and only manual to visit.

Lesson #2 (sort of obvious): Make sure that both your Network ACL and Security Groups allow all the necessary traffic in and out.  Double-check the IP addresses in the rules.  Make sure you are not using a proxy server, when looking up your external IP address on or similar.

Lesson #3: Do not use ICMP traffic (ping and such) as a troubleshooting tool.  It looks like Amazon RDS won’t be ping-able even if you allow it in your firewalls.  Try with “telnet your-rds-end-point-server your-rds-end-point-port” (example: “telnet 3306” or with a real database client, like the command-line MySQL one.

Lesson #4: Make sure your routing is setup properly.  Check that the subnet in which your RDS instance resides has the correct routing table attached to it, and that the routing table has the default gateway ( route configured to either the Internet Gateway or to some sort of NAT.  Chances are your subnet is only dealing with private IP range and has no way of sending traffic outside.

Lesson #5: When confused, disoriented, and stuck, assume it’s not Amazon’s fault.  Keep calm and troubleshoot like any other remote connection issue.  Double-check your assumptions.

There’s probably lesson 6 somewhere there, about contacting support or something along those lines.  But in this particular case it didn’t get to that.  Amazon AWS support is excellent though.  I had to deal with those guys twice in the last two-something years, and they were awesome.

Taking the Pain Out of MySQL Schema Changes

Taking the Pain Out of MySQL Schema Changes” covers the following approaches to deploying MySQL schema changes:

  1. Schema Change in Downtime
  2. Role Swap (cluster setup)
  3. pt-online-schema-change

The last one is the usage of pt-online-schema-change tool developed by Percona guys, as part of their Percona Toolkit – an Open Source set of command-line tools for MySQL.