10 Favorite Job Interview Questions for Linux System Administrators

As someone who interviews a lot of people (mostly for the web development positions though, not system administration), I’m always looking for more ideas on what to ask the candidates.  Today I came across “10 Favorite Job Interview Questions for Linux System Administrators“, which has a few of bits that I liked.

First of all, this GitHub repository is super awesomeness.  It also links to a few other resources with more questions and ideas.  Not only for sysadmin interviews.

Then, this one is funny, yet somewhat challenging:

2. Name and describe a different Linux/Unix command for each letter of the alphabet. But also, describe how a common flush toilet works.

It also checks that you know the alphabet.

9. Print the content of a file backwards.

“I like broad questions where each person could give a different answer depending on their depth of knowledge. My personal answer is 8 characters not including the filename.” – Marc Merlin, Google.

This one caught me by surprise.  My immediate thought was “tac some_file“, but that’s obviously not enough.  tac only prints the lines in reverse order.  Which is not the same as reversing the file.  Perl to the rescue, but I wonder what’s the most elegant way to do it without the scripting language.

As always, interview questions are not only useful for the interviews.  They are a good measure of your own knowledge gaps and habit pitfalls.  This time was no exception.

Test your backups!

You can read all the books in the world and know all there is to know, but if you don’t follow the wisdom and practice the knowledge, then it’s all useless.  That’s my lesson from yesterday.

The Tao of Backup, which I linked to before, says:

backup testing

So, what happened?  Well, as I was preparing for the Fedora 24 installation, I wanted to backup some of my files, as the partition would be formatted.  I’ve connected an external USB drive with plenty of space and ZIP-archived a few of the vital directories on to it.

That was a very simple backup procedure and I saw the resulting files on the volume.  What else should I do, right?  Wrong!  I should have tested the restore.  I didn’t.

Most of the directories that I backed up were small – /etc, /opt, /root.  But my /home directory was about 20 GBs.  The external USB disk used the FAT-32 file system, which has a 4 GB file size limit.  So only the first 4 GBs of my /home folder were backed up.  Funny enough, those files were mostly browser cache and image thumbnails – stuff that should be ignored from backups.  The main two folders that I wanted – Desktop and .ssh were not part of the backup.  And I only realized that after the partition has been formatted.

So, yeah, I should have tested the backup.

P.S.: Gladly, I do have backups elsewhere, and most of my work is committed to GitHub/BitBucket anyways.

Fedora 24 : the day of 64-bit has come

I’ve been using 64-bit Linux distributions on the servers for a while now, but was reluctant to put one on my laptop.  I’ve tried a couple of times many years ago, and found that there were all sorts of weird issues.

Yesterday, with a little push from my brother, Google, and Slashdot, I’ve decided to give it another go.  64-bit Fedora 24 is now on my laptop and I am carefully exploring it.  So far, so good.

I guess I won’t have to worry about Year 2038 Problem after all.

Forcing Amazon Linux AMI compatibility with CentOS in Ansible

One of the things that makes Ansible so awesome is a huge collection of shared roles over at Ansible Galaxy.  These bring you best practices, flexible configurations and in general save hours and hours of hardcore swearing and hair pulling.

Each role usually supports multiple versions of multiple Linux distributions.  However, you’ll find that the majority of the supported distributions are Ubuntu, Debian, Red Hat Enterprise Linux, CentOS, and Fedora.  The rest aren’t as popular.

Which brings me to the point with Amazon Linux AMI.  Amazon Linux AMI is mostly compatible with CentOS, but it uses a different version approach, which means that most of those Ansible roles will ignore or complain about not supporting Amazon AMI.

Here is an example I came across yesterday from the dj-wasabi.zabbix-server role.  The template for the Yum repository uses ansible_os_major_version variable, which is expected to be similar to Red Hat / CentOS version number – 5, 6, 7, etc.  Amazon Linux AMI’s major version is reported as “NA” – not available.   That’s probably because Amazon Linux AMI versions are date-based – with the latest one being 2016.03.

[zabbix]
name=Zabbix Official Repository - $basearch
baseurl=http://repo.zabbix.com/zabbix/{{ zabbix_version }}/rhel/{{ ansible_distribution_major_version }}/$basearch/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX

Officially, Amazon Linux AMI is not CentOS or Red Hat Enterprise Linux.  But if you don’t care about such little nuances, and you are brave enough to experiment and assume things, than you can make that role work, by simply setting the appropriate variables to the values that you want.

First, here is a standalone test.yml playbook to try things out:

- name: Test
hosts: localhost
pre_tasks:
- set_fact: ansible_distribution_major_version=6
when: ansible_distribution == "Amazon"
tasks:
- debug: msg={{ ansible_distribution_major_version }}

Let’s run it and look at the output:

$ ansible-playbook test.yml

PLAY [Test] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [set_fact ansible_distribution_major_version=6] *************************
ok: [localhost]

TASK: [debug msg={{ ansible_distribution_major_version }}] ********************
ok: [localhost] => {
"msg": "6"
}

PLAY RECAP ********************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0

So far so good.  Now we need to integrate this into our playbook in such a way that the variable is set before the third-party role is executed.  For that, we’ll use pre_tasks.  Here is an example:

---
- name: Zabbix Server
hosts: zabbix.server
sudo: yes
pre_tasks:
- set_fact: ansible_distribution_major_version=6
when: ansible_distribution == "Amazon" and ansible_distribution_major_version == "NA"
roles:
- role: dj-wasabi.zabbix-server

A minor twist here is also checking if the major version is not set yet. You can skip that, or you can change it, for example, to examine the Amazon Linux AMI version and set corresponding CentOS version.

Amazon Elastic File System

Here are some great news from the Amazon AWS blog – the announcement of the Elastic File System (EFS):

EFS lets you create POSIX-compliant file systems and attach them to one or more of your EC2 instances via NFS. The file system grows and shrinks as necessary (there’s no fixed upper limit and you can grow to petabyte scale) and you don’t pre-provision storage space or bandwidth. You pay only for the storage that you use.

EFS protects your data by storing copies of your files, directories, links, and metadata in multiple Availability Zones.

In order to provide the performance needed to support large file systems accessed by multiple clients simultaneously,Elastic File System performance scales with storage (I’ll say more about this later).

I think this might have been the most requested feature/service from Amazon AWS since EC2 launch.  Sure, one could have built an NFS file server before, but with the variety of storage options, availability zones, and the dynamic nature of the cloud setup itself, that was quite a challenge.  Now – all that and more in just a few clicks.

Thank you Amazon!