Category: Sysadmin

System administration is a special are of IT. It also has a special place in my heart. It is an interesting mixture of all the other disciplines, both common across the whole industry, and at the same time unique for each person, company, and geographical location. When I have something to say or share about system administration, I use this category.

10 Favorite Job Interview Questions for Linux System Administrators

As someone who interviews a lot of people (mostly for the web development positions though, not system administration), I’m always looking for more ideas on what to ask the candidates.  Today I came across “10 Favorite Job Interview Questions for Linux System Administrators“, which has a few of bits that I liked.

First of all, this GitHub repository is super awesomeness.  It also links to a few other resources with more questions and ideas.  Not only for sysadmin interviews.

Then, this one is funny, yet somewhat challenging:

2. Name and describe a different Linux/Unix command for each letter of the alphabet. But also, describe how a common flush toilet works.

It also checks that you know the alphabet.

9. Print the content of a file backwards.

“I like broad questions where each person could give a different answer depending on their depth of knowledge. My personal answer is 8 characters not including the filename.” – Marc Merlin, Google.

This one caught me by surprise.  My immediate thought was “tac some_file“, but that’s obviously not enough.  tac only prints the lines in reverse order.  Which is not the same as reversing the file.  Perl to the rescue, but I wonder what’s the most elegant way to do it without the scripting language.

As always, interview questions are not only useful for the interviews.  They are a good measure of your own knowledge gaps and habit pitfalls.  This time was no exception.

Test your backups!

You can read all the books in the world and know all there is to know, but if you don’t follow the wisdom and practice the knowledge, then it’s all useless.  That’s my lesson from yesterday.

The Tao of Backup, which I linked to before, says:

backup testing

So, what happened?  Well, as I was preparing for the Fedora 24 installation, I wanted to backup some of my files, as the partition would be formatted.  I’ve connected an external USB drive with plenty of space and ZIP-archived a few of the vital directories on to it.

That was a very simple backup procedure and I saw the resulting files on the volume.  What else should I do, right?  Wrong!  I should have tested the restore.  I didn’t.

Most of the directories that I backed up were small – /etc, /opt, /root.  But my /home directory was about 20 GBs.  The external USB disk used the FAT-32 file system, which has a 4 GB file size limit.  So only the first 4 GBs of my /home folder were backed up.  Funny enough, those files were mostly browser cache and image thumbnails – stuff that should be ignored from backups.  The main two folders that I wanted – Desktop and .ssh were not part of the backup.  And I only realized that after the partition has been formatted.

So, yeah, I should have tested the backup.

P.S.: Gladly, I do have backups elsewhere, and most of my work is committed to GitHub/BitBucket anyways.

Fedora 24 : the day of 64-bit has come

I’ve been using 64-bit Linux distributions on the servers for a while now, but was reluctant to put one on my laptop.  I’ve tried a couple of times many years ago, and found that there were all sorts of weird issues.

Yesterday, with a little push from my brother, Google, and Slashdot, I’ve decided to give it another go.  64-bit Fedora 24 is now on my laptop and I am carefully exploring it.  So far, so good.

I guess I won’t have to worry about Year 2038 Problem after all.

SugarCRM cache directory – it is NOT a cache directory!

Here is a useful reminder from a few years back – “SugarCRM cache directory – it is NOT a cache directory!“.   Unlike most modern day web applications, which use cache/ folder for temporary files, which are safe to delete, SugarCRM keeps a bunch of stuff in there, which, if disappeared, would leave you in a very uncomfortable and confused stay.

Things have obviously improved over the years, but it’s still far from perfect.  And while we are on the subject of surprising issues with SugarCRM, make sure check my other post about working with encrypted values.  Basically, the summary is: backup, backup, backup!  If you want to sleep well at night, backup SugarCRM’s full file system (files, configurations, temporary files, caches, etc) and its database.  And never ever change anything.

Forcing Amazon Linux AMI compatibility with CentOS in Ansible

One of the things that makes Ansible so awesome is a huge collection of shared roles over at Ansible Galaxy.  These bring you best practices, flexible configurations and in general save hours and hours of hardcore swearing and hair pulling.

Each role usually supports multiple versions of multiple Linux distributions.  However, you’ll find that the majority of the supported distributions are Ubuntu, Debian, Red Hat Enterprise Linux, CentOS, and Fedora.  The rest aren’t as popular.

Which brings me to the point with Amazon Linux AMI.  Amazon Linux AMI is mostly compatible with CentOS, but it uses a different version approach, which means that most of those Ansible roles will ignore or complain about not supporting Amazon AMI.

Here is an example I came across yesterday from the dj-wasabi.zabbix-server role.  The template for the Yum repository uses ansible_os_major_version variable, which is expected to be similar to Red Hat / CentOS version number – 5, 6, 7, etc.  Amazon Linux AMI’s major version is reported as “NA” – not available.   That’s probably because Amazon Linux AMI versions are date-based – with the latest one being 2016.03.

[zabbix]
name=Zabbix Official Repository - $basearch
baseurl=http://repo.zabbix.com/zabbix/{{ zabbix_version }}/rhel/{{ ansible_distribution_major_version }}/$basearch/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX

Officially, Amazon Linux AMI is not CentOS or Red Hat Enterprise Linux.  But if you don’t care about such little nuances, and you are brave enough to experiment and assume things, than you can make that role work, by simply setting the appropriate variables to the values that you want.

First, here is a standalone test.yml playbook to try things out:

- name: Test
hosts: localhost
pre_tasks:
- set_fact: ansible_distribution_major_version=6
when: ansible_distribution == "Amazon"
tasks:
- debug: msg={{ ansible_distribution_major_version }}

Let’s run it and look at the output:

$ ansible-playbook test.yml

PLAY [Test] *******************************************************************

GATHERING FACTS ***************************************************************
ok: [localhost]

TASK: [set_fact ansible_distribution_major_version=6] *************************
ok: [localhost]

TASK: [debug msg={{ ansible_distribution_major_version }}] ********************
ok: [localhost] => {
"msg": "6"
}

PLAY RECAP ********************************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0

So far so good.  Now we need to integrate this into our playbook in such a way that the variable is set before the third-party role is executed.  For that, we’ll use pre_tasks.  Here is an example:

---
- name: Zabbix Server
hosts: zabbix.server
sudo: yes
pre_tasks:
- set_fact: ansible_distribution_major_version=6
when: ansible_distribution == "Amazon" and ansible_distribution_major_version == "NA"
roles:
- role: dj-wasabi.zabbix-server

A minor twist here is also checking if the major version is not set yet. You can skip that, or you can change it, for example, to examine the Amazon Linux AMI version and set corresponding CentOS version.