Category: Sysadmin

System administration is a special are of IT. It also has a special place in my heart. It is an interesting mixture of all the other disciplines, both common across the whole industry, and at the same time unique for each person, company, and geographical location. When I have something to say or share about system administration, I use this category.

CentOS 7.3 released

CentOS 7.3 was released rather quietly a couple of days ago.  Or maybe it wasn’t quietly, but I still somehow missed it.  Here is a list of major changes:

  • Since release 1503 (abrt>= 2.1.11-19.el7.centos.0.1) CentOS-7 can report bugs directly to bugs.centos.org.
  • Various new packages include among others: python-gssapi, python-netifaces, mod_auth_openidc, pidgin and Qt5.
  • Support for the 7th-generation Core i3, i5, and i7 Intel processors and I2C on 6th-generation Core Processors has been added.
  • Various packages have been rebased. Some of those are samba, squid, systemd, krb5, gcc-libraries, binutils, gfs-utils, libreoffice, GIMP,SELinux, firewalld, libreswan, tomcat and open-vm-tools.
  • SHA2 is now supported by OpenLDAP.
  • ECC-support has been added to OPenJDK-8, PerlNet:SSLeay and PerlIO::Socket::SSL.
  • Bluetooth LE is now supported.
  • virt-p2v is now fully supported. virt-v2v and virt-p2v add support for the latest windows releases.
  • Lots of updated storage, network and graphics drivers.
  • Technology Preview: Among others support of Btrfs, OverlayFS, CephFS, DNSSEC, kpatch, the Cisco VIC and usNIC kernel driver, nested virtualization with KVM and multi-threaded xz compression with rpm-builds.

More information is here.

Also, make sure you read the Known Issues section, as it might surprise you:

  • SElinux received major changes in this release, which might break certain functionality on your system. You might want to take a look at this bugzilla entry for further information.
  • The initramfs files are now significantly bigger than in CentOS-7 (1503). You may want to consider lowering installonly_limit in /etc/yum.conf to reduce the number of installed kernels if your /boot partition is smaller than 400MB. New installations should consider using 1GB as the size of the /boot partition.
  • The newer version of openssh in this release does not exit on the first match in the .ssh/config file as the older version did. This means if you have multiple host sections that match in your config for a given host, ALL will be applied. As an example, if you have a “host1.example.com” entry and a “*.example.com” entry, it will apply BOTH sets of instructions to “host1.example.com” but only the “*.example.com” section for “host2.example.com”.
  • Many people have complained that Ethernet interfaces are not started with the new default NetworkManager tool/have to be explicitly enabled during installation. See CentOS-7 FAQ#2.
  • At least 1024 MB RAM is required to install and use CentOS-7 (1611). When using the Live ISOs for install, 1024 MB RAM produces very slow results and even some install failures. At least 1344 MB RAM is recommend for LiveGNOME or LiveKDE installs.
  • If your screen resolution is 800×600 or lower, parts of the images shown at the bottom during install are clipped.
  • VMware Workstation/VMware ESXi allow to install two different virtual SCSI adapters: BusLogic and LsiLogic. However the default kernel from CentOS-7 does not include the corresponding driver for any of them thus resulting in an unbootable system if you install on a SCSI disk using the defaults for CentOS Linux. If you select ‘Red Hat Enterprise Linux’ as OS, the paravirtualized SCSI adapter is used, which works.
  • Commonly used utilities such as ifconfig/netstat have been marked as deprecated for some considerable time and the ‘net-tools’ package is no longer part of the @core group so will not be installed by default. Use nmcli c up ifname <interfacename> to get your network up and running and use yum to install the package if you really need it. Kickstart users can pull in the net-tools package as part of the install.
  • The AlpsPS/2 ‘ALPS DualPoint TouchPad’ edge scrolling does not work by default on CentOS-7. See bug 7403 for the command to make this feature work.
    After the update, some NICs may change their name from something like enoxxxxxxxx to something like ensxxx. This is due to the updated systemd package.
  • The 4 STIG Security Profiles in the anaconda installer produce a broken sshd_config that must be edited before sshd will start (BZ 1401069)

What does Operations *do*?

SysAdvent runs a blog post about Operations (in IT sense of a word), explaining what the department (hopefully it’s a department and not a single guy who doesn’t remember the meaning of the word “sleep”) does, and how the job complexity silently escalates with time.

An operations team, almost by definition, focuses on the steady-state “run” of whatever that team is responsible for … well, operating. This could be an electrical plant, an HR department, or an IT operations team; anything which needs to function day-in and day-out, reliably and without fuss. Those things the rest of us assume “just work”.

But just because we don’t notice doesn’t mean there isn’t anything happening. Much like an iceberg, where 90% of its mass sits below the waterline, nobody outside the team is aware of what the team actually does in order to make sure everything “just works”.

Anybody who’s been involved in Operations, at some point goes through this bit:

We can see how Oscar’s responsibilities grew over just two years. At first, it was just 6–8 laptops, office wifi, and a third-party office solution. Then it’s a cobbled-together server. Then development environments. Within a year, it’s 20 laptops, two application environments in the cloud, monitoring, alerting, and backups. After another 12 months, it’s a dozen third-party services, 40 laptops, 2 team members, offboarding processes, and monthly security audits.

Over beers one evening, Oscar makes the comment to a teammate that even he didn’t understand just how profoundly the company depended on the operations team; how much impact his work had on everyone else’s ability to do their jobs.

This blog post actually goes well with the one I am planning to write shortly about software features being like babies.  Stay tuned.

100 Favorite Programming, Computer and Science Books

Peteris Krumins, of the Browserling fame, has a series of blog posts on his top favorite programming, computer and science books.  It’s an excellent selection of titles, from which I’ve read only a fraction.  Good timing for the Christmas shopping too.  Here are the blog posts in the series so far (5 books per post):

Even with the 30 books mentioned so far, there are new things to read and learn.  I wonder how many of the notes to self I’ll have by the time the whole 100 are listed.

Amazon RDS and Amazon Virtual Private Cloud (VPC)

Yesterday I helped a friend to figure out why he couldn’t connect to his Amazon RDS database inside the Amazon VPC (Virtual Private Cloud).  It was the second time someone asked me to help with the Amazon Web Services (AWS), and it was the first time I was actually helpful.  Yey!

While I do use quite a few of the Amazon Web Services, I don’t have any experience with the Amazon RDS yet, as I’m managing my own MySQL instances.  It was interesting to get my toes wet in the troubleshooting.

Here are a few things I’ve learned in the process.

Lesson #1: Amazon supports two different ways of accessing the RDS service.  Make sure you know which one you are using and act accordingly.

gs-vpc-network

If you run an Amazon RDS instance in the VPC, you’ll have to setup your networking and security access properly.  This page – Connecting to a DB Instance Running the MySQL Database Engine – will only be useful once everything else is taken care of.  It’s not your first and only manual to visit.

Lesson #2 (sort of obvious): Make sure that both your Network ACL and Security Groups allow all the necessary traffic in and out.  Double-check the IP addresses in the rules.  Make sure you are not using a proxy server, when looking up your external IP address on WhatIsMyIP.com or similar.

Lesson #3: Do not use ICMP traffic (ping and such) as a troubleshooting tool.  It looks like Amazon RDS won’t be ping-able even if you allow it in your firewalls.  Try with “telnet your-rds-end-point-server your-rds-end-point-port” (example: “telnet 1.2.3.4 3306” or with a real database client, like the command-line MySQL one.

Lesson #4: Make sure your routing is setup properly.  Check that the subnet in which your RDS instance resides has the correct routing table attached to it, and that the routing table has the default gateway (0.0.0.0/0) route configured to either the Internet Gateway or to some sort of NAT.  Chances are your subnet is only dealing with private IP range and has no way of sending traffic outside.

Lesson #5: When confused, disoriented, and stuck, assume it’s not Amazon’s fault.  Keep calm and troubleshoot like any other remote connection issue.  Double-check your assumptions.

There’s probably lesson 6 somewhere there, about contacting support or something along those lines.  But in this particular case it didn’t get to that.  Amazon AWS support is excellent though.  I had to deal with those guys twice in the last two-something years, and they were awesome.

Taking the Pain Out of MySQL Schema Changes

“Taking the Pain Out of MySQL Schema Changes” covers the following approaches to deploying MySQL schema changes:

  1. Schema Change in Downtime
  2. Role Swap (cluster setup)
  3. pt-online-schema-change

The last one is the usage of pt-online-schema-change tool developed by Percona guys, as part of their Percona Toolkit – an Open Source set of command-line tools for MySQL.