Category: Sysadmin

System administration is a special are of IT. It also has a special place in my heart. It is an interesting mixture of all the other disciplines, both common across the whole industry, and at the same time unique for each person, company, and geographical location. When I have something to say or share about system administration, I use this category.

Go Monorepo

As a maintainer and contributor to multiple, and often related, git repositories, I often find myself thinking that there must be a simpler way.

Consider, for example, a variety of CakePHP plugins that we develop and maintain at work.  Sure, each one provides a completely separate bit of functionality and has its purpose as a standalone project.  But, at the same time, they all have enough overlap that we use a common cakephp-plugin-template repository to keep things in sync, and also require cakephp-utils plugin from all of them.

It’s only natural to consider other ways of doing things.  Sure, I tried git submodules before, but they don’t quite cut it.  Something is still missing.

Today I came across the Go Monorepo website.  Which makes me wonder even further down this road.  There are also a few other tools and alternatives mentioned in this article.

That’s not something that I’ll jump into right now, but it’s definitely something I’ll consider to try out in the nearest future.

AnsibleFest Austin 2018 via Jeff Geerling

If you missed the opportunity to attend the recent AnsibleFest Austin 2018 event,  here are a couple of interesting links for you, via Jeff Geerling’s blog (aka geerlingguy):

There’s plenty of stuff to play with over the next weekend or two.

HipChat to Slack migration

Back in July, Slack acquired Hipchat with the intent of migrating all the users and shutting down the service.  For us at work, these were very sad news.  We’ve been using HipChat for years, and relied on it heavily.  We have also evaluated Slack in its early days and didn’t quite like it at the time.

Once it was clear that the HipChat is going away, we had a couple of options to consider.  One was to move to Slack.  Another – to migrate to a different solution, preferably to a self-hosted one, like Rocket.Chat.

So, first thing we did, was another test trial of Slack.  We were glad to see that it matured and improved a lot over the last few years, and that it would be faster and easier for us to move to Slack rather than to roll out our own self-hosted solution.

Here are a few things that we had to keep in mind for this migration:

  • We have a rather small team – about 15-20 people altogether, but with a very mixed backgrounds.  Some are very technical, some are not.  Some are on Windows, some on Linux, and some on MacOS.  Mobile use is also heavy across the board.
  • We are working with a lot of rooms/channels.  We have a few generic ones, like Support, Sales, Management, etc.  And we have a separate channel for every project.  And those we have over a hundred of.
  • We have a number of integrations connected – GitHub and BitBucket repositories, Zabbix monitoring system, RSS feeds, deployment automation, etc.
  • HipChat would only be around for the next month.  Not a tight deadline by our means, but the timeline is important here.

Since the time was limited from the beginning, we decided to start with the migration pretty much immediately, rather than waiting until the last moment.  In case something wasn’t working, we’d have some space to maneuver.

Instead of a once off switch, we also decided to do a gradual migration, shifting from HipChat to Slack bit by bit, using both services in parallel.

Creating the new workspace and registering all the users was, as you can imagine, the easiest part.  After all we do have only a small team, and that many accounts can be done manually, without any automation what so ever.

That was an option for the rooms/channels though, as we have hundreds of those.  Gladly, both Slack and HipChat provide an API which helps with automation, and we have awesome DevOps who handle things like that.

Here comes the first issue that we came across.  HipChat allows for much more when it comes to room names, than Slack.  Slack channels are limited to 21 characters (which is not a lot), and the selection of the characters is much less also (dashes are in, but slashes and other special characters are out).

That meant that we had to come up with the new naming convention, which we did.  We also used this opportunity to consolidate some of the HipChat rooms into fewer Slack channels.   In HipChat, as I mentioned, we had a separate room for each project.  Which was handy for busy projects, but somewhat confusing for smaller projects that belonged to the same client.  For example, it’s not a rare occurrence for us to have two, three or more projects for a single client.  And while they are all different, often they overlap a lot (websites for a group of company, or a website integrated with the CRM, etc).

Once we decided on the adjustments to the naming convention and consolidation of some rooms, the migration wasn’t too difficult, given the API of the both services.

Next came the integrations.  This is where, I have to say, Slack shines.  While HipChat integrates with both BitBucket and GitHub too, the functionality provided by Slack is a lot better and easier.  Both GitHub and BitBucket are available out of the box, so we quickly connected all the project channels to their appropriate repositories.

RSS feeds weren’t a problem either.  So the next one that we had to do was the Zabbix monitoring system.  Again, Zabbix is pretty good with integrating to other systems, Slack API is flexible, and our DevOps are amazing.  So that took no time at all. In fact, we even made it better, since Slack allows for richer and better notifications.

The last, but the most important one, was our deployment automation.  It wasn’t painful, but it did require us to do some adjustments.  Some of these we wanted to make for a while now anyway.  The biggest change was that HipChat didn’t care about the long running tasks, while Slack expects an answer from your webhook within 5 seconds.  Many of out deployments take longer than that (some are up to 20 minutes), so we had to do some changes.

When the chat webhook request for deployment is received, we couldn’t wait for the deployment to finish anymore to send the response back.  We had to respond within 5 seconds.  That meant that we had to disconnect the actual deployment run from the chat communications.  As I said, we wanted to do that for a while anyway.  A quick proof of concept with the Linux “at” command worked quite well, so stayed with that for the time being.  Now when the request is received, the deployment is scheduled with the “at” command for the next minute, and the response is sent to the chat within a second.

That was about it.  Since we didn’t care about migrating the actual chat histories or uploaded files or anything like that (we use Redmine for anything that needs persistence), the migration overall was smooth.

Once all the users were registered, channels created, and integrations configured, we started to phase out HipChat rooms one-by-one, archiving and disabling them on HipChat and pushing users more and more towards Slack.

Overall, the setup and configuration took us about three weeks.  The rest of the migration with the users and phasing out of HipChat took approximately another two weeks.

So, now that we’ve been using Slack exclusively for about a month and a half, how do we find it?  I think overall it’s a lot better now that it was with HipChat.  Here are some of the benefits:

  • The mobile app is amazing.  It supports multiple workspaces (some of our clients are also using Slack and we need to be in multiple places at the same time), has a great user interface, and, most importantly, the push notifications are working A LOT BETTER.  HipChat had a tendency of going offline after about 10 minutes and being completely silenced until the next manual check.  Slack even provides a Snooze option for 20 minutes, 1 hours, etc, which makes it easy to silence it during a meeting.
  • Slack desktop and web applications seem to be better enjoyed by non-technical users, than their HipChat alternatives were.  For me personally, HipChat was easier on the eye, but Slack provides a lot of functionality which wasn’t in HipChat.
  • Threads!  I think this is one of the greatest features that HipChat was missing.  Project rooms especially can get quite noisy.  It’s often difficult to follow up with all the chatter.  Threads make it a lot easier.  With threads there can be multiple concurrent conversations in the same channel without any confusion or overlap.
  • Reminders!  Again, for those of us who are busy with a million things, reminders in Slack are a killer feature.  They help to find that message that needs a response, half an our later, in a hundred rooms with dozens of conversations going on at the same time.
  • Slash commands.  While both HipChat and Slack have them, Slack allows to send the same slash command from any room (handy for deployments).  With HipChat we were a lot more limited, which pushed us into a separate room for all deployments.
  • Emojis.  Usually, I’m the last person to use emojis.  I’m the old school smiley guy.  But in this particular context, I find emojis to be very useful.  They often save a lot of typing and help keep the atmosphere nice and friendly.  HipChat had a very weird approach to emojis.  While they were there, they were very few, and very custom to the HipChat team.  We did upload a lot of our own over time, but with Slack that wasn’t necessary.
  • Stats and analytics.  Slack provides a great insight into how the workspace is being used.  Stats are available both as an admin menu item, and as a quick weekly email.
  • Integrations and API.  Slack provides a much more flexible API and integration options.  While we were doing fine with HipChat, I think all of our integrations are much better now because of the move to Slack.  (slash commands, updated messages, message replies, file attachments, rich messages, etc).

As I said, overall the migration went quite smooth and we are in a much better place now.  This is quite surprising and refreshing, especially after us considering the news of HipChat acquisition as a sad story.  In fact, I can’t remember, from the top of my head, when a sad story like this turned out to be such an improvement.

Thank you HipChat team for all the effort – you’ve been great for the last few years.  Thank you Slack for making things even better!

Highlights from Git 2.19 and PHP syntax

GitHub blog covers the highlights from Git 2.19, which was recently released.

One particular change that I want to highlight (which GitHub skipped in their highlights, but which exists in the Git release notes for this version) is the syntax pattern update for the PHP files.

Let’s have a look at the full diff of this release.  The particular change I am referring to is in the userdiff.c file.   Git now understands final, abstract, interface, and trait keywords.

If you are not sure where and how it is used, here are a couple of useful links for you:

Now that you have it all configured, there are a couple of ways to benefit from this functionality.  First, you’ll see a more useful context comment in the diffs.  Here’s the screenshot (using an older git version still).  On the left, the diff without the PHP syntax configured, and on the right is the diff with the PHP syntax configured:

As you can see, on the left, the context shows that the change was done somewhere in the ChangeLogTrait trait.  On the right, the context is much more specific – it shows that the change was done in the public function changelog.

This makes reviewing code changes a lot easier.  But there is also one other place where this is useful – in reviewing the history of a particular function.  For example, running “git log -L :changelog:ChangelogTrait.php” will produce the git log output containing only the commits and diffs on the function changelog in the file ChangelogTrait.php.  Very handy!

Vim tweaks and updates

Over the last few weeks, I have significantly changed and updated my Vim configuration.  I’ve been using the editor for decades, and yet I every time I revisit my setup, I am amazed at how far the editor progressed and how wisdom the Vim community shares via themes, plugins, configuration tweaks, etc.

Here are some of the highlights this time around:

  • Switched from Pathogen plugin manager (and Ansible bits) to Vundle.
  • Added vim-devicons plugin and a patched font that supports them.
  • Switched from Syntastic to ALE for faster and better syntax checks.
  • Improved the code completion configuration, tags, and such.
  • Added a whole bunch of plugins for developers.

I am still getting used to some new shortcuts, catching up on the documentation, and trying things out.  But if it’s been a while since you’ve looked at your own Vim configuration, I suggest you do so.  Things are moving and evolving faster than you might think.