Using aws-cli –query Option To Simplify Output

Eric Hammond shares a super-handy tip for those of us who work with Amazon AWS via the command line:

I just learned about a recent addition to aws-cli: The --query option lets you specify what parts of the response data structure you want output.

Instead of wading through pages of JSON output, you can select a few specific values and output them as JSON, table, or simple text. The new --query option is far easier to use than jqgrep+cut, or Perl, my other fallback tools for parsing the output.

Read the rest of his blog post for a few examples of how to use it.

Go Monorepo

As a maintainer and contributor to multiple, and often related, git repositories, I often find myself thinking that there must be a simpler way.

Consider, for example, a variety of CakePHP plugins that we develop and maintain at work.  Sure, each one provides a completely separate bit of functionality and has its purpose as a standalone project.  But, at the same time, they all have enough overlap that we use a common cakephp-plugin-template repository to keep things in sync, and also require cakephp-utils plugin from all of them.

It’s only natural to consider other ways of doing things.  Sure, I tried git submodules before, but they don’t quite cut it.  Something is still missing.

Today I came across the Go Monorepo website.  Which makes me wonder even further down this road.  There are also a few other tools and alternatives mentioned in this article.

That’s not something that I’ll jump into right now, but it’s definitely something I’ll consider to try out in the nearest future.

The Tool that Will Help You Choose Better Product Ideas

Itamar Gilad, of ex-Google fame, has an excellent post – The Tool that Will Help You Choose Better Product Ideas – which describes a process for selecting better product ideas and implementing them with confidence, minimizing the risks and maximizing the team efficiency.

Unlike many other similar posts, this one is very practical, detailed, and relies on iterations, feedback, communications, and ties together different roles within the company or the team.

HipChat to Slack migration

Back in July, Slack acquired Hipchat with the intent of migrating all the users and shutting down the service.  For us at work, these were very sad news.  We’ve been using HipChat for years, and relied on it heavily.  We have also evaluated Slack in its early days and didn’t quite like it at the time.

Once it was clear that the HipChat is going away, we had a couple of options to consider.  One was to move to Slack.  Another – to migrate to a different solution, preferably to a self-hosted one, like Rocket.Chat.

So, first thing we did, was another test trial of Slack.  We were glad to see that it matured and improved a lot over the last few years, and that it would be faster and easier for us to move to Slack rather than to roll out our own self-hosted solution.

Here are a few things that we had to keep in mind for this migration:

  • We have a rather small team – about 15-20 people altogether, but with a very mixed backgrounds.  Some are very technical, some are not.  Some are on Windows, some on Linux, and some on MacOS.  Mobile use is also heavy across the board.
  • We are working with a lot of rooms/channels.  We have a few generic ones, like Support, Sales, Management, etc.  And we have a separate channel for every project.  And those we have over a hundred of.
  • We have a number of integrations connected – GitHub and BitBucket repositories, Zabbix monitoring system, RSS feeds, deployment automation, etc.
  • HipChat would only be around for the next month.  Not a tight deadline by our means, but the timeline is important here.

Since the time was limited from the beginning, we decided to start with the migration pretty much immediately, rather than waiting until the last moment.  In case something wasn’t working, we’d have some space to maneuver.

Instead of a once off switch, we also decided to do a gradual migration, shifting from HipChat to Slack bit by bit, using both services in parallel.

Creating the new workspace and registering all the users was, as you can imagine, the easiest part.  After all we do have only a small team, and that many accounts can be done manually, without any automation what so ever.

That was an option for the rooms/channels though, as we have hundreds of those.  Gladly, both Slack and HipChat provide an API which helps with automation, and we have awesome DevOps who handle things like that.

Here comes the first issue that we came across.  HipChat allows for much more when it comes to room names, than Slack.  Slack channels are limited to 21 characters (which is not a lot), and the selection of the characters is much less also (dashes are in, but slashes and other special characters are out).

That meant that we had to come up with the new naming convention, which we did.  We also used this opportunity to consolidate some of the HipChat rooms into fewer Slack channels.   In HipChat, as I mentioned, we had a separate room for each project.  Which was handy for busy projects, but somewhat confusing for smaller projects that belonged to the same client.  For example, it’s not a rare occurrence for us to have two, three or more projects for a single client.  And while they are all different, often they overlap a lot (websites for a group of company, or a website integrated with the CRM, etc).

Once we decided on the adjustments to the naming convention and consolidation of some rooms, the migration wasn’t too difficult, given the API of the both services.

Next came the integrations.  This is where, I have to say, Slack shines.  While HipChat integrates with both BitBucket and GitHub too, the functionality provided by Slack is a lot better and easier.  Both GitHub and BitBucket are available out of the box, so we quickly connected all the project channels to their appropriate repositories.

RSS feeds weren’t a problem either.  So the next one that we had to do was the Zabbix monitoring system.  Again, Zabbix is pretty good with integrating to other systems, Slack API is flexible, and our DevOps are amazing.  So that took no time at all. In fact, we even made it better, since Slack allows for richer and better notifications.

The last, but the most important one, was our deployment automation.  It wasn’t painful, but it did require us to do some adjustments.  Some of these we wanted to make for a while now anyway.  The biggest change was that HipChat didn’t care about the long running tasks, while Slack expects an answer from your webhook within 5 seconds.  Many of out deployments take longer than that (some are up to 20 minutes), so we had to do some changes.

When the chat webhook request for deployment is received, we couldn’t wait for the deployment to finish anymore to send the response back.  We had to respond within 5 seconds.  That meant that we had to disconnect the actual deployment run from the chat communications.  As I said, we wanted to do that for a while anyway.  A quick proof of concept with the Linux “at” command worked quite well, so stayed with that for the time being.  Now when the request is received, the deployment is scheduled with the “at” command for the next minute, and the response is sent to the chat within a second.

That was about it.  Since we didn’t care about migrating the actual chat histories or uploaded files or anything like that (we use Redmine for anything that needs persistence), the migration overall was smooth.

Once all the users were registered, channels created, and integrations configured, we started to phase out HipChat rooms one-by-one, archiving and disabling them on HipChat and pushing users more and more towards Slack.

Overall, the setup and configuration took us about three weeks.  The rest of the migration with the users and phasing out of HipChat took approximately another two weeks.

So, now that we’ve been using Slack exclusively for about a month and a half, how do we find it?  I think overall it’s a lot better now that it was with HipChat.  Here are some of the benefits:

  • The mobile app is amazing.  It supports multiple workspaces (some of our clients are also using Slack and we need to be in multiple places at the same time), has a great user interface, and, most importantly, the push notifications are working A LOT BETTER.  HipChat had a tendency of going offline after about 10 minutes and being completely silenced until the next manual check.  Slack even provides a Snooze option for 20 minutes, 1 hours, etc, which makes it easy to silence it during a meeting.
  • Slack desktop and web applications seem to be better enjoyed by non-technical users, than their HipChat alternatives were.  For me personally, HipChat was easier on the eye, but Slack provides a lot of functionality which wasn’t in HipChat.
  • Threads!  I think this is one of the greatest features that HipChat was missing.  Project rooms especially can get quite noisy.  It’s often difficult to follow up with all the chatter.  Threads make it a lot easier.  With threads there can be multiple concurrent conversations in the same channel without any confusion or overlap.
  • Reminders!  Again, for those of us who are busy with a million things, reminders in Slack are a killer feature.  They help to find that message that needs a response, half an our later, in a hundred rooms with dozens of conversations going on at the same time.
  • Slash commands.  While both HipChat and Slack have them, Slack allows to send the same slash command from any room (handy for deployments).  With HipChat we were a lot more limited, which pushed us into a separate room for all deployments.
  • Emojis.  Usually, I’m the last person to use emojis.  I’m the old school smiley guy.  But in this particular context, I find emojis to be very useful.  They often save a lot of typing and help keep the atmosphere nice and friendly.  HipChat had a very weird approach to emojis.  While they were there, they were very few, and very custom to the HipChat team.  We did upload a lot of our own over time, but with Slack that wasn’t necessary.
  • Stats and analytics.  Slack provides a great insight into how the workspace is being used.  Stats are available both as an admin menu item, and as a quick weekly email.
  • Integrations and API.  Slack provides a much more flexible API and integration options.  While we were doing fine with HipChat, I think all of our integrations are much better now because of the move to Slack.  (slash commands, updated messages, message replies, file attachments, rich messages, etc).

As I said, overall the migration went quite smooth and we are in a much better place now.  This is quite surprising and refreshing, especially after us considering the news of HipChat acquisition as a sad story.  In fact, I can’t remember, from the top of my head, when a sad story like this turned out to be such an improvement.

Thank you HipChat team for all the effort – you’ve been great for the last few years.  Thank you Slack for making things even better!

Vim tweaks and updates

Over the last few weeks, I have significantly changed and updated my Vim configuration.  I’ve been using the editor for decades, and yet I every time I revisit my setup, I am amazed at how far the editor progressed and how wisdom the Vim community shares via themes, plugins, configuration tweaks, etc.

Here are some of the highlights this time around:

  • Switched from Pathogen plugin manager (and Ansible bits) to Vundle.
  • Added vim-devicons plugin and a patched font that supports them.
  • Switched from Syntastic to ALE for faster and better syntax checks.
  • Improved the code completion configuration, tags, and such.
  • Added a whole bunch of plugins for developers.

I am still getting used to some new shortcuts, catching up on the documentation, and trying things out.  But if it’s been a while since you’ve looked at your own Vim configuration, I suggest you do so.  Things are moving and evolving faster than you might think.

git worktree – a better way for git stash abusers

If you constantly find yourself using “git stash” while working on a project, or, even worse – have multiple copies of the same project cloned on the same machine, “git worktree” might be a much better alternative for you.

Manage multiple working trees attached to the same repository.

A git repository can support multiple working trees, allowing you to check out more than one branch at a time. With git worktree add a new working tree is associated with the repository. This new working tree is called a “linked working tree” as opposed to the “main working tree” prepared by “git init” or “git clone”. A repository has one main working tree (if it’s not a bare repository) and zero or more linked working trees.

When you are done with a linked working tree you can simply delete it.

Here are a few links to get you started:


11 Best Programming Fonts

It’s that time of the year once again, when you should take a five minute break from whatever it is you are doing.  Here are “11 Best Programming Fonts” that might help you out in being a little bit more productive.  If not that, than at least bring you a slight change and some eye candy, when staring at your code.

For me personally, Source Code Pro by Adobe (featured in the screenshot above) is still the best option.  I have it setup in the Terminator as Source Code Pro Semibold at size 11, which makes it large enough to work with the code comfortably and small enough to eat too much screen space.

If you want to find and compare more fonts, have a look at these two resources:


Why programmers are not paid in proportion to their productivity

Here are a couple of bits that I liked in “Why programmers are not paid in proportion to their productivity” blog post:

How can someone be 10x more productive than his peers without being noticed? In some professions such a difference would be obvious. A salesman who sells 10x as much as his peers will be noticed, and compensated accordingly. Sales are easy to measure, and some salesmen make orders of magnitude more money than others. If a bricklayer were 10x more productive than his peers this would be obvious too, but it doesn’t happen: the best bricklayers cannot lay 10x as much brick as average bricklayers. Software output cannot be measured as easily as dollars or bricks. The best programmers do not necessarily write 10x as many lines of code and they certainly do not work 10x longer hours.

Programmers are most effective when they avoid writing code.

… and:

The romantic image of an über-programmer is someone who fires up Emacs, types like a machine gun, and delivers a flawless final product from scratch. A more accurate image would be someone who stares quietly into space for a few minutes and then says “Hmm. I think I’ve seen something like this before.”