Building the Right Alerting System

Here’s something I wanted to get into for a while now, but haven’t had the time yet – switching the monitoring / alerting system from server-oriented to business-oriented.  The gist of the story is:

If it’s not actionable and business critical, then it shouldn’t ring.

The article has some statistics and summaries as well.  The reasoning behind the switch is obvious, but it’s good to have it formulated:

After a few months, I can tell reducing our alerting rate should have been a top priority before things got out of hands, for a few reasons.

  • Constant alerts prevented the team to focus on what was important. Being interrupted even for things that can wait for a few hours lowers our productivity when we work on things that can’t wait.
  • Being awaken every night, several times a night exhausts a team and make people less productive at day, and more prone to do errors.
  • Too many off hours interventions cost the company a lot of money that could be invested in hardening the infrastructure or hiring someone else instead.

Is group chat making you sweat?

Jason Fried has an excellent write-up on the pros and cons of using group chat for the team communications, and some of the ways to make it better. We use HipChat in the company and while it’s vital to our operations and I can’t even begin to think how we could do what we do without it, it does have some negative side effects – exactly as James describes them.

The most valuable advice out of that long article is this one (I’ve heard it before a few times, but it’s worth repeating):

Think about it like sleep. If someone was interrupted every 15 minutes while they were trying to sleep, you wouldn’t think they’d be getting a good night’s sleep. So how can getting interrupted all day long lead to a good day’s work?

 

Programmer Interrupted

Slashdot runs a thread on “Are Remote Software Teams More Productive?“.  The original post links to a few research references that, unsurprisingly, show how expensive interruptions are to programmers, and how unprepared we are, as an industry, to deal with this problem.  I particularly liked a rather in-depth look at the issue in “Programmer Interrupted” article.

Like you, I am programmer, interrupted. Unfortunately, our understanding of interruption and remedies for them are not too far from homeopathic cures and bloodletting leeches.

Here are a few points, if the article is too long for you to handle:

Based on a analysis of 10,000 programming sessions recorded from 86 programmers using Eclipse and Visual Studio and a survey of 414 programmers (Parnin:10), we found:

  • A programmer takes between 10-15 minutes to start editing code after resuming work from an interruption.
  • When interrupted during an edit of a method, only 10% of times did a programmer resume work in less than a minute.
  • A programmer is likely to get just one uninterrupted 2-hour session in a day

And also this bit on the worst time to interrupt a programmer:

If an interrupted person is allowed to suspend their working state or reach a “good breakpoint”, then the impact of the interruption can be reduced (Trafton:03). However, programmers often need at least 7 minutes before they transition from a high memory state to low memory state (Iqbal:07). An experiment evaluating which state a programmer less desired an interruption found these states to be especially problematic (Fogarty:05):

  • During an edit, especially with concurrent edits in multiple locations.
  • Navigation and search activities.
  • Comprehending data flow and control flow in code.
  • IDE window is out of focus.

Overall, not surprising at all, but it’s nice to have some numbers and research papers to point to…

i3 window manager – a week later

A week ago I blogged about i3 window manager and my attempt to use it instead of MATE.  So, how am I am doing so far?

The long story short: I love i3.  It’s awesome.  But I still switch back to MATE once in a while.

What’s good about i3?  It’s super fast.  Even faster than a pretty fast MATE.  It’s keyboard navigated, and it only takes about a day to get used to enough keyboard shortcuts to feel comfortable and productive.  It’s super efficient.  Until I tried i3 I didn’t recognize how much time I spend moving windows around.  It is unexcusable amount of time spent needlessly.

What’s bad about i3?  It’s low level.  In order to make it work right with multiple screens, one need to get really familiar with xrandr, the tool I last used years ago.  If you are on a laptop, with a dynamic setup for the second screen (one monitor at the office, one at home, and an occasionally different project at client’s premises), you’ll need a bunch of helper scripts to assist you in quick change between these setups.

And then there is an issue of flickering desktop.  The web is full of questions about how to solve a variety of flickering issues when using i3.  The one that I see most often is the screen going black once in a while.  Sometimes it takes a second to come back, sometimes a few seconds, and sometimes and it doesn’t come back at all.  The more windows I have, spread across more workspaces, with more connected monitors – the more often I see the issues.  It’s annoying, and it’s difficult to troubleshoot or even report, as I haven’t found a pattern yet, or how to reproduce the problem.

With that said though, I am now about 80% time using i3.  I like the simplicity and efficiency of it.  It’s so good, that I work better even without a second monitor.  But when I do need a second monitor (paired programming, demos, etc), or when I have a projector connected, I switch to MATE.  That’s about 20% of my time.

i3 – tiling window manager

In the last few days my attention was unfairly distributed between a whole lot of tasks.  The fragmentation and constant context switching affected my productivity, so I briefly revisited my toolbox setup, in hopes to find something that I didn’t know about, forgot about, or have greatly underutilized.

One of the things that came (again) on my radar was terminal multiplexer tmux.  I’ve blogged about it before.  I used it for a while, but at some point, it faded away from my daily routine.  The two most useful features of tmux are:

  1. Persistent sessions, where you can work on a remote machine, detach your terminal, disconnect from the machine entirely, and then, at some point later, connect again and continue from where you left off.  With simpler workloads and reliable Internet connection, this became less useful to me.  When I do need this functionality, I use screen, which is more often installed on the machines that I work with.
  2. Terminal multiplexer, where you can split your terminal screen into a number of panels and work with each one like it’s a separate terminal.  This is still useful, but can be done by a number of different tools these days.  I use Terminator, which supports both horizontal and vertical screen split.  Terminology is another option from a choice of many.

I thought, let me find something that people who used tmux have moved on to.  That search led me, among other things, to “ditching tmux” thread on HackerNews, where in the comments a few people were talking about i3 tiling window manager.

Continue reading “i3 — tiling window manager”

Is It Worth the Time?

I’ve seen this chart before, but completely forgot where it was from.  Tried to find it a couple of times in a hurry of a conversation, but couldn’t.  Thanks to this SysAdvent blog post, I now have it permanently engraved into the archives of my blog.

xkcd figured out how long can you work on making a routine task more efficient before you’re spending more time than you save.  The crossing of how much time you shave off and how often you do the task gets progressively less obvious when move from the top left corner of this char to the bottom right corner.

Never judge a programmer by their commit history

In a comment to another post, Andrey sent in a link to this blog post, titled “Never judge a programmer by their commit history”.  It’s very similar to something that I wanted to write for quite some time now.

It’s been a very long time since I judged any programmer based on their commit history and I believe if you think you can judge a programmer’s ability by reading his/her code YOU ARE WRONG.

As technical folk, we are often fast to judge an implementation purely on its technical merits, forgetting, that there are other factors often at play.  Mehdi Khalili, the author of the post, goes over just some of them, including:

  • Abiding by bad coding standards
  • Bad leader and project manager
  • Junior devs
  • Business reality
  • Brain fart
  • Personal issues
  • Synergy or lack thereof
  • Physical issues (which is similar to the Personal issues above)
  • Imposters! (which is funny and, something I didn’t think about)

I’ve seen (and done) almost all of these.  Business reality and junior devs are the two I’ve come across the most.

Pull Request focused dashboards for BitBucket

PR-focused-dashboard

A few days ago BitBucket announced the re-worked dashboards, which are now much more focused on the Pull Requests that you’ve created or need to review, rather than lists of repositories that you have access to.  I’ve enabled the feature for my team and it looks super awesome!

If you’ve been suffering from being lost in dozens or hundreds of projects and missing out on the Pull Requests activity, check them out.  You’d be surprised.

“I’ve never had a goal”

I found the blog post “I’ve never had a goal” over at Jason Kottke blog interesting.  There is a quote from Jason Fried, the founder of Basecamep (aka 37signals):

I can’t remember having a goal. An actual goal.

There are things I’ve wanted to do, but if I didn’t do them I’d be fine with that too. There are targets that would have been nice to hit, but if I didn’t hit them I wouldn’t look back and say I missed them.

I don’t aim for things that way.

I do things, I try things, I build things, I want to make progress, I want to make things better for me, my company, my family, my neighborhood, etc. But I’ve never set a goal. It’s just not how I approach things.

Also, Jason Kottke’s therapist advice:

For the longest time, I thought I was wrong to not have goals. Setting goals is the only way of achieving things, right? When I was criticizing my goalless approach to my therapist a few years ago, he looked at me and said, “It seems like you’ve done pretty well for yourself so far without worrying about goals. That’s just the way you are and it’s working for you. You don’t have to change.”

I myself don’t set goals either.  But I’m yet to reach that “you’ve done pretty well for yourself” part. Wink.