Why software projects take longer than you think – a statistical model

Why software projects take longer than you think – a statistical model” is an interesting take on the problem of bad estimations in software projects. I’m not that great with math, but even then the article is very interesting. And there is a lot that I agree with.

Here’s a quote for those of you who couldn’t make it through:

Why software tasks always take longer than you think

Assuming this dataset is representative of software development (questionable!), we can infer some more numbers. We have the parameters for the t-distribution, so we can compute the mean time it takes to complete a task, without knowing the σ for that task is.

While the median blowup factor imputed from this fit is 1x (as before), the 99% percentile blowup factor is 32x, but if you go to 99.99% percentile, it’s a whopping 55 million! One (hand wavy) interpretation is that some tasks end up being essentially impossible to do. In fact, these extreme edge cases have such an outsize impact on the mean, that the mean blowup factor of any task ends up being infinite. This is pretty bad news for people trying to hit deadlines!

Our Software Dependency Problem

Our Software Dependency Problem” is a great article going in-depth into the subject of the dependency management during software engineering.

Dependency managers have scaled this open-source code reuse model down: now, developers can share code at the granularity of individual functions of tens of lines. This is a major technical accomplishment. There are myriad available packages, and writing code can involve such a large number of them, but the commercial, legal, and reputational support mechanisms for trusting the code have not carried over. We are trusting more code with less justification for doing so.

Not only it nicely describes the problem in simple terms, but also provides practical examples and solutions to it. In particular, I enjoyed the section that suggests how to improve dependency evaluation in terms of design, code quality, testing, debugging, maintenance, usage, security, and licensing.

How to Slow Down to Go Faster Than Ever in Software Development

While reading through “How to Slow Down to Go Faster Than Ever in Software Development” I couldn’t help but nod my head in agreement continuously. The article goes over the well-known and familiar challenges in software development, but it also words them in a very simple and straight-forward way. It’s difficult to disagree with.

Here are a few quotes to get you started.

As Robert C. Martin mentions on the primary value of software at CleanCoders, “The ability of a software system to tolerate and facilitate such ongoing change is the primary value of software”. Rushing is evil in software development. Any attempt to rush causes dramatic damage in productivity, focus, people’s effectiveness, adaptation capability, and tolerance of software.
For instance, we always have time for fixing bugs, but no time for writing tests. We don’t refactor and write tests because we don’t have enough time. But we have time for debugging, hacking code and fixing bugs.
We focus on processes so much that we often forget the main asset in software development: people. Processes help people to improve the way they build products, increase motivation and cultivate a healthy environment. In the end, the efficiency of processes is important, but people are crucial.

Processes and tools do not build products, but people do. We have to admit, “talent hiring” is the most important functionality of an organization. It has direct impact on the future of the company and the product itself.
Hire the best talent for your organization. By saying “the best”, I do not mean the smartest, or most experienced people around. I look for passion, discipline and motivation at a minimum. If all three exists in a talent, the other skills can grow with ease. Hiring is a win-win process, so both sides should gain from the process. So you should slow down your hiring process and invest on improving it. People join companies in which they believe. So model the behavior you want to see. And through your company culture, your vision and people, make talent believe in you.

One thing is clear. Without having a quality codebase, you cannot be agile, sorry. The first thing you need to do is eliminate technical debt and resolve bugs. If you need to stop building the features for a while, and focus on eliminating bugs.
“Fixing bugs and deploying to servers afterwards” is not a proper procedure today. It contains risks and danger. We need a better and more disciplined way of doing it. When you want to fix a bug, first write a test and reproduce the problem programmatically. Then fix the bug and see that the tests are passing. Deploying to production is safe afterwards.

Reading postmortems

Once in a while a seemingly straightforward article turns into a goldmine of links and resources. This happened to me today with this one – “Reading postmortems“.

Not only this article itself is a very nice roundup of common sources for system failures, but it also links to a couple of awesome references:

  • Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. This is both a talk and a paper.
  • danluu/post-mortems – a GitHub repository with a collection of publicly available postmortems from a variety of organizations, like Google, Amazon, Facebook, NASA, GitHub, and more.

If you still have no idea what postmortem is, Wikipedia explains.

Monorepos: Please don’t!

I have only recently discovered the concept of monorepos. It does sound interesting and I am yet to try it out, or, at least, dig deeper into the subject. But then, there are people who have strong opinions against them, and I the reasoning makes all the sense to me. Here’s a nice one: Monorepos: Please don’t!

sr.ht (“sir hat”) – open source software suite for managing your software development projects

sr.ht – pronounced “sir hat” – is a new competitor in the world of GitHub, BitBucket, and GitLab.  Much like all of these, you can either self-hosted it or use a managed service.  It might not yet be as fancy, polished, and cool (I think they need a better name and the domain) as its competitors, but there are a couple of reasons that might make a difference when making a choice:

  1. Open Source.  From a quick look, sr.ht is distributed under the GNU Afero GPL.
  2. Modular.  The suite consists of the following components:
    1. git – git repository hosting service
    2. build – continuous integration service
    3. lists – mailing lists service
    4. todo – ticketing system / bug tracker
    5. dispatch – task dispatcher and integration service
    6. man – markdown and git-based wiki service
    7. meta – account management service


Microservices Architecture : Best Practices

The other day I came across this article: “Microservices Architecture: All the Best Practices You Need to Know“.  There’s been a lot said and written about the microservices architectures around the web.  But I like this article in particular, because it paints a more realistic picture, in my opinion.  Big parts of it are covering the “why?” part of the whole conversation, and it presents a balanced view of pros and cons, as well as several approaches to solving the problems.

This is very refreshing after tonnes of “Microservices are amazing, and are the best thing since sliced bread” and “Microservices is nothing but hype and bubble” coverage out there.

Periodic Table of Software Engineering

I came across this Periodic Table of Software Engineering, and I think it’s an excellent visualization.  For those working their way to become software engineers, it provides a nice map of skills, topics, and knowledge areas to cover.  For the rest, it clearly explains why software is so much more complex than anybody thinks.

Multiple Perspectives On Technical Problems and Solutions

Multiple Perspectives On Technical Problems and Solutions” is an interesting take on engineering in general and software architecture in particular.  It starts off with:

Fundamental: engineering decision-making is a socially constructed activity


In other words, engineering (as an activity) does not have “correct” solutions to problems. As an aside, if you’re looking for correct solutions to problems, I’d suggest that you go work in a different field (like mathematics); engineering will likely frustrate you.

It then goes into dialogues and discussions, architecture review meetings, and provides a few pointers on how to get the best of those.