HTTPS on Stack Overflow: The End of a Long Road

Way too often I hear rants from random people (unfortunately, many of them are also from the IT industry, with the deep understanding of the underlying issues) complaining about why company X or product Y doesn’t implement this or that feature.  As someone who has been involved a dozens, if not hundreds, of projects, I pretty much always can think of a number of reasons why even seemingly the simplest of features aren’t implemented for years.  These can vary from business side of things – insufficient budgets, strategic goals, and the like – to technical, such as architectural limitations, insufficient expertise, insufficient resources, etc.

One of the recent frequent rant that keeps coming up is “Why don’t they just enable HTTPS?”.  Again, as someone being involved in HTTPS setup for several different environments I can think of a number of reasons why.  SSL certificates used to cost money and were quite cumbersome to install until very recently.  Thanks to Let’s Encrypt effort, SSL certificates are now free and quite easy to issue and renew.  But that’s only part of the problem.  Enabling HTTPS requires infrastructural changes, and the more complex your infrastructure, the more changes are needed.  Just think of a few points here – web server configuration (especially when you have multiple web servers, with varied software (Apache, Nginx, IIS) and varied versions of that software), load balancers, web application firewalls, reverse proxies, caching servers, and so on.

Apart from the infrastructural changes, HTTPS often needs changes on the application level.  Caching, cookies, headers, making sure that all your resources are HTTPS-only, redirects, and the like.

All of the above issues are multiplied by a gadzillion, when your project is publicly available, used by tonnes of people, and provides embeddable content or APIs to third-party (hello, backward compatibility).

This is not to mention that HTTPS itself is a complex subject, not well understood by even the most experienced system administrators and developers.  There are different protocols and versions (SSL vs. TLS), cipher suites, handshakes, and protocol details.  Just have a look at the variety of checks and the report length done by Qualys’ SSL Labs Server Test.  Even giants like Google, who employ thousands of smart people, can’t get it all right.

But for some reason, people either don’t know or prefer to ignore all this complexities, and whine and cry anyway.

Recently, Stack Overflow – a well known collection of sites on a variety of technical subjects, has completed the migration to HTTPS everywhere.  These are also people with a lot of knowledge and expertise and with access to all the information.  Just have a look at their long way, which took not months, but years: HTTPS on Stack Overflow: The End of a Long Road.

Today, we deployed HTTPS by default on Stack Overflow. All traffic is now redirected to https:// and Google links will change over the next few weeks. The activation of this is quite literally flipping a switch (feature flag), but getting to that point has taken years of work. As of now, HTTPS is the default on all Q&A websites.

We’ve been rolling it out across the Stack Exchange network for the past 2 months. Stack Overflow is the last site, and by far the largest. This is a huge milestone for us, but by no means the end. There’s still more work to do, which we’ll get to. But the end is finally in sight, hooray!

So next time you are about to start crying about somebody not having feature X or Y, just give it a minute first.  Try to imagine what goes on on the other side.  You aren’t the only one with low budgets, pressing deadlines, insufficient knowledge, bad colleagues and horrible bosses…

Introducing Moby Project: a new open-source project to advance the software containerization movement

Docker Blog is introducing the Moby Project:

The Moby Project is a new open-source project to advance the software containerization movement and help the ecosystem take containers mainstream. It provides a library of components, a framework for assembling them into custom container-based systems and a place for all container enthusiasts to experiment and exchange ideas.

This just had to happen, given the nature of the Open Source and the importance of the container technology for the modern infrastructure.

Stack Overflow: Helping One Million Developers Exit Vim

OK, this one is socially funny and statistically cool – Stack Overflow question on how to exit Vim editor was viewed over a million times in the last few years.  Now, there’s a breakdown of all sorts of statistics about who gets stuck in Vim the most.  It’s pretty amazing the kind of questions and answers one can ponder at when having access to a lot of statistical data.

:wq

Using the Strict-Transport-Security header

Julia Evans has an excellent write-up on “Using the Strict-Transport-Security header” – what it is, why you’d want to use it, and what are some of the consequences of using one.

As always with her blog posts, this one is very focused on one particular subject, easy to read, and explains things simply, so that the reader’s technical level is always irrelevant (OK, OK, you do need a basic understanding of how HTTP works, but not more than that).

Linux utils that you might not know

Linux utils that you might not know covers a few Linux command line utilities that aren’t very famous:

  • column, for “columnating” lists, which is very useful for display of table-like data (think CSV, for example);
  • cal, for displaying calendars;
  • factor, for calculating factors;
  • numfmt, for formatting numbers and converting them to/from human-readable formats;
  • shred, for overwriting the content of a deleted file, making it much more difficult to recover.

HAProxy and Nginx abuse and rate limiting

My brother wrote a follow-up – HAProxy abuse filtering and rate limiting – to his previous post – Nginx rate limit by user agent (control bots).  This is just a tip of the iceberg that we are working with at the office, but it’s pretty cool.

Hopefully, soon enough our Ansible playbooks will be up to date and shareable…

HAProxy SNI

HAProxy SNI” is pure gold! If you want to have a load balancer for HTTPS traffic, without managing SSL certificates on the said load balancer, there is a way to do so.

The approach is utilizing the Server Name Indication (SNI) extension to the TLS protocol.  I knew about it and I was already using it on the web server side, but it didn’t occur to me that it’ll be utilized on the load balancer.  Here’s the configuration bit:

frontend https *:443
  description Incoming traffic to port 443
  mode tcp
  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  use_backend backend-ssl-foobar if { req_ssl_sni -i foobar.com }
  use_backend backend-ssl-example if { req_ssl_sni -i example.com }
  default_backend backend-ssl-default

The above will make HAProxy listen on port 443, and then send all traffic for foobar.com to one backend, all traffic for example.com to another backend, and the rest to the third, default backend.

Writing systemd Units

Vidar Hokstad explains what systemd units are and how to write them.  Very useful for that day when I will stop hating systemd and will try to embrace it.

Systemd has become the defacto new standard init for Linux-based systems. While not everyone has made the switch yet, pretty much all the major distros have made the decision to switch.

For most people this has not meant all that much yet, other than a lot of controversy. Systemd has built in SysV init system compatibility, and so it’s possible to avoid dealing with it quite well.

But there is much to be gained from picking up some basics. Systemd is very poweful.

I’m not going to deal with the basics of interacting with systemd as that’s well covered elsewhere. You can find a number of basic tips and tricks here.

Instead I want to talk about how to write systemd units.

Deprecated Linux networking commands and their replacements

Doug Vitale Tech Blog runs a post with a collection of the deprecated Linux networking commands and their replacements. Pretty handy if you want update some of your old bash scripts.

Deprecated command Replacement command(s)
arp ip n (ip neighbor)
ifconfig ip a (ip addr), ip link, ip -s (ip -stats)
iptunnel ip tunnel
iwconfig iw
nameif ip link, ifrename
netstat ss, ip route (for netstat-r), ip -s link (for netstat -i), ip maddr (for netstat-g)
route ip r (ip route)

Docker Image Vulnerability Research

Federacy has an interesting research in Docker image vulnerabilities.  The bottom line is:

24% of latest Docker images have significant vulnerabilities

This can and should be improved, especially given the whole hierarchical structure of Docker images.  It’s not like improving security of all those random GitHub repositories.