![\"wtf\"](\"https:\/\/i0.wp.com\/mamchenkov.net\/wordpress\/wp-content\/uploads\/2016\/02\/wtf.png?ssl=1\")
<\/a><\/p>\nAs you can see, we have an office network with NAT on the gateway. \u00a0We have an Amazon VPC with NAT on the bastion host. \u00a0And then there’s the rest of the Internet.<\/p>\n
The setup is pretty straight forward. \u00a0There are no outgoing firewalls anywhere, no VLANs, no network equipment – all of the involved machines are a variety of Linux boxes. \u00a0The whole thing has been working fine for a while now.<\/p>\n
A couple of weeks ago we had an issue with our ISP in the office. \u00a0The Internet connection was alive, but we were getting extremely high packet loss – around 80%. \u00a0The technician passed by, changed the cables, rebooted the ADSL modem, and we’ve also rebooted the gateway. \u00a0The problem was fixed, except for one annoying bit. \u00a0We could access all of the Internet just fine, except our Amazon VPC bastion host. \u00a0Here’s where it gets interesting.<\/p>\n
<\/p>\n
Consider the following:<\/p>\n
- \n
- Amazon VPC bastion host was perfectly available from all Internet hosts.<\/li>\n
- Amazon VPC bastion host was perfectly available from the gateway host.<\/li>\n
- All Internet hosts were perfectly available from all office computers.<\/li>\n
- Amazon VPC bastion host was pingable from all office computers … but<\/li>\n
- Amazon VPC service (ssh, https, etc) were unavailable from all office computers (“Connection timeout” errors).<\/li>\n<\/ul>\n
The following were checked and confirmed:<\/p>\n
- \n
- Office gateway NAT configuration was fine.<\/li>\n
- Amazon VPC bastion NAT configuration was fine.<\/li>\n
- No outgoing firewall was present on the office gateway.<\/li>\n
- Incoming firewall on the Amazon VPC bastion was fine.<\/li>\n
- All necessary services on Amazon VPC bastion (ssh, https, etc) were working fine.<\/li>\n<\/ul>\n
And yet it was impossible to SSH into the bastion host from any computer on the office network. \u00a0The outgoing IP address of the office gateway was the same as the outgoing IP of the machines going through NAT. \u00a0The traceroute<\/em> was the same from the office gateway and the machines on the office network…<\/p>\n
Let’s where lots of swearing and hair pulling happened.<\/p>\n
Until I gave up and called for help. \u00a0My younger brother<\/a>, who is a sysadmin genius, jumped on the case. \u00a0Within the first hour of checking everything and troubleshooting the issue, he confirmed that I wasn’t a complete idiot – all the settings looked right and it should be working.<\/p>\n
A few more hours later, with tcpdump<\/em> in his hands, he managed to locate and fix the problem. \u00a0The solution was very simple. \u00a0Add the following line to the \/etc\/sysctl.conf<\/em> of the Amazon VPC bastion host (running Amazon Linux AMI 2015.09):<\/p>\n
\r\nnet.ipv4.tcp_tw_recycle=0\r\n<\/pre>\n
I’ll leave it to his discretion to explain how he found the problem<\/a>. \u00a0How does the fix work? \u00a0Well, here is a handy article – “Copying with the TCP TIME-WAIT state on busy Linux servers<\/a>“. \u00a0I read the article twice, and the best way to describe how I understand it is with this scene<\/a> from the IT Crowd TV series – season 1, episode 1, Jen zones out on Moss’ explanation:<\/p>\n