“SSH Examples, Tips & Tunnels” is a nice collection of tips and examples for Secure Shell (ssh) users. It covers a variety of scenarios from simple remote connections, to file copying, to tunnels and jump hosts.
SSH vs OpenVPN for Tunneling
I have never particularly liked Virtual Private Networking (VPN). From the old days, when there were a gadzillion of proprietary implementations, each being super slow, resource hungry, and requiring a mess of versions specific requirements, like Java and Firefox. Secure Shell (SSH) has always been my choice for remote connections and tunneling.
Today I came across this article, which also shows that SSH tunnels are much faster than OpenVPN (if one has to use VPN, OpenVPN is probably the best choice around). Needless to say they are also much easier to setup, both manually and automatically.
This adds yet another argument to my SSH vs VPN toolbox.
SSH via bastion host
A while back I wrote this blog post on the subject of using SSH via bastion hosts. If you are into this sort of thing, have a look at this blog post by my brother. He is providing a few more explanations and clarifications, as well as covers a tricky to troubleshoot case with non-default location of your SSH configuration files and keys.
sshrc – bring your .bashrc, .vimrc, etc. with you when you ssh
sshrc looks like a handy tool, for those quick SSH sessions to machines, where you can’t setup your full environment for whatever reason (maybe a shared account or automated templating or restricted access). Here’s a description from the project page:
sshrc works just like ssh, but it also sources the ~/.sshrc on your local computer after logging in remotely.$ echo "echo welcome" >> ~/.sshrc $ sshrc me@myserver welcome $ echo "alias ..='cd ..'" >> ~/.sshrc $ sshrc me@myserver $ type .. .. is aliased to `cd ..'
You can use this to set environment variables, define functions, and run post-login commands. It’s that simple, and it won’t impact other users on the server – even if they use sshrc too. This makes sshrc very useful if you share a server with multiple users and can’t edit the server’s ~/.bashrc without affecting them, or if you have several servers that you don’t want to configure independently.
I’ve discovered it by accident when searching through packages in the Fedora repositories. So, yes, you can install it with yum/dnf.
SSH multiplexing and Ansible via bastion host
It never ceases to amaze me how even after years and years of working with some technologies I keep finding out about super useful features in those technologies, that could have saved me lots of time if I knew about them earlier. Today was a day just like that.
I was working on the Ansible setup for a new hosting environment. One particular thing I wanted to utilize more was a bastion host – a single Linux machine with exposed secure shell (SSH) port, which will be used for managing the configurations of all the servers within the environment. I sort of done that before, but the solution wasn’t as elegant as I wanted it to be.
So, I came across this article – Running Ansible Through an SSH Bastion Host. Which, among other things taught me about a feature that I didn’t know nothing about. Literally. Haven’t even heard about it. Multiplexing in OpenSSH:
Multiplexing is the ability to send more than one signal over a single line or connection. With multiplexing, OpenSSH can re-use an existing TCP connection for multiple concurrent SSH sessions rather than creating a new one each time.
This doesn’t sound too useful for when you are working in command line, one server at a time. Who cares how many TCP connections do you need? It’ll be one, or two, or five. Ten, if you are really involved. But by that time you’ll probably be running background processes, and screen or tmux (which are apparently called “terminal multiplexers“).
It’s when you are going deeper into automation, such as in my case with Ansible, when you’ll need OpenSSH multiplexing. Ansible, being a configuration manager, can run a whole lot of commands one after another. It can run them on multiple servers in parallel as well. That’s where reusing the connections can make quite a bit of a difference. If every command you run connects to the remote server, executes, and then disconnects, you can benefit from not needing to connect and disconnect multiple times (tens or hundreds of times, every playbook run). Reusing connection for parallel jobs is even better – and that’s a case with bastion host, for example.
Here are a few useful links from that article, just in case the ether eats it one day:
- Empowering OpenSSH
- Using SSH Multiplexing
- Managing OpenStack instances with Ansible through an SSH bastion host
Armed with those, I had my setup running in no time. The only minor correction I had to do for my case was the SSH configuration for the bastion host. The example in the article is NOT wrong:
Host 10.10.10.* ProxyCommand ssh -W %h:%p bastion.example.com IdentityFile ~/.ssh/private_key.pem Host bastion.example.com Hostname bastion.example.com User ubuntu IdentityFile ~/.ssh/private_key.pem ForwardAgent yes ControlMaster auto ControlPath ~/.ssh/ansible-%r@%h:%p ControlPersist 5m
It’s just that in my case, I use hostnames both for the bastion host and the hosts which are managed through it. So I had to adjust it as so:
Host *.example.com !bastion.example.com ProxyCommand ssh -W %h:%p bastion.example.com IdentityFile ~/.ssh/private_key.pem Host bastion.example.com Hostname bastion.example.com User ubuntu IdentityFile ~/.ssh/private_key.pem ForwardAgent yes ControlMaster auto ControlPath ~/.ssh/ansible-%r@%h:%p ControlPersist 5m
Notice the two changes:
- Switch of the first block from IP addresses to host names, with a mask.
- Negation of the bastion host configuration.
The reason for the second change is that if there are multiple Host matches in the configuration file, OpenSSH will combine all options from the matched configurations (something I didn’t find in the ssh_config manual). Try this example ssh.conf with some real hosts of yours:
Host bastion.example.com User someuser Host *.example.com Port 2222
You’ll see the output similar to this:
$ ssh -F ssh.conf bastion.example.com -v OpenSSH_7.2p2, OpenSSL 1.0.2h-fips 3 May 2016 debug1: Reading configuration data ssh.conf debug1: ssh.conf line 1: Applying options for bastion.example.com debug1: ssh.conf line 4: Applying options for *.example.com debug1: Connecting to bastion.example.com [220.127.116.11] port 2222. ^C
Once you negate the bastion host from the wildcard configuration, everything works as expected.
You might also try using “%r@%h:%p” for the socket to be different for each remote username that you will concurrently connect with, but that’s just nit-picking.