I am running a Linux server at a remote, off the grid, site that I am calling “cave” for the purposes of this post.
The internet access is provided by Mint Mobile, which uses T-Mobile cell towers. T-Mobile does not provide users with publicly addressable IP address. Instead, multiple uses share a single IP address using NAT. I want to SSH into the cave server from anywhere. To do that, I use Reverse SSH tunnelling.
What is Reverse SSH Tunneling?
The concept of reverse SSH tunneling is simple. For this, I needed another host (so-called “relay host”) outside the restrictive cave network. The Raspberry Pi I will use is located at my home. It has an IP address that is publicly visible and unique to me. I can connect to it from anywhere on the internet. I will refer to it as the “Pi.”
Then I set up a persistent SSH tunnel from the server in the cave network to the Pi. With that, I can connect “back” to the cave server from the Pi. You can see why this configuration is called “reverse” tunnel.
As long as the Pi is reachable, I can connect to the cave server from wherever I am regardless of how restrictive T-Mobile’s NAT is – and regardless of how restrictive my in-bound firewall rules are on the cave’s network.
Set up a Reverse SSH Tunnel on Linux
Let’s see how we can create and use a reverse SSH tunnel. We assume the following. We will be setting up a reverse SSH tunnel from the cave server to the Pi so that we can SSH to cave server via the Pi from another computer, such as a laptop. Let’s call that computer “laptop.”
Assume the public IP address of the Pi is 173.173.63.132.
On cave server, open an SSH connection to the Pi as follows:
caveserver~$ ssh -fN -R 12001:localhost:22 [email protected]
Here the port 12001 is an arbitrary port number that is not used by other programs on the Pi.
The “-R 12001:localhost:22” option defines a reverse tunnel. It forwards traffic on port 12001 of the Pi to port 22 of the cave server.
With “-fN” option, SSH will go into the background once successfully authenticated with an SSH server. This option is useful since we do not want to execute any commands on a remote SSH server. We just want to forward ports.
After running the above command, you will be right back to the command prompt of the cave server.
Now, log into the Pi and verify that 127.0.0.1:12001 is bound to sshd. If so, that means a reverse tunnel is set up correctly.
pi~$ sudo netstat -nap | grep 12001
tcp 0 0 127.0.0.1:12001 0.0.0.0:* LISTEN 8493/sshd
Now from any other computer (e.g., the laptop), log in to the Pi. Then access the cave server as follows:
pi~$ ssh -p 12001 CaveUser@localhost
One thing to take note is that the SSH login/password you type for localhost should be for the cave server, not the Pi, since you are logging into the cave server via the tunnel’s local endpoint. After successful login, you will be on cave server.
Connect Directly to a NATed Server via a Reverse SSH Tunnel
While the above method allows you to reach the cave server behind T-Mobile’s NAT, you need to log in twice: first to the Pi and then to the cave server. This is because the end point of an SSH tunnel on the Pi is binding to loopback address (127.0.0.1).
But in fact, there is a way to reach the cave server directly with a single login to the Pi. For this, you will need to let sshd on the Pi forward a port not only from loopback address, but also from an external host. This is achieved by specifying GatewayPorts option in sshd running on the Pi.
Open /etc/ssh/sshd_conf of the Pi and add the following line:
pi~$ vi /etc/ssh/sshd_conf
GatewayPorts clientspecified
Restart sshd.
pi~$ sudo /etc/init.d/ssh restart
Now let’s initiate a reverse SSH tunnel from the cave server as follows:
caveserver~$ ssh -fN -R 173.173.63.132:12001:localhost:22 [email protected]
Log into the Pi and confirm with netstat command that a reverse SSH tunnel is established successfully.
Pi~$ sudo netstat -nap | grep 12001
tcp 0 0 173.173.63.132:12001 0.0.0.0:* LISTEN 1538/sshd: dev
Unlike a previous case, the end point of the tunnel is now at 173.173.63.132:12001 (the Pi’s public IP address), not 127.0.0.1:12001. This means that the end point of the tunnel is reachable from an external host.
Now from any other computer (e.g., the laptop), type the following command to gain access to NATed cave server.
clientcomputer~$ ssh -p 12001 [email protected]
Set up a Persistent Reverse SSH Tunnel on Linux
Now, after proving that this concept works, I made the tunnel “persistent.” This means that the tunnel is active all of the time and can reestablish itself in the case of temporary network congestion, SSH timeout, the Pi rebooting, etc..
For a persistent tunnel, I used a tool called autossh. As the name implies, this program automatically restarts an SSH session should it breaks for any reason.
As the first step, I set up the ability for the caveserver to log into the Pi without a password. That way, autossh can restart a broken reverse SSH tunnel without user involvement.
Next, install autossh on the caveserver..
From caveserver, run autossh with the following arguments to create a persistent SSH tunnel destined to the Pi.173.173.63.132
caveserver~$ autossh -M 12002 -fN -o "PubkeyAuthentication=yes" -o "StrictHostKeyChecking=false" -o "PasswordAuthentication=no" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 173.173.63.132:12001:localhost:22 [email protected]
The “-M 12001” option specifies a monitoring port on Pi which will be used to exchange test data to monitor an SSH session. This port should not be used by any program on Pi.
The “-fN” option is passed to ssh command, which will let the SSH tunnel run in the background.
The “-o XXXX” options tell ssh to:
- Use key authentication, not password authentication.
- Automatically accept (unknown) SSH host keys.
- Exchange keep-alive messages every 60 seconds.
- Send up to 3 keep-alive messages without receiving any response back.
The rest of reverse SSH tunneling related options remain the same as before.
To establish an SSH tunnel automatically up upon boot, I added the above autossh command to the /etc/rc.local on the caveserver.
