Accessing a Linux Server Behind NAT via Reverse SSH Tunnel

I am running a Linux server at a remote, off the grid, site that I am calling “cave” for the purposes of this post.  

The internet access is provided by Mint Mobile, which uses T-Mobile cell towers.  T-Mobile does not provide users with publicly addressable IP address. Instead, multiple uses share a single IP address using NAT.  I want to SSH into the cave server from anywhere. To do that, I use Reverse SSH tunnelling.

What is Reverse SSH Tunneling?

The concept of reverse SSH tunneling is simple. For this, I needed another host (so-called “relay host”) outside the restrictive cave network.  The Raspberry Pi I will use is located at my home. It has an IP address that is publicly visible and unique to me. I can connect to it from anywhere on the internet. I will refer to it as the “Pi.”

Then I set up a persistent SSH tunnel from the server in the cave network to the Pi. With that, I can connect “back” to the cave server from the Pi.  You can see why this configuration is called “reverse” tunnel. 

As long as the Pi is reachable, I can connect to the cave server from wherever I am regardless of how restrictive T-Mobile’s NAT is – and regardless of how restrictive my in-bound firewall rules are on the cave’s network.

Set up a Reverse SSH Tunnel on Linux

Let’s see how we can create and use a reverse SSH tunnel. We assume the following. We will be setting up a reverse SSH tunnel from the cave server to the Pi so that we can SSH to cave server via the Pi from another computer, such as a laptop.  Let’s call that computer “laptop.”  

Assume the public IP address of the Pi is 173.173.63.132.  

On cave server,  open an SSH connection to the Pi as follows:

caveserver~$ ssh -fN -R 12001:localhost:22 PiUser@173.173.63.132

Here the port 12001 is an arbitrary port number that is not used by other programs on the Pi. 

The “-R 12001:localhost:22” option defines a reverse tunnel. It forwards traffic on port 12001 of the Pi to port 22 of the cave server.

With “-fN” option, SSH will go into the background once successfully authenticated with an SSH server. This option is useful since we do not want to execute any commands on a remote SSH server.  We just want to forward ports.

After running the above command, you will be right back to the command prompt of the cave server.

Now, log into the Pi and verify that 127.0.0.1:12001 is bound to sshd. If so, that means a reverse tunnel is set up correctly.

pi~$ sudo netstat -nap | grep 12001

tcp      0 0 127.0.0.1:12001          0.0.0.0:* LISTEN     8493/sshd           

 

Now from any other computer (e.g., the laptop), log in to the Pi. Then access the cave server as follows:

pi~$ ssh -p 12001 CaveUser@localhost

One thing to take note is that the SSH login/password you type for localhost should be for the cave server, not the Pi, since you are logging into the cave server via the tunnel’s local endpoint.  After successful login, you will be on cave server.

Connect Directly to a NATed Server via a Reverse SSH Tunnel

While the above method allows you to reach the cave server behind T-Mobile’s NAT, you need to log in twice: first to the Pi and then to the cave server.  This is because the end point of an SSH tunnel on the Pi is binding to loopback address (127.0.0.1).

But in fact, there is a way to reach the cave server directly with a single login to the Pi. For this, you will need to let sshd on the Pi forward a port not only from loopback address, but also from an external host. This is achieved by specifying GatewayPorts option in sshd running on the Pi.

Open /etc/ssh/sshd_conf of the Pi and add the following line:

pi~$ vi /etc/ssh/sshd_conf

GatewayPorts clientspecified


Restart sshd.

pi~$ sudo /etc/init.d/ssh restart

Now let’s initiate a reverse SSH tunnel from the cave server as follows:

caveserver~$ ssh -fN -R 173.173.63.132:12001:localhost:22 PiUser@173.173.63.132

Log into the Pi and confirm with netstat command that a reverse SSH tunnel is established successfully.

Pi~$ sudo netstat -nap | grep 12001

tcp      0 0 173.173.63.132:12001     0.0.0.0:* LISTEN 1538/sshd: dev  

 

Unlike a previous case, the end point of the tunnel is now at 173.173.63.132:12001 (the Pi’s public IP address), not 127.0.0.1:12001. This means that the end point of the tunnel is reachable from an external host.

Now from any other computer (e.g., the laptop), type the following command to gain access to NATed cave server.

clientcomputer~$ ssh -p 12001 CaveUser@173.173.63.132

Set up a Persistent Reverse SSH Tunnel on Linux

Now, after proving that this concept works, I made the tunnel “persistent.”  This means that the tunnel is active all of the time and can reestablish itself in the case of temporary network congestion, SSH timeout, the Pi rebooting, etc.. 

For a persistent tunnel, I used a tool called autossh. As the name implies, this program automatically restarts an SSH session should it breaks for any reason. 

As the first step, I set up the ability for the caveserver to log into the Pi without a password. That way, autossh can restart a broken reverse SSH tunnel without user involvement.

Next, install autossh on the caveserver..

From caveserver, run autossh with the following arguments to create a persistent SSH tunnel destined to the Pi.173.173.63.132

caveserver~$ autossh -M 12002 -fN -o "PubkeyAuthentication=yes" -o "StrictHostKeyChecking=false" -o "PasswordAuthentication=no" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 173.173.63.132:12001:localhost:22 PiUser@173.173.63.132

The “-M 12001” option specifies a monitoring port on Pi which will be used to exchange test data to monitor an SSH session. This port should not be used by any program on Pi.

The “-fN” option is passed to ssh command, which will let the SSH tunnel run in the background.

The “-o XXXX” options tell ssh to:

  • Use key authentication, not password authentication.
  • Automatically accept (unknown) SSH host keys.
  • Exchange keep-alive messages every 60 seconds.
  • Send up to 3 keep-alive messages without receiving any response back.

The rest of reverse SSH tunneling related options remain the same as before.

To establish an SSH tunnel automatically up upon boot, I added the above autossh command to the /etc/rc.local on the caveserver.

 

IPTABLES Personal Firewall

Personal Firewall is a tidy little set of scripts for implementing a pretty tight firewall on a machine that is connecting to an open (or otherwise untrusted) network.  It is a set of scripts for allowing a minimal set of traffic (HTTP, DNS, DHCP, NTP, etc) immediately and then save the configuration to be persistent upon reboot.

Of course, add or delete services you do not need.  For example, do you need Skype?  If not, delete the Skype configuration from the script.

https://github.com/meetrp/personalfirewall

This link also contains a script to convert URLs to CIDR addresses.  For example, the script will identify the IP address ranges of an unwanted website so that you can block it with the Personal Firewall.

The Personal Firewall uses iptables.  So, it only works on a Linux machine.  However, it could be of use for a home router running a Linux based firmware build such as OpenWRT.

Chocolatey Package Manager for Windows

Chocolatey is a package manager for Windows (like apt-get or yum but for Windows). It is a single, unified interface designed to easily work with all aspects of managing Windows software (installers, zip archives, runtime binaries, internal and 3rd party software) using a packaging framework that understands software versions as well as dependencies.

Chocolatey packages encapsulate everything required to manage a particular piece of software into one application by wrapping installers, executables, zips, and scripts into a single package.  This makes applications easy to install.  It simplifies the process of checking for updates.  And it makes installing those updates easy.  Installing updates on a regular basis is crucial for maintaining the security of your Windows machine.

Thousands of packages are available through Chocolatey.  Most, if not all, of them are free and open source.

Open a command prompt in Windows under administrator privileges:

Copy and paste the following command into the command prompt:

@”%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe” -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command “iex ((New-Object System.Net.WebClient).DownloadString(‘https://chocolatey.org/install.ps1’))” && SET “PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin”

After the installation finishes, type the following into the command prompt:

choco upgrade chocolatey

Finally, type the following into the command prompt if you want to use the GUI interface:

choco install chocolateygui

The installer will ask if you want to run the script.  The answer is Yes.

Once the installation has completed, close the command prompt.  Chocolatey should now be visible on your Start menu.