Accessing a Linux Server Behind NAT via Reverse SSH Tunnel

I am running a Linux server at a remote, off the grid, site that I am calling “cave” for the purposes of this post.  

The internet access is provided by Mint Mobile, which uses T-Mobile cell towers.  T-Mobile does not provide users with publicly addressable IP address. Instead, multiple uses share a single IP address using NAT.  I want to SSH into the cave server from anywhere. To do that, I use Reverse SSH tunnelling.

What is Reverse SSH Tunneling?

The concept of reverse SSH tunneling is simple. For this, I needed another host (so-called “relay host”) outside the restrictive cave network.  The Raspberry Pi I will use is located at my home. It has an IP address that is publicly visible and unique to me. I can connect to it from anywhere on the internet. I will refer to it as the “Pi.”

Then I set up a persistent SSH tunnel from the server in the cave network to the Pi. With that, I can connect “back” to the cave server from the Pi.  You can see why this configuration is called “reverse” tunnel. 

As long as the Pi is reachable, I can connect to the cave server from wherever I am regardless of how restrictive T-Mobile’s NAT is – and regardless of how restrictive my in-bound firewall rules are on the cave’s network.

Set up a Reverse SSH Tunnel on Linux

Let’s see how we can create and use a reverse SSH tunnel. We assume the following. We will be setting up a reverse SSH tunnel from the cave server to the Pi so that we can SSH to cave server via the Pi from another computer, such as a laptop.  Let’s call that computer “laptop.”  

Assume the public IP address of the Pi is 173.173.63.132.  

On cave server,  open an SSH connection to the Pi as follows:

caveserver~$ ssh -fN -R 12001:localhost:22 [email protected]

Here the port 12001 is an arbitrary port number that is not used by other programs on the Pi. 

The “-R 12001:localhost:22” option defines a reverse tunnel. It forwards traffic on port 12001 of the Pi to port 22 of the cave server.

With “-fN” option, SSH will go into the background once successfully authenticated with an SSH server. This option is useful since we do not want to execute any commands on a remote SSH server.  We just want to forward ports.

After running the above command, you will be right back to the command prompt of the cave server.

Now, log into the Pi and verify that 127.0.0.1:12001 is bound to sshd. If so, that means a reverse tunnel is set up correctly.

pi~$ sudo netstat -nap | grep 12001

tcp      0 0 127.0.0.1:12001          0.0.0.0:* LISTEN     8493/sshd           

 

Now from any other computer (e.g., the laptop), log in to the Pi. Then access the cave server as follows:

pi~$ ssh -p 12001 CaveUser@localhost

One thing to take note is that the SSH login/password you type for localhost should be for the cave server, not the Pi, since you are logging into the cave server via the tunnel’s local endpoint.  After successful login, you will be on cave server.

Connect Directly to a NATed Server via a Reverse SSH Tunnel

While the above method allows you to reach the cave server behind T-Mobile’s NAT, you need to log in twice: first to the Pi and then to the cave server.  This is because the end point of an SSH tunnel on the Pi is binding to loopback address (127.0.0.1).

But in fact, there is a way to reach the cave server directly with a single login to the Pi. For this, you will need to let sshd on the Pi forward a port not only from loopback address, but also from an external host. This is achieved by specifying GatewayPorts option in sshd running on the Pi.

Open /etc/ssh/sshd_conf of the Pi and add the following line:

pi~$ vi /etc/ssh/sshd_conf

GatewayPorts clientspecified


Restart sshd.

pi~$ sudo /etc/init.d/ssh restart

Now let’s initiate a reverse SSH tunnel from the cave server as follows:

caveserver~$ ssh -fN -R 173.173.63.132:12001:localhost:22 [email protected]

Log into the Pi and confirm with netstat command that a reverse SSH tunnel is established successfully.

Pi~$ sudo netstat -nap | grep 12001

tcp      0 0 173.173.63.132:12001     0.0.0.0:* LISTEN 1538/sshd: dev  

 

Unlike a previous case, the end point of the tunnel is now at 173.173.63.132:12001 (the Pi’s public IP address), not 127.0.0.1:12001. This means that the end point of the tunnel is reachable from an external host.

Now from any other computer (e.g., the laptop), type the following command to gain access to NATed cave server.

clientcomputer~$ ssh -p 12001 [email protected]

Set up a Persistent Reverse SSH Tunnel on Linux

Now, after proving that this concept works, I made the tunnel “persistent.”  This means that the tunnel is active all of the time and can reestablish itself in the case of temporary network congestion, SSH timeout, the Pi rebooting, etc.. 

For a persistent tunnel, I used a tool called autossh. As the name implies, this program automatically restarts an SSH session should it breaks for any reason. 

As the first step, I set up the ability for the caveserver to log into the Pi without a password. That way, autossh can restart a broken reverse SSH tunnel without user involvement.

Next, install autossh on the caveserver..

From caveserver, run autossh with the following arguments to create a persistent SSH tunnel destined to the Pi.173.173.63.132

caveserver~$ autossh -M 12002 -fN -o "PubkeyAuthentication=yes" -o "StrictHostKeyChecking=false" -o "PasswordAuthentication=no" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 173.173.63.132:12001:localhost:22 [email protected]

The “-M 12001” option specifies a monitoring port on Pi which will be used to exchange test data to monitor an SSH session. This port should not be used by any program on Pi.

The “-fN” option is passed to ssh command, which will let the SSH tunnel run in the background.

The “-o XXXX” options tell ssh to:

  • Use key authentication, not password authentication.
  • Automatically accept (unknown) SSH host keys.
  • Exchange keep-alive messages every 60 seconds.
  • Send up to 3 keep-alive messages without receiving any response back.

The rest of reverse SSH tunneling related options remain the same as before.

To establish an SSH tunnel automatically up upon boot, I added the above autossh command to the /etc/rc.local on the caveserver.

 

Leaving the Country to Escape Student Loan Debt

There are so many things wrong with this report that it leaves me speechless.

For starters, if a person is only willing to pay $50 per month for a hammock in the jungle in India, similar accommodations are available in the United States.

Secondly, note the article discusses loans with payments that are $700 per month or less.  I bet these same people paid at least that much between their unnecessary car payments, cell phones, internet access, etc.

And thirdly, $700 per month (the maximum amount mentioned in this article) is only $8400 per year.  Yet the average starting salary for the LOWEST paying college degree (elementary education) is $14000 per year more than the average salary of a plain old high school graduate.

I could go on and on.  It is sad that these types of articles are written because, unfortunately, there are people who read articles like this and then decide to commit to a lifetime of minimum wage jobs after high school.

Notes from Robert Kaplan’s Presentation to Rice Alumni (19-Feb-2019)

On February 19, 2019, I went to a Rice Alumni event at the Federal Reserve Bank in Dallas.  The Dallas Fed is one of twelve Federal Reserve Banks in the United States.  The CEO’s of the Fed banks meet every six weeks or so to set national monetary policy.

Robert Kaplan, the CEO of the Dallas Fed was speaking about the state of the economy.  Peter Rodriguez, the Dean of the Jones School of Business at Rice University, was asking Dr. Kaplan questions.  Late in the presentation, Rice Alumni and their guests asked questions.

Here are some key points that I took away:

  1. Business Pricing Power:  In the current market, consumers have real-time access to prices of products.  They have never had such comprehensive access before.  Because of this, price competition is extremely stiff and businesses are not able to pass on cost increases to consumers.  Dr. Kaplan specifically mentioned the recent steel and aluminum tariffs as costs that businesses are not able to pass on to consumers.
  2. The number of people in the labor force and productivity are the primary drivers of GDP growth.  More people in the labor force yields higher GDP growth.  Likewise, higher productivity yields higher GDP growth.  Technology and education drive productivity increases.  Education is a big problem for Texas because Texas lags the US in general.  And, likewise, the US lags many countries in the developed world.  (More on this below.) Texas’ growth looks positive for at least the short and medium term, however, because Texas is experiencing large population growth.  This is dampening the education disadvantage that Texas has.  Likewise, Texas’ population growth – which results in higher GDP – allows Texas to solve its education problem in the future easier than states with stagnant or declining populations.  Dr. Kaplan specifically mentioned Kansas, where he is originally from, and Illinois.Dr. Kaplan mentioned immigration as a component of population growth in Texas.  But, he did not explore the subject.
  3. Technology helps people with post-secondary educations or specialized training in the trades.  It hurts those with a high school (or less) education.  This is the biggest driver of the growing divide between the wealthy and the poor.  Dr. Kaplan mentioned STEM many times as the path to growth.
  4. Dr. Kaplan specifically mentioned Khan Academy and edX as potentially disruptive forces in the post-secondary education arena.  He stated that education delivery needs to be significantly revamped.  He believes this should be done with steps such as busting teachers’ unions, eliminating tenure for college professors, and financially separating teaching from research at the university level.
  5. The media concentrates on the U3 unemployment rate.  However, the Fed primarily looks like the U6 unemployment rate which considers people who have given up looking for work and part-time workers who would prefer to be full-time.  The U6 unemployment rate is about 8%.  This paints a much less positive picture of the economy than the U3 rate that the media reports.
  6. The US owes $54 Trillion in unfunded entitlements such as social security, medicare, and retiree pensions.  This is a ticking time bomb.  It does not show up in the deficit or national debt numbers that we hear.
  7. 2018 saw GDP grow at a rate of 3%.  That was largely due to the tax overhaul and increases in government spending.  The effects of the tax overhaul are likely over for individuals.  The effects of the tax overhaul for businesses may be longer lasting.  But, that is not clear.  The effects of increases in government spending last year are probably over.  Therefore, GDP will likely grow at 2% or less in 2019.
  8. Dr. Kaplan believes that education spending will yield productivity increases (which will yield GDP increases).  He specifically mentioned early childhood education and that 60% of children who enter 1st Grade behind will never catch up.  He also believes that college prep should have more emphasis at the high school level.
  9. Shale oil/fracking:   The petroleum yield from this kind of oil production is short-lived.  That likely means that the US’ production levels will struggle to maintain their historic high levels and that the oil industry will profit from high oil prices in the future.

Short Commentary

During the question and answer session from Rice Alumni and their guests, one gentleman told Dr. Kaplan that there is either no correlation, or negative correlation, between spending on education and educational achievement levels.  If you think about it, such an assertion is absurd because, if true, the best thing for society (including individuals) would be to spend absolutely nothing on literacy.  Obviously, Dr. Kaplan disagreed.

After the presentation, I spoke with Dr. Rodriguez, the Jones Business School Dean.  I told Dr. Rodriguez that I was surprised by this question and was also surprised that Robert Kaplan did not point out that much of the spending labeled as “education” spending really goes for athletic programs which do not increase literacy.  Dr. Rodriguez agreed and mentioned the huge high school stadium in Allen as a great example.

I told Dr. Rodriguez that I would like to understand the research about the relative effect of employment growth and productivity on GDP.  For example, immigration increases the number of workers in the workforce.  But, immigrants come with various educational levels depending on their circumstances.  So, more immigrants cause GDP to grow.  And highly educated immigrants increase productively and that causes GDP to grow too.  But, illiterate immigrants have the opposite effect on productivity – at least initially.  That means that the positive effects of increased workers are dampened if those workers have lower than average productivity.  Unfortunately, Dr. Rodriguez seemed to take this as a political question and brushed it aside.  It was not meant to be.  I am sure that the Fed has data on this question.  I would love to see it.

 

SSH Access to OpenWRT without a Password

OpenWRT is an open source router firmware that can be installed on most consumer WiFi routers for increased security, functionality, and performance.

One way to configure OpenWRT is through a web interface.  The other, more powerful, way to configure it is through SSH.  OpenWRT comes with dropbear for SSH.  Dropbear is a optimized, reduced functionality, SSH server.  So, the typical methodology of creating public/private key pairs for authentication does not always work.  Here is what I have found to work:

  1. Assume that the OpenWRT router has an assigned IP address of 192.168.1.1
  2. In Cygwin, or in a Linux terminal, run the following commands:

The first command creates a 2048 RSA key, which is the strength recommended by NIST for RSA.  To login without a password, just choose the defaults by pressing enter at each prompt.  The second command copies the public key to the OpenWRT router.  Now, log in to the OpenWRT router with SSH:

ssh [email protected]

You will be prompted for a password.  Use the password that you set up for the OpenWRT web interface.

Once logged into the router, execute the following command:

cp /root/.ssh/authorized_keys /etc/dropbear/authorized_keys

This will copy the public key to the location expected by dropbear.  This has to be done because ssh-keygen puts the key in the directory expected by openssh, not dropbear.

Now, exit from the router:

exit

Back that the Cygwin or Linux terminal, try logging into the OpenWRT router again with SSH:

ssh [email protected]

This time, you should be able to get in without a password.

Finally, use the OpenWRT GUI (under System->Administration) to turn off SSH password authentication and disallow the root user to login with a password.  Test to verify password authentication is turned off by typing the following in Cygwin or a Linux terminal:

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no [email protected]

You should get an error that says [email protected]: Permission denied (publickey).

Two things to note:

  1. Unfortunately, OpenWRT makes it difficult to create anything but a root user.  There are ways to do so, however.  Google it.
  2. I tried generating RSA keys larger than 2048 bits.  However, they did not seem to work with dropbear.

Remote Desktop for Raspberry Pi

To access your Raspberry Pi through remote desktop, type the following commands at the Raspberry Pi terminal prompt:

sudo apt-get install tightvncserver

When installation of tightvncserver is complete, execute this command:

sudo apt-get install xrdp

When this is complete, the Pi should be running a remote desktop server.  To access the Pi on Windows, choose “Remote Desktop Connection” from the start menu.  The remote desktop client will appear and ask you for the IP address of the Pi.  My Pi is at 10.0.0.129.

Hit Connect.

A login screen will appear and ask for your username and password.  The default username for the Pi is “pi”  — and the default password is “raspberry”

Congratulations!  The Raspberry Pi’s desktop should appear.  Now, you can access the Pi from anywhere on your network and there is no need to lug around a monitor, keyboard, mouse, cables, etc.

 

Browser Add-ons — Read Terms and Conditions

I found this wonderful sounding add-on to Chrome this morning. It lets you select one or more Gmail emails to convert to PDF. Then, you can download the PDF or put it on Google Drive. I was quite excited and downloaded the add-on.
 
When I installed it, I learned that I have to create a CloudHQ account and give CloudHQ permissions to have read-only access to my emails.
 
No. Just no.
 
Always read Terms and Conditions when installing anything on your computer. The more companies that have access to your information, the more likely you will become a victim to a data breach and identity theft.
 

Cost of Goods (COGS) – or Cost of Sales

The terms Cost of Goods Sold (COGS), Cost of Sales, and Cost of Revenue are synonymous.  They describe the direct costs of producing a good or service that is sold to customers.   In this post, let’s just refer to this as COGS.

Direct costs include direct labor and materials, and facility or plant overhead that is directly tied to producing the good or service.  For example, the salary of a person assembling a television would be a direct cost.  Extra electricity used to run a machine used only to produce the good or service would also be included.

But, the salary of the janitor at a plant that makes televisions, phones, and alarm clocks would be an indirect cost.   The reason is that the cost of the janitor does not increase or decrease as a result of making more or less televisions.  The amount of floor space to sweep in the facility is the same regardless of the number of televisions produced (within reason.)  The janitor’s salary is an example of SG&A costs.  SG&A stands for selling, general, and administrative.  SG&A expenses occur when the company incurs an expense for

  • Promoting, selling, or delivering products and services
  • Managing the overall company

These types of costs will appear on the company’s quarterly (or annual) income statement for the period they were incurred.  More specific examples of indirect SG&A costs include sales commissions, advertising and promotional materials, management compensation, compensation for support staff, rent, utilities, and office supplies.

The general rule is that direct costs do not include general overhead or administrative expenses.  These expenses are not part of the COGS calculation.

COGS is key metric for cost analysis because shows the operational costs of producing a good and service. If cost of sales is rising while gross revenue is flat, net earnings (gross profit) will decrease.  Remember that:

(Gross Revenue) – (COGS) = (Gross Profit)

Note that for a service business without a tangible, physical, product, COGS is a bit of a misnomer since there is not a “good.”  That is why the term Cost of Sales is often used.  But, the terms mean the same thing.