The Hidden Magic Behind SSH, Remote Access & Port Forwarding

When I took over DevOps responsibilities at my current company, I inherited something beautiful: a fully configured Terminus setup with a long list of servers and databases, most of which were made accessible via port forwarding and .pem files.

It felt magical.

One click and I was inside a remote EC2 instance. Another quick config and I was securely accessing a production PostgreSQL database — right from my laptop.

When it was time to move on from the company, I found myself passing on that same setup to the next person. All the right .pem files, SSH aliases, and database forwarding rules — just like they were once handed to me.

That's when I realized: I had used SSH daily, but never truly appreciated how it worked under the hood.

So this post is both a thank-you note to SSH and a deep dive into how it does what it does — especially around remote access and port forwarding.

🔐 What SSH Is, Really

SSH (Secure Shell) is a cryptographic network protocol that lets you securely communicate with remote machines over an insecure network like the internet.

Think of it as:

"A secure tunnel that encrypts everything — login, commands, files — between your machine and the remote system."

Under the hood, SSH uses:

• Asymmetric encryption for authentication (your SSH key pair)
• Symmetric encryption for actual data transfer (after the handshake)
• MAC (Message Authentication Codes) to ensure message integrity

🗝️ How SSH Authentication Works

Let's say you're using a .pem file or an SSH key to connect.

Here's what's happening step-by-step:

1. Key Pair: You have a private key (id_rsa or .pem) and the remote machine has the corresponding public key (~/.ssh/authorized_keys).
2. Challenge: The server sends a challenge encrypted with your public key.
3. Proof: Your client decrypts it with the private key and proves identity.
4. Secure Tunnel: Once authenticated, a symmetric encryption session begins for all future communication.

This handshake ensures that no passwords are sent over the wire and no one can impersonate you unless they have your private key.

🖥️ Remote SSH — Accessing Machines Over the Internet

When you run a command like this:

ssh -i "my-key.pem" ec2-user@ec2-12-34-56-78.compute-1.amazonaws.com

You're telling your machine to:

• Use the private key at my-key.pem
• Connect via port 22 (default SSH)
• Use ec2-user as the login
• Trust the public IP (or DNS) of the EC2 instance
• Negotiate a secure connection using SSH protocol

The server checks your public key in ~/.ssh/authorized_keys. If it matches, you're in — and you can now treat that remote machine like it's your own terminal.

🌉 What About Port Forwarding?

Port forwarding lets you tunnel traffic from your local machine to a remote service, like a database or a dashboard, through the SSH connection.

🧭 Local Port Forwarding

This is the most common. It forwards traffic from a local port to a remote host/port.

ssh -i "my-key.pem" -L 5432:localhost:5432 ec2-user@remote-host

What this does:

• Binds localhost:5432 on your laptop
• Forwards all traffic to localhost:5432 on the remote EC2 instance
• You can now connect to the remote Postgres DB via localhost:5432 on your local machine

It feels like the database is running locally — but it's actually remote.

🧭 Remote Port Forwarding

Less common, but powerful in reverse. It allows the remote machine to access a service on your local machine.

ssh -R 9000:localhost:3000 ec2-user@remote-host

Now localhost:9000 on the remote machine will forward to your local localhost:3000.

Useful when:

• You want to expose your local development server to a remote environment (e.g., testing a webhook endpoint on a cloud server).

🧭 Dynamic Port Forwarding

This creates a SOCKS proxy that can route traffic to multiple destinations through your SSH connection:

ssh -D 8080 ec2-user@remote-host

What this does:

• Creates a SOCKS proxy server on your local machine at port 8080
• Any applications configured to use this proxy will route their traffic through the SSH tunnel
• Perfect for browsing internal websites or accessing multiple services without setting up individual forwards

🧰 How I Used It in Practice

At work, I had a dozen databases across different environments (dev, staging, prod), each set up using:

ssh -i key.pem -L <local-port>:localhost:<remote-db-port> ec2-user@bastion-host

Sometimes I chained bastion hosts using ProxyCommand, like:

Host prod-db
  HostName internal-db
  User ec2-user
  IdentityFile ~/.ssh/key.pem
  ProxyCommand ssh -W %h:%p bastion-host

All this was stored in my .ssh/config, and configured in Terminus so that it just worked.

And it was easy to hand over — just zip up .pem files, share .ssh/config, and document the port mappings.

🧠 Why This Matters More Than We Realize

SSH feels like an invisible convenience, but it's a gateway to:

• Security: Password-less, encrypted access
• Productivity: Quick terminal access, effortless port forwarding
• Flexibility: Tunnels, proxies, chained access across environments
• Simplicity: Everything from Git access to database admin goes through SSH

🛑 A Few Gotchas to Watch For

• Don't share .pem or private keys carelessly — they're as powerful as passwords.
• Always use chmod 400 key.pem to restrict file permissions.
• Expired or rotated keys can silently break access (good to automate reminders).
• Monitor for zombie port-forwarding sessions that never get used.
• Be careful with the -g flag in port forwarding, which makes your forwarded ports accessible to other machines on your network.
• Remember that frequent disconnections might require using ServerAliveInterval and ServerAliveCountMax in your SSH config.

📋 Advanced SSH Configuration

Your .ssh/config file can make all the difference in your SSH experience. Here are some powerful configurations:

# Default settings for all hosts
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    AddKeysToAgent yes
    IdentitiesOnly yes

# Jump host configuration
Host bastion
    HostName bastion.example.com
    User jumpuser
    IdentityFile ~/.ssh/jump_key.pem

# Internal hosts accessed through the jump host
Host internal-*
    ProxyJump bastion
    User ec2-user
    IdentityFile ~/.ssh/internal_key.pem

# Database access with automatic port forwarding
Host db-prod
    HostName db-prod.internal
    LocalForward 5432:localhost:5432
    LocalForward 6379:redis-prod:6379
    User dbadmin
    IdentityFile ~/.ssh/db_key.pem
    ProxyJump bastion

With this configuration, running ssh db-prod automatically:

• Connects to the bastion host first
• Jumps to the db-prod server
• Sets up port forwarding for both PostgreSQL and Redis
• All with a single command!

🔍 Troubleshooting Common SSH Issues

When things go wrong, these are the common culprits:

1. Permission issues: Always check file permissions with ls -la ~/.ssh/ and ensure private keys are set to 600 or 400.

2. Host key verification failures: If you see the dreaded "Host key verification failed", it might be due to:

   • Server IP changed but hostname remained the same
   • Server reinstalled with new keys
   • Man-in-the-middle attack (rare but possible)

   To fix: ssh-keygen -R hostname to remove the old key, then reconnect.

3. Connection timeouts: Check for:

   • Network connectivity issues
   • Firewall rules blocking port 22
   • SSH service not running on the target host

4. Authentication failures: Verify:

   • You're using the correct key
   • The server has your public key in .ssh/authorized_keys
   • The key format is compatible with the server

5. Port forwarding failures: Common causes include:

   • Target port already in use
   • Insufficient permissions on the remote server
   • Binding to port < 1024 without root privileges

The -v flag (or -vv or -vvv for more verbosity) is your best friend when troubleshooting:

ssh -vvv -i key.pem user@hostname

🔚 Final Thoughts

SSH might feel like a quiet background utility, but it's one of the most elegant and powerful tools we use as engineers.

Whether it's opening 10+ Postgres instances from your laptop, or handing over the keys to infrastructure with a single file — it all just works. And that's beautiful.

If you've never looked deeper into how SSH works, take a moment this week to try ssh -v, explore .ssh/config, or forward a port manually. You'll come away with a new appreciation for this unsung hero of DevOps.

Sometimes, understanding the tools you use every day is the best kind of magic.

📊 Diagrams

SSH Authentication Flow

Client                                 Server
------                                 ------
  |                                      |
  |------- SSH Protocol Version -------->|
  |<------ SSH Protocol Version ---------|
  |                                      |
  |------- Algorithm Negotiation ------->|
  |<------ Algorithm Negotiation --------|
  |                                      |
  |------- Diffie-Hellman Key Exchange ->|
  |<------ Diffie-Hellman Key Exchange --|
  |                                      |
  |        [Session Keys Established]    |
  |                                      |
  |------- User Authentication Request ->|
  |<------ Authentication Challenge -----|
  |------- Signed Response ------------->|
  |<------ Success ----------------------|
  |                                      |
  |------- Channel Open Request -------->|
  |<------ Channel Confirmation ---------|
  |                                      |
  |        [Secure Session Established]  |
  |                                      |

SSH Port Forwarding Types

Local Port Forwarding:
--------------------
localhost:8080 -> SSH Client -> SSH Server -> target:80
                    |             |
                 Your PC        Remote Server

Remote Port Forwarding:
--------------------
                    |             |
                 Your PC        Remote Server
target:80 <- SSH Client <- SSH Server <- remotehost:8080

Dynamic Port Forwarding:
--------------------
Application -> SOCKS Proxy -> SSH Client -> SSH Server -> Various Destinations
                               |             |
                            Your PC        Remote Server

SSH Key Authentication Process

┌──────────┐                            ┌──────────┐
│  Client  │                            │  Server  │
└────┬─────┘                            └────┬─────┘
     │                                       │
     │ 1. Connection request                 │
     │──────────────────────────────────────▶│
     │                                       │
     │ 2. Server sends challenge             │
     │◀──────────────────────────────────────│
     │                                       │
     │ 3. Client signs challenge with        │
     │    private key                        │
     │                                       │
     │ 4. Client sends signature             │
     │──────────────────────────────────────▶│
     │                                       │
     │                                       │ 5. Server verifies signature
     │                                       │    using public key in
     │                                       │    authorized_keys
     │                                       │
     │ 6. Authentication successful          │
     │◀──────────────────────────────────────│
     │                                       │
┌────┴─────┐                            ┌────┴─────┐
│  Client  │                            │  Server  │
└──────────┘                            └──────────┘

SSH Encrypted Tunnel Architecture

┌───────────────┐              ┌───────────────┐              ┌────────────────┐
│                │              │               │              │                │
│  Local        │              │  SSH Tunnel   │              │  Remote        │
│  Applications ├──────────────┤  (Encrypted)  ├──────────────┤  Services      │
│               │              │               │              │                │
└───────┬───────┘              └───────────────┘              └────────┬───────┘
        │                                                              │
        │                                                              │
┌───────┴───────┐                                              ┌───────┴───────┐
│  Local Ports  │                                              │  Remote Ports │
│               │                                              │               │
│  - Web: 8080  │                                              │  - Web: 80    │
│  - DB: 5432   │                                              │  - DB: 5432   │
│  - SOCKS: 1080│                                              │  - Redis: 6379│
└───────────────┘                                              └───────────────┘

📚 Additional Resources

If you're interested in learning more about SSH, here are some excellent resources:

• OpenSSH Documentation - The official documentation for OpenSSH
• SSH Mastery by Michael W. Lucas - An excellent book on SSH
• SSH Config File Examples - More advanced configuration examples
• SSH Port Forwarding Tutorial - More examples of SSH tunneling
• OpenSSH Security Guidelines - Best practices for secure SSH configuration

Remember, SSH is more than just a way to log into remote servers. It's a powerful tool for secure networking that can simplify your entire infrastructure management workflow.

May your connections always be secure, and your port forwards never conflict!