courses

SSH: Secure Shell

More than a remote login

You will spend this entire course inside ssh — connecting to the course VM, the lab machines, and ada. Most people learn just enough to type ssh user@host and stop there, and in doing so miss that SSH is one of the most capable tools on the system: an encrypted transport that can carry interactive shells, file transfers, arbitrary TCP connections (tunnels), X11, and agent credentials, all multiplexed over a single authenticated connection.

SSH matters for security on both sides of the fence. Defensively, key-based authentication is the single biggest upgrade you can make over passwords (no password to phish or brute-force), and ~/.ssh/authorized_keys is a file defenders must watch because adding a key is the quietest persistence an attacker can establish. Offensively, SSH tunnels are the classic pivoting mechanism for lateral movement — a -D SOCKS proxy turns one foothold into access to an entire internal network.

This page covers key generation and use, the ~/.ssh/config file (patterns, placeholders, and Match), forward/reverse/dynamic tunnels, locking down authorized_keys, and the often-confusing question of which setting wins when options are specified in several places at once.

ℹ️ SSH has three layers: a transport layer that negotiates encryption and authenticates the server (via its host key), a user-authentication layer (password, public key, etc.), and a connection layer that multiplexes channels (your shell, tunnels, agent forwarding) over the one encrypted pipe. The host key is why you see “The authenticity of host … can’t be established” on first connect — that is trust-on-first-use, recorded in ~/.ssh/known_hosts.

Keys: generate, distribute, use

Public-key authentication replaces a password with a key pair: a private key that never leaves your machine and a public key you place on every server you want to reach. The server challenges you to prove you hold the private key; nothing reusable crosses the wire. This is the digital-signature / asymmetric-crypto primitive applied to login.

Generating a key

Use Ed25519 — it is fast, small, and modern (prefer it over RSA; if you must use RSA, use 4096-bit):

$ ssh-keygen -t ed25519 -C "dmcgrath@laptop-2026"
Generating public/private ed25519 key pair.
Enter file in which to save the key (~/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):      ← USE a passphrase
...
Your identification has been saved in ~/.ssh/id_ed25519       (private — guard this)
Your public key has been saved in ~/.ssh/id_ed25519.pub       (shareable)

Distributing the public key

The public key must end up in ~/.ssh/authorized_keys on the server. The painless way:

$ ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

Without ssh-copy-id, append it manually (note the append >> so you don’t clobber existing keys):

$ cat ~/.ssh/id_ed25519.pub | ssh user@server 'cat >> ~/.ssh/authorized_keys'

⚠️ Permissions are enforced by sshd. SSH refuses to use keys if the permissions are too open: ~/.ssh must be 700, and ~/.ssh/authorized_keys and your private key 600. A silently failing key-auth is, nine times out of ten, a permissions problem (chmod 700 ~/.ssh; chmod 600 ~/.ssh/*).

The agent: type the passphrase once

ssh-agent holds your decrypted keys in memory so you aren’t prompted on every connection:

$ eval "$(ssh-agent -s)"          # start an agent (usually already running)
$ ssh-add ~/.ssh/id_ed25519       # add a key (prompts for the passphrase once)
$ ssh-add -l                      # list loaded keys
$ ssh-add --apple-use-keychain ~/.ssh/id_ed25519   # macOS: store passphrase in Keychain

The config option AddKeysToAgent yes (used in the example config below) does this automatically the first time a key is used.

Windows: enable the ssh-agent service

Windows ships OpenSSH (since Windows 10 1809 / Server 2019), but unlike Linux and macOS its ssh-agent is a Windows service that is disabled by default — so ssh-add fails with “Error connecting to agent” until you turn it on. Enable it once, from an administrator PowerShell prompt:

# Run as Administrator. Set the agent to start automatically on boot, then start it now.
Get-Service ssh-agent | Set-Service -StartupType Automatic
Start-Service ssh-agent
Get-Service ssh-agent            # should report Status: Running

After that one-time setup, use ssh-add from a normal (non-elevated) prompt exactly as on Linux — note the $env:USERPROFILE path style instead of ~:

ssh-add $env:USERPROFILE\.ssh\id_ed25519   # prompts for the passphrase once
ssh-add -l                                 # list loaded keys

The Windows agent stores keys in your Windows account’s security context (backed by the registry, in DPAPI-protected form), so they persist across reboots and the service reloads them automatically — you are not prompted again.

ℹ️ This applies to the native Windows OpenSSH in PowerShell/Command Prompt. If you instead use Git Bash, it runs its own MSYS2 ssh-agent (start it with eval "$(ssh-agent -s)", as on Linux), and WSL is a separate Linux environment with its own agent again — three different agents that do not share keys. Pick one workflow per machine to avoid confusion.

⚠️ Microsoft’s docs suggest backing up the private key elsewhere and deleting it from disk once it is loaded into the agent, since the agent can serve it without the file present. Be careful: if the agent’s stored copy is ever lost (profile reset, re-image) and you kept no backup, the key is gone — you would have to generate a new pair and re-distribute the public key everywhere.

⚠️ “Error connecting to agent: No such file or directory” — even though the service is Running. The native Windows client reaches the agent over a named pipe (\\.\pipe\openssh-ssh-agent) and ignores the Unix-style SSH_AUTH_SOCK. But if something has set SSH_AUTH_SOCK, the client tries that socket path instead, and since it does not exist as a Windows file you get this error. Diagnose and fix in PowerShell:

$env:SSH_AUTH_SOCK          # if this prints a path, that's the problem
Remove-Item Env:\SSH_AUTH_SOCK   # clear it for this session
ssh-add -l                  # should now connect (lists keys, or "no identities")

The usual culprit is a terminal that forwards its own agent. WezTerm is the common one: its mux_enable_ssh_agent option (on by default) sets SSH_AUTH_SOCK in every pane to a WezTerm-managed socket — fine on Linux/macOS, but it shadows the Windows named pipe. Disable it in ~/.wezterm.lua with config.mux_enable_ssh_agent = false, then fully restart the terminal (existing panes keep the old environment). WSL↔Windows agent-sharing tools (npiperelay/wsl-ssh-agent) and some dotfiles set SSH_AUTH_SOCK for the same reason — check those if it reappears in a fresh shell.

⚠️ Agent forwarding (ForwardAgent) is convenient and dangerous. It lets a remote host use your local keys to authenticate onward — but root on that host can hijack your agent and impersonate you everywhere. Leave it no by default and enable it per-host only when you trust the box. (Note the example config sets ForwardAgent no.) Prefer ProxyJump for reaching machines through a bastion.

The client config file: ~/.ssh/config

Typing -i, -p, -l, and ProxyJump flags every time is miserable. ~/.ssh/config lets you name hosts and attach settings to them. This is where SSH gets powerful, so the examples below are drawn from a real working config.

Host blocks, patterns, and placeholders

A Host line introduces a block whose options apply to any connection whose target name matches the pattern. Patterns use glob wildcards: * (any run of characters), ? (one character), and ! (negation).

# A single named host with its own key
Host ada
    Hostname linux.cs.pdx.edu
    IdentityFile ~/.ssh/id_ed25519

# '??' matches exactly two characters: systemsec-01 ... systemsec-99
Host systemsec-??
    Hostname %h.cs.pdx.edu          # %h = the host you typed
    Port 22
    IdentitiesOnly yes              # only offer the key below, not every agent key
    IdentityFile ~/.ssh/lab/proxmox/%n   # %n = the original name on the command line
    User root

The % tokens are placeholders expanded at connect time — they let one block serve many hosts:

Token Expands to
%h the host name being connected to (after Hostname substitution)
%n the original name you typed on the command line
%p the port
%r the remote username
%% a literal %

So ssh systemsec-07 connects to systemsec-07.cs.pdx.edu as root using the key ~/.ssh/lab/proxmox/systemsec-07 — all from one block.

Wildcard groups and ProxyJump

ProxyJump (-J on the command line) routes a connection through a bastion/jump host — the traffic to the final host is tunneled inside the connection to the jump host, so the final host never needs to be directly reachable. Combine it with a wildcard group:

# Every host starting 'pp' is reached by jumping through the 'pandaprox' gateway
Host pp*
    ProxyJump pandaprox
    IdentityFile ~/.ssh/pandaprox

Host ppkali
    Hostname 10.20.100.105
    HostKeyAlias ppkali             # see note below
    User kali

Now ssh ppkali transparently hops you → pandaprox → ppkali. You can chain jumps too: ProxyJump ada,imcgrath goes through two bastions in order (equivalent to ssh -J ada,imcgrath …).

ℹ️ HostKeyAlias is the fix for a real lab headache: when several different VMs reuse the same internal IP (e.g. each student rack hands out 10.20.100.105), their host keys collide in known_hosts and SSH screams about a possible attack. HostKeyAlias ppkali tells SSH to file this host’s key under the name ppkali instead of the shared IP, keeping them separate.

The Match keyword: conditional configuration

Host only matches on the target name. Match matches on richer conditions — and crucially can run a command and branch on its exit status with exec. This config uses it to only jump through a bastion when off-campus:

# If we are NOT already on the cecs network (resolv.conf has no cecs search
# domain), reach 'panda' by proxying through 'ada'. On campus, connect directly.
Match host panda !exec "grep -q 'search.*cecs.pdx.edu' /etc/resolv.conf"
    ProxyJump ada

Match host panda restricts it to the panda target; !exec "..." adds the condition “and the command fails” (the ! negates) — i.e. we are not on campus. When both hold, the ProxyJump applies; otherwise it’s skipped and you connect directly. This is how one config Just Works from the coffee shop and from the lab without edits.

The catch-all Host * and why it goes last

A block matching Host * applies to everything. It is the right place for global defaults — but it must go at the bottom of the file, for the precedence reason in the next section:

Host *
    IdentitiesOnly yes        # only use specified/loaded keys, not every key (avoids "too many auth failures")
    AddKeysToAgent yes        # auto-add keys to the agent on first use
    IgnoreUnknown UseKeychain # don't error on the next line where UseKeychain is unsupported (Windows)
    UseKeychain yes           # macOS: pull passphrases from the Keychain
    ForwardAgent no           # safe default (see agent-forwarding warning above)
    ServerAliveInterval 30    # send a keepalive every 30s …
    ServerAliveCountMax 3     # … and drop after 3 missed, so dead connections don't hang
    User dmcgrath             # default username almost everywhere

Copying files over SSH: scp, sftp, and rsync

The same authenticated channel that carries your shell can carry files. Three tools ride on top of SSH, and they share its keys, its ~/.ssh/config host aliases, and its authorized_keys restrictions.

scp — quick file copy

scp copies files between hosts with cp-like syntax, where a remote path is host:path (the host can be any alias from your config):

$ scp report.pdf ada:~/uploads/           # local → remote (uses the 'ada' config block)
$ scp ada:/etc/motd ./                     # remote → local
$ scp -r ./site ada:/var/www/             # -r: copy a directory recursively
$ scp -P 2222 file user@host:~/           # -P (CAPITAL) sets the port — NOT -p like ssh!
$ scp -i ~/.ssh/other_key file host:~/    # -i: pick an identity, same as ssh
$ scp -C bigfile.tar ada:~/               # -C: compress in transit

⚠️ Two scp gotchas. The port flag is -P (capital) — lowercase -p means “preserve timestamps,” the opposite of ssh’s -p. And as of OpenSSH 9.0 (2022), scp uses the SFTP protocol under the hood by default; its old RCP-based protocol had surprising filename-glob and quoting behavior and is now deprecated (-O forces the legacy mode). For anything beyond a one-off copy, prefer sftp or rsync.

sftp — interactive or scripted transfer

sftp speaks a richer file-transfer protocol over the same SSH connection, with an FTP-like interactive prompt (get, put, ls, cd, mkdir) or batch mode:

$ sftp ada
sftp> put localfile.txt          # upload
sftp> get remotefile.txt         # download
sftp> ls -l                      # browse the remote side
sftp> bye

$ sftp -b commands.txt ada       # -b: run a batch script non-interactively

rsync — the right tool for big or repeated copies

For large trees, mirrors, or anything you’ll copy more than once, rsync over SSH transfers only the changed portions and preserves attributes — far more efficient than scp. It’s covered on the Working with Files page; in brief:

$ rsync -avz --progress ./build/ ada:/var/www/html/    # sync over SSH (rsync uses ssh by default)
$ rsync -avz -e "ssh -p 2222" src/ host:dst/           # -e to pass custom ssh options

Because all three run over SSH, they obey the server’s policy: a key locked to a command="…" forced command (below) won’t run scp unless your wrapper allows it, and the sftp subsystem can be disabled or chrooted in sshd_config to offer file transfer without shell access.

Tunnels: forwarding ports over SSH

Because the SSH connection layer multiplexes channels, you can push arbitrary TCP connections through the encrypted pipe. There are three directions, and the trick to keeping them straight is to ask which machine opens the listening port.

The common flags: -N (do not run a remote command — just hold the tunnel), -f (background after auth), and -T (no terminal).

Local forwarding (-L) — pull a remote service to you

-L [bind:]localport:target:targetport opens a listener on your machine; connections to it pop out from the SSH server and go to target:targetport. Use it to reach something only the server can see — a database bound to localhost, an internal web UI, a VPN-less path to an internal host.

# Reach a database that only listens on the server's localhost:5432.
# Now connecting to localhost:5432 on YOUR box hits the remote Postgres.
$ ssh -N -L 5432:localhost:5432 ada

# Reach an internal-only web app (intranet.internal:80) via the bastion 'ada':
$ ssh -N -L 8080:intranet.internal:80 ada     # browse http://localhost:8080

Remote forwarding (-R) — expose your service to the far side

-R [bind:]remoteport:target:targetport opens a listener on the SSH server; connections there are tunneled back and emerge from your machine toward target:targetport. Use it to expose a local service to a network you can only reach outbound — the basis of the Windows-RDP-over-SSH guide.

# Expose your laptop's RDP (localhost:3389) as port 1222 on the server:
$ ssh -N -R 1222:localhost:3389 ada

⚠️ Reverse forwarding is also how attackers establish reverse shells and exfiltration paths that beat egress firewalls — an outbound SSH session that opens an inbound door. Seeing unexpected -R tunnels on a host is a red flag for SOC/IR. By default the server-side listener binds only to its loopback; GatewayPorts must be enabled to expose it more widely.

Dynamic forwarding (-D) — a SOCKS proxy / poor man’s VPN

-D localport opens a SOCKS proxy on your machine; any app pointed at it has its traffic emerge from the SSH server, to any destination. One command turns a single SSH foothold into network-wide access — which is exactly why it’s a favorite pivoting tool.

$ ssh -N -D 1080 ada              # SOCKS5 proxy on localhost:1080
$ curl --socks5-hostname localhost:1080 http://internal-only.host/   # routed through ada

You can make a tunnel permanent in ~/.ssh/config with LocalForward, RemoteForward, and DynamicForward directives attached to a host.

authorized_keys: controlling what a key can do

On the server, ~/.ssh/authorized_keys lists the public keys allowed to log in as that user. The basic format is one key per line. But each line can be prefixed with options that constrain what the key may do — turning a key from “full shell access” into a narrowly scoped capability.

Forced commands: a key that can only do one thing

The command="…" option forces that command to run on login, ignoring whatever the client asked for. This is the standard way to grant automated, least-privilege access — a backup job, a git pull, a status check — without handing over a shell:

# In the server's ~/.ssh/authorized_keys:
command="/usr/local/bin/backup-readonly.sh",restrict ssh-ed25519 AAAA...key... backup@ci

# 'restrict' (OpenSSH 7.2+) denies EVERYTHING — no port forwarding, no agent
# forwarding, no X11, no pty — the safest baseline. Opt back in explicitly if needed,
# e.g.:  restrict,pty   or   restrict,port-forwarding

The command the client tried to run is available to your script in $SSH_ORIGINAL_COMMAND, so a single forced command can dispatch among a few allowed actions:

#!/bin/bash
# backup-readonly.sh — only permit specific rsync/backup invocations
case "$SSH_ORIGINAL_COMMAND" in
  "rsync --server --sender"*) exec $SSH_ORIGINAL_COMMAND ;;
  *) echo "Denied: this key may only run backups." >&2; exit 1 ;;
esac

Other useful per-key restrictions

Option Effect
restrict Deny all features (the recommended baseline); re-enable selectively
from="198.51.100.0/24,*.cs.pdx.edu" Only accept this key from matching source addresses/hosts
no-port-forwarding This key cannot set up -L/-R/-D tunnels
no-agent-forwarding, no-X11-forwarding, no-pty Disable those specific features
permitopen="host:port" Restrict any forwarding to a single destination
expiry-time="20261231" Key stops working after a date (OpenSSH 8.2+)

⚠️ Because authorized_keys grants standing access, it is a top persistence mechanism. Auditing it — and watching it with file-integrity monitoring (auditd/AIDE) — is basic host hardening. An unexplained key, or one with no from=/restrict, deserves investigation.

Where options come from: the precedence rules

When the same option can be set on the command line, in your user config, in the system config, and (server-side) in sshd_config, which wins? Two different rule-sets are at play — get them straight and the rest of SSH stops surprising you.

Client side — “first value wins”

For the client, options are gathered in this order, and for each option the first value obtained is used and later ones are ignored:

  1. Command-line -o, -i, -p, -l, etc. (highest priority — read first)
  2. ~/.ssh/config (your per-user file)
  3. /etc/ssh/ssh_config (system-wide defaults, lowest priority)

Two consequences trip everyone up at least once:

# One-off override: force a different key and user for a single connection,
# beating whatever ~/.ssh/config says, without editing any file:
$ ssh -i ~/.ssh/other_key -o User=root -o IdentitiesOnly=yes systemsec-07

# Inspect exactly which options SSH resolved for a host (no connection made):
$ ssh -G systemsec-07 | grep -iE 'user|identityfile|proxyjump|hostname'

ssh -G <host> prints the fully-resolved configuration SSH would use — the definitive way to debug “why is it picking that key/user?”

Server side — sshd_config and Match

The server independently decides what it will allow, in /etc/ssh/sshd_config. Here the precedence is different: the first matching Match block wins, and a Match overrides the global settings for connections that match it. This is how an admin says “passwords are off everywhere, but this one bastion account may only port-forward”:

# /etc/ssh/sshd_config (server)
PasswordAuthentication no          # global: keys only
PermitRootLogin prohibit-password  # root may log in by key, never by password

Match User backup                  # for the 'backup' user only:
    ForceCommand /usr/local/bin/backup-readonly.sh
    PermitTTY no

The client config requests; the server config disposes. If the server says PasswordAuthentication no, no amount of client configuration brings it back. When something is refused, check both sides — and sshd -T on the server prints its fully-resolved effective config, the server-side analogue of ssh -G.

Key takeaways

References

Related course pages: SSH Tunnel for Windows RDP · Cryptography · Identity and Access Management · Host Security · VPNs and IPSec · Working with Files · Shell and Other Basics

🛠️ Maintenance note: SSH syntax is very stable, but per-key options evolve — restrict (7.2), expiry-time (8.2), and the deprecation of ssh-rsa (SHA-1) signatures are all relatively recent. Re-verify against the OpenSSH version on the course VM, and confirm the example ~/.ssh/config patterns still reflect the current lab addressing each term.