courses

SSH: Secure Shell

More than a remote login

You will spend this entire course inside ssh — connecting to the course VM, the lab machines, and ada. Most people learn just enough to type ssh user@host and stop there, and in doing so miss that SSH is one of the most capable tools on the system: an encrypted transport that can carry interactive shells, file transfers, arbitrary TCP connections (tunnels), X11, and agent credentials, all multiplexed over a single authenticated connection.

SSH matters for security on both sides of the fence. Defensively, key-based authentication is the single biggest upgrade you can make over passwords (no password to phish or brute-force), and ~/.ssh/authorized_keys is a file defenders must watch because adding a key is the quietest persistence an attacker can establish. Offensively, SSH tunnels are the classic pivoting mechanism for lateral movement — a -D SOCKS proxy turns one foothold into access to an entire internal network.

This page covers key generation and use, the ~/.ssh/config file (patterns, placeholders, and Match), forward/reverse/dynamic tunnels, locking down authorized_keys, and the often-confusing question of which setting wins when options are specified in several places at once.

ℹ️ SSH has three layers: a transport layer that negotiates encryption and authenticates the server (via its host key), a user-authentication layer (password, public key, etc.), and a connection layer that multiplexes channels (your shell, tunnels, agent forwarding) over the one encrypted pipe. The host key is why you see “The authenticity of host … can’t be established” on first connect — that is trust-on-first-use, recorded in ~/.ssh/known_hosts.

Keys: generate, distribute, use

Public-key authentication replaces a password with a key pair: a private key that never leaves your machine and a public key you place on every server you want to reach. The server challenges you to prove you hold the private key; nothing reusable crosses the wire. This is the digital-signature / asymmetric-crypto primitive applied to login.

Generating a key

Use Ed25519 — it is fast, small, and modern (prefer it over RSA; if you must use RSA, use 4096-bit):

$ ssh-keygen -t ed25519 -C "dmcgrath@laptop-2026"
Generating public/private ed25519 key pair.
Enter file in which to save the key (~/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):      ← USE a passphrase
...
Your identification has been saved in ~/.ssh/id_ed25519       (private — guard this)
Your public key has been saved in ~/.ssh/id_ed25519.pub       (shareable)

Hardware-backed keys (FIDO/security keys)

A passphrase protects a private key at rest, but the key is still a file that malware running as you can copy. A security key (YubiKey, SoLoKey, or a platform authenticator) removes that: the secret lives in the token’s hardware and cannot be exported, so an attacker with full access to your laptop still cannot take it with them. OpenSSH has supported these natively since 8.2:

$ ssh-keygen -t ed25519-sk -C "dmcgrath@yubikey"
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter file in which to save the key (~/.ssh/id_ed25519_sk):

The -sk types produce a small key handle on disk plus the real secret in the token; the file alone is useless without the hardware. Authentication requires a physical touch by default, which also stops silent use by anything running on your machine. Useful variants:

$ ssh-keygen -t ed25519-sk -O resident       # store on the token; recover with ssh-keygen -K
$ ssh-keygen -t ed25519-sk -O verify-required # also require the token's PIN or biometric
$ ssh-keygen -t ecdsa-sk                      # for tokens too old for Ed25519

On the server side, verify-required is also an authorized_keys option, so the server can insist a key was used with PIN/biometric verification rather than trusting the client to have asked.

ℹ️ Both ends need OpenSSH 8.2 or newer, and the server must accept the sk key types — so check before you rely on one for your only route into a machine. Keep a second key (and a second token, or a passphrase-protected software key kept offline) enrolled, or you can lock yourself out of everything at once.

Distributing the public key

The public key must end up in ~/.ssh/authorized_keys on the server. The painless way:

$ ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

Without ssh-copy-id, append it manually (note the append >> so you don’t clobber existing keys):

$ cat ~/.ssh/id_ed25519.pub | ssh user@server 'cat >> ~/.ssh/authorized_keys'

⚠️ Permissions are enforced by sshd. SSH refuses to use keys if the permissions are too open: ~/.ssh must be 700, and ~/.ssh/authorized_keys and your private key 600. A silently failing key-auth is, nine times out of ten, a permissions problem (chmod 700 ~/.ssh; chmod 600 ~/.ssh/*).

The agent: type the passphrase once

ssh-agent holds your decrypted keys in memory so you aren’t prompted on every connection:

$ eval "$(ssh-agent -s)"          # start an agent (usually already running)
$ ssh-add ~/.ssh/id_ed25519       # add a key (prompts for the passphrase once)
$ ssh-add -l                      # list loaded keys
$ ssh-add --apple-use-keychain ~/.ssh/id_ed25519   # macOS: store passphrase in Keychain

The config option AddKeysToAgent yes (used in the example config below) does this automatically the first time a key is used.

Windows: enable the ssh-agent service

Windows ships OpenSSH (since Windows 10 1809 / Server 2019), but unlike Linux and macOS its ssh-agent is a Windows service that is disabled by default — so ssh-add fails with “Error connecting to agent” until you turn it on. Enable it once, from an administrator PowerShell prompt:

# Run as Administrator. Set the agent to start automatically on boot, then start it now.
Get-Service ssh-agent | Set-Service -StartupType Automatic
Start-Service ssh-agent
Get-Service ssh-agent            # should report Status: Running

After that one-time setup, use ssh-add from a normal (non-elevated) prompt exactly as on Linux — note the $env:USERPROFILE path style instead of ~:

ssh-add $env:USERPROFILE\.ssh\id_ed25519   # prompts for the passphrase once
ssh-add -l                                 # list loaded keys

The Windows agent stores keys in your Windows account’s security context (backed by the registry, in DPAPI-protected form), so they persist across reboots and the service reloads them automatically — you are not prompted again.

ℹ️ This applies to the native Windows OpenSSH in PowerShell/Command Prompt. If you instead use Git Bash, it runs its own MSYS2 ssh-agent (start it with eval "$(ssh-agent -s)", as on Linux), and WSL is a separate Linux environment with its own agent again — three different agents that do not share keys. Pick one workflow per machine to avoid confusion.

⚠️ Microsoft’s docs suggest backing up the private key elsewhere and deleting it from disk once it is loaded into the agent, since the agent can serve it without the file present. Be careful: if the agent’s stored copy is ever lost (profile reset, re-image) and you kept no backup, the key is gone — you would have to generate a new pair and re-distribute the public key everywhere.

⚠️ “Error connecting to agent: No such file or directory” — even though the service is Running. The native Windows client reaches the agent over a named pipe (\\.\pipe\openssh-ssh-agent) and ignores the Unix-style SSH_AUTH_SOCK. But if something has set SSH_AUTH_SOCK, the client tries that socket path instead, and since it does not exist as a Windows file you get this error. Diagnose and fix in PowerShell:

$env:SSH_AUTH_SOCK          # if this prints a path, that's the problem
Remove-Item Env:\SSH_AUTH_SOCK   # clear it for this session
ssh-add -l                  # should now connect (lists keys, or "no identities")

The usual culprit is a terminal that forwards its own agent. WezTerm is the common one: its mux_enable_ssh_agent option (on by default) sets SSH_AUTH_SOCK in every pane to a WezTerm-managed socket — fine on Linux/macOS, but it shadows the Windows named pipe. Disable it in ~/.wezterm.lua with config.mux_enable_ssh_agent = false, then fully restart the terminal (existing panes keep the old environment). WSL↔Windows agent-sharing tools (npiperelay/wsl-ssh-agent) and some dotfiles set SSH_AUTH_SOCK for the same reason — check those if it reappears in a fresh shell.

⚠️ Agent forwarding (ForwardAgent) is convenient and dangerous. It lets a remote host use your local keys to authenticate onward — but root on that host can hijack your agent and impersonate you everywhere. Leave it no by default and enable it per-host only when you trust the box. (Note the example config sets ForwardAgent no.) Prefer ProxyJump for reaching machines through a bastion.

Everyday ssh-keygen operations

Beyond generating keys, ssh-keygen answers most “what is this key?” questions:

$ ssh-keygen -lf ~/.ssh/id_ed25519.pub     # fingerprint (-l) of a key file (-f)
256 SHA256:HDwq/kt0GCL7Q5rQ8ZY... dmcgrath@laptop-2026 (ED25519)

$ ssh-keygen -lvf ~/.ssh/id_ed25519.pub    # add -v for the ASCII-art randomart
$ ssh-keygen -y -f ~/.ssh/id_ed25519       # re-derive the PUBLIC key from a private key
$ ssh-keygen -p -f ~/.ssh/id_ed25519       # change (or add) the passphrase, keeping the key
$ ssh-keygen -c -f ~/.ssh/id_ed25519       # change the comment

-y is the one worth remembering: a lost .pub file is not a lost key pair, because the public key can always be recomputed from the private one. A lost private key is unrecoverable — generate a new pair and redistribute.

Host keys and known_hosts: authenticating the server

Everything above authenticates you to the server. The host key runs the other way — it is how you know the machine answering on port 22 is the one you meant. Without it, every key you offer and every keystroke you type could be going to somebody in the middle.

The server holds a host key pair in /etc/ssh/ssh_host_*_key. On first connect you are shown its fingerprint and asked to accept it; that is trust on first use (TOFU), and the answer is recorded in ~/.ssh/known_hosts. Every later connection checks the presented key against that record.

$ ssh ada
The authenticity of host 'ada (131.252.208.30)' can't be established.
ED25519 key fingerprint is SHA256:R0m1aSbWMUDD0Fw8zvBzMPmm1eXrqI8/D5RBbFWFK5w.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

The honest answer to that prompt is “I don’t know yet.” To actually verify it, get the fingerprint out of band — from the machine’s console, or from whoever built it:

# On the server (or its console), print the fingerprint to compare:
$ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:R0m1aSbWMUDD0Fw8zvBzMPmm1eXrqI8/D5RBbFWFK5w root@ada (ED25519)

Note the third option at that prompt: you can paste the expected SHA256:… fingerprint instead of typing yes, and SSH accepts the connection only if it matches.

When the host key changes

This is the warning every student meets sooner or later, usually after a VM is rebuilt:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
...
Offending ECDSA key in /home/you/.ssh/known_hosts:42

It means exactly what it says: the key does not match the one you recorded. There are two possible explanations, and they look identical from here —

Do not reflexively delete the line. Confirm out of band that the host really was rebuilt; then remove the stale entry:

$ ssh-keygen -F ada          # show the known_hosts entry for a host
$ ssh-keygen -R ada          # remove it (rewrites known_hosts, keeps a .old backup)
$ ssh ada                    # reconnect and verify the NEW fingerprint

⚠️ The advice you will find on the internet — StrictHostKeyChecking no, or piping ssh-keyscan straight into known_hosts — silences the warning by turning off server authentication. That converts SSH’s strongest guarantee into decoration. If you need automation to connect without a prompt, use accept-new, which trusts a first sighting but still refuses a changed key:

Host lab-*
    StrictHostKeyChecking accept-new

The values are yes (never add automatically, refuse changes), accept-new (add new, refuse changes), ask (the default — prompt for new, refuse changes), and no/off (add new and accept changes: the dangerous one).

Pre-seeding and scaling known_hosts

ssh-keyscan collects host keys over the network. It is fine for bootstrapping a list you then verify, and not a substitute for verification:

$ ssh-keyscan -t ed25519 ada >> ~/.ssh/known_hosts     # fetch and append
$ ssh-keyscan -t ed25519 ada | ssh-keygen -lf -        # print fingerprints to compare

Two more things you will run into:

The client config file: ~/.ssh/config

Typing -i, -p, -l, and ProxyJump flags every time is miserable. ~/.ssh/config lets you name hosts and attach settings to them. This is where SSH gets powerful, so the examples below are drawn from a real working config.

Host blocks, patterns, and placeholders

A Host line introduces a block whose options apply to any connection whose target name matches the pattern. Patterns use glob wildcards: * (any run of characters), ? (one character), and ! (negation).

# A single named host with its own key
Host ada
    Hostname linux.cs.pdx.edu
    IdentityFile ~/.ssh/id_ed25519

# '??' matches exactly two characters: systemsec-01 ... systemsec-99
Host systemsec-??
    Hostname %h.cs.pdx.edu          # %h = the host you typed
    Port 22
    IdentitiesOnly yes              # only offer the key below, not every agent key
    IdentityFile ~/.ssh/lab/proxmox/%n   # %n = the original name on the command line
    User root

The % tokens are placeholders expanded at connect time — they let one block serve many hosts:

Token Expands to
%h the host name being connected to (after Hostname substitution)
%n the original name you typed on the command line
%p the port
%r the remote username
%% a literal %

So ssh systemsec-07 connects to systemsec-07.cs.pdx.edu as root using the key ~/.ssh/lab/proxmox/systemsec-07 — all from one block.

⚠️ Not every token works in every directive, which is a quiet source of “why didn’t that expand?”. Hostname accepts only %% and %h — Hostname %n.cs.pdx.edu will not work. ProxyCommand and ProxyJump accept %%, %h, %n, %p, and %r. The full set above is accepted by IdentityFile, ControlPath, CertificateFile, IdentityAgent, Include, LocalForward, RemoteForward, UserKnownHostsFile, and Match exec. LocalCommand accepts all of them. The authoritative list is the TOKENS section of ssh_config(5).

Wildcard groups and ProxyJump

ProxyJump (-J on the command line) routes a connection through a bastion/jump host — the traffic to the final host is tunneled inside the connection to the jump host, so the final host never needs to be directly reachable. Combine it with a wildcard group:

# Every host starting 'pp' is reached by jumping through the 'pandaprox' gateway
Host pp*
    ProxyJump pandaprox
    IdentityFile ~/.ssh/pandaprox

Host ppkali
    Hostname 10.20.100.105
    HostKeyAlias ppkali             # see note below
    User kali

Now ssh ppkali transparently hops you → pandaprox → ppkali. You can chain jumps too: ProxyJump ada,imcgrath goes through two bastions in order (equivalent to ssh -J ada,imcgrath …).

ℹ️ HostKeyAlias is the fix for a real lab headache: when several different VMs reuse the same internal IP (e.g. each student rack hands out 10.20.100.105), their host keys collide in known_hosts and SSH screams about a possible attack. HostKeyAlias ppkali tells SSH to file this host’s key under the name ppkali instead of the shared IP, keeping them separate.

The Match keyword: conditional configuration

Host only matches on the target name. Match matches on richer conditions — and crucially can run a command and branch on its exit status with exec. This config uses it to only jump through a bastion when off-campus:

# If we are NOT already on the cecs network (resolv.conf has no cecs search
# domain), reach 'panda' by proxying through 'ada'. On campus, connect directly.
Match host panda !exec "grep -q 'search.*cecs.pdx.edu' /etc/resolv.conf"
    ProxyJump ada

Match host panda restricts it to the panda target; !exec "..." adds the condition “and the command fails” (the ! negates) — i.e. we are not on campus. When both hold, the ProxyJump applies; otherwise it’s skipped and you connect directly. This is how one config Just Works from the coffee shop and from the lab without edits.

The full criteria list is canonical, final, exec, localnetwork, host, originalhost, tagged, command, user, localuser, and version; any of them can be negated with !, and all matches everything.

ℹ️ Match localnetwork (OpenSSH 9.4+) does the same job without shelling out. It matches on the addresses actually assigned to your interfaces, which is both faster and more robust than grepping /etc/resolv.conf — exec runs a subprocess on every config parse, and the resolv.conf trick breaks the moment a VPN or systemd-resolved rewrites the file. On a new enough client, prefer:

Match host panda !localnetwork 131.252.0.0/16
    ProxyJump ada

Keep the exec form if you need to support clients older than 9.4 — Ubuntu 22.04 ships 8.9, for instance.

The catch-all Host * and why it goes last

A block matching Host * applies to everything. It is the right place for global defaults — but it must go at the bottom of the file, for the precedence reason in the next section:

Host *
    IdentitiesOnly yes        # only use specified/loaded keys, not every key (avoids "too many auth failures")
    AddKeysToAgent yes        # auto-add keys to the agent on first use
    IgnoreUnknown UseKeychain # don't error on the next line where UseKeychain is unsupported (Windows)
    UseKeychain yes           # macOS: pull passphrases from the Keychain
    ForwardAgent no           # safe default (see agent-forwarding warning above)
    ServerAliveInterval 30    # send a keepalive every 30s …
    ServerAliveCountMax 3     # … and drop after 3 missed, so dead connections don't hang
    User dmcgrath             # default username almost everywhere

Splitting the config with Include

Once a config grows past a screen or two — or once some of it is work-specific, machine-specific, or not yours to commit — split it:

# At the TOP of ~/.ssh/config, because first-match-wins (see the precedence
# section): whatever the included files set takes effect before anything below.
Include ~/.ssh/config.d/*.conf

Each fragment is an ordinary config file. This is how you keep a lab config that changes every term separate from the personal settings that don’t, and how tools that manage SSH access (cloud CLIs, gh, corporate device management) add host blocks without editing your file. Relative paths are taken as relative to ~/.ssh.

Reusing one connection: multiplexing

Every new SSH connection pays for a TCP handshake, a key exchange, and an authentication round trip — noticeable when a script runs ssh host in a loop, or when you open a second window to a host across a slow link. Connection multiplexing makes later connections ride the channel layer of the first one, which costs nothing.

Host *
    ControlMaster auto                       # reuse an existing connection, or become the master
    ControlPath ~/.ssh/cm/%C                 # one socket per connection; %C is a hash of host/port/user
    ControlPersist 10m                       # keep the master alive 10 minutes after the last session

Create the directory first (mkdir -p ~/.ssh/cm), since SSH will not make it for you. %C is used rather than %h:%p:%r because the literal form produces long socket paths that can exceed the ~104-character Unix socket limit.

The difference is easy to see:

$ time ssh ada true       # first connection: full handshake + auth
real    0m1.284s
$ time ssh ada true       # second: rides the existing master
real    0m0.071s

Manage the master with -O:

$ ssh -O check ada        # is a master running?
Master running (pid=48213)
$ ssh -O exit ada         # tear it down (drops every session using it)

⚠️ Two caveats. Multiplexed sessions share the master’s fate — kill the master and every session over it dies, and a master established with agent forwarding shares that too. And the socket in ControlPath is a live, already-authenticated channel to the remote host: anyone who can read it gets in without a key, so it belongs under your own 700 directory and never on shared storage. Some servers also disable it (MaxSessions 1).

Copying files over SSH: scp, sftp, and rsync

The same authenticated channel that carries your shell can carry files. Three tools ride on top of SSH, and they share its keys, its ~/.ssh/config host aliases, and its authorized_keys restrictions.

scp — quick file copy

scp copies files between hosts with cp-like syntax, where a remote path is host:path (the host can be any alias from your config):

$ scp report.pdf ada:~/uploads/           # local → remote (uses the 'ada' config block)
$ scp ada:/etc/motd ./                     # remote → local
$ scp -r ./site ada:/var/www/             # -r: copy a directory recursively
$ scp -P 2222 file user@host:~/           # -P (CAPITAL) sets the port — NOT -p like ssh!
$ scp -i ~/.ssh/other_key file host:~/    # -i: pick an identity, same as ssh
$ scp -C bigfile.tar ada:~/               # -C: compress in transit

⚠️ Two scp gotchas. The port flag is -P (capital) — lowercase -p means “preserve timestamps,” the opposite of ssh’s -p. And as of OpenSSH 9.0 (2022), scp uses the SFTP protocol under the hood by default; its old RCP-based protocol had surprising filename-glob and quoting behavior and is now deprecated (-O forces the legacy mode). For anything beyond a one-off copy, prefer sftp or rsync.

sftp — interactive or scripted transfer

sftp speaks a richer file-transfer protocol over the same SSH connection, with an FTP-like interactive prompt (get, put, ls, cd, mkdir) or batch mode:

$ sftp ada
sftp> put localfile.txt          # upload
sftp> get remotefile.txt         # download
sftp> ls -l                      # browse the remote side
sftp> bye

$ sftp -b commands.txt ada       # -b: run a batch script non-interactively

rsync — the right tool for big or repeated copies

For large trees, mirrors, or anything you’ll copy more than once, rsync over SSH transfers only the changed portions and preserves attributes — far more efficient than scp. It’s covered on the Working with Files page; in brief:

$ rsync -avz --progress ./build/ ada:/var/www/html/    # sync over SSH (rsync uses ssh by default)
$ rsync -avz -e "ssh -p 2222" src/ host:dst/           # -e to pass custom ssh options

Because all three run over SSH, they obey the server’s policy: a key locked to a command="…" forced command (below) won’t run scp unless your wrapper allows it, and the sftp subsystem can be disabled or chrooted in sshd_config to offer file transfer without shell access.

Tunnels: forwarding ports over SSH

Because the SSH connection layer multiplexes channels, you can push arbitrary TCP connections through the encrypted pipe. There are three directions, and the trick to keeping them straight is to ask which machine opens the listening port.

The common flags: -N (do not run a remote command — just hold the tunnel), -f (background after auth), and -T (no terminal).

Local forwarding (-L) — pull a remote service to you

-L [bind:]localport:target:targetport opens a listener on your machine; connections to it pop out from the SSH server and go to target:targetport. Use it to reach something only the server can see — a database bound to localhost, an internal web UI, a VPN-less path to an internal host.

# Reach a database that only listens on the server's localhost:5432.
# Now connecting to localhost:5432 on YOUR box hits the remote Postgres.
$ ssh -N -L 5432:localhost:5432 ada

# Reach an internal-only web app (intranet.internal:80) via the bastion 'ada':
$ ssh -N -L 8080:intranet.internal:80 ada     # browse http://localhost:8080

Remote forwarding (-R) — expose your service to the far side

-R [bind:]remoteport:target:targetport opens a listener on the SSH server; connections there are tunneled back and emerge from your machine toward target:targetport. Use it to expose a local service to a network you can only reach outbound — the basis of the Windows-RDP-over-SSH guide.

# Expose your laptop's RDP (localhost:3389) as port 1222 on the server:
$ ssh -N -R 1222:localhost:3389 ada

⚠️ Reverse forwarding is also how attackers establish reverse shells and exfiltration paths that beat egress firewalls — an outbound SSH session that opens an inbound door. Seeing unexpected -R tunnels on a host is a red flag for SOC/IR. By default the server-side listener binds only to its loopback; GatewayPorts must be enabled to expose it more widely.

Dynamic forwarding (-D) — a SOCKS proxy / poor man’s VPN

-D localport opens a SOCKS proxy on your machine; any app pointed at it has its traffic emerge from the SSH server, to any destination. One command turns a single SSH foothold into network-wide access — which is exactly why it’s a favorite pivoting tool.

$ ssh -N -D 1080 ada              # SOCKS5 proxy on localhost:1080
$ curl --socks5-hostname localhost:1080 http://internal-only.host/   # routed through ada

You can make a tunnel permanent in ~/.ssh/config with LocalForward, RemoteForward, and DynamicForward directives attached to a host.

authorized_keys: controlling what a key can do

On the server, ~/.ssh/authorized_keys lists the public keys allowed to log in as that user. The basic format is one key per line. But each line can be prefixed with options that constrain what the key may do — turning a key from “full shell access” into a narrowly scoped capability.

Forced commands: a key that can only do one thing

The command="…" option forces that command to run on login, ignoring whatever the client asked for. This is the standard way to grant automated, least-privilege access — a backup job, a git pull, a status check — without handing over a shell:

# In the server's ~/.ssh/authorized_keys:
command="/usr/local/bin/backup-readonly.sh",restrict ssh-ed25519 AAAA...key... backup@ci

# 'restrict' (OpenSSH 7.2+) denies EVERYTHING — no port forwarding, no agent
# forwarding, no X11, no pty — the safest baseline. Opt back in explicitly if needed,
# e.g.:  restrict,pty   or   restrict,port-forwarding

The command the client tried to run is available to your script in $SSH_ORIGINAL_COMMAND, so a single forced command can dispatch among a few allowed actions:

#!/bin/bash
# backup-readonly.sh — only permit specific rsync/backup invocations
case "$SSH_ORIGINAL_COMMAND" in
  "rsync --server --sender"*) exec $SSH_ORIGINAL_COMMAND ;;
  *) echo "Denied: this key may only run backups." >&2; exit 1 ;;
esac

Other useful per-key restrictions

Option Effect
restrict Deny all features (the recommended baseline); re-enable selectively
from="198.51.100.0/24,*.cs.pdx.edu" Only accept this key from matching source addresses/hosts
no-port-forwarding This key cannot set up -L/-R/-D tunnels
no-agent-forwarding, no-X11-forwarding, no-pty Disable those specific features
permitopen="host:port" Restrict -L/-D forwarding to a single destination
permitlisten="host:port" Restrict -R forwarding to a single listen address (OpenSSH 7.8+)
expiry-time="20261231" Key stops working after a date (OpenSSH 7.7+; append Z for UTC, 9.1+)

⚠️ Because authorized_keys grants standing access, it is a top persistence mechanism. Auditing it — and watching it with file-integrity monitoring (auditd/AIDE) — is basic host hardening. An unexplained key, or one with no from=/restrict, deserves investigation.

Certificates: keys that expire and scale

authorized_keys works until you have many users and many servers, at which point it becomes N × M files to edit — and, crucially, there is no way to make a key stop working everywhere at once. Removing someone’s access means visiting every host. SSH certificates solve both problems, and they are the mechanism behind every large SSH deployment you will meet in industry.

A certificate is a public key plus metadata — who it belongs to, which principals it may log in as, and when it expires — signed by a certificate authority (CA), which is just another SSH key kept somewhere safe. A host that trusts the CA accepts any unexpired certificate it signed, without knowing the individual key.

# 1. Create a CA key (once; guard the private half like a crown jewel)
$ ssh-keygen -t ed25519 -f ~/ca/user_ca -C "CS dept user CA"

# 2. Sign a user's public key: -I is an identifier that shows up in logs,
#    -n the principals (usernames) it may log in as, -V the validity window
$ ssh-keygen -s ~/ca/user_ca -I "jsmith@pdx" -n jsmith,devuser -V +8h \
      ~/keys/jsmith_ed25519.pub
Signed user key ~/keys/jsmith_ed25519-cert.pub: id "jsmith@pdx" serial 0 for jsmith,devuser
valid from 2026-09-22T09:00:00 to 2026-09-22T17:00:00

# 3. Inspect any certificate
$ ssh-keygen -Lf ~/keys/jsmith_ed25519-cert.pub

The server trusts the CA with one line in sshd_config, and then needs no per-user key at all:

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/user_ca.pub

The same idea runs in the other direction for host keys, and it retires the TOFU problem entirely. Sign each host’s key with a host CA, and put one line in known_hosts:

# In ~/.ssh/known_hosts — trust any host presenting a cert signed by this CA
@cert-authority *.cs.pdx.edu ssh-ed25519 AAAA...host-CA-public-key...

Now a freshly rebuilt VM presents a certificate signed by the CA, and it just works — no “REMOTE HOST IDENTIFICATION HAS CHANGED”, no prompt, and no habit of clicking through warnings.

Two authorized_keys options relate to this: cert-authority marks a line as a CA to trust rather than a key to accept, and principals="..." limits which certificate principals that CA may authorise.

ℹ️ The real win is expiry. An 8-hour certificate means offboarding is automatic: stop issuing, and access ends on its own. Compare that with authorized_keys, where a forgotten line grants access forever. Revoking before expiry needs a revocation list (ssh-keygen -k, pointed at by RevokedKeys), which is why short lifetimes are preferred over long ones plus revocation.

Where options come from: the precedence rules

When the same option can be set on the command line, in your user config, in the system config, and (server-side) in sshd_config, which wins? Two different rule-sets are at play — get them straight and the rest of SSH stops surprising you.

Client side — “first value wins”

For the client, options are gathered in this order, and for each option the first value obtained is used and later ones are ignored:

  1. Command-line -o, -i, -p, -l, etc. (highest priority — read first)
  2. ~/.ssh/config (your per-user file)
  3. /etc/ssh/ssh_config (system-wide defaults, lowest priority)

Two consequences trip everyone up at least once:

# One-off override: force a different key and user for a single connection,
# beating whatever ~/.ssh/config says, without editing any file:
$ ssh -i ~/.ssh/other_key -o User=root -o IdentitiesOnly=yes systemsec-07

# Inspect exactly which options SSH resolved for a host (no connection made):
$ ssh -G systemsec-07 | grep -iE 'user|identityfile|proxyjump|hostname'

ssh -G <host> prints the fully-resolved configuration SSH would use — the definitive way to debug “why is it picking that key/user?”

Server side — sshd_config and Match

The server independently decides what it will allow, in /etc/ssh/sshd_config. Here the precedence is different: the first matching Match block wins, and a Match overrides the global settings for connections that match it. This is how an admin says “passwords are off everywhere, but this one bastion account may only port-forward”:

# /etc/ssh/sshd_config (server)
PasswordAuthentication no          # global: keys only
KbdInteractiveAuthentication no    # close the other password-shaped door (PAM)
PermitRootLogin prohibit-password  # root may log in by key, never by password
AllowGroups sshusers               # only members of this group may log in at all
MaxAuthTries 3                     # disconnect after 3 failed attempts
LoginGraceTime 30                  # 30s to authenticate before the connection is dropped
X11Forwarding no                   # off unless a lab actually needs it
AllowTcpForwarding no              # no -L/-R/-D from this server (see the tunnels section)

Match User backup                  # for the 'backup' user only:
    ForceCommand /usr/local/bin/backup-readonly.sh
    PermitTTY no

AllowGroups/AllowUsers are the highest-value line there: they turn SSH from “every account on the box” into an explicit allowlist, so a service account created by some package cannot be logged into even if it somehow acquires a password. Note PasswordAuthentication no alone is not enough — KbdInteractiveAuthentication (called ChallengeResponseAuthentication before OpenSSH 8.7) can still let PAM prompt for a password.

The client config requests; the server config disposes. If the server says PasswordAuthentication no, no amount of client configuration brings it back. When something is refused, check both sides — and sshd -T on the server prints its fully-resolved effective config, the server-side analogue of ssh -G.

When it doesn’t work: debugging

SSH fails quietly by design — a server that explained why it rejected you would be helping an attacker. So the information you need is on the client, behind -v, and on the server, in its logs.

Client side: -v, -vv, -vvv

Each v adds a level. One is usually enough to see which keys were offered and in what order:

$ ssh -v ada
debug1: Reading configuration data /home/you/.ssh/config
debug1: /home/you/.ssh/config line 12: Applying options for ada
debug1: Connecting to linux.cs.pdx.edu [131.252.208.30] port 22.
debug1: Server host key: ssh-ed25519 SHA256:R0m1aSbWMUDD0Fw8...
debug1: Offering public key: /home/you/.ssh/id_ed25519 ED25519
debug1: Server accepts key: /home/you/.ssh/id_ed25519 ED25519
debug1: Authentication succeeded (publickey).

Read it for three things: which config blocks applied (Applying options for … — this catches a Host * that matched sooner than you meant), which keys were offered in which order, and where it stopped. Offering public key with no Server accepts key after it means the server did not find that key in authorized_keys.

The most common failures and what they look like:

Symptom Usual cause
Permission denied (publickey) after offering the right key Key not in the server’s authorized_keys, or ~/.ssh / authorized_keys permissions too open (server-side StrictModes)
Too many authentication failures The agent offered every loaded key and hit MaxAuthTries before the right one; fix with IdentitiesOnly yes + an explicit IdentityFile
Offers the wrong key, ignores IdentityFile Another block matched first — check ssh -G and the Applying options for lines
Hangs after Connecting to … Filtered port or wrong host, not an auth problem
no matching host key type found Legacy server — see below

Server side

The client is only ever told “no”. The reason is in the server’s log:

$ sudo journalctl -u ssh -f              # Debian/Kali/Ubuntu (the unit is 'ssh')
$ sudo journalctl -u sshd -f             # Arch, Fedora, RHEL
$ sudo tail -f /var/log/auth.log         # systems still logging to files

Authentication refused: bad ownership or modes for directory /home/you/.ssh is the permissions problem from earlier, stated plainly — and it appears only in the server log, which is why key auth that “silently fails” is so often diagnosed in the wrong place.

Before restarting sshd after an edit, check the config — a syntax error plus a restart can lock you out of a remote machine:

$ sudo sshd -t                           # test the config; silent = OK
$ sudo sshd -T                           # print the fully-resolved effective config
$ sudo sshd -T -C user=backup,host=ci.example.edu,addr=198.51.100.7 | grep -i forcecommand
forcecommand /usr/local/bin/backup-readonly.sh

That last form is the one worth knowing: -C supplies a hypothetical connection so sshd -T evaluates the Match blocks, letting you confirm a rule fires for the user you meant and not for anyone else.

⚠️ Keep a second session open whenever you edit sshd_config on a remote host. Restart the service in one window and test with the other — if the new config locks you out, the already-authenticated session is your way back in. Recovering otherwise means console access.

Talking to old servers: the ssh-rsa problem

OpenSSH 8.8 disabled RSA signatures using the SHA-1 hash by default, because SHA-1 is no longer collision-resistant. Connecting to elderly gear — an old switch, an unpatched appliance, a legacy jump box — now fails like this:

$ ssh admin@old-switch
Unable to negotiate with 198.51.100.9 port 22: no matching host key type found.
Their offer: ssh-rsa

The fix is to re-enable the algorithm for that host only, never globally:

Host old-switch
    HostkeyAlgorithms +ssh-rsa
    PubkeyAcceptedAlgorithms +ssh-rsa

The leading + appends to the default set rather than replacing it, so everything else stays modern. Note this is about the SHA-1 signature algorithm, not RSA keys as such — an RSA key used with rsa-sha2-256/rsa-sha2-512 (supported since OpenSSH 7.2) is fine and needs no change. Treat the override as a dated workaround and a prompt to get the far end upgraded.

Escape sequences: rescuing a frozen session

When the network drops, an SSH session hangs and Ctrl-C does nothing — it is being sent to the remote side, which is not listening. SSH has an escape character, ~, recognised only immediately after a newline:

Sequence Effect
~. Disconnect — the one to remember for a dead session
~? List all escape sequences
~^Z Suspend SSH to the local shell (fg to return)
~# List forwarded connections
~C Open a command line to add/cancel forwards on a live session
~B / ~R Send a BREAK / request rekeying

So: press Enter, then ~, then . — the session closes immediately. ~C is the other useful one, letting you add a tunnel you forgot without reconnecting:

ssh> -L 8080:intranet.internal:80
Forwarding port.

ℹ️ On a nested session (you SSH’d from a machine you SSH’d to), each hop consumes one ~, so use ~~. to close the second hop and ~. to close the outermost. ServerAliveInterval in the example config makes most of this unnecessary by dropping dead connections on its own.

Key takeaways

References

Related course pages: SSH Tunnel for Windows RDP · Cryptography · Identity and Access Management · Host Security · VPNs and IPSec · Working with Files · Shell and Other Basics

🛠️ Maintenance note: SSH syntax is very stable, but features arrive with specific releases and this page names them: restrict (7.2), expiry-time (7.7), permitlisten (7.8), FIDO -sk key types (8.2), KbdInteractiveAuthentication replacing ChallengeResponseAuthentication (8.7), RSA/SHA-1 disabled by default (8.8), scp over SFTP (9.0), and Match localnetwork (9.4). Check these against the OpenSSH on the course VM and on student laptops — a client older than a named version is the usual reason an example here does not work (Ubuntu 22.04 still ships 8.9, so Match localnetwork is unavailable there). Confirm the example ~/.ssh/config patterns still reflect the current lab addressing each term, and re-check the ssh-rsa workaround section once the legacy gear it exists for is retired.