OS Hardening

• OS Hardening
  • Shrinking what an attacker can reach
  • Attack Surface Reduction
    • Auditing installed packages
    • Auditing running services
  • SSH Hardening
    • Key changes in /etc/ssh/sshd_config
    • Verifying your changes
  • User and Group Security
    • Service accounts
    • Auditing sudo access
  • File Permission Audit
    • SUID and SGID binaries
    • World-writable files and directories
    • Sensitive file permissions
    • Automated permission audit script
  • fail2ban
    • Installation
    • Configuration
    • Verifying a ban
  • Key takeaways
  • References

Shrinking what an attacker can reach

OS hardening is the process of reducing a system’s attack surface by removing what isn’t needed, locking down what remains, and making the result auditable. A freshly installed Ubuntu server is configured for convenience, not security — the defaults assume a trusted environment that doesn’t exist once the machine is network-accessible.

The goal is not to make the system impossible to use. It is to ensure that every open service, every permitted login method, and every writable path is there for an explicit reason.

ℹ️ Harden against a baseline, not a checklist. The value of hardening comes from being able to prove the system still matches a known-good state. Capture the baseline — installed packages, open ports, the SUID set, the sshd config — right after install, and treat any later drift as something to explain rather than a box you tick once.

Attack Surface Reduction

The attack surface is the sum of all entry points an attacker can try to reach. Reducing it means:

• Removing unnecessary packages — every installed package is code that can have vulnerabilities
• Stopping unnecessary services — a stopped service cannot be exploited
• Closing unused ports — a closed port cannot be scanned, probed, or attacked
• Removing unnecessary users and groups — fewer accounts means fewer credential targets

Auditing installed packages

# List explicitly installed packages (not pulled in as dependencies)
apt-mark showmanual | sort

# Find packages that are no longer needed
apt autoremove --dry-run

# Remove a package and its config files
apt purge <package>

Auditing running services

# List all active services
systemctl list-units --type=service --state=running

# List all enabled services (start at boot)
systemctl list-unit-files --type=service --state=enabled

# Disable and stop a service you don't need
systemctl disable --now <service>

# Mask a service to prevent it from being started even manually
systemctl mask <service>

Common services to review on a headless Ubuntu server:

SSH Hardening

SSH is almost always the first thing an attacker probes. A default SSH configuration accepts password authentication, allows root login, and doesn’t limit connection attempts — all of which make brute-force trivial.

Key changes in /etc/ssh/sshd_config

# Disable password authentication entirely — key auth only
PasswordAuthentication no
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no

# Never allow root to log in via SSH
PermitRootLogin no

# Restrict which users can SSH in
AllowUsers student

# Reduce the window for incomplete connections
LoginGraceTime 30

# Limit authentication attempts per connection
MaxAuthTries 3

# Limit concurrent unauthenticated connections
MaxStartups 3:50:10

# Disable unused auth methods
PermitEmptyPasswords no
X11Forwarding no

After editing, test the config before restarting:

sshd -t          # syntax check only — exits 0 if valid
systemctl reload sshd

Never restart sshd with an untested config while connected over SSH — a bad config will lock you out on reconnect.

Verifying your changes

# Confirm root login is denied
ssh root@localhost          # should fail with "Permission denied"

# Confirm password auth is disabled
ssh -o PreferredAuthentications=password student@localhost  # should fail

User and Group Security

Service accounts

Every long-running service should run as its own dedicated system account with:

• No login shell (/usr/sbin/nologin or /bin/false)
• No home directory (or a restricted one)
• Membership only in groups it needs

# Create a service account for DVWA
useradd --system --no-create-home --shell /usr/sbin/nologin dvwa-svc

# Verify
grep dvwa-svc /etc/passwd
# dvwa-svc:x:999:999::/home/dvwa-svc:/usr/sbin/nologin

Auditing sudo access

# Show all sudo rules
cat /etc/sudoers
ls /etc/sudoers.d/

# Show what a specific user can run with sudo
sudo -l -U dvwa-svc

# Show who is in the sudo group
getent group sudo

Avoid NOPASSWD for accounts that interact with the network. If you need it for automation, restrict it to a specific command:

# Good: restrict to a single command
dvwa-svc ALL=(root) NOPASSWD: /usr/bin/systemctl restart dvwa

# Bad: unrestricted sudo without password
dvwa-svc ALL=(root) NOPASSWD: ALL

File Permission Audit

SUID and SGID binaries

SUID binaries run as their owner (often root) regardless of who executes them. A surprising SUID binary is a privilege escalation vector.

# Find all SUID binaries
find / -xdev -perm -4000 -type f -ls 2>/dev/null

# Find all SGID binaries
find / -xdev -perm -2000 -type f -ls 2>/dev/null

# Find both at once
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -ls 2>/dev/null

Compare the output against a known-good list. Unexpected entries (especially anything in /tmp or /home) warrant investigation.

World-writable files and directories

# World-writable files (anyone can modify)
find / -xdev -type f -perm -0002 -ls 2>/dev/null

# World-writable directories (more common and often legitimate)
find / -xdev -type d -perm -0002 -ls 2>/dev/null | grep -v '/tmp\|/var/tmp'

Sensitive file permissions

# /etc/shadow should be readable only by root and shadow group
stat /etc/shadow
# Should show: 640 root:shadow

# /etc/passwd should be world-readable (programs need it) but not writable
stat /etc/passwd
# Should show: 644 root:root

# SSH authorized_keys must be owner-only
stat ~/.ssh/authorized_keys
# Should show: 600 student:student

Automated permission audit script

The script below checks the most common hardening failures. Run it before and after making changes to confirm progress:

#!/usr/bin/env bash
# scripts/harden-audit.sh — basic hardening checks
set -euo pipefail

PASS=0; WARN=0; FAIL=0

check() {
    local label="$1" result="$2"
    if [[ "$result" == "pass" ]]; then
        echo "  PASS  $label"; ((PASS++))
    elif [[ "$result" == "warn" ]]; then
        echo "  WARN  $label"; ((WARN++))
    else
        echo "  FAIL  $label"; ((FAIL++))
    fi
}

echo "=== SSH checks ==="
val=$(sshd -T 2>/dev/null | grep -i "^permitrootlogin" | awk '{print $2}')
[[ "$val" == "no" ]] && r=pass || r=fail
check "PermitRootLogin no" "$r"

val=$(sshd -T 2>/dev/null | grep -i "^passwordauthentication" | awk '{print $2}')
[[ "$val" == "no" ]] && r=pass || r=fail
check "PasswordAuthentication no" "$r"

val=$(sshd -T 2>/dev/null | grep -i "^maxauthtries" | awk '{print $2}')
[[ "$val" -le 3 ]] 2>/dev/null && r=pass || r=warn
check "MaxAuthTries <= 3" "$r"

echo "=== File permission checks ==="
perms=$(stat -c '%a' /etc/shadow 2>/dev/null)
[[ "$perms" == "640" || "$perms" == "000" ]] && r=pass || r=fail
check "/etc/shadow permissions (640)" "$r"

count=$(find /etc -xdev -type f -perm -0002 2>/dev/null | wc -l)
[[ "$count" -eq 0 ]] && r=pass || r=fail
check "No world-writable files in /etc ($count found)" "$r"

echo "=== SUID/SGID checks ==="
count=$(find /tmp /var/tmp -xdev \( -perm -4000 -o -perm -2000 \) -type f 2>/dev/null | wc -l)
[[ "$count" -eq 0 ]] && r=pass || r=fail
check "No SUID/SGID in /tmp or /var/tmp ($count found)" "$r"

echo ""
echo "Results: $PASS passed, $WARN warnings, $FAIL failures"
[[ "$FAIL" -eq 0 ]]

fail2ban

fail2ban watches log files for repeated authentication failures and temporarily bans the source IP using the system firewall. It is the simplest effective defense against SSH brute-force.

Installation

apt install fail2ban

Configuration

fail2ban reads /etc/fail2ban/jail.conf but you should override it in /etc/fail2ban/jail.local — the .conf file is overwritten on package upgrades.

# /etc/fail2ban/jail.local

[DEFAULT]
# Ban for 1 hour
bantime  = 3600
# Look back 10 minutes
findtime = 600
# 3 failures triggers a ban
maxretry = 3
# Use nftables backend (Ubuntu 22.04+)
banaction = nftables-multiport
banaction_allports = nftables-allports

[sshd]
enabled  = true
port     = ssh
logpath  = %(sshd_log)s
backend  = systemd
maxretry = 3

systemctl enable --now fail2ban

# Check status
fail2ban-client status sshd

# Manually unban an IP
fail2ban-client set sshd unbanip 10.10.10.10

# Test your config without applying it
fail2ban-client --test

Verifying a ban

# From Kali, attempt 5 failed SSH logins
for i in {1..5}; do ssh -o ConnectTimeout=3 wronguser@10.10.10.20; done

# On ubuntu-server, confirm the ban
fail2ban-client status sshd
# Should show the Kali IP in the banned list

# Confirm nftables rule was added
nft list table inet fail2ban

Key takeaways

• Hardening is attack-surface reduction: remove unused packages, mask services you don’t need, and close every port that isn’t there for a reason.
• SSH is the first thing probed — disable password auth and root login, cap MaxAuthTries, and always sshd -t before reloading so a bad config can’t lock you out.
• Run each service as a dedicated nologin system account, and scope any NOPASSWD sudo rule to a single command rather than ALL.
• Audit the SUID/SGID set, world-writable paths, and sensitive file permissions against a known-good baseline; unexpected entries are escalation vectors.
• fail2ban turns repeated auth failures into automatic, time-limited firewall bans — the simplest effective defense against brute force.

References

• NIST SP 800-123 — Guide to General Server Security — https://csrc.nist.gov/publications/detail/sp/800/123/final
• CIS Ubuntu Linux Benchmark — https://www.cisecurity.org/benchmark/ubuntu_linux
• OpenSSH Manual — sshd_config(5) — https://man.openbsd.org/sshd_config
• fail2ban documentation — https://github.com/fail2ban/fail2ban/wiki
• Linux Audit — SSH hardening guide — https://linux-audit.com/audit-and-harden-your-ssh-configuration/

Related course pages: IAM & access control · Firewalls · DevSecOps fundamentals

🛠️ Maintenance note: Ubuntu 22.04+ uses the nftables firewall backend, so the jail.local here sets banaction = nftables-multiport; on older iptables hosts use iptables-multiport instead. ChallengeResponseAuthentication was renamed to KbdInteractiveAuthentication in OpenSSH 8.7 — both are set here for compatibility, but drop the old name once every target runs 8.7+. Re-check the CIS Ubuntu Benchmark version against the LTS release in use.