Incident Response

Incident Response

What you do when detection fires

Incident response (IR) is the organized approach to handling security incidents — from detection through containment, eradication, recovery, and post-incident review. A well-defined IR process reduces dwell time (how long an attacker is in the environment), limits damage, and produces evidence for forensic analysis and legal proceedings.

⚠️ The single most common IR mistake is destroying evidence to stop the bleeding — powering off a host wipes RAM, where running processes, network state, and encryption keys live. Contain by isolating (firewall/VLAN), capture volatile data in order of volatility, then eradicate.

Incident Response Lifecycle

The NIST Computer Security Incident Handling Guide (SP 800-61) defines four phases:

Preparation → Detection & Analysis → Containment, Eradication & Recovery → Post-Incident Activity
     ↑                                                                              │
     └──────────────────────── Lessons Learned ────────────────────────────────────┘

Phase 1: Preparation

Preparation happens before any incident occurs. Without it, every incident is chaotic.

Key preparation activities:

IR plan — documented, approved, tested process for responding to incident types
Runbooks — step-by-step playbooks for common incident types (ransomware, data breach, account compromise)
Communication plan — who to notify, when, and through what channels (avoid compromised channels)
Asset inventory — know what systems exist and what data they hold
Logging — ensure all systems generate and centralize security-relevant logs
IR toolkit — pre-staged tools for forensic collection and analysis
Legal and compliance contacts — know when a breach requires regulatory notification
Tabletop exercises — regular drills to test the plan without a real incident

Phase 2: Detection and Analysis

Detection comes from SIEM alerts, IDS/IPS, EDR, user reports, or threat intelligence. Not every alert is an incident — analysis determines severity.

# Triage checklist: is this a real incident?
triage_questions = [
    "Is this a confirmed malicious event or could it be a false positive?",
    "What systems/data are affected?",
    "Is the attacker still active, or is this a historical event?",
    "What is the business impact? (production, PII, financial data?)",
    "Does this meet the threshold for a declared incident?",
    "What is the initial severity classification?",
]

# Incident severity classification
SEVERITY = {
    "P1 - Critical": "Active breach, data exfiltration, ransomware, critical system down",
    "P2 - High":     "Confirmed compromise without confirmed exfiltration, lateral movement",
    "P3 - Medium":   "Suspicious activity, policy violation, potential compromise",
    "P4 - Low":      "Informational, unconfirmed, low impact",
}

Phase 3: Containment, Eradication, and Recovery

Short-term containment stops the bleeding while preserving evidence:

# Network containment: isolate a compromised host (Linux)
# Do NOT simply power off — this destroys volatile memory evidence

# Option 1: block at the host firewall (preserves the running system for forensics)
# Current Kali/Ubuntu default to nftables — create a dedicated default-drop table:
nft add table inet ir
nft add chain inet ir input  '{ type filter hook input  priority 0 ; policy drop ; }'
nft add chain inet ir output '{ type filter hook output priority 0 ; policy drop ; }'
nft add rule  inet ir input  iif lo accept        # keep loopback up for local tools
nft add rule  inet ir output oif lo accept
nft add rule  inet ir input  ip saddr 10.0.1.50 accept   # allow only the forensics box
nft add rule  inet ir output ip daddr 10.0.1.50 accept
# Legacy iptables equivalent (older hosts, or iptables-nft):
#   iptables -I INPUT -j DROP;  iptables -I OUTPUT -j DROP
#   iptables -I INPUT  -s 10.0.1.50 -j ACCEPT
#   iptables -I OUTPUT -d 10.0.1.50 -j ACCEPT

# Option 2: isolate via network switch (if you have switch management access)
# ssh switch-mgmt "interface GigabitEthernet0/1; shutdown"

# Option 3: Kubernetes — cordon and drain the node
kubectl cordon node01           # prevent new pod scheduling
kubectl drain node01 --force    # evict pods
# Or delete a specific compromised pod
kubectl delete pod suspicious-pod --grace-period=0

Evidence preservation before eradication:

# Capture volatile memory FIRST (before containment if safe to do so)
# LiME (Linux Memory Extractor) — kernel module for memory capture
# Load module, dump to network socket
insmod lime-$(uname -r).ko "path=tcp:4444 format=lime"
# On the forensics workstation:
nc -l -p 4444 > memory.lime

# Capture running processes, network connections, and open files
ps auxf > /evidence/ps-auxf.txt
ss -tlnpu > /evidence/connections.txt
lsof -n > /evidence/lsof.txt
last -a > /evidence/last.txt
who > /evidence/who.txt
w > /evidence/w.txt
cat /proc/*/cmdline 2>/dev/null | tr '\0' ' ' > /evidence/cmdlines.txt

# Capture disk image (after memory, preserves filesystem timeline)
dd if=/dev/sda bs=4M status=progress | gzip > /evidence/sda.img.gz
# Or with dcfldd (adds hashing)
dcfldd if=/dev/sda of=/evidence/sda.img hash=sha256 hashlog=/evidence/sda.sha256

Phase 4: Post-Incident Activity

The post-incident review (also called after-action review or lessons learned) is mandatory:

Post-Incident Review Agenda:
Timeline reconstruction — when did each event occur?
Root cause — how did the attacker get in? What vulnerability was exploited?
Detection gap — why did it take this long to detect?
What worked — what parts of the IR process functioned correctly?
What failed — what slowed the response or was missing?
Action items — specific, assigned, time-bound improvements
Metrics — MTTD (mean time to detect), MTTR (mean time to respond/recover)

Digital Forensics

Digital forensics is the collection, preservation, analysis, and reporting of digital evidence in a manner that maintains its integrity and admissibility.

Forensic principles

Order of volatility — collect evidence from most volatile (registers, RAM) to least volatile (disk, backups) because volatile data is lost on power-off
Chain of custody — document every person who handles evidence, when, and why
Write blockers — use hardware or software write blockers when imaging drives to prevent modifying evidence
Hashing — compute SHA-256 of every evidence item immediately after collection; verify at each subsequent step

Order of volatility

Priority	Evidence type	Survives power-off?
1	CPU registers, cache	No
2	RAM (running processes, network connections, encryption keys)	No
3	Swap / pagefile	Usually no
4	Network state (ARP cache, routing table)	No
5	Running processes	No
6	Disk (filesystem, deleted files, log files)	Yes
7	Remote logging (SIEM)	Yes (off-host)
8	Backups	Yes

Linux forensic collection

# Timeline: when were files accessed/modified/created?
# Use The Sleuth Kit (TSK) and Autopsy for full forensic analysis

# Quick timeline from a mounted (read-only) filesystem
mount -o ro,noexec /dev/sda1 /mnt/evidence

# Generate filesystem timeline (MAC times: Modified, Accessed, Changed)
find /mnt/evidence -printf '%T@ %Tc %p\n' | sort -n > /evidence/fs-timeline.txt

# Find recently modified files (last 24 hours)
find /mnt/evidence -newer /mnt/evidence/var/log/syslog -not -newer /mnt/evidence \
    -type f -ls 2>/dev/null

# Recover deleted files with Autopsy or foremost
foremost -i /evidence/sda.img -o /evidence/recovered-files/ -t all

# String searching in a disk image
strings /evidence/sda.img | grep -E '(password|passwd|secret|api_key)' \
    > /evidence/strings-sensitive.txt

Log analysis for forensics

# Authentication log analysis (Debian/Ubuntu)
grep "Accepted\|Failed\|Invalid" /var/log/auth.log | \
    awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c | sort -rn

# Find what a specific user did
grep "alice" /var/log/auth.log
grep "sudo.*alice" /var/log/auth.log

# Bash history (often tampered with — check anyway)
cat /home/alice/.bash_history
cat /root/.bash_history

# Journal entries for a specific timeframe
journalctl --since "2026-04-01 00:00:00" --until "2026-04-02 00:00:00" \
    -u sshd -u sudo > /evidence/journal-april1.txt

# Python: parse auth.log for brute force pattern
import re
from collections import defaultdict
from datetime import datetime

FAILED_RE = re.compile(
    r'(\w{3}\s+\d+\s[\d:]+).*Failed password for (?:invalid user )?(\S+) from ([\d.]+)'
)

def analyze_auth_log(logfile: str) -> dict:
    failures = defaultdict(list)
    with open(logfile) as f:
        for line in f:
            m = FAILED_RE.search(line)
            if m:
                timestamp, user, ip = m.groups()
                failures[ip].append({'user': user, 'time': timestamp})
    return dict(failures)

results = analyze_auth_log('/var/log/auth.log')
for ip, attempts in sorted(results.items(), key=lambda x: len(x[1]), reverse=True)[:10]:
    users = {a['user'] for a in attempts}
    print(f"{ip}: {len(attempts)} failures, users: {', '.join(users)}")

Containment Strategies

Network segmentation for containment

# Emergency VLAN reassignment (conceptual — depends on switch/SDN)
# Move a compromised host to an isolated quarantine VLAN

# Linux: create an isolated network namespace for a suspected process
# (does not stop the process but cuts its network access)
PID=1234
ip netns add quarantine
ip link add veth0 type veth peer name veth1
ip link set veth1 netns quarantine
nsenter -t $PID -n -- ip link show   # inspect the process's current network
# Reassign process to quarantine namespace
nsenter -t $PID -n ip link set dev eth0 netns quarantine

Kubernetes containment

# Immediately block all egress from a compromised namespace
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine-deny-all
  namespace: compromised-ns
spec:
  podSelector: {}        # matches all pods
  policyTypes:
    - Egress
    - Ingress            # deny all ingress too
EOF

# Delete a compromised pod (do not use --grace-period=0 if forensics are needed)
kubectl delete pod compromised-pod -n production

# Disable a compromised service account
kubectl patch serviceaccount compromised-sa -n production \
    -p '{"automountServiceAccountToken": false}'

Root Cause Analysis

Root cause analysis (RCA) identifies the underlying cause of an incident, not just the immediate trigger. Without RCA, the same incident recurs.

The 5 Whys

The 5 Whys technique iteratively asks “why” until the root cause is found:

Incident: Attacker exfiltrated customer data

Why 1: Why could they exfiltrate it?
  → Database was accessible from the internet

Why 2: Why was the database internet-accessible?
  → Terraform config had 0.0.0.0/0 in the security group ingress rule

Why 3: Why was that configuration deployed?
  → The IaC PR was approved without a security review

Why 4: Why was there no security review?
  → The team's PR checklist did not include an IaC security step

Why 5: Why was there no IaC security step in the checklist?
  → No one owned the process of updating the checklist after adding Terraform

Root cause: No process for maintaining the PR security checklist as infrastructure
            practices evolve

Corrective actions:
  1. Add Checkov/tfsec to CI pipeline (automated, immediate)
  2. Add IaC security item to PR checklist (process, this week)
  3. Assign ownership of checklist maintenance (people, this sprint)
  4. Restrict database security group to internal subnets only (immediate)

Response strategy framework

# Incident response tracking
from dataclasses import dataclass, field
from datetime import datetime
from typing import List, Optional
from enum import Enum

class IncidentStatus(Enum):
    DETECTED = "Detected"
    TRIAGED = "Triaged"
    CONTAINED = "Contained"
    ERADICATED = "Eradicated"
    RECOVERING = "Recovering"
    CLOSED = "Closed"

@dataclass
class TimelineEntry:
    timestamp: datetime
    actor: str
    action: str

@dataclass
class Incident:
    id: str
    title: str
    severity: str
    detected_at: datetime
    status: IncidentStatus = IncidentStatus.DETECTED
    affected_systems: List[str] = field(default_factory=list)
    timeline: List[TimelineEntry] = field(default_factory=list)
    root_cause: Optional[str] = None
    contained_at: Optional[datetime] = None
    resolved_at: Optional[datetime] = None

    def add_event(self, actor: str, action: str):
        self.timeline.append(TimelineEntry(datetime.now(), actor, action))

    def mttd(self) -> Optional[float]:
        """Mean time to detect: minutes from incident start to detection."""
        # In practice, 'incident_start' comes from forensic analysis of logs
        return None

    def mttr(self) -> Optional[float]:
        """Mean time to respond/recover: detection to resolution in minutes."""
        if self.resolved_at:
            delta = self.resolved_at - self.detected_at
            return delta.total_seconds() / 60
        return None

incident = Incident(
    id="INC-2026-042",
    title="SSH brute force leading to unauthorized access",
    severity="P2 - High",
    detected_at=datetime(2026, 4, 15, 14, 32),
)
incident.add_event("SIEM", "Alert triggered: 500 failed SSH logins from 203.0.113.42")
incident.add_event("SOC-analyst", "Confirmed successful login at 14:28 — incident declared")
incident.status = IncidentStatus.TRIAGED

Key takeaways

IR follows the NIST SP 800-61 lifecycle: Preparation → Detection & Analysis → Containment, Eradication & Recovery → Post-Incident Activity, looping back through lessons learned.
Preparation is the phase that decides the outcome — plans, runbooks, comms channels, asset inventory, centralized logging, and tabletop drills before anything happens.
Contain without destroying evidence: isolate via firewall/VLAN/NetworkPolicy, then collect in order of volatility (RAM → swap → network state → disk → backups) with hashing and chain of custody.
Forensics is disciplined evidence handling: write blockers, immediate SHA-256, read-only mounts, filesystem timelines, and log analysis (auth.log, journald, bash history).
Triage assigns severity (P1–P4) so response effort matches impact, and every incident ends with a root-cause analysis (e.g. the 5 Whys) plus assigned, time-bound corrective actions — otherwise it recurs.
Track MTTD/MTTR: shrinking time-to-detect and time-to-respond is how you measure whether the program is improving.

References

NIST SP 800-61 Rev. 2 — Computer Security Incident Handling Guide. https://csrc.nist.gov/publications/detail/sp/800/61/rev-2/final
SANS — Incident Handler’s Handbook. https://www.sans.org/white-papers/33901/
CISA — Federal incident & vulnerability response playbooks. https://www.cisa.gov/sites/default/files/publications/Federal_Government_Cybersecurity_Incident_and_Vulnerability_Response_Playbooks_508C.pdf
The Sleuth Kit and Autopsy — forensic tooling. https://www.sleuthkit.org/
MITRE ATT&CK — technique classification. https://attack.mitre.org/
LiME — Linux Memory Extractor. https://github.com/504ensicsLabs/LiME

Related course pages: Capturing Packets with tcpdump · Wireshark · Technical Writing

🛠️ Maintenance note: NIST SP 800-61 Rev. 3 was finalized in 2025 and reframes IR around the CSF 2.0 functions — verify which revision your course tracks, as the four-phase model above is the long-standing Rev. 2 framing. Re-check the LiME and TSK release state each term.