Advanced Dynamic Analysis
- Debugger-Based Analysis
- Defeating Anti-Analysis in the Debugger
- Runtime Unpacking
- Dynamic Instrumentation
- Kernel-Level Monitoring
- Detecting Process Injection at Runtime
- Memory Forensics During Execution
- Correlating Dynamic and Static Findings
- Behavioral Correlation with MITRE ATT&CK
- Workflow Summary
- Tools Quick Reference
- Further Reading
This page picks up where basic dynamic analysis leaves off. The foundation — sandboxes, strace, network capture — reveals what a sample does. Advanced dynamic analysis answers how and why: it lets you control execution, intercept at arbitrary points, defeat anti-analysis tricks, and extract artifacts that never exist on disk. It requires a working debugger and an isolated analysis VM with a clean snapshot.
Debugger-Based Analysis
A debugger gives you the ability to pause execution at any point, inspect registers and memory, modify values, and step through individual instructions. For malware analysis, this is the primary tool for understanding obfuscated logic, decryption routines, and conditional C2 check-ins.
Choosing a Debugger
| Target | Debugger | Notes |
|---|---|---|
| Linux ELF (user-mode) | GDB + pwndbg | pwndbg adds a context pane, heap inspection, memory search, and cyclic patterns |
| Linux ELF (kernel module) | KGDB / GDB over serial | Requires a second VM as the debug host |
| Windows PE (user-mode) | x64dbg | Open source; best plugin ecosystem for malware work |
| Windows PE (kernel) | WinDbg | Required for rootkit/driver analysis; TTD support |
| Cross-platform scripting | GDB + Python / x64dbg + xdbg64 script | Both support full scripting APIs |
GDB + pwndbg for Linux Samples
The malware-env playbook already installs pwndbg and deploys a ~/.gdbinit that loads it, so on a course VM gdb comes up with the pwndbg prompt and you can skip this.
To install it by hand elsewhere, use upstream’s install script — this is the method pwndbg currently recommends:
curl --proto '=https' --tlsv1.2 -LsSf 'https://install.pwndbg.re' | sh -s -- -t pwndbg-gdb
Add -u to install for your user only, without root. The older git clone plus ./setup.sh route still works but is documented upstream as the legacy method.
Start or attach:
gdb --args ./sample arg1 arg2 # run directly with arguments
gdb -p $(pgrep sample) # attach to a running process
Essential GDB commands:
| Command | Effect |
|---|---|
b *0xdeadbeef |
Breakpoint at an absolute address |
b mmap |
Breakpoint on the mmap PLT entry |
run / r |
Start execution |
continue / c |
Resume after a break |
ni / si |
Step over / step into (one instruction) |
x/32xb $rsp |
Examine 32 bytes at RSP as hex |
x/10i $rip |
Disassemble 10 instructions from RIP |
info proc mappings |
Print all mapped memory regions |
dump binary memory out.bin 0x400000 0x401000 |
Dump a memory region to a file |
set $rip = 0x40107a |
Force-redirect execution |
set *(char*)0x601020 = 0 |
Patch a byte in memory |
pwndbg enhancements:
| Command | Effect |
|---|---|
context |
Register, stack, disassembly, and backtrace pane |
vmmap |
Color-coded virtual memory layout |
telescope $rsp 20 |
Dereference pointer chain for 20 words from RSP |
search -s "http" |
Scan mapped memory for a string |
search -x 4d5a |
Scan for MZ headers (injected PEs) |
cyclic 200 / cyclic -l $rsp |
Generate / find offset in a De Bruijn pattern |
heap / bins |
Heap chunk list and bin contents (glibc malloc) |
Filtering command output: pipe
GDB can send any command’s output into a shell pipeline, which matters as soon
as you are looking at a real process — info functions on a dynamically linked
sample runs to hundreds of lines, and vmmap on a browser to hundreds more.
pipe, or | for short:
(gdb) pipe vmmap | grep rwx # writable AND executable only
(gdb) pipe info functions | grep -c "^0x" # how many symbols are there
(gdb) | search -x 4d5a | wc -l # count MZ headers in memory
(gdb) pipe info proc mappings | grep -E "rwxp" # the same, without pwndbg
It captures pwndbg’s output as well as gdb’s own, so context, vmmap,
telescope, and search all pipe. Spaces around the | are optional.
Two forms worth knowing:
| Form | Use |
|---|---|
pipe \| SHELL_COMMAND |
Repeat the last command and filter its output — interactive sessions only, since a script has no command history |
pipe -d DELIM COMMAND DELIM SHELL_COMMAND |
When the gdb command itself contains a \|, e.g. pipe -d ! print 5 \| 2 ! grep . |
$_shell_exitcode holds the pipeline’s exit status, so a script can branch on
whether the filter matched anything:
(gdb) pipe vmmap | grep -q rwx
(gdb) print $_shell_exitcode
$1 = 0
⚠️ Colour escapes reach
grep. pwndbg’s start before the leading whitespace on each line, so an anchored pattern silently matches nothing —pipe vmmap | grep -c "^ *0x"returns0on a process with two dozen mappings. It is not an error, just an empty result. Fix it withset disable-colors on(pwndbg),set style enabled off(gdb’s own styling), or by stripping the escapes in the pipeline itself:pipe vmmap | sed 's/\x1b\[[0-9;]*m//g' | grep "^ *0x".
Where a filter is all you need, several info commands take a regular
expression directly and produce no colour to fight — info functions ^decrypt,
info variables, info types, and info registers rsp rip for a named
subset.
pipe,|, and$_shell_exitcodearrived in GDB 9.1 (February 2020). On anything older — Ubuntu 18.04 shipped GDB 8.1 — the fallback isset logging file /tmp/gdb.log,set logging enabled on, then grep the file afterwards. Note that those are subcommands as of GDB 11; the older spelling wasset logging on.
Breakpoint strategy for unpacking
Packed samples call mmap or mprotect to create executable memory, then transfer control there. Set breakpoints on both and inspect what was written before execution reaches the new region:
(gdb) catch syscall mmap mprotect
(gdb) run
# when stopped at mprotect with PROT_EXEC:
(gdb) finish # let the call complete
(gdb) vmmap # find the new executable region
(gdb) x/10i <new_region_addr>
Conditionals and mid-session patches:
# Break only when a register holds a specific value
break *0x401234 if $rdi == 0x0
# Patch a JZ to NOP+NOP so a branch is always taken
set *(short*)0x401a30 = 0x9090
Multi-Pane Layouts: GDB TUI
By default pwndbg reprints its whole context block after every si, and a long unpacking loop scrolls the thing you were reading off the top of the terminal. Both fixes below pin each section to a fixed region of the screen so disassembly, registers, and the stack stay put while you step. The TUI costs nothing to set up and works over a plain SSH session; splitmind needs tmux but gives you real panes.
pwndbg registers every context section as a native GDB TUI window called pwndbg_<section>, and ships two ready-made layouts:
(gdb) layout pwndbg # disassembly-led
(gdb) layout pwndbg_code # source/decompiler pane beside the disassembly
The windows available to a layout (pwndbg 2026.09; the list is generated from pwndbg’s registered context sections, so it grows as upstream adds them):
| Window | Content |
|---|---|
pwndbg_disasm |
Disassembly around RIP |
pwndbg_code |
Source lines, when the binary has them |
pwndbg_regs |
Register values, with dereferences |
pwndbg_stack |
Stack words, telescoped |
pwndbg_backtrace |
Call stack |
pwndbg_args |
Arguments of the current function |
pwndbg_expressions |
Watched expressions (contextwatch) |
pwndbg_threads |
Thread list |
pwndbg_heap_tracker |
glibc heap activity |
pwndbg_last_signal |
Most recent signal — useful on a sample that traps deliberately |
pwndbg_legend |
The colour key for the panes |
pwndbg_control |
pwndbg’s own control pane |
cmd, status |
GDB’s command window and status line |
Moving around once you are in it:
| Action | Effect |
|---|---|
focus cmd |
Put the arrow keys back on command history instead of scrolling a pane |
focus pwndbg_stack |
Point the arrow keys at a named pane to scroll it |
tui disable |
Drop back to the plain CLI, for a command whose output is long |
ctrl-x then a |
Toggle TUI/CLI without typing a command |
| hold Shift while dragging | Bypass TUI mouse handling so your terminal’s select-and-copy works |
Worked example — a custom layout. Layouts are ordinary GDB tui new-layout definitions, so the quickest route to your own is to start from the one pwndbg ships. This is pwndbg_code with the heap tracker dropped and the stack given more room, which is roughly what you want when stepping an unpacker that is building code on the heap:
tui new-layout malware \
{-horizontal \
{ \
{ pwndbg_code 1 pwndbg_disasm 1 } 2 \
{ \
{ -horizontal pwndbg_legend 8 pwndbg_control 2 } 0 pwndbg_regs 1 pwndbg_stack 2 \
} 3 \
} 7 cmd 3 \
} 3 { pwndbg_backtrace 2 pwndbg_expressions 2 } 1 status 1
(gdb) layout malware
The numbers are relative weights, not rows: pwndbg_stack 2 beside pwndbg_regs 1 gives the stack twice the height. Put the tui new-layout line in ~/.gdbinit (see the caveat about bare set lines below — tui new-layout is a real GDB command and is safe unguarded) and the layout is there every session.
TUI windows need a GDB built with the Python window API, which is GDB 10 and later. Check before you debug why nothing appears:
gdb -q -nx -ex 'python print(hasattr(gdb, "register_window_type"))' -ex quit
If that prints False, pwndbg silently registers no windows and layout pwndbg will fail — use splitmind instead.
Multi-Pane Layouts: splitmind + tmux
splitmind takes the same context sections and sends each one to its own tmux pane, by handing pwndbg a separate tty per section. Panes are resizable, independently scrollable, and selectable with the mouse, and you can give the debuggee its own pane so its output never interleaves with pwndbg’s. It supports only pwndbg as the information provider and only tmux as the splitter.
git clone https://github.com/jerdna-regeiz/splitmind ~/clones/splitmind
The malware-env playbook expects the clone at exactly that path — it deploys its gdbinit.py into the clone and prints a SKIP splitmind not cloned line if you have not cloned it first. On a course VM, clone it, then re-run the deploy script.
Worked example. This is the layout the course VM deploys: disassembly and code sharing the top left, registers over the stack on the right, a pane for the debuggee’s own input and output, and a Python REPL under the backtrace.
python
import splitmind
(splitmind.Mind()
.tell_splitter(show_titles=True)
.tell_splitter(set_title="Main")
.right(display="backtrace", size="25%")
.above(of="main", display="disasm", size="80%", banner="top")
.show("code", on="disasm", banner="none")
.right(cmd='tty; tail -f /dev/null', size="65%", clearing=False)
.tell_splitter(set_title='Input / Output')
.above(display="stack", size="75%")
.above(display="legend", size="25")
.show("regs", on="legend")
.below(of="backtrace", cmd="ipython3", size="30%")
).build(nobanner=True)
end
Each split is created relative to the previous one unless you name a different parent with of=. show() adds a second section to a pane that already exists instead of splitting again — that is how code lands on top of disasm and regs on top of legend. A split with cmd= and no display= is just a pane running a command, which is how the I/O and REPL panes get there.
Inside tmux, start pwndbg and confirm the panes came up:
$ tmux list-panes -F '#{pane_index} #{pane_title} #{pane_width}x#{pane_height}'
0 disasm, code 57x46
1 legend, regs 106x24
2 stack 106x8
3 Input / Output 106x12
4 Main 164x12
5 backtrace 55x59
Six panes with those titles means splitmind bound every section. If you get one pane and a normal pwndbg prompt, the layout never ran — check the gotchas below.
To route the sample’s own stdin/stdout into the Input / Output pane, read the tty that pane prints at startup and tell GDB to use it:
(gdb) tty /dev/pts/7
GDB warns about the redirection; ignore it. This matters for malware work more than it sounds: a sample that prints a prompt or reads from stdin becomes unreadable when its output is interleaved with a context redraw after every step.
Gotchas, all of which produce confusing rather than obvious failures
-
Outside tmux, splitmind raises. It drives
tmux split-window, so it needs a tmux session to split. Guard it, or every non-tmuxgdbinvocation starts with a traceback:if not os.environ.get("TMUX"): print("gdbinit: not inside tmux — skipping the splitmind layout.") else: gdb.execute("source " + os.path.expanduser("~/clones/splitmind/gdbinit.py")) # ... build the Mind() here -
A bare pwndbg
setline in~/.gdbinitbreaks plaingdb.set context-sections ...is a pwndbg parameter, so plain GDB parses it as an expression, fails withNo symbol table is loaded, and — the part that bites — abandons the rest of the init file at that point. Anything below it, splitmind included, silently never runs. Set pwndbg’s parameters from a Python block guarded on"pwndbg" in sys.modules, onegdb.execute()per parameter in its owntry, so a parameter upstream renames costs you one warning instead of the whole layout. -
context-source-code-linesno longer exists. splitmind’s own examplegdbinitstill sets it; current pwndbg answersUndefined show command.context-code-linescovers it. Combined with the previous gotcha, that one stale line in a copied config takes the layout down with it. -
cmd="ipython"fails on Debian and Ubuntu, which package only/usr/bin/ipython3— and the course VMs are Debian-family. The pane stays open showingcommand not found, which is easy to miss. Useipython3, or fall back withshutil.which. -
Do not re-source pwndbg’s
gdbinit.pyfrom~/.gdbinitif you launch with thepwndbgwrapper. It has already loaded, and pwndbg’s own double-load check will tell you to remove the line.
Which to use: the TUI if you are on an unfamiliar box, over SSH, or inside a VM console where tmux is not set up — it is built in and needs no clone. splitmind once a layout is worth keeping: real panes you can resize and copy from, and a dedicated pane for the sample’s own I/O.
Both depend on pwndbg’s context-section names and parameter names, and both drift between pwndbg releases — the renamed context-source-code-lines above is one example. If a layout that used to work comes up empty after an upgrade, check the section names with show context-sections and pwndbg [filter] before rewriting the config.
x64dbg for Windows Samples
x64dbg is the standard open-source Windows malware debugger. Core workflow:
- Load the sample: File → Open. Let the loader place it.
- Set the entry-point breakpoint: it pauses at the PE entry point automatically.
- Pause on API calls: Right-click in the Symbols pane → “Set breakpoint on all calls to [function]”
- Search for strings: Right-click in the disassembly → Search for → All Referenced Strings
Keyboard shortcuts:
| Shortcut | Action |
|---|---|
| F2 | Toggle breakpoint on selected instruction |
| F7 / F8 | Step into / step over |
| F9 | Run |
| Ctrl+G | Go to address or symbol |
| Ctrl+F | Find pattern in current region |
| Alt+M | Memory map window |
| Ctrl+B | Set hardware breakpoint on memory access |
Essential plugins:
| Plugin | Purpose |
|---|---|
| ScyllaHide | Transparent anti-anti-debugging; patches NtQueryInformationProcess, timing, and PEB checks |
| Scylla | PE dump + import reconstruction for packed samples |
| xAnalyzer | Annotates call arguments inline in the disassembly |
| ret-sync | Synchronizes a debugger (x64dbg, WinDbg, GDB) with Ghidra, IDA, or Binary Ninja in real time |
WinDbg for Kernel-Mode Malware
For rootkits and malicious drivers, WinDbg over a kernel debug connection is required. Set up kdnet between the analysis VM and a debugger host:
On the VM (run once, then reboot):
bcdedit /dbgsettings net hostip:<debugger_host_ip> port:50000
bcdedit /debug on
On the debugger host:
windbg -k net:port=50000,key=<key>
Essential kernel commands:
| Command | Effect |
|---|---|
!process 0 0 |
List all processes (walks ActiveProcessLinks) |
!process <addr> 7 |
Detailed process info including threads |
!drvobj \Driver\<name> |
Inspect a driver object |
!devobj <addr> |
Inspect a device object |
bp <module>!<function> |
Breakpoint on an exported symbol |
dt nt!_EPROCESS <addr> |
Overlay EPROCESS structure at address |
lm |
List loaded modules |
!idt |
Dump the Interrupt Descriptor Table |
eb @$peb+2 0 |
Clear the BeingDebugged byte in PEB |
To detect DKOM-hidden processes, compare !process 0 0 (linked-list walk) against a pool-tag scan in Volatility — processes hidden from the active list will still appear in windows.psscan.
Scripting the Debugger
Both debuggers support Python scripting for automated analysis:
GDB + pwndbg/Python — log all connect() calls
import gdb
class ConnectBreak(gdb.Breakpoint):
def stop(self):
rdi = int(gdb.parse_and_eval("$rdi")) # sockfd
rsi = int(gdb.parse_and_eval("$rsi")) # struct sockaddr*
# read the sockaddr from memory
raw = gdb.selected_inferior().read_memory(rsi, 16)
import socket, struct
port = struct.unpack_from(">H", bytes(raw), 2)[0]
addr = socket.inet_ntoa(bytes(raw)[4:8])
print(f"[connect] {addr}:{port}")
return False # don't stop, just log
ConnectBreak("connect")
Load with source log_connect.py inside GDB.
x64dbg script — conditional NOP patch at runtime
// In the x64dbg scripting console
// Patch a JNZ at 0x401A30 to NOP+NOP (always fall through)
mov [0x401A30], 0x9090
Defeating Anti-Analysis in the Debugger
When a sample exits silently or behaves differently under a debugger, work through this checklist before investing time in deeper analysis:
-
Enable ScyllaHide before loading the sample. It patches
NtQueryInformationProcess, PEB flags, timing checks, and heap flags transparently. -
Patch
IsDebuggerPresentto return 0 (x64dbg): find it in the Symbols pane → right-click → Patch → set return value to 0. - Clear the PEB
BeingDebuggedbyte (offset +0x2 from PEB base):- x64dbg: Memory Map → navigate to PEB → offset +2 → Binary Edit →
00 - WinDbg:
eb @$peb+2 0 - Frida:
Interceptor.replace(Module.getExportByName("kernel32.dll", "IsDebuggerPresent"), new NativeCallback(() => 0, "int", []))
- x64dbg: Memory Map → navigate to PEB → offset +2 → Binary Edit →
-
Suppress
OutputDebugStringanti-debug: this technique raises exception0x40010006to detect a live debugger. In x64dbg: Options → Preferences → Exceptions → add0x40010006to the pass-through list. -
For timing checks (
RDTSC,GetTickCount,QueryPerformanceCounter): set a breakpoint immediately after the call and manually zero the elapsed value. -
For VM detection: remove or spoof VMware Tools, use a non-default snapshot name, ensure a realistic process list (browser, office apps), and screen resolution ≥ 1024×768.
- For connectivity checks: run INetSim or FakeNet-NG so the sample believes it has real network access and proceeds past its C2 check-in gate.
For Linux samples using ptrace-based self-detection:
catch syscall ptrace
commands
silent
set $rax = 0
continue
end
This intercepts the ptrace(PTRACE_TRACEME) call the sample uses to detect a debugger and forces it to return success.
See Anti-Analysis Techniques for a full taxonomy of evasion primitives and their detection signatures.
Runtime Unpacking
Most malware in the wild is packed or encrypted. The executable you receive does not contain the real code — a small stub decrypts and loads it at runtime. The goal of runtime unpacking is to capture the decrypted payload from memory.
How Packing Works
Disk:
[ packer stub ][ encrypted payload ]
↓ run
Memory (after stub executes):
[ packer stub ][ decrypted PE in new allocation ]
EIP → OEP (Original Entry Point of the real malware)
The stub decrypts the payload, often calls VirtualAlloc + WriteProcessMemory or maps a new region with PROT_EXEC, then jumps to the Original Entry Point (OEP) of the real code.
Identifying a Packed Binary
Before hunting the OEP, confirm the binary is actually packed:
- Very few imports (typically only
VirtualAlloc,LoadLibrary,GetProcAddress) - High-entropy sections:
binwalk -E sample.exeordiec sample.exe - Section names like
.UPX0,UPX1,.packed, or random strings - On-disk size much smaller than reported virtual size
Finding the OEP
Method 1 — Hardware breakpoint on VirtualProtect/mprotect
Set a breakpoint on the call that makes a memory region executable. When it fires, the decrypted payload is in the just-protected region. Single-step until you reach a jmp eax/jmp [reg] or an unusual-looking PUSH sequence (common OEP patterns).
break mprotect
commands
silent
printf "mprotect addr=%p len=%lu prot=%d\n", $rdi, $rsi, $rdx
continue
end
Method 2 — Memory access breakpoints (watch for self-writing)
In x64dbg: right-click on a memory region → “Hardware, on access”. The debugger breaks the instant any instruction reads from or writes to that page — catching the decryptor as it fills in the real code.
Method 3 — ESP trick (Windows, simple packers)
At the packer entry point, note the value of ESP. Set a hardware breakpoint on that memory address. When the packer finishes and returns to the OEP, it will restore the stack to approximately its original value, triggering your watchpoint.
Dumping the Unpacked Image
Linux
# While the process is paused at OEP, dump its full memory map
gdb -p <pid> -batch -ex "dump memory /tmp/unpacked.bin 0x400000 0x410000"
# Or use /proc
cp /proc/<pid>/exe /tmp/unpacked_exe
# Read the maps to find the correct region
cat /proc/<pid>/maps
Windows — Scylla (via x64dbg plugin)
- Run to OEP.
- Plugins → Scylla → “IAT Autosearch” (finds the Import Address Table of the unpacked binary).
- “Get Imports” — resolves all imports.
- “Dump” — writes the reconstructed PE to disk.
- “Fix Dump” — patches the dumped file’s IAT so it loads correctly.
The result is a reconstructable PE that Ghidra can analyze statically.
Dynamic Instrumentation
Dynamic instrumentation inserts your analysis code into a running process without modifying the binary on disk. It works on packed samples, encrypted samples, and samples that detect debuggers.
API Monitor (Windows)
API Monitor intercepts thousands of Windows API functions and logs every call with arguments and return values — without a debugger attached.
- Start API Monitor as administrator.
- Select API categories to monitor. For malware: Crypto, Networking, Process/Thread, File I/O, Registry.
- Monitor New Process: enter the path and arguments, or attach to a running PID.
- Filter by API name or return value to isolate anomalous calls.
Key APIs to watch: VirtualAllocEx, WriteProcessMemory, CreateRemoteThread, NtUnmapViewOfSection, RegSetValueEx, WinExec, ShellExecute, CryptEncrypt, InternetConnect.
Frida
Frida injects a JavaScript engine into the target process. You write hooks in JavaScript (or Python), attach from the command line, and intercept calls in real time. It runs on Windows, Linux, macOS, Android, and iOS — no recompile needed.
Install
pip install frida-tools
Intercept connect() on Linux
// hook_connect.js
Interceptor.attach(Module.getExportByName(null, "connect"), {
onEnter(args) {
const sockfd = args[0].toInt32();
const sockaddr = args[1];
const family = sockaddr.readU16();
if (family === 2) { // AF_INET
// sin_port is big-endian (network byte order); read byte-by-byte
const port = (sockaddr.add(2).readU8() << 8) | sockaddr.add(3).readU8();
const a = sockaddr.add(4);
const ip = `${a.readU8()}.${a.add(1).readU8()}.${a.add(2).readU8()}.${a.add(3).readU8()}`;
console.log(`[connect] ${ip}:${port}`);
}
}
});
frida -l hook_connect.js -f ./sample --no-pause
# Attach to running process by name
frida -l hook_connect.js -n sample
Intercept a function by absolute address
Interceptor.attach(ptr("0x7f1234005678"), {
onEnter(args) {
console.log("arg0 = " + args[0]);
},
onLeave(retval) {
console.log("returned " + retval);
}
});
Force a function to return a fixed value (anti-debug bypass)
const IsDebuggerPresent = Module.getExportByName("kernel32.dll", "IsDebuggerPresent");
Interceptor.replace(IsDebuggerPresent, new NativeCallback(() => 0, "int", []));
Scripts can be loaded from a file (frida -p <PID> -l script.js) and reloaded live, making bypass iteration much faster than patching and rerunning.
Dump a decrypted buffer when a function returns
// Intercept a custom decrypt() function and print its output
const base = Module.getBaseAddress("sample");
const decryptFn = base.add(0x1234); // RVA of decrypt routine
Interceptor.attach(decryptFn, {
onLeave(retval) {
// assume return value is a pointer to the decrypted buffer
// and it's null-terminated
const s = retval.readUtf8String();
console.log("[decrypt output]", s);
}
});
Stalker — execution tracing
// Trace every basic block the target executes
Stalker.follow(Process.getCurrentThreadId(), {
events: { block: true },
onReceive(events) {
const blocks = Stalker.parse(events, { annotate: true, stringify: true });
for (const b of blocks)
console.log(b);
}
});
Stalker output can be fed directly into coverage visualizers — Cartographer for Ghidra, or Lighthouse for IDA Pro and Binary Ninja.
PIN (Intel)
PIN is a dynamic binary instrumentation framework from Intel. It is more powerful than Frida for fine-grained x86/x64 analysis but requires writing C++ Pintool plugins.
Count instruction executions by address
pin -t source/tools/ManualExamples/obj-intel64/inscount0.so -- ./sample
cat inscount.out
A common malware use case: write a Pintool that records every unique address executed (coverage), then compare coverage between runs to see what new code paths the malware takes after a given input.
DynamoRIO
DynamoRIO is an open-source DBI framework. The drcov client collects basic-block coverage in the DRCOV format, readable by Cartographer in Ghidra (and by Lighthouse in IDA Pro and Binary Ninja):
drrun -t drcov -- ./sample arg
ls *.log # drcov.sample.<pid>.0000.proc.log
Coverage diffing workflow
drrun -t drcov -- ./sample benign_input→ save asrun_benign.logdrrun -t drcov -- ./sample trigger_input→ save asrun_trigger.log- In Ghidra: load each log with Cartographer (Window → Code Coverage)
- Use Lighthouse’s Coverage Overview to subtract: find basic blocks in
run_triggerbut not inrun_benign— these are newly activated paths - Set a breakpoint at the first newly covered block and rerun with the triggering input to inspect context
Unexplored branches are often where additional capability is hidden — triggered only by specific C2 responses, time conditions, or configuration values.
DBI limitations: DBI imposes 10–100× execution overhead and can break self-modifying code, packed stubs, and samples that detect instrumentation via timing or explicit checks. Apply DBI after unpacking and defeating major anti-analysis checks.
Kernel-Level Monitoring
Userspace tools miss kernel-level malware behavior (rootkits, driver-based persistence). Two complementary approaches cover Linux and Windows.
eBPF (Linux)
eBPF programs attach to kernel tracepoints and perf events without modifying kernel source. Unlike strace, eBPF has near-zero overhead, can filter in-kernel before surfacing events, and survives against malware that detects ptrace.
Install bpftrace
sudo apt install bpftrace
Trace all execve calls system-wide
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s exec: ", comm); join(args->argv); }'
Trace file opens by a specific PID
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat /pid == $1/ { printf("open: %s\n", str(args->filename)); }' -- <pid>
Watch for mprotect calls that add PROT_EXEC
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_mprotect
/args->prot & 4/ # PROT_EXEC = 0x4
{
printf("[%s pid=%d] mprotect addr=0x%lx len=%lu prot=0x%x\n",
comm, pid, args->start, args->len, args->prot);
}'
This catches memory-based unpacking and shellcode injection without touching the process at all.
Caveat on connect address decoding: args->uservaddr in sys_enter_connect is a user-space pointer — dereferencing it directly in a bpftrace tracepoint produces unreliable results. To decode destination IP/port, use strace -e connect -p <PID> or intercept connect() with Frida in user context instead.
⚠️
strace -pon an already-running process needs ptrace permission. Most current distributions shipkernel.yama.ptrace_scope = 1, which allows attaching only to descendants of the tracer, so the command above fails:$ strace -e connect -p 760334 strace: attach: ptrace(PTRACE_SEIZE, 760334): Operation not permitted strace: sysctl kernel.yama.ptrace_scope = 1 may be restricting this attachEither start the sample under strace in the first place — which needs no special privilege, because it is then a child — or lift the restriction inside the analysis VM with
sudo sysctl -w kernel.yama.ptrace_scope=0. Lift it in the VM, never on your host.
Full syscall argument logging with bcc
# trace_malware.py — using BCC Python bindings
from bcc import BPF
prog = r"""
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
TRACEPOINT_PROBE(syscalls, sys_enter_connect) {
char comm[TASK_COMM_LEN];
bpf_get_current_comm(&comm, sizeof(comm));
bpf_trace_printk("connect: %s fd=%d\n", comm, args->fd);
return 0;
}
"""
b = BPF(text=prog)
b.trace_print()
Sysmon + ETW (Windows)
ETW (Event Tracing for Windows) is the kernel’s built-in structured event bus. Every driver, OS component, and many applications publish events to ETW providers. Sysmon subscribes to the relevant providers and writes them to the Windows Event Log in a parsed, queryable format.
Install Sysmon with a comprehensive config
# Recommended config from SwiftOnSecurity or Olaf Hartong
.\Sysmon64.exe -accepteula -i sysmonconfig.xml
Key Sysmon event IDs for malware
| Event ID | Description |
|---|---|
| 1 | Process Create (full command line, parent, hashes) |
| 3 | Network Connection (process, remote IP/port, DNS name) |
| 7 | Image Loaded (DLL loaded into a process) |
| 8 | CreateRemoteThread (classic injection indicator) |
| 10 | Process Access (OpenProcess — precursor to injection) |
| 11 | File Create |
| 12/13 | Registry Create/Set |
| 17/18 | Pipe Create/Connect (lateral movement) |
| 22 | DNS Query (process name, domain, result) |
| 25 | Process Tampering (hollowing, herpaderping detected) |
Query Sysmon logs with PowerShell
Get-WinEvent -LogName "Microsoft-Windows-Sysmon/Operational" |
Where-Object { $_.Id -eq 3 } |
Select-Object -First 20 |
Format-List
Direct ETW consumption (without Sysmon)
Tools like SilkETW and krabsetw let you subscribe to arbitrary ETW providers in real time. This is harder to evade than Sysmon because you choose the providers yourself:
# SilkETW: capture Microsoft-Windows-Kernel-Process provider
.\SilkETW.exe -t kernel -pn Microsoft-Windows-Kernel-Process `
-ot file -p C:\analysis\etw_out.json
Detecting Process Injection at Runtime
Process injection techniques (DLL injection, process hollowing, reflective loading) are among the most common malware behaviors. Dynamic detection differs from static detection: you look at the live state of a process’s memory rather than its on-disk binary.
Comparing Loaded DLLs Against Disk
An injected DLL may appear in the process’s module list but not on disk (reflective loading), or may appear at a path that does not match its image (DLL hijacking).
Windows — compare loaded modules to disk hashes
$pid = (Get-Process notepad).Id
$modules = (Get-Process -Id $pid).Modules
foreach ($m in $modules) {
$hash = (Get-FileHash $m.FileName -Algorithm SHA256 -ErrorAction SilentlyContinue).Hash
Write-Host "$($m.ModuleName) $($m.FileName) $hash"
}
Flag any module where FileName is in %TEMP%, %APPDATA%, or a non-standard path.
Linux — check for anonymous executable mappings
# Print memory mappings that are executable but have no backing file
awk '$2 ~ /x/ && $6 == ""' /proc/<pid>/maps
A legitimate process rarely has anonymous executable pages. Any result here warrants investigation — it may be shellcode, a reflectively loaded ELF, or an unpacked payload.
Thread Hijacking
The malware suspends a thread in the target process, rewrites its CONTEXT structure to redirect RIP/EIP to injected shellcode, then resumes it.
Detection
- Break on
SetThreadContextin API Monitor - In WinDbg:
!threadon all threads of a suspicious process — check whetherRippoints outside any known module
DLL Injection
Classic: the injector calls WriteProcessMemory to write a DLL path into the target, then CreateRemoteThread(LoadLibraryA).
Reflective: the DLL carries its own loader; CreateRemoteThread points directly into the injected buffer — no LoadLibraryA call, the DLL does not appear in the normal module list.
Detection
WriteProcessMemory→CreateRemoteThreadpair in API Monitor- For reflective DLL:
VirtualAllocExwithEXECUTE_READWRITE→WriteProcessMemory→CreateRemoteThreadto an address outside any module
Process Doppelgänging
Uses Windows NTFS transactions (TxF) to write malicious code to a transacted file that is never committed to disk, then create a process image section from it. The process appears legitimate because it references the phantom transacted file. MITRE: T1055.013.
Detection: Catch the NtCreateTransaction → NtCreateSection → NtCreateProcessEx call sequence in API Monitor or kernel callbacks.
Detecting Hollowing
Process hollowing creates a new process in suspended state, unmaps its original image, and writes malicious code into the vacated address space. The tell-tale sign is an executable page at the expected image base address that does not match the on-disk binary.
Windows — check image base vs. disk
# Read the PEB's ImageBaseAddress and compare with the on-disk PE
$proc = Get-Process -Name svchost | Select-Object -First 1
$addr = [System.Diagnostics.Process]::GetProcessById($proc.Id).MainModule.BaseAddress
# Compare with what's at that address using a tool like PE-bear or Moneta
Moneta is the current standard tool for this class of detection:
Moneta64.exe -m ioc -p <pid>
It identifies: modified PE headers, unsigned code, PE files mapped from non-disk paths, executable sections with no backing image, and regions with RWX permissions.
Other Injection Variants
| Technique | Key APIs / Mechanism | Detection Cue |
|---|---|---|
| APC Injection / Early Bird | QueueUserAPC or NtQueueApcThread into a suspended thread before it runs |
QueueUserAPC targeting a remote thread, paired with WriteProcessMemory |
| Manual Mapping | Malware parses and maps a PE by hand — resolve imports, apply relocations — without LoadLibrary |
Executable anonymous VAD region with a PE header; no backing module in lm or dlllist |
| Section Mapping | NtCreateSection + NtMapViewOfSection across process boundaries; bypasses WriteProcessMemory |
NtCreateSection/NtMapViewOfSection pair targeting a different process; Volatility handles |
| Process Herpaderping / Ghosting | Create section from a malicious file, then modify or delete the file before process creation; AV scans the stale disk image | Section object from a file subsequently modified or deleted; detected by kernel callbacks on file modification after section creation |
MITRE: T1055 covers the full injection family.
Hunting Injected Shellcode with Volatility
If you have a memory image taken mid-execution (or from a hypervisor snapshot), Volatility’s malfind plugin automates the hunt:
vol -f memdump.raw windows.malfind
It flags every executable, non-module-backed memory region and shows the first bytes — look for the 4D 5A (MZ) header that indicates an injected PE, or \x55\x48\x89\xe5 (x64 function prologue) indicating shellcode.
Full Volatility 3 triage:
vol -f memory.raw windows.pslist # walks ActiveProcessLinks
vol -f memory.raw windows.psscan # pool-tag scan — finds DKOM-hidden processes
vol -f memory.raw windows.malfind # executable anonymous VAD regions
vol -f memory.raw windows.dlllist # loaded modules per process
vol -f memory.raw windows.handles # open handles (mutexes, named pipes, files)
vol -f memory.raw windows.netscan # network connections including recently closed
Cross-referencing pslist against psscan is the fastest way to identify DKOM-hidden processes: any entry in psscan absent from pslist has been unlinked from the active process list.
Memory Forensics During Execution
You do not have to wait until after the fact to apply memory forensics techniques. Taking a snapshot of a VM mid-execution and analyzing the snapshot offline is a powerful combination.
Capturing a Live Memory Image
Linux
sudo avml /tmp/memdump.lime
# or with LiME kernel module
sudo insmod lime-$(uname -r).ko "path=/tmp/memdump.lime format=lime"
Windows
# WinPmem
winpmem_mini_x64_rc2.exe memdump.raw
Extracting Decrypted Strings and Config
Many malware families decrypt their C2 configuration (IPs, domains, beacon interval, mutex names) at runtime and store it in a heap allocation. Once you’ve identified the allocation — via a breakpoint on the decryption return, or via search in pwndbg — dump it:
# In GDB: dump 4096 bytes starting at a heap address
dump memory /tmp/config_block.bin 0x55a3b2c0d000 0x55a3b2c0e000
# Inspect for strings
strings -n 8 /tmp/config_block.bin
Carving PE Files from a Process Dump
# Use foremost against a process memory dump to find embedded PEs
foremost -t exe -i memdump.raw -o carved_pes/
# Or bulk_extractor for broader artifact recovery
bulk_extractor -o output/ memdump.raw
Correlating Dynamic and Static Findings
Advanced dynamic analysis is most powerful when cross-referenced against static findings:
| Finding | Origin | Implication |
|---|---|---|
| Encrypted buffer at a high-entropy data section address | Static | Break on first write there; collect the decryption key |
LoadLibraryA and GetProcAddress as only imports |
Static | Packed or shellcode-like; hunt OEP before any other analysis |
CreateRemoteThread in IAT |
Static | Expect injection; monitor with API Monitor; compare memory maps before and after |
Module not in lm but executable pages visible |
Dynamic (WinDbg / Volatility) | Reflective injection or manual mapping |
NtUnmapViewOfSection in API trace |
Dynamic | Process hollowing; compare target process on-disk vs. in-memory image |
| Divergent execution path at a basic block | Dynamic (Lighthouse coverage) | Untriggered branch; examine condition with debugger and force it |
Behavioral Correlation with MITRE ATT&CK
Dynamic analysis output maps directly to ATT&CK techniques. Build a table as you work:
| Observed behavior | ATT&CK technique |
|---|---|
CreateRemoteThread into explorer.exe |
T1055.001 — Process Injection: DLL Injection |
| Anonymous RWX memory region | T1055.004 — Asynchronous Procedure Call |
| Process image hollowing detected by Moneta | T1055.012 — Process Hollowing |
mprotect(PROT_EXEC) on heap region |
T1620 — Reflective Code Loading |
| Beaconing to hard-coded IP on port 443, self-signed cert | T1071.001 — Application Layer Protocol: Web Protocols |
| DGA-like hostnames in DNS queries | T1568.002 — Dynamic Resolution: DGA |
reg add HKCU\...\Run |
T1547.001 — Boot or Logon Autostart: Registry Run Keys |
Writes to %APPDATA%\Microsoft\Windows\Start Menu\Startup |
T1547.001 |
Elevated ptrace call on another process |
T1055 — Process Injection (general) |
The MITRE ATT&CK Navigator lets you highlight techniques as you find them, producing a heat-map for the final report.
Workflow Summary
A structured advanced dynamic analysis session:
- Revert to clean snapshot; start monitoring (Sysmon/bpftrace/Wireshark) before running the sample.
- Run the sample under a debugger with anti-anti-debugging bypass (pwndbg
catch syscall ptrace+return 0/ ScyllaHide on Windows). - Break on
VirtualAlloc/mprotectwith EXEC — observe packing/decryption. - At OEP, dump the unpacked image (Scylla/GDB dump memory).
- Attach Frida to log API calls of interest; use Stalker if code coverage is needed.
- Take a VM snapshot at peak activity for offline memory forensics.
- Run Moneta/
malfindagainst the snapshot to enumerate injected regions. - Correlate every observation to an ATT&CK technique.
- Restore snapshot; repeat with network simulation (INetSim) enabled.
Tools Quick Reference
| Tool | Platform | Purpose |
|---|---|---|
| GDB + pwndbg | Linux | Step-through debugging, heap inspection, memory search |
| splitmind + tmux | Linux | Each pwndbg context section in its own resizable tmux pane |
| WinDbg | Windows | User-mode and kernel-mode debugging; TTD support |
| x64dbg + ScyllaHide | Windows | Step-through, anti-anti-debug bypass |
| Scylla | Windows | PE dump + IAT reconstruction |
| API Monitor | Windows | GUI API call interception with arguments and return values |
| Frida | Cross-platform | Scriptable API hooking, execution tracing, bypass injection |
| Intel PIN | Linux/Windows | Low-level instrumentation, coverage |
| DynamoRIO + drcov | Linux/Windows | Basic-block coverage for Lighthouse diffing |
| bpftrace / BCC | Linux | Kernel-level syscall/event tracing without ptrace |
| Sysmon + ETW | Windows | Structured kernel event capture |
| SilkETW / krabsetw | Windows | Raw ETW provider consumption |
| Moneta | Windows | Live process memory IOC scanner |
Volatility malfind |
Offline image | Shellcode and injected PE hunter |
| MITRE ATT&CK Navigator | Web | Technique mapping and report visualization |
Further Reading
- M. Sikorski & A. Honig. (2012). Practical Malware Analysis, Ch. 9–11. No Starch Press.
- M. Ligh et al. (2014). The Art of Memory Forensics. Wiley.
- pwndbg GDB TUI tutorial — the shipped
pwndbglayouts and thepwndbg_*windows - GDB TUI reference —
tui new-layoutsyntax and the TUI key bindings - GDB shell commands —
pipe, the-ddelimiter, and$_shell_exitcode - splitmind — routes pwndbg context sections to separate tmux panes
- Frida documentation — JavaScript API reference
- Intel PIN user guide
- DynamoRIO drcov documentation
- Cartographer — code coverage visualization for Ghidra
- Lighthouse plugin — the equivalent for IDA Pro and Binary Ninja
- bpftrace reference guide
- Sysmon configuration guide
- MITRE ATT&CK T1055 — Process Injection (all sub-techniques)
- MITRE ATT&CK Navigator
- Moneta — live process memory scanner
- al-khaser — anti-analysis technique PoC collection