Advanced Dynamic Analysis

• Advanced Dynamic Analysis
  • Debugger-Based Analysis
    • GDB / pwndbg (Linux)
    • x64dbg (Windows)
    • WinDbg (Windows kernel and user-mode)
  • Defeating Anti-Analysis in the Debugger
  • Dynamic Unpacking
    • Identifying a Packed Binary
    • Finding the OEP
    • Dumping the Unpacked Image
  • Process Injection Detection
    • Process Hollowing (RunPE)
    • DLL Injection (Classic and Reflective)
    • Process Doppelgänging
    • Thread Hijacking
    • Other Injection Variants
    • Memory Forensics Triage
  • API Monitoring and Hooking
    • API Monitor (Windows)
    • Frida (Linux and Windows)
  • Dynamic Binary Instrumentation (DBI)
    • Intel PIN
    • DynamoRIO
    • Code Coverage with Lighthouse
  • Kernel-Level Analysis
    • eBPF (Linux)
    • Kernel Debugging with WinDbg
  • Correlating Results
  • Useful Resources

This page assumes familiarity with basic dynamic analysis (strace, ltrace, sandbox tools) and extends into techniques that reveal behavior hidden behind packers, process injection, and anti-analysis checks. For a catalogue of anti-analysis primitives and evasion signatures, see Anti-Analysis Techniques; for shellcode-specific analysis workflows, see Shellcode Analysis.

Debugger-Based Analysis

A debugger gives you interactive control over execution: you can pause at any instruction, inspect and modify registers and memory, and force execution paths that the malware would otherwise suppress. This makes it the primary tool for unpacking, injection analysis, and defeating anti-analysis checks.

GDB / pwndbg (Linux)

The pwndbg plugin (installed via course setup scripts) augments GDB with malware-analysis-friendly output: automatic stack/register display, heap introspection, and pattern searching.

Attach to a running process:

$ gdb -p <PID>

Run with arguments:

$ gdb --args ./sample arg1 arg2

Essential commands:

Useful pwndbg commands:

Breakpoint strategy for unpacking:

Packed samples call mmap or mprotect to create executable memory, then transfer control there. Set breakpoints on both and inspect what gets written before execution reaches the new region:

(gdb) catch syscall mmap mprotect
(gdb) run
# when stopped at mprotect with PROT_EXEC:
(gdb) finish    # let the call complete
(gdb) vmmap     # find the new executable region
(gdb) x/10i <new_region_addr>

x64dbg (Windows)

x64dbg is the open-source equivalent of OllyDbg, with a modern UI and active plugin ecosystem. Key navigation:

ScyllaHide plugin: Automatically patches the most common IsDebuggerPresent, PEB flags, and timing check anti-analysis techniques. Install it and enable all relevant options before loading a sample.

WinDbg (Windows kernel and user-mode)

For kernel-mode malware (rootkits, drivers), use WinDbg over a serial or network (kdnet) connection between an analysis VM and a debugger host.

Attach to kernel via network (kdnet):

On the VM (run once):

bcdedit /dbgsettings net hostip:<debugger_host_ip> port:50000
bcdedit /debug on

On the debugger host:

windbg -k net:port=50000,key=<key>

Useful kernel commands:

Defeating Anti-Analysis in the Debugger

When a sample exits silently or behaves differently under a debugger, work through this checklist before investing time in deeper analysis:

1. Enable ScyllaHide or equivalent plugin before loading the sample.
2. Patch IsDebuggerPresent to return 0:

   (x64dbg) Find IsDebuggerPresent in the imports; right-click → Patch → set return value
   

3. Clear the PEB BeingDebugged byte (Windows only; offset +0x2 from PEB base):
   • In x64dbg: Open Memory Map, navigate to the PEB (shown in the status bar), go to offset +2, right-click → Binary Edit → set to 00.
   • In WinDbg: eb @$peb+2 0
   • Via Frida (shown in API Monitoring below): patch IsDebuggerPresent to return 0 rather than directly writing the PEB.
4. Suppress OutputDebugString anti-debug: an exception-based technique raises DBG_PRINTEXCEPTION_C to detect a live debugger. In x64dbg: Options → Preferences → Exceptions → Add exception code 0x40010006 to the pass-through list.
5. For timing checks: set a breakpoint immediately after the RDTSC/GetTickCount call and manually set the elapsed value to 0.
6. For VM detection: ensure the analysis VM has VMware Tools removed or spoofed, no default snapshot names, a realistic process list (browser, office apps), and screen resolution ≥ 1024×768.
7. For connectivity checks: run INetSim or FakeNet-NG to provide simulated services so the malware believes it has real network access.

See Anti-Analysis Techniques for a full taxonomy of evasion primitives and their detection signatures.

Dynamic Unpacking

Packers compress or encrypt the original binary and use a stub to restore it at runtime. Static analysis of a packed binary gives you the stub, not the payload. You must let the stub run and then extract the unpacked code from memory.

Identifying a Packed Binary

• Very few imports (typically only VirtualAlloc, LoadLibrary, GetProcAddress)
• High entropy sections (binwalk -E sample.exe or diec sample.exe)
• Section names like .UPX0, UPX1, .packed, or random strings

Finding the OEP

The Original Entry Point (OEP) is where the real code begins executing after the unpacking stub finishes. Two reliable techniques:

1. ESP trick (stack-based OEP detection)

The stub always saves and restores the context it inherited. Immediately after the packed entry point:

(in debugger, at entry point of packed binary)
- note the current ESP value
- set a hardware breakpoint on memory access (write) at ESP
- run; when the stub restores the saved context, the breakpoint fires just before it jumps to OEP

The JMP EAX or POPAD; JMP <addr> instruction you land on is typically the stub finishing and handing control to the OEP.

2. VirtualAlloc / VirtualProtect breakpoint

- Set a breakpoint on VirtualAlloc or VirtualProtect (PROT_EXEC)
- Let the stub allocate executable memory and write the payload
- Set a memory execution breakpoint on the allocated region
- The first execution hit in that region is at or near the OEP

Dumping the Unpacked Image

Once at the OEP:

Linux — with GDB:

(gdb) info proc mappings       # find the base address of the image
(gdb) dump binary memory unpacked.bin 0x400000 0x500000

Windows — Scylla (x64dbg plugin):

1. Pause at the OEP
2. Open Scylla (Ctrl+I)
3. “IAT Autosearch” → “Get Imports” → “Fix Dump”

Scylla rebuilds the Import Address Table from the live process, producing a PE binary that loads correctly and can be analyzed statically.

Process Injection Detection

Many malware families inject code into legitimate host processes to evade process-based detections and hide network activity behind trusted executables. The following are the most common injection techniques and how to detect them dynamically.

Process Hollowing (RunPE)

The malware creates a suspended legitimate process (CreateProcess with CREATE_SUSPENDED), unmaps its image (NtUnmapViewOfSection), maps its own payload — preferring the original image base but relocating if necessary — then resumes execution.

Detection:

• Break on NtUnmapViewOfSection in the analysis debugger
• Check CreateProcess calls with CREATE_SUSPENDED in API Monitor or ProcMon
• After the injection, compare the on-disk image of the host process with its in-memory image using Volatility’s malfind or procdump plugins

DLL Injection (Classic and Reflective)

Classic: The injector calls WriteProcessMemory to write a DLL path into the target, then CreateRemoteThread(LoadLibraryA).

Reflective: The DLL carries its own loader; CreateRemoteThread points directly into the injected buffer. No call to LoadLibraryA — the DLL does not appear in the normal module list.

Detection:

• WriteProcessMemory → CreateRemoteThread: look for this pair in API Monitor
• For reflective DLL: VirtualAllocEx in the target with EXECUTE_READWRITE permissions, followed by WriteProcessMemory, followed by CreateRemoteThread to an address outside any module
• In Volatility: malfind to find executable memory regions with no backing module

Process Doppelgänging

Uses Windows NTFS transactions (TxF) to write malicious code to a transacted file that is never committed to disk, then create a process image section from it. The process appears legitimate because it points to the “phantom” transacted file.

Detection: User-mode API tracing can catch the NtCreateTransaction → NtCreateSection → NtCreateProcessEx call sequence; kernel-level visibility helps confirm and correlate. MITRE: T1055.013.

Thread Hijacking

The malware suspends a thread in the target process, rewrites its CONTEXT structure to redirect RIP/EIP to injected shellcode, then resumes it.

Detection:

• Break on SetThreadContext in API Monitor
• In WinDbg: !thread on all threads of a suspicious process to check if Rip points outside any known module

Other Injection Variants

MITRE: T1055 covers the full injection family.

Memory Forensics Triage

Once you have a memory image, Volatility provides a cross-view perspective that passive monitoring cannot match.

Capture:

# Windows — WinPMEM:
winpmem_mini_x64_rc2.exe memory.raw

# Linux — LiME kernel module:
sudo insmod lime.ko "path=/mnt/usb/memory.lime format=lime"

Triage with Volatility 3:

vol -f memory.raw windows.pslist    # walks ActiveProcessLinks
vol -f memory.raw windows.psscan    # pool-tag scan — finds DKOM-hidden processes
vol -f memory.raw windows.malfind   # executable anonymous VAD regions (injection indicator)
vol -f memory.raw windows.dlllist   # loaded modules per process
vol -f memory.raw windows.handles   # open handles (mutexes, named pipes, files)
vol -f memory.raw windows.netscan   # network connections including recently closed

Cross-referencing pslist against psscan is the fastest way to identify processes hidden by DKOM: any entry in psscan absent from pslist is unlinked from the active process list.

API Monitoring and Hooking

Monitoring API calls at runtime reveals what a sample does without requiring full debugger control.

API Monitor (Windows)

API Monitor intercepts thousands of Windows API functions and logs calls with arguments and return values.

1. Start API Monitor as administrator
2. Select the APIs to monitor (preset collections for crypto, networking, process creation, and file I/O are most useful for malware)
3. Monitor a new process: enter the path and arguments
4. Filter the call log by API category or return value to find anomalies

Key APIs to watch: VirtualAllocEx, WriteProcessMemory, CreateRemoteThread, NtUnmapViewOfSection, RegSetValueEx, WinExec, ShellExecute, CryptEncrypt, InternetConnect.

Frida (Linux and Windows)

Frida injects a JavaScript engine into a running process and lets you intercept and modify function calls programmatically. It does not require a debugger or source code.

Attach to a running process:

$ frida -p <PID>

Intercept a function by address:

Interceptor.attach(ptr("0x7f1234005678"), {
    onEnter(args) {
        console.log("called with arg0 = " + args[0]);
    },
    onLeave(retval) {
        console.log("returned " + retval);
    }
});

Intercept an exported function by name:

const openFunc = Module.getExportByName("libc.so.6", "open");
Interceptor.attach(openFunc, {
    onEnter(args) {
        console.log("open(" + args[0].readUtf8String() + ")");
    }
});

Force a function to return a specific value (bypass):

const IsDebuggerPresent = Module.getExportByName("kernel32.dll", "IsDebuggerPresent");
Interceptor.replace(IsDebuggerPresent, new NativeCallback(() => 0, "int", []));

Frida scripts can be loaded from a file (frida -p <PID> -l script.js) and reloaded live, making it far faster to iterate on bypasses than patching and rerunning.

Dynamic Binary Instrumentation (DBI)

DBI frameworks instrument every instruction (or selected instructions) as a binary executes, without modifying the file on disk. This enables:

• Complete instruction traces
• Taint tracking (following data derived from network input)
• Code coverage collection

Intel PIN

PIN is Intel’s production DBI framework. Analysis code is written in C/C++ as a Pintool and loaded as a shared library.

Count all instructions executed:

$ pin -t $PIN_ROOT/source/tools/ManualExamples/inscount0.so -- ./sample
$ cat inscount.out
Count 9347812

Generate an instruction trace:

$ pin -t $PIN_ROOT/source/tools/ManualExamples/itrace.so -- ./sample
$ head itrace.out
0x400557
0x40055a
...

For malware, trace differences between runs (e.g., when injecting different inputs) reveal which code paths are newly activated — useful for triggering dormant payloads.

DynamoRIO

DynamoRIO is an open-source alternative to PIN. The drcov client collects basic-block coverage in a format readable by the Lighthouse plugin for IDA/Ghidra:

$ drrun -t drcov -- ./sample arg
$ ls *.log    # drcov.sample.<pid>.0000.proc.log

Code Coverage with Lighthouse

Once you have a drcov log:

1. Open the binary in IDA Pro or Ghidra
2. In IDA: File → Load file → Code coverage file (Lighthouse plugin)
3. Covered basic blocks are highlighted in the graph view

This immediately shows which code the sample executed — and, crucially, which branches it did not take. Unexplored branches are often where additional capability is hidden (triggered by specific conditions like time, C2 response, or configuration).

Coverage diffing workflow:

1. Run drrun -t drcov -- ./sample benign_input → save as run_benign.log
2. Run drrun -t drcov -- ./sample trigger_input → save as run_trigger.log
3. Load both logs in IDA via Lighthouse (File → Load file → Code coverage file for each)
4. Use Lighthouse’s Coverage Overview to subtract: find basic blocks covered in run_trigger but not in run_benign — these are newly activated paths
5. Set a breakpoint at the first newly covered block and rerun with the triggering input to inspect context

DBI limitations:

DBI imposes 10–100× execution overhead and can break self-modifying code, packed stubs, and code that detects instrumentation via timing or explicit checks. Apply DBI after unpacking and defeating major anti-analysis checks, not before.

Kernel-Level Analysis

User-mode monitoring has a fundamental limitation: malware with kernel code (rootkits, bootkits, malicious drivers) can hide from it entirely. Kernel analysis is necessary when:

• A sample installs a driver
• Process listings / file listings appear manipulated (DKOM)
• System calls return different results from what memory shows

eBPF (Linux)

eBPF programs run in the kernel and can attach to system call entry/exit points, kprobes, and tracepoints without modifying the kernel. bpftrace provides a high-level scripting interface:

Trace all execve calls with arguments:

# bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s %s\n", comm, str(args->filename)); }'

Trace connect syscalls (pid and file descriptor only):

# bpftrace -e 'tracepoint:syscalls:sys_enter_connect { printf("%s (pid %d) connect fd=%d\n", comm, pid, args->fd); }'

Note: args->uservaddr in sys_enter_connect is a user-space pointer — dereferencing it directly in a bpftrace tracepoint will produce unreliable results. To decode destination addresses, use strace -e connect -p <PID> or intercept connect() with Frida in user context instead.

eBPF runs at the kernel level, so it cannot be hidden from by user-space rootkits (though kernel rootkits can still tamper with eBPF output).

Kernel Debugging with WinDbg

With a kernel debugger attached (see Debugger-Based Analysis above):

Detect DKOM (Direct Kernel Object Manipulation):

DKOM rootkits hide processes by unlinking them from the ActiveProcessLinks list in _EPROCESS. The process still exists but ps/Task Manager skips it.

kd> !process 0 0           # walk ActiveProcessLinks — misses unlinked (DKOM-hidden) processes

For cross-view DKOM detection, pool-tag scanning is most reliably done offline. Take a memory snapshot with WinPMEM and compare in Volatility:

vol -f memory.raw windows.pslist    # linked-list walk (matches !process 0 0)
vol -f memory.raw windows.psscan    # pool-tag scan for all EPROCESS objects

Any process in psscan that is absent from pslist has been unlinked by DKOM. See Memory Forensics Triage above for capture instructions.

Inspect a driver’s dispatch table for hooks:

kd> lm m <driver_name>
kd> dt nt!_DRIVER_OBJECT <addr>
kd> dqs <MajorFunction array addr> L1c    # look for unexpected dispatch pointers

Correlating Results

Advanced dynamic analysis is most powerful when cross-referenced against static findings:

Useful Resources

• The Art of Memory Forensics — Ligh et al.; memory forensics and injection detection
• Frida documentation — JavaScript API reference
• Intel PIN user guide
• DynamoRIO documentation — drcov tool
• Lighthouse plugin — code coverage in IDA/Ghidra
• bpftrace reference guide
• MITRE ATT&CK T1055 — Process Injection (all sub-techniques)
• al-khaser — anti-analysis technique PoC collection
• Practical Malware Analysis (Sikorski & Honig), Chapters 9–11 — OllyDbg and dynamic analysis techniques