Advanced Dynamic Analysis

• Advanced Dynamic Analysis
  • Debugger-Based Analysis
    • Choosing a Debugger
    • GDB + pwndbg for Linux Samples
    • x64dbg for Windows Samples
    • WinDbg for Kernel-Mode Malware
    • Scripting the Debugger
  • Defeating Anti-Analysis in the Debugger
  • Runtime Unpacking
    • How Packing Works
    • Identifying a Packed Binary
    • Finding the OEP
    • Dumping the Unpacked Image
  • Dynamic Instrumentation
    • API Monitor (Windows)
    • Frida
    • PIN (Intel)
    • DynamoRIO
  • Kernel-Level Monitoring
    • eBPF (Linux)
    • Sysmon + ETW (Windows)
  • Detecting Process Injection at Runtime
    • Comparing Loaded DLLs Against Disk
    • Thread Hijacking
    • DLL Injection
    • Process Doppelgänging
    • Detecting Hollowing
    • Other Injection Variants
    • Hunting Injected Shellcode with Volatility
  • Memory Forensics During Execution
    • Capturing a Live Memory Image
    • Extracting Decrypted Strings and Config
    • Carving PE Files from a Process Dump
  • Correlating Dynamic and Static Findings
  • Behavioral Correlation with MITRE ATT&CK
  • Workflow Summary
  • Tools Quick Reference
  • Further Reading

This page picks up where basic dynamic analysis leaves off. The foundation — sandboxes, strace, network capture — reveals what a sample does. Advanced dynamic analysis answers how and why: it lets you control execution, intercept at arbitrary points, defeat anti-analysis tricks, and extract artifacts that never exist on disk. It requires a working debugger and an isolated analysis VM with a clean snapshot.

Debugger-Based Analysis

A debugger gives you the ability to pause execution at any point, inspect registers and memory, modify values, and step through individual instructions. For malware analysis, this is the primary tool for understanding obfuscated logic, decryption routines, and conditional C2 check-ins.

Choosing a Debugger

GDB + pwndbg for Linux Samples

Install pwndbg once (it replaces the default GDB prompt):

git clone https://github.com/pwndbg/pwndbg
cd pwndbg
./setup.sh

Start or attach:

gdb --args ./sample arg1 arg2   # run directly with arguments
gdb -p $(pgrep sample)          # attach to a running process

Essential GDB commands:

pwndbg enhancements:

Breakpoint strategy for unpacking:

Packed samples call mmap or mprotect to create executable memory, then transfer control there. Set breakpoints on both and inspect what was written before execution reaches the new region:

(gdb) catch syscall mmap mprotect
(gdb) run
# when stopped at mprotect with PROT_EXEC:
(gdb) finish    # let the call complete
(gdb) vmmap     # find the new executable region
(gdb) x/10i <new_region_addr>

Conditionals and mid-session patches:

# Break only when a register holds a specific value
break *0x401234 if $rdi == 0x0

# Patch a JZ to NOP+NOP so a branch is always taken
set *(short*)0x401a30 = 0x9090

x64dbg for Windows Samples

x64dbg is the standard open-source Windows malware debugger. Core workflow:

1. Load the sample: File → Open. Let the loader place it.
2. Set the entry-point breakpoint: it pauses at the PE entry point automatically.
3. Pause on API calls: Right-click in the Symbols pane → “Set breakpoint on all calls to [function]”
4. Search for strings: Right-click in the disassembly → Search for → All Referenced Strings

Keyboard shortcuts:

Essential plugins:

WinDbg for Kernel-Mode Malware

For rootkits and malicious drivers, WinDbg over a kernel debug connection is required. Set up kdnet between the analysis VM and a debugger host:

On the VM (run once, then reboot):

bcdedit /dbgsettings net hostip:<debugger_host_ip> port:50000
bcdedit /debug on

On the debugger host:

windbg -k net:port=50000,key=<key>

Essential kernel commands:

To detect DKOM-hidden processes, compare !process 0 0 (linked-list walk) against a pool-tag scan in Volatility — processes hidden from the active list will still appear in windows.psscan.

Scripting the Debugger

Both debuggers support Python scripting for automated analysis:

GDB + pwndbg/Python — log all connect() calls:

import gdb

class ConnectBreak(gdb.Breakpoint):
    def stop(self):
        rdi = int(gdb.parse_and_eval("$rdi"))   # sockfd
        rsi = int(gdb.parse_and_eval("$rsi"))   # struct sockaddr*
        # read the sockaddr from memory
        raw = gdb.selected_inferior().read_memory(rsi, 16)
        import socket, struct
        port = struct.unpack_from(">H", bytes(raw), 2)[0]
        addr = socket.inet_ntoa(bytes(raw)[4:8])
        print(f"[connect] {addr}:{port}")
        return False   # don't stop, just log

ConnectBreak("connect")

Load with source log_connect.py inside GDB.

x64dbg script — conditional NOP patch at runtime:

// In the x64dbg scripting console
// Patch a JNZ at 0x401A30 to NOP+NOP (always fall through)
mov [0x401A30], 0x9090

Defeating Anti-Analysis in the Debugger

When a sample exits silently or behaves differently under a debugger, work through this checklist before investing time in deeper analysis:

1. Enable ScyllaHide before loading the sample. It patches NtQueryInformationProcess, PEB flags, timing checks, and heap flags transparently.

2. Patch IsDebuggerPresent to return 0 (x64dbg): find it in the Symbols pane → right-click → Patch → set return value to 0.

3. Clear the PEB BeingDebugged byte (offset +0x2 from PEB base):
   • x64dbg: Memory Map → navigate to PEB → offset +2 → Binary Edit → 00
   • WinDbg: eb @$peb+2 0
   • Frida: Interceptor.replace(Module.getExportByName("kernel32.dll", "IsDebuggerPresent"), new NativeCallback(() => 0, "int", []))

4. Suppress OutputDebugString anti-debug: this technique raises exception 0x40010006 to detect a live debugger. In x64dbg: Options → Preferences → Exceptions → add 0x40010006 to the pass-through list.

5. For timing checks (RDTSC, GetTickCount, QueryPerformanceCounter): set a breakpoint immediately after the call and manually zero the elapsed value.

6. For VM detection: remove or spoof VMware Tools, use a non-default snapshot name, ensure a realistic process list (browser, office apps), and screen resolution ≥ 1024×768.

7. For connectivity checks: run INetSim or FakeNet-NG so the sample believes it has real network access and proceeds past its C2 check-in gate.

For Linux samples using ptrace-based self-detection:

catch syscall ptrace
commands
  silent
  set $rax = 0
  continue
end

This intercepts the ptrace(PTRACE_TRACEME) call the sample uses to detect a debugger and forces it to return success.

See Anti-Analysis Techniques for a full taxonomy of evasion primitives and their detection signatures.

Runtime Unpacking

Most malware in the wild is packed or encrypted. The executable you receive does not contain the real code — a small stub decrypts and loads it at runtime. The goal of runtime unpacking is to capture the decrypted payload from memory.

How Packing Works

Disk:
  [ packer stub ][ encrypted payload ]
                        ↓ run
Memory (after stub executes):
  [ packer stub ][ decrypted PE in new allocation ]
  EIP → OEP (Original Entry Point of the real malware)

The stub decrypts the payload, often calls VirtualAlloc + WriteProcessMemory or maps a new region with PROT_EXEC, then jumps to the Original Entry Point (OEP) of the real code.

Identifying a Packed Binary

Before hunting the OEP, confirm the binary is actually packed:

• Very few imports (typically only VirtualAlloc, LoadLibrary, GetProcAddress)
• High-entropy sections: binwalk -E sample.exe or diec sample.exe
• Section names like .UPX0, UPX1, .packed, or random strings
• On-disk size much smaller than reported virtual size

Finding the OEP

Method 1 — Hardware breakpoint on VirtualProtect/mprotect:

Set a breakpoint on the call that makes a memory region executable. When it fires, the decrypted payload is in the just-protected region. Single-step until you reach a jmp eax/jmp [reg] or an unusual-looking PUSH sequence (common OEP patterns).

break mprotect
commands
  silent
  printf "mprotect addr=%p len=%lu prot=%d\n", $rdi, $rsi, $rdx
  continue
end

Method 2 — Memory access breakpoints (watch for self-writing):

In x64dbg: right-click on a memory region → “Hardware, on access”. The debugger breaks the instant any instruction reads from or writes to that page — catching the decryptor as it fills in the real code.

Method 3 — ESP trick (Windows, simple packers):

At the packer entry point, note the value of ESP. Set a hardware breakpoint on that memory address. When the packer finishes and returns to the OEP, it will restore the stack to approximately its original value, triggering your watchpoint.

Dumping the Unpacked Image

Linux:

# While the process is paused at OEP, dump its full memory map
gdb -p <pid> -batch -ex "dump memory /tmp/unpacked.bin 0x400000 0x410000"

# Or use /proc
cp /proc/<pid>/exe /tmp/unpacked_exe
# Read the maps to find the correct region
cat /proc/<pid>/maps

Windows — Scylla (via x64dbg plugin):

1. Run to OEP.
2. Plugins → Scylla → “IAT Autosearch” (finds the Import Address Table of the unpacked binary).
3. “Get Imports” — resolves all imports.
4. “Dump” — writes the reconstructed PE to disk.
5. “Fix Dump” — patches the dumped file’s IAT so it loads correctly.

The result is a reconstructable PE that IDA or Ghidra can analyze statically.

Dynamic Instrumentation

Dynamic instrumentation inserts your analysis code into a running process without modifying the binary on disk. It works on packed samples, encrypted samples, and samples that detect debuggers.

API Monitor (Windows)

API Monitor intercepts thousands of Windows API functions and logs every call with arguments and return values — without a debugger attached.

1. Start API Monitor as administrator.
2. Select API categories to monitor. For malware: Crypto, Networking, Process/Thread, File I/O, Registry.
3. Monitor New Process: enter the path and arguments, or attach to a running PID.
4. Filter by API name or return value to isolate anomalous calls.

Key APIs to watch: VirtualAllocEx, WriteProcessMemory, CreateRemoteThread, NtUnmapViewOfSection, RegSetValueEx, WinExec, ShellExecute, CryptEncrypt, InternetConnect.

Frida

Frida injects a JavaScript engine into the target process. You write hooks in JavaScript (or Python), attach from the command line, and intercept calls in real time. It runs on Windows, Linux, macOS, Android, and iOS — no recompile needed.

Install:

pip install frida-tools

Intercept connect() on Linux:

// hook_connect.js
Interceptor.attach(Module.getExportByName(null, "connect"), {
    onEnter(args) {
        const sockfd  = args[0].toInt32();
        const sockaddr = args[1];
        const family  = sockaddr.readU16();
        if (family === 2) {   // AF_INET
            // sin_port is big-endian (network byte order); read byte-by-byte
            const port = (sockaddr.add(2).readU8() << 8) | sockaddr.add(3).readU8();
            const a = sockaddr.add(4);
            const ip = `${a.readU8()}.${a.add(1).readU8()}.${a.add(2).readU8()}.${a.add(3).readU8()}`;
            console.log(`[connect] ${ip}:${port}`);
        }
    }
});

frida -l hook_connect.js -f ./sample --no-pause
# Attach to running process by name
frida -l hook_connect.js -n sample

Intercept a function by absolute address:

Interceptor.attach(ptr("0x7f1234005678"), {
    onEnter(args) {
        console.log("arg0 = " + args[0]);
    },
    onLeave(retval) {
        console.log("returned " + retval);
    }
});

Force a function to return a fixed value (anti-debug bypass):

const IsDebuggerPresent = Module.getExportByName("kernel32.dll", "IsDebuggerPresent");
Interceptor.replace(IsDebuggerPresent, new NativeCallback(() => 0, "int", []));

Scripts can be loaded from a file (frida -p <PID> -l script.js) and reloaded live, making bypass iteration much faster than patching and rerunning.

Dump a decrypted buffer when a function returns:

// Intercept a custom decrypt() function and print its output
const base  = Module.getBaseAddress("sample");
const decryptFn = base.add(0x1234);   // RVA of decrypt routine

Interceptor.attach(decryptFn, {
    onLeave(retval) {
        // assume return value is a pointer to the decrypted buffer
        // and it's null-terminated
        const s = retval.readUtf8String();
        console.log("[decrypt output]", s);
    }
});

Stalker — execution tracing:

// Trace every basic block the target executes
Stalker.follow(Process.getCurrentThreadId(), {
    events: { block: true },
    onReceive(events) {
        const blocks = Stalker.parse(events, { annotate: true, stringify: true });
        for (const b of blocks)
            console.log(b);
    }
});

Stalker output can be fed directly into tools like Lighthouse (a coverage visualization plugin for IDA/Ghidra).

PIN (Intel)

PIN is a dynamic binary instrumentation framework from Intel. It is more powerful than Frida for fine-grained x86/x64 analysis but requires writing C++ Pintool plugins.

Count instruction executions by address:

pin -t source/tools/ManualExamples/obj-intel64/inscount0.so -- ./sample
cat inscount.out

A common malware use case: write a Pintool that records every unique address executed (coverage), then compare coverage between runs to see what new code paths the malware takes after a given input.

DynamoRIO

DynamoRIO is an open-source DBI framework. The drcov client collects basic-block coverage in a format readable by the Lighthouse plugin for IDA Pro and Ghidra:

drrun -t drcov -- ./sample arg
ls *.log    # drcov.sample.<pid>.0000.proc.log

Coverage diffing workflow:

1. drrun -t drcov -- ./sample benign_input → save as run_benign.log
2. drrun -t drcov -- ./sample trigger_input → save as run_trigger.log
3. In IDA: File → Load file → Code coverage file (Lighthouse) for each log
4. Use Lighthouse’s Coverage Overview to subtract: find basic blocks in run_trigger but not in run_benign — these are newly activated paths
5. Set a breakpoint at the first newly covered block and rerun with the triggering input to inspect context

Unexplored branches are often where additional capability is hidden — triggered only by specific C2 responses, time conditions, or configuration values.

DBI limitations: DBI imposes 10–100× execution overhead and can break self-modifying code, packed stubs, and samples that detect instrumentation via timing or explicit checks. Apply DBI after unpacking and defeating major anti-analysis checks.

Kernel-Level Monitoring

Userspace tools miss kernel-level malware behavior (rootkits, driver-based persistence). Two complementary approaches cover Linux and Windows.

eBPF (Linux)

eBPF programs attach to kernel tracepoints and perf events without modifying kernel source. Unlike strace, eBPF has near-zero overhead, can filter in-kernel before surfacing events, and survives against malware that detects ptrace.

Install bpftrace:

sudo apt install bpftrace

Trace all execve calls system-wide:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s exec: ", comm); join(args->argv); }'

Trace file opens by a specific PID:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat /pid == $1/ { printf("open: %s\n", str(args->filename)); }' -- <pid>

Watch for mprotect calls that add PROT_EXEC:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_mprotect
/args->prot & 4/    # PROT_EXEC = 0x4
{
    printf("[%s pid=%d] mprotect addr=0x%lx len=%lu prot=0x%x\n",
           comm, pid, args->start, args->len, args->prot);
}'

This catches memory-based unpacking and shellcode injection without touching the process at all.

Caveat on connect address decoding: args->uservaddr in sys_enter_connect is a user-space pointer — dereferencing it directly in a bpftrace tracepoint produces unreliable results. To decode destination IP/port, use strace -e connect -p <PID> or intercept connect() with Frida in user context instead.

Full syscall argument logging with bcc:

# trace_malware.py — using BCC Python bindings
from bcc import BPF

prog = r"""
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

TRACEPOINT_PROBE(syscalls, sys_enter_connect) {
    char comm[TASK_COMM_LEN];
    bpf_get_current_comm(&comm, sizeof(comm));
    bpf_trace_printk("connect: %s fd=%d\n", comm, args->fd);
    return 0;
}
"""
b = BPF(text=prog)
b.trace_print()

Sysmon + ETW (Windows)

ETW (Event Tracing for Windows) is the kernel’s built-in structured event bus. Every driver, OS component, and many applications publish events to ETW providers. Sysmon subscribes to the relevant providers and writes them to the Windows Event Log in a parsed, queryable format.

Install Sysmon with a comprehensive config:

# Recommended config from SwiftOnSecurity or Olaf Hartong
.\Sysmon64.exe -accepteula -i sysmonconfig.xml

Key Sysmon event IDs for malware:

Query Sysmon logs with PowerShell:

Get-WinEvent -LogName "Microsoft-Windows-Sysmon/Operational" |
    Where-Object { $_.Id -eq 3 } |
    Select-Object -First 20 |
    Format-List

Direct ETW consumption (without Sysmon):

Tools like SilkETW and krabsetw let you subscribe to arbitrary ETW providers in real time. This is harder to evade than Sysmon because you choose the providers yourself:

# SilkETW: capture Microsoft-Windows-Kernel-Process provider
.\SilkETW.exe -t kernel -pn Microsoft-Windows-Kernel-Process `
    -ot file -p C:\analysis\etw_out.json

Detecting Process Injection at Runtime

Process injection techniques (DLL injection, process hollowing, reflective loading) are among the most common malware behaviors. Dynamic detection differs from static detection: you look at the live state of a process’s memory rather than its on-disk binary.

Comparing Loaded DLLs Against Disk

An injected DLL may appear in the process’s module list but not on disk (reflective loading), or may appear at a path that does not match its image (DLL hijacking).

Windows — compare loaded modules to disk hashes:

$pid = (Get-Process notepad).Id
$modules = (Get-Process -Id $pid).Modules
foreach ($m in $modules) {
    $hash = (Get-FileHash $m.FileName -Algorithm SHA256 -ErrorAction SilentlyContinue).Hash
    Write-Host "$($m.ModuleName)  $($m.FileName)  $hash"
}

Flag any module where FileName is in %TEMP%, %APPDATA%, or a non-standard path.

Linux — check for anonymous executable mappings:

# Print memory mappings that are executable but have no backing file
awk '$2 ~ /x/ && $6 == ""' /proc/<pid>/maps

A legitimate process rarely has anonymous executable pages. Any result here warrants investigation — it may be shellcode, a reflectively loaded ELF, or an unpacked payload.

Thread Hijacking

The malware suspends a thread in the target process, rewrites its CONTEXT structure to redirect RIP/EIP to injected shellcode, then resumes it.

Detection:

• Break on SetThreadContext in API Monitor
• In WinDbg: !thread on all threads of a suspicious process — check whether Rip points outside any known module

DLL Injection

Classic: the injector calls WriteProcessMemory to write a DLL path into the target, then CreateRemoteThread(LoadLibraryA).

Reflective: the DLL carries its own loader; CreateRemoteThread points directly into the injected buffer — no LoadLibraryA call, the DLL does not appear in the normal module list.

Detection:

• WriteProcessMemory → CreateRemoteThread pair in API Monitor
• For reflective DLL: VirtualAllocEx with EXECUTE_READWRITE → WriteProcessMemory → CreateRemoteThread to an address outside any module

Process Doppelgänging

Uses Windows NTFS transactions (TxF) to write malicious code to a transacted file that is never committed to disk, then create a process image section from it. The process appears legitimate because it references the phantom transacted file. MITRE: T1055.013.

Detection: Catch the NtCreateTransaction → NtCreateSection → NtCreateProcessEx call sequence in API Monitor or kernel callbacks.

Detecting Hollowing

Process hollowing creates a new process in suspended state, unmaps its original image, and writes malicious code into the vacated address space. The tell-tale sign is an executable page at the expected image base address that does not match the on-disk binary.

Windows — check image base vs. disk:

# Read the PEB's ImageBaseAddress and compare with the on-disk PE
$proc = Get-Process -Name svchost | Select-Object -First 1
$addr = [System.Diagnostics.Process]::GetProcessById($proc.Id).MainModule.BaseAddress
# Compare with what's at that address using a tool like PE-bear or Moneta

Moneta is the current standard tool for this class of detection:

Moneta64.exe -m ioc -p <pid>

It identifies: modified PE headers, unsigned code, PE files mapped from non-disk paths, executable sections with no backing image, and regions with RWX permissions.

Other Injection Variants

MITRE: T1055 covers the full injection family.

Hunting Injected Shellcode with Volatility

If you have a memory image taken mid-execution (or from a hypervisor snapshot), Volatility’s malfind plugin automates the hunt:

vol -f memdump.raw windows.malfind

It flags every executable, non-module-backed memory region and shows the first bytes — look for the 4D 5A (MZ) header that indicates an injected PE, or \x55\x48\x89\xe5 (x64 function prologue) indicating shellcode.

Full Volatility 3 triage:

vol -f memory.raw windows.pslist    # walks ActiveProcessLinks
vol -f memory.raw windows.psscan    # pool-tag scan — finds DKOM-hidden processes
vol -f memory.raw windows.malfind   # executable anonymous VAD regions
vol -f memory.raw windows.dlllist   # loaded modules per process
vol -f memory.raw windows.handles   # open handles (mutexes, named pipes, files)
vol -f memory.raw windows.netscan   # network connections including recently closed

Cross-referencing pslist against psscan is the fastest way to identify DKOM-hidden processes: any entry in psscan absent from pslist has been unlinked from the active process list.

Memory Forensics During Execution

You do not have to wait until after the fact to apply memory forensics techniques. Taking a snapshot of a VM mid-execution and analyzing the snapshot offline is a powerful combination.

Capturing a Live Memory Image

Linux:

sudo avml /tmp/memdump.lime
# or with LiME kernel module
sudo insmod lime-$(uname -r).ko "path=/tmp/memdump.lime format=lime"

Windows:

# WinPmem
winpmem_mini_x64_rc2.exe memdump.raw

Extracting Decrypted Strings and Config

Many malware families decrypt their C2 configuration (IPs, domains, beacon interval, mutex names) at runtime and store it in a heap allocation. Once you’ve identified the allocation — via a breakpoint on the decryption return, or via search in pwndbg — dump it:

# In GDB: dump 4096 bytes starting at a heap address
dump memory /tmp/config_block.bin 0x55a3b2c0d000 0x55a3b2c0e000
# Inspect for strings
strings -n 8 /tmp/config_block.bin

Carving PE Files from a Process Dump

# Use foremost against a process memory dump to find embedded PEs
foremost -t exe -i memdump.raw -o carved_pes/

# Or bulk_extractor for broader artifact recovery
bulk_extractor -o output/ memdump.raw

Correlating Dynamic and Static Findings

Advanced dynamic analysis is most powerful when cross-referenced against static findings:

Behavioral Correlation with MITRE ATT&CK

Dynamic analysis output maps directly to ATT&CK techniques. Build a table as you work:

The MITRE ATT&CK Navigator lets you highlight techniques as you find them, producing a heat-map for the final report.

Workflow Summary

A structured advanced dynamic analysis session:

1. Revert to clean snapshot; start monitoring (Sysmon/bpftrace/Wireshark) before running the sample.
2. Run the sample under a debugger with anti-anti-debugging bypass (pwndbg catch syscall ptrace + return 0 / ScyllaHide on Windows).
3. Break on VirtualAlloc/mprotect with EXEC — observe packing/decryption.
4. At OEP, dump the unpacked image (Scylla/GDB dump memory).
5. Attach Frida to log API calls of interest; use Stalker if code coverage is needed.
6. Take a VM snapshot at peak activity for offline memory forensics.
7. Run Moneta/malfind against the snapshot to enumerate injected regions.
8. Correlate every observation to an ATT&CK technique.
9. Restore snapshot; repeat with network simulation (INetSim) enabled.

Tools Quick Reference

Further Reading

• M. Sikorski & A. Honig. (2012). Practical Malware Analysis, Ch. 9–11. No Starch Press.
• M. Ligh et al. (2014). The Art of Memory Forensics. Wiley.
• Frida documentation — JavaScript API reference
• Intel PIN user guide
• DynamoRIO drcov documentation
• Lighthouse plugin — code coverage visualization for IDA/Ghidra
• bpftrace reference guide
• Sysmon configuration guide
• MITRE ATT&CK T1055 — Process Injection (all sub-techniques)
• MITRE ATT&CK Navigator
• Moneta — live process memory scanner
• al-khaser — anti-analysis technique PoC collection