courses

Advanced Dynamic Analysis

This page picks up where basic dynamic analysis leaves off. The foundation — sandboxes, strace, network capture — reveals what a sample does. Advanced dynamic analysis answers how and why: it lets you control execution, intercept at arbitrary points, defeat anti-analysis tricks, and extract artifacts that never exist on disk. It requires a working debugger and an isolated analysis VM with a clean snapshot.


Debugger-Based Analysis

A debugger gives you the ability to pause execution at any point, inspect registers and memory, modify values, and step through individual instructions. For malware analysis, this is the primary tool for understanding obfuscated logic, decryption routines, and conditional C2 check-ins.

Choosing a Debugger

Target Debugger Notes
Linux ELF (user-mode) GDB + pwndbg pwndbg adds a context pane, heap inspection, memory search, and cyclic patterns
Linux ELF (kernel module) KGDB / GDB over serial Requires a second VM as the debug host
Windows PE (user-mode) x64dbg Open source; best plugin ecosystem for malware work
Windows PE (kernel) WinDbg Required for rootkit/driver analysis; TTD support
Cross-platform scripting GDB + Python / x64dbg + xdbg64 script Both support full scripting APIs

GDB + pwndbg for Linux Samples

The malware-env playbook already installs pwndbg and deploys a ~/.gdbinit that loads it, so on a course VM gdb comes up with the pwndbg prompt and you can skip this.

To install it by hand elsewhere, use upstream’s install script — this is the method pwndbg currently recommends:

curl --proto '=https' --tlsv1.2 -LsSf 'https://install.pwndbg.re' | sh -s -- -t pwndbg-gdb

Add -u to install for your user only, without root. The older git clone plus ./setup.sh route still works but is documented upstream as the legacy method.

Start or attach:

gdb --args ./sample arg1 arg2   # run directly with arguments
gdb -p $(pgrep sample)          # attach to a running process

Essential GDB commands:

Command Effect
b *0xdeadbeef Breakpoint at an absolute address
b mmap Breakpoint on the mmap PLT entry
run / r Start execution
continue / c Resume after a break
ni / si Step over / step into (one instruction)
x/32xb $rsp Examine 32 bytes at RSP as hex
x/10i $rip Disassemble 10 instructions from RIP
info proc mappings Print all mapped memory regions
dump binary memory out.bin 0x400000 0x401000 Dump a memory region to a file
set $rip = 0x40107a Force-redirect execution
set *(char*)0x601020 = 0 Patch a byte in memory

pwndbg enhancements:

Command Effect
context Register, stack, disassembly, and backtrace pane
vmmap Color-coded virtual memory layout
telescope $rsp 20 Dereference pointer chain for 20 words from RSP
search -s "http" Scan mapped memory for a string
search -x 4d5a Scan for MZ headers (injected PEs)
cyclic 200 / cyclic -l $rsp Generate / find offset in a De Bruijn pattern
heap / bins Heap chunk list and bin contents (glibc malloc)

Filtering command output: pipe

GDB can send any command’s output into a shell pipeline, which matters as soon as you are looking at a real process — info functions on a dynamically linked sample runs to hundreds of lines, and vmmap on a browser to hundreds more. pipe, or | for short:

(gdb) pipe vmmap | grep rwx                    # writable AND executable only
(gdb) pipe info functions | grep -c "^0x"      # how many symbols are there
(gdb) | search -x 4d5a | wc -l                 # count MZ headers in memory
(gdb) pipe info proc mappings | grep -E "rwxp" # the same, without pwndbg

It captures pwndbg’s output as well as gdb’s own, so context, vmmap, telescope, and search all pipe. Spaces around the | are optional.

Two forms worth knowing:

Form Use
pipe \| SHELL_COMMAND Repeat the last command and filter its output — interactive sessions only, since a script has no command history
pipe -d DELIM COMMAND DELIM SHELL_COMMAND When the gdb command itself contains a \|, e.g. pipe -d ! print 5 \| 2 ! grep .

$_shell_exitcode holds the pipeline’s exit status, so a script can branch on whether the filter matched anything:

(gdb) pipe vmmap | grep -q rwx
(gdb) print $_shell_exitcode
$1 = 0

⚠️ Colour escapes reach grep. pwndbg’s start before the leading whitespace on each line, so an anchored pattern silently matches nothing — pipe vmmap | grep -c "^ *0x" returns 0 on a process with two dozen mappings. It is not an error, just an empty result. Fix it with set disable-colors on (pwndbg), set style enabled off (gdb’s own styling), or by stripping the escapes in the pipeline itself: pipe vmmap | sed 's/\x1b\[[0-9;]*m//g' | grep "^ *0x".

Where a filter is all you need, several info commands take a regular expression directly and produce no colour to fight — info functions ^decrypt, info variables, info types, and info registers rsp rip for a named subset.

pipe, |, and $_shell_exitcode arrived in GDB 9.1 (February 2020). On anything older — Ubuntu 18.04 shipped GDB 8.1 — the fallback is set logging file /tmp/gdb.log, set logging enabled on, then grep the file afterwards. Note that those are subcommands as of GDB 11; the older spelling was set logging on.

Breakpoint strategy for unpacking

Packed samples call mmap or mprotect to create executable memory, then transfer control there. Set breakpoints on both and inspect what was written before execution reaches the new region:

(gdb) catch syscall mmap mprotect
(gdb) run
# when stopped at mprotect with PROT_EXEC:
(gdb) finish    # let the call complete
(gdb) vmmap     # find the new executable region
(gdb) x/10i <new_region_addr>

Conditionals and mid-session patches:

# Break only when a register holds a specific value
break *0x401234 if $rdi == 0x0

# Patch a JZ to NOP+NOP so a branch is always taken
set *(short*)0x401a30 = 0x9090

Multi-Pane Layouts: GDB TUI

By default pwndbg reprints its whole context block after every si, and a long unpacking loop scrolls the thing you were reading off the top of the terminal. Both fixes below pin each section to a fixed region of the screen so disassembly, registers, and the stack stay put while you step. The TUI costs nothing to set up and works over a plain SSH session; splitmind needs tmux but gives you real panes.

pwndbg registers every context section as a native GDB TUI window called pwndbg_<section>, and ships two ready-made layouts:

(gdb) layout pwndbg        # disassembly-led
(gdb) layout pwndbg_code   # source/decompiler pane beside the disassembly

The windows available to a layout (pwndbg 2026.09; the list is generated from pwndbg’s registered context sections, so it grows as upstream adds them):

Window Content
pwndbg_disasm Disassembly around RIP
pwndbg_code Source lines, when the binary has them
pwndbg_regs Register values, with dereferences
pwndbg_stack Stack words, telescoped
pwndbg_backtrace Call stack
pwndbg_args Arguments of the current function
pwndbg_expressions Watched expressions (contextwatch)
pwndbg_threads Thread list
pwndbg_heap_tracker glibc heap activity
pwndbg_last_signal Most recent signal — useful on a sample that traps deliberately
pwndbg_legend The colour key for the panes
pwndbg_control pwndbg’s own control pane
cmd, status GDB’s command window and status line

Moving around once you are in it:

Action Effect
focus cmd Put the arrow keys back on command history instead of scrolling a pane
focus pwndbg_stack Point the arrow keys at a named pane to scroll it
tui disable Drop back to the plain CLI, for a command whose output is long
ctrl-x then a Toggle TUI/CLI without typing a command
hold Shift while dragging Bypass TUI mouse handling so your terminal’s select-and-copy works

Worked example — a custom layout. Layouts are ordinary GDB tui new-layout definitions, so the quickest route to your own is to start from the one pwndbg ships. This is pwndbg_code with the heap tracker dropped and the stack given more room, which is roughly what you want when stepping an unpacker that is building code on the heap:

tui new-layout malware \
  {-horizontal \
    { \
      { pwndbg_code 1 pwndbg_disasm 1 } 2 \
      { \
        { -horizontal pwndbg_legend 8 pwndbg_control 2 } 0 pwndbg_regs 1 pwndbg_stack 2 \
      } 3 \
    } 7 cmd 3 \
  } 3 { pwndbg_backtrace 2 pwndbg_expressions 2 } 1 status 1
(gdb) layout malware

The numbers are relative weights, not rows: pwndbg_stack 2 beside pwndbg_regs 1 gives the stack twice the height. Put the tui new-layout line in ~/.gdbinit (see the caveat about bare set lines below — tui new-layout is a real GDB command and is safe unguarded) and the layout is there every session.

TUI windows need a GDB built with the Python window API, which is GDB 10 and later. Check before you debug why nothing appears:

gdb -q -nx -ex 'python print(hasattr(gdb, "register_window_type"))' -ex quit

If that prints False, pwndbg silently registers no windows and layout pwndbg will fail — use splitmind instead.

Multi-Pane Layouts: splitmind + tmux

splitmind takes the same context sections and sends each one to its own tmux pane, by handing pwndbg a separate tty per section. Panes are resizable, independently scrollable, and selectable with the mouse, and you can give the debuggee its own pane so its output never interleaves with pwndbg’s. It supports only pwndbg as the information provider and only tmux as the splitter.

git clone https://github.com/jerdna-regeiz/splitmind ~/clones/splitmind

The malware-env playbook expects the clone at exactly that path — it deploys its gdbinit.py into the clone and prints a SKIP splitmind not cloned line if you have not cloned it first. On a course VM, clone it, then re-run the deploy script.

Worked example. This is the layout the course VM deploys: disassembly and code sharing the top left, registers over the stack on the right, a pane for the debuggee’s own input and output, and a Python REPL under the backtrace.

python
import splitmind
(splitmind.Mind()
  .tell_splitter(show_titles=True)
  .tell_splitter(set_title="Main")
  .right(display="backtrace", size="25%")
  .above(of="main", display="disasm", size="80%", banner="top")
  .show("code", on="disasm", banner="none")
  .right(cmd='tty; tail -f /dev/null', size="65%", clearing=False)
  .tell_splitter(set_title='Input / Output')
  .above(display="stack", size="75%")
  .above(display="legend", size="25")
  .show("regs", on="legend")
  .below(of="backtrace", cmd="ipython3", size="30%")
).build(nobanner=True)
end

Each split is created relative to the previous one unless you name a different parent with of=. show() adds a second section to a pane that already exists instead of splitting again — that is how code lands on top of disasm and regs on top of legend. A split with cmd= and no display= is just a pane running a command, which is how the I/O and REPL panes get there.

Inside tmux, start pwndbg and confirm the panes came up:

$ tmux list-panes -F '#{pane_index} #{pane_title} #{pane_width}x#{pane_height}'
0 disasm, code 57x46
1 legend, regs 106x24
2 stack 106x8
3 Input / Output 106x12
4 Main 164x12
5 backtrace 55x59

Six panes with those titles means splitmind bound every section. If you get one pane and a normal pwndbg prompt, the layout never ran — check the gotchas below.

To route the sample’s own stdin/stdout into the Input / Output pane, read the tty that pane prints at startup and tell GDB to use it:

(gdb) tty /dev/pts/7

GDB warns about the redirection; ignore it. This matters for malware work more than it sounds: a sample that prints a prompt or reads from stdin becomes unreadable when its output is interleaved with a context redraw after every step.

Gotchas, all of which produce confusing rather than obvious failures

Which to use: the TUI if you are on an unfamiliar box, over SSH, or inside a VM console where tmux is not set up — it is built in and needs no clone. splitmind once a layout is worth keeping: real panes you can resize and copy from, and a dedicated pane for the sample’s own I/O.

Both depend on pwndbg’s context-section names and parameter names, and both drift between pwndbg releases — the renamed context-source-code-lines above is one example. If a layout that used to work comes up empty after an upgrade, check the section names with show context-sections and pwndbg [filter] before rewriting the config.

x64dbg for Windows Samples

x64dbg is the standard open-source Windows malware debugger. Core workflow:

  1. Load the sample: File → Open. Let the loader place it.
  2. Set the entry-point breakpoint: it pauses at the PE entry point automatically.
  3. Pause on API calls: Right-click in the Symbols pane → “Set breakpoint on all calls to [function]”
  4. Search for strings: Right-click in the disassembly → Search for → All Referenced Strings

Keyboard shortcuts:

Shortcut Action
F2 Toggle breakpoint on selected instruction
F7 / F8 Step into / step over
F9 Run
Ctrl+G Go to address or symbol
Ctrl+F Find pattern in current region
Alt+M Memory map window
Ctrl+B Set hardware breakpoint on memory access

Essential plugins:

Plugin Purpose
ScyllaHide Transparent anti-anti-debugging; patches NtQueryInformationProcess, timing, and PEB checks
Scylla PE dump + import reconstruction for packed samples
xAnalyzer Annotates call arguments inline in the disassembly
ret-sync Synchronizes a debugger (x64dbg, WinDbg, GDB) with Ghidra, IDA, or Binary Ninja in real time

WinDbg for Kernel-Mode Malware

For rootkits and malicious drivers, WinDbg over a kernel debug connection is required. Set up kdnet between the analysis VM and a debugger host:

On the VM (run once, then reboot):

bcdedit /dbgsettings net hostip:<debugger_host_ip> port:50000
bcdedit /debug on

On the debugger host:

windbg -k net:port=50000,key=<key>

Essential kernel commands:

Command Effect
!process 0 0 List all processes (walks ActiveProcessLinks)
!process <addr> 7 Detailed process info including threads
!drvobj \Driver\<name> Inspect a driver object
!devobj <addr> Inspect a device object
bp <module>!<function> Breakpoint on an exported symbol
dt nt!_EPROCESS <addr> Overlay EPROCESS structure at address
lm List loaded modules
!idt Dump the Interrupt Descriptor Table
eb @$peb+2 0 Clear the BeingDebugged byte in PEB

To detect DKOM-hidden processes, compare !process 0 0 (linked-list walk) against a pool-tag scan in Volatility — processes hidden from the active list will still appear in windows.psscan.

Scripting the Debugger

Both debuggers support Python scripting for automated analysis:

GDB + pwndbg/Python — log all connect() calls

import gdb

class ConnectBreak(gdb.Breakpoint):
    def stop(self):
        rdi = int(gdb.parse_and_eval("$rdi"))   # sockfd
        rsi = int(gdb.parse_and_eval("$rsi"))   # struct sockaddr*
        # read the sockaddr from memory
        raw = gdb.selected_inferior().read_memory(rsi, 16)
        import socket, struct
        port = struct.unpack_from(">H", bytes(raw), 2)[0]
        addr = socket.inet_ntoa(bytes(raw)[4:8])
        print(f"[connect] {addr}:{port}")
        return False   # don't stop, just log

ConnectBreak("connect")

Load with source log_connect.py inside GDB.

x64dbg script — conditional NOP patch at runtime

// In the x64dbg scripting console
// Patch a JNZ at 0x401A30 to NOP+NOP (always fall through)
mov [0x401A30], 0x9090

Defeating Anti-Analysis in the Debugger

When a sample exits silently or behaves differently under a debugger, work through this checklist before investing time in deeper analysis:

  1. Enable ScyllaHide before loading the sample. It patches NtQueryInformationProcess, PEB flags, timing checks, and heap flags transparently.

  2. Patch IsDebuggerPresent to return 0 (x64dbg): find it in the Symbols pane → right-click → Patch → set return value to 0.

  3. Clear the PEB BeingDebugged byte (offset +0x2 from PEB base):
    • x64dbg: Memory Map → navigate to PEB → offset +2 → Binary Edit → 00
    • WinDbg: eb @$peb+2 0
    • Frida: Interceptor.replace(Module.getExportByName("kernel32.dll", "IsDebuggerPresent"), new NativeCallback(() => 0, "int", []))
  4. Suppress OutputDebugString anti-debug: this technique raises exception 0x40010006 to detect a live debugger. In x64dbg: Options → Preferences → Exceptions → add 0x40010006 to the pass-through list.

  5. For timing checks (RDTSC, GetTickCount, QueryPerformanceCounter): set a breakpoint immediately after the call and manually zero the elapsed value.

  6. For VM detection: remove or spoof VMware Tools, use a non-default snapshot name, ensure a realistic process list (browser, office apps), and screen resolution ≥ 1024×768.

  7. For connectivity checks: run INetSim or FakeNet-NG so the sample believes it has real network access and proceeds past its C2 check-in gate.

For Linux samples using ptrace-based self-detection:

catch syscall ptrace
commands
  silent
  set $rax = 0
  continue
end

This intercepts the ptrace(PTRACE_TRACEME) call the sample uses to detect a debugger and forces it to return success.

See Anti-Analysis Techniques for a full taxonomy of evasion primitives and their detection signatures.


Runtime Unpacking

Most malware in the wild is packed or encrypted. The executable you receive does not contain the real code — a small stub decrypts and loads it at runtime. The goal of runtime unpacking is to capture the decrypted payload from memory.

How Packing Works

Disk:
  [ packer stub ][ encrypted payload ]
                        ↓ run
Memory (after stub executes):
  [ packer stub ][ decrypted PE in new allocation ]
  EIP → OEP (Original Entry Point of the real malware)

The stub decrypts the payload, often calls VirtualAlloc + WriteProcessMemory or maps a new region with PROT_EXEC, then jumps to the Original Entry Point (OEP) of the real code.

Identifying a Packed Binary

Before hunting the OEP, confirm the binary is actually packed:

Finding the OEP

Method 1 — Hardware breakpoint on VirtualProtect/mprotect

Set a breakpoint on the call that makes a memory region executable. When it fires, the decrypted payload is in the just-protected region. Single-step until you reach a jmp eax/jmp [reg] or an unusual-looking PUSH sequence (common OEP patterns).

break mprotect
commands
  silent
  printf "mprotect addr=%p len=%lu prot=%d\n", $rdi, $rsi, $rdx
  continue
end

Method 2 — Memory access breakpoints (watch for self-writing)

In x64dbg: right-click on a memory region → “Hardware, on access”. The debugger breaks the instant any instruction reads from or writes to that page — catching the decryptor as it fills in the real code.

Method 3 — ESP trick (Windows, simple packers)

At the packer entry point, note the value of ESP. Set a hardware breakpoint on that memory address. When the packer finishes and returns to the OEP, it will restore the stack to approximately its original value, triggering your watchpoint.

Dumping the Unpacked Image

Linux

# While the process is paused at OEP, dump its full memory map
gdb -p <pid> -batch -ex "dump memory /tmp/unpacked.bin 0x400000 0x410000"

# Or use /proc
cp /proc/<pid>/exe /tmp/unpacked_exe
# Read the maps to find the correct region
cat /proc/<pid>/maps

Windows — Scylla (via x64dbg plugin)

  1. Run to OEP.
  2. Plugins → Scylla → “IAT Autosearch” (finds the Import Address Table of the unpacked binary).
  3. “Get Imports” — resolves all imports.
  4. “Dump” — writes the reconstructed PE to disk.
  5. “Fix Dump” — patches the dumped file’s IAT so it loads correctly.

The result is a reconstructable PE that Ghidra can analyze statically.


Dynamic Instrumentation

Dynamic instrumentation inserts your analysis code into a running process without modifying the binary on disk. It works on packed samples, encrypted samples, and samples that detect debuggers.

API Monitor (Windows)

API Monitor intercepts thousands of Windows API functions and logs every call with arguments and return values — without a debugger attached.

  1. Start API Monitor as administrator.
  2. Select API categories to monitor. For malware: Crypto, Networking, Process/Thread, File I/O, Registry.
  3. Monitor New Process: enter the path and arguments, or attach to a running PID.
  4. Filter by API name or return value to isolate anomalous calls.

Key APIs to watch: VirtualAllocEx, WriteProcessMemory, CreateRemoteThread, NtUnmapViewOfSection, RegSetValueEx, WinExec, ShellExecute, CryptEncrypt, InternetConnect.

Frida

Frida injects a JavaScript engine into the target process. You write hooks in JavaScript (or Python), attach from the command line, and intercept calls in real time. It runs on Windows, Linux, macOS, Android, and iOS — no recompile needed.

Install

pip install frida-tools

Intercept connect() on Linux

// hook_connect.js
Interceptor.attach(Module.getExportByName(null, "connect"), {
    onEnter(args) {
        const sockfd  = args[0].toInt32();
        const sockaddr = args[1];
        const family  = sockaddr.readU16();
        if (family === 2) {   // AF_INET
            // sin_port is big-endian (network byte order); read byte-by-byte
            const port = (sockaddr.add(2).readU8() << 8) | sockaddr.add(3).readU8();
            const a = sockaddr.add(4);
            const ip = `${a.readU8()}.${a.add(1).readU8()}.${a.add(2).readU8()}.${a.add(3).readU8()}`;
            console.log(`[connect] ${ip}:${port}`);
        }
    }
});
frida -l hook_connect.js -f ./sample --no-pause
# Attach to running process by name
frida -l hook_connect.js -n sample

Intercept a function by absolute address

Interceptor.attach(ptr("0x7f1234005678"), {
    onEnter(args) {
        console.log("arg0 = " + args[0]);
    },
    onLeave(retval) {
        console.log("returned " + retval);
    }
});

Force a function to return a fixed value (anti-debug bypass)

const IsDebuggerPresent = Module.getExportByName("kernel32.dll", "IsDebuggerPresent");
Interceptor.replace(IsDebuggerPresent, new NativeCallback(() => 0, "int", []));

Scripts can be loaded from a file (frida -p <PID> -l script.js) and reloaded live, making bypass iteration much faster than patching and rerunning.

Dump a decrypted buffer when a function returns

// Intercept a custom decrypt() function and print its output
const base  = Module.getBaseAddress("sample");
const decryptFn = base.add(0x1234);   // RVA of decrypt routine

Interceptor.attach(decryptFn, {
    onLeave(retval) {
        // assume return value is a pointer to the decrypted buffer
        // and it's null-terminated
        const s = retval.readUtf8String();
        console.log("[decrypt output]", s);
    }
});

Stalker — execution tracing

// Trace every basic block the target executes
Stalker.follow(Process.getCurrentThreadId(), {
    events: { block: true },
    onReceive(events) {
        const blocks = Stalker.parse(events, { annotate: true, stringify: true });
        for (const b of blocks)
            console.log(b);
    }
});

Stalker output can be fed directly into coverage visualizers — Cartographer for Ghidra, or Lighthouse for IDA Pro and Binary Ninja.

PIN (Intel)

PIN is a dynamic binary instrumentation framework from Intel. It is more powerful than Frida for fine-grained x86/x64 analysis but requires writing C++ Pintool plugins.

Count instruction executions by address

pin -t source/tools/ManualExamples/obj-intel64/inscount0.so -- ./sample
cat inscount.out

A common malware use case: write a Pintool that records every unique address executed (coverage), then compare coverage between runs to see what new code paths the malware takes after a given input.

DynamoRIO

DynamoRIO is an open-source DBI framework. The drcov client collects basic-block coverage in the DRCOV format, readable by Cartographer in Ghidra (and by Lighthouse in IDA Pro and Binary Ninja):

drrun -t drcov -- ./sample arg
ls *.log    # drcov.sample.<pid>.0000.proc.log

Coverage diffing workflow

  1. drrun -t drcov -- ./sample benign_input → save as run_benign.log
  2. drrun -t drcov -- ./sample trigger_input → save as run_trigger.log
  3. In Ghidra: load each log with Cartographer (Window → Code Coverage)
  4. Use Lighthouse’s Coverage Overview to subtract: find basic blocks in run_trigger but not in run_benign — these are newly activated paths
  5. Set a breakpoint at the first newly covered block and rerun with the triggering input to inspect context

Unexplored branches are often where additional capability is hidden — triggered only by specific C2 responses, time conditions, or configuration values.

DBI limitations: DBI imposes 10–100× execution overhead and can break self-modifying code, packed stubs, and samples that detect instrumentation via timing or explicit checks. Apply DBI after unpacking and defeating major anti-analysis checks.


Kernel-Level Monitoring

Userspace tools miss kernel-level malware behavior (rootkits, driver-based persistence). Two complementary approaches cover Linux and Windows.

eBPF (Linux)

eBPF programs attach to kernel tracepoints and perf events without modifying kernel source. Unlike strace, eBPF has near-zero overhead, can filter in-kernel before surfacing events, and survives against malware that detects ptrace.

Install bpftrace

sudo apt install bpftrace

Trace all execve calls system-wide

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s exec: ", comm); join(args->argv); }'

Trace file opens by a specific PID

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat /pid == $1/ { printf("open: %s\n", str(args->filename)); }' -- <pid>

Watch for mprotect calls that add PROT_EXEC

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_mprotect
/args->prot & 4/    # PROT_EXEC = 0x4
{
    printf("[%s pid=%d] mprotect addr=0x%lx len=%lu prot=0x%x\n",
           comm, pid, args->start, args->len, args->prot);
}'

This catches memory-based unpacking and shellcode injection without touching the process at all.

Caveat on connect address decoding: args->uservaddr in sys_enter_connect is a user-space pointer — dereferencing it directly in a bpftrace tracepoint produces unreliable results. To decode destination IP/port, use strace -e connect -p <PID> or intercept connect() with Frida in user context instead.

⚠️ strace -p on an already-running process needs ptrace permission. Most current distributions ship kernel.yama.ptrace_scope = 1, which allows attaching only to descendants of the tracer, so the command above fails:

$ strace -e connect -p 760334
strace: attach: ptrace(PTRACE_SEIZE, 760334): Operation not permitted
strace: sysctl kernel.yama.ptrace_scope = 1 may be restricting this attach

Either start the sample under strace in the first place — which needs no special privilege, because it is then a child — or lift the restriction inside the analysis VM with sudo sysctl -w kernel.yama.ptrace_scope=0. Lift it in the VM, never on your host.

Full syscall argument logging with bcc

# trace_malware.py — using BCC Python bindings
from bcc import BPF

prog = r"""
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

TRACEPOINT_PROBE(syscalls, sys_enter_connect) {
    char comm[TASK_COMM_LEN];
    bpf_get_current_comm(&comm, sizeof(comm));
    bpf_trace_printk("connect: %s fd=%d\n", comm, args->fd);
    return 0;
}
"""
b = BPF(text=prog)
b.trace_print()

Sysmon + ETW (Windows)

ETW (Event Tracing for Windows) is the kernel’s built-in structured event bus. Every driver, OS component, and many applications publish events to ETW providers. Sysmon subscribes to the relevant providers and writes them to the Windows Event Log in a parsed, queryable format.

Install Sysmon with a comprehensive config

# Recommended config from SwiftOnSecurity or Olaf Hartong
.\Sysmon64.exe -accepteula -i sysmonconfig.xml

Key Sysmon event IDs for malware

Event ID Description
1 Process Create (full command line, parent, hashes)
3 Network Connection (process, remote IP/port, DNS name)
7 Image Loaded (DLL loaded into a process)
8 CreateRemoteThread (classic injection indicator)
10 Process Access (OpenProcess — precursor to injection)
11 File Create
12/13 Registry Create/Set
17/18 Pipe Create/Connect (lateral movement)
22 DNS Query (process name, domain, result)
25 Process Tampering (hollowing, herpaderping detected)

Query Sysmon logs with PowerShell

Get-WinEvent -LogName "Microsoft-Windows-Sysmon/Operational" |
    Where-Object { $_.Id -eq 3 } |
    Select-Object -First 20 |
    Format-List

Direct ETW consumption (without Sysmon)

Tools like SilkETW and krabsetw let you subscribe to arbitrary ETW providers in real time. This is harder to evade than Sysmon because you choose the providers yourself:

# SilkETW: capture Microsoft-Windows-Kernel-Process provider
.\SilkETW.exe -t kernel -pn Microsoft-Windows-Kernel-Process `
    -ot file -p C:\analysis\etw_out.json

Detecting Process Injection at Runtime

Process injection techniques (DLL injection, process hollowing, reflective loading) are among the most common malware behaviors. Dynamic detection differs from static detection: you look at the live state of a process’s memory rather than its on-disk binary.

Comparing Loaded DLLs Against Disk

An injected DLL may appear in the process’s module list but not on disk (reflective loading), or may appear at a path that does not match its image (DLL hijacking).

Windows — compare loaded modules to disk hashes

$pid = (Get-Process notepad).Id
$modules = (Get-Process -Id $pid).Modules
foreach ($m in $modules) {
    $hash = (Get-FileHash $m.FileName -Algorithm SHA256 -ErrorAction SilentlyContinue).Hash
    Write-Host "$($m.ModuleName)  $($m.FileName)  $hash"
}

Flag any module where FileName is in %TEMP%, %APPDATA%, or a non-standard path.

Linux — check for anonymous executable mappings

# Print memory mappings that are executable but have no backing file
awk '$2 ~ /x/ && $6 == ""' /proc/<pid>/maps

A legitimate process rarely has anonymous executable pages. Any result here warrants investigation — it may be shellcode, a reflectively loaded ELF, or an unpacked payload.

Thread Hijacking

The malware suspends a thread in the target process, rewrites its CONTEXT structure to redirect RIP/EIP to injected shellcode, then resumes it.

Detection

DLL Injection

Classic: the injector calls WriteProcessMemory to write a DLL path into the target, then CreateRemoteThread(LoadLibraryA).

Reflective: the DLL carries its own loader; CreateRemoteThread points directly into the injected buffer — no LoadLibraryA call, the DLL does not appear in the normal module list.

Detection

Process Doppelgänging

Uses Windows NTFS transactions (TxF) to write malicious code to a transacted file that is never committed to disk, then create a process image section from it. The process appears legitimate because it references the phantom transacted file. MITRE: T1055.013.

Detection: Catch the NtCreateTransaction → NtCreateSection → NtCreateProcessEx call sequence in API Monitor or kernel callbacks.

Detecting Hollowing

Process hollowing creates a new process in suspended state, unmaps its original image, and writes malicious code into the vacated address space. The tell-tale sign is an executable page at the expected image base address that does not match the on-disk binary.

Windows — check image base vs. disk

# Read the PEB's ImageBaseAddress and compare with the on-disk PE
$proc = Get-Process -Name svchost | Select-Object -First 1
$addr = [System.Diagnostics.Process]::GetProcessById($proc.Id).MainModule.BaseAddress
# Compare with what's at that address using a tool like PE-bear or Moneta

Moneta is the current standard tool for this class of detection:

Moneta64.exe -m ioc -p <pid>

It identifies: modified PE headers, unsigned code, PE files mapped from non-disk paths, executable sections with no backing image, and regions with RWX permissions.

Other Injection Variants

Technique Key APIs / Mechanism Detection Cue
APC Injection / Early Bird QueueUserAPC or NtQueueApcThread into a suspended thread before it runs QueueUserAPC targeting a remote thread, paired with WriteProcessMemory
Manual Mapping Malware parses and maps a PE by hand — resolve imports, apply relocations — without LoadLibrary Executable anonymous VAD region with a PE header; no backing module in lm or dlllist
Section Mapping NtCreateSection + NtMapViewOfSection across process boundaries; bypasses WriteProcessMemory NtCreateSection/NtMapViewOfSection pair targeting a different process; Volatility handles
Process Herpaderping / Ghosting Create section from a malicious file, then modify or delete the file before process creation; AV scans the stale disk image Section object from a file subsequently modified or deleted; detected by kernel callbacks on file modification after section creation

MITRE: T1055 covers the full injection family.

Hunting Injected Shellcode with Volatility

If you have a memory image taken mid-execution (or from a hypervisor snapshot), Volatility’s malfind plugin automates the hunt:

vol -f memdump.raw windows.malfind

It flags every executable, non-module-backed memory region and shows the first bytes — look for the 4D 5A (MZ) header that indicates an injected PE, or \x55\x48\x89\xe5 (x64 function prologue) indicating shellcode.

Full Volatility 3 triage:

vol -f memory.raw windows.pslist    # walks ActiveProcessLinks
vol -f memory.raw windows.psscan    # pool-tag scan — finds DKOM-hidden processes
vol -f memory.raw windows.malfind   # executable anonymous VAD regions
vol -f memory.raw windows.dlllist   # loaded modules per process
vol -f memory.raw windows.handles   # open handles (mutexes, named pipes, files)
vol -f memory.raw windows.netscan   # network connections including recently closed

Cross-referencing pslist against psscan is the fastest way to identify DKOM-hidden processes: any entry in psscan absent from pslist has been unlinked from the active process list.


Memory Forensics During Execution

You do not have to wait until after the fact to apply memory forensics techniques. Taking a snapshot of a VM mid-execution and analyzing the snapshot offline is a powerful combination.

Capturing a Live Memory Image

Linux

sudo avml /tmp/memdump.lime
# or with LiME kernel module
sudo insmod lime-$(uname -r).ko "path=/tmp/memdump.lime format=lime"

Windows

# WinPmem
winpmem_mini_x64_rc2.exe memdump.raw

Extracting Decrypted Strings and Config

Many malware families decrypt their C2 configuration (IPs, domains, beacon interval, mutex names) at runtime and store it in a heap allocation. Once you’ve identified the allocation — via a breakpoint on the decryption return, or via search in pwndbg — dump it:

# In GDB: dump 4096 bytes starting at a heap address
dump memory /tmp/config_block.bin 0x55a3b2c0d000 0x55a3b2c0e000
# Inspect for strings
strings -n 8 /tmp/config_block.bin

Carving PE Files from a Process Dump

# Use foremost against a process memory dump to find embedded PEs
foremost -t exe -i memdump.raw -o carved_pes/

# Or bulk_extractor for broader artifact recovery
bulk_extractor -o output/ memdump.raw

Correlating Dynamic and Static Findings

Advanced dynamic analysis is most powerful when cross-referenced against static findings:

Finding Origin Implication
Encrypted buffer at a high-entropy data section address Static Break on first write there; collect the decryption key
LoadLibraryA and GetProcAddress as only imports Static Packed or shellcode-like; hunt OEP before any other analysis
CreateRemoteThread in IAT Static Expect injection; monitor with API Monitor; compare memory maps before and after
Module not in lm but executable pages visible Dynamic (WinDbg / Volatility) Reflective injection or manual mapping
NtUnmapViewOfSection in API trace Dynamic Process hollowing; compare target process on-disk vs. in-memory image
Divergent execution path at a basic block Dynamic (Lighthouse coverage) Untriggered branch; examine condition with debugger and force it

Behavioral Correlation with MITRE ATT&CK

Dynamic analysis output maps directly to ATT&CK techniques. Build a table as you work:

Observed behavior ATT&CK technique
CreateRemoteThread into explorer.exe T1055.001 — Process Injection: DLL Injection
Anonymous RWX memory region T1055.004 — Asynchronous Procedure Call
Process image hollowing detected by Moneta T1055.012 — Process Hollowing
mprotect(PROT_EXEC) on heap region T1620 — Reflective Code Loading
Beaconing to hard-coded IP on port 443, self-signed cert T1071.001 — Application Layer Protocol: Web Protocols
DGA-like hostnames in DNS queries T1568.002 — Dynamic Resolution: DGA
reg add HKCU\...\Run T1547.001 — Boot or Logon Autostart: Registry Run Keys
Writes to %APPDATA%\Microsoft\Windows\Start Menu\Startup T1547.001
Elevated ptrace call on another process T1055 — Process Injection (general)

The MITRE ATT&CK Navigator lets you highlight techniques as you find them, producing a heat-map for the final report.


Workflow Summary

A structured advanced dynamic analysis session:

  1. Revert to clean snapshot; start monitoring (Sysmon/bpftrace/Wireshark) before running the sample.
  2. Run the sample under a debugger with anti-anti-debugging bypass (pwndbg catch syscall ptrace + return 0 / ScyllaHide on Windows).
  3. Break on VirtualAlloc/mprotect with EXEC — observe packing/decryption.
  4. At OEP, dump the unpacked image (Scylla/GDB dump memory).
  5. Attach Frida to log API calls of interest; use Stalker if code coverage is needed.
  6. Take a VM snapshot at peak activity for offline memory forensics.
  7. Run Moneta/malfind against the snapshot to enumerate injected regions.
  8. Correlate every observation to an ATT&CK technique.
  9. Restore snapshot; repeat with network simulation (INetSim) enabled.

Tools Quick Reference

Tool Platform Purpose
GDB + pwndbg Linux Step-through debugging, heap inspection, memory search
splitmind + tmux Linux Each pwndbg context section in its own resizable tmux pane
WinDbg Windows User-mode and kernel-mode debugging; TTD support
x64dbg + ScyllaHide Windows Step-through, anti-anti-debug bypass
Scylla Windows PE dump + IAT reconstruction
API Monitor Windows GUI API call interception with arguments and return values
Frida Cross-platform Scriptable API hooking, execution tracing, bypass injection
Intel PIN Linux/Windows Low-level instrumentation, coverage
DynamoRIO + drcov Linux/Windows Basic-block coverage for Lighthouse diffing
bpftrace / BCC Linux Kernel-level syscall/event tracing without ptrace
Sysmon + ETW Windows Structured kernel event capture
SilkETW / krabsetw Windows Raw ETW provider consumption
Moneta Windows Live process memory IOC scanner
Volatility malfind Offline image Shellcode and injected PE hunter
MITRE ATT&CK Navigator Web Technique mapping and report visualization

Further Reading