Week 7: Anti-Analysis Techniques
- Overview
- Anti-Disassembly
- Anti-Debugging
- Anti-Virtual Machine
- Packers and Unpacking
- Bypassing Checks: General Strategy
- In-Class Activity
- Lab
Overview
Anti-analysis techniques are the malware author’s counter-moves against the analyst’s tools. The goal is to slow down or prevent analysis — every extra hour of analyst time is an hour the malware can continue operating undetected. Hundreds of distinct techniques exist; this week covers the major categories.
The analyst’s response is always the same: identify the check, understand what it detects, and bypass it — either by patching the binary, manipulating the debugger, or configuring the analysis environment to look “real.”
Anti-Disassembly
Disassemblers (Ghidra, Cutter/Rizin/radare2, objdump) use heuristics to determine where code begins and ends. These heuristics can be exploited.
Linear vs. Flow-Oriented Disassembly
- Linear: decode instructions sequentially from start to end, ignoring jumps. Simple but wrong — it decodes data as code and misses code after jumps.
- Flow-oriented: follow branches, build a graph of reachable code. More accurate but susceptible to tricks that confuse branch analysis.
Fake Conditional Jumps
Two back-to-back conditional jumps with opposite conditions guarantee one always taken — effectively an unconditional jump, but the disassembler may decode the byte after the second jump as an instruction:
xor eax, eax ; always sets ZF
jz next ; always taken
jnz next ; never taken — but disassembler may decode byte after this
db 0xE8 ; bogus byte — disassembled as part of a "call" by linear disassembler
next:
; real code here
Fix: in Ghidra, press C on the bogus byte to clear it, then D at the real instruction boundary to disassemble. In Cutter, right-click → Set as Data, then right-click → Disassemble.
Inward Jumps (Impossible Disassembly)
A jump targets the middle of a two-byte instruction:
jmp label+1 ; jumps to the second byte of the next instruction
db 0xFF ; first byte of a two-byte sequence
label:
db 0xE0 ; jmp eax — the real instruction starts here
The disassembler decodes 0xFF 0xE0 as one instruction; the jump uses only the 0xE0 byte. The disassembler cannot represent both interpretations simultaneously.
Fix: NOP out the initial byte, or manually redefine the bytes.
call/pop Anti-Disassembly
A call to the next instruction pushes the return address and jumps — but if the target is just the next byte, the call is really just pushing the current EIP. If the “return” is never executed, the disassembler may treat the bytes after the call target as part of the called “function.”
call next
db 0x41 ; bogus byte the disassembler decodes as a "return address"
next:
pop eax ; eax = address of the db byte
Control Flow Flattening (OLLVM)
All code paths are routed through a central switch dispatcher. A state variable encodes which basic block to execute next. The call graph becomes a single central dispatcher, and every block jumps back to it — making the structure unreadable.
Tools like Sophos’s emotet_unflatten_poc and the IDA plugin d-810 can deobfuscate flattened code. For Cutter/Rizin, the rz-pipe scripting interface allows writing custom deobfuscation scripts (use r2pipe for radare2).
Misusing SEH for Hidden Execution
An exception handler is pushed onto the stack, then a divide-by-zero (or other) exception is triggered deliberately. The malicious payload is the exception handler. The disassembler, seeing a call/ret pair around the triggering code, stops analysis there and never disassembles the handler.
push handler_addr ; push SEH record
push fs:[0]
mov fs:[0], esp ; install exception handler
xor eax, eax
div eax ; divide by zero → exception → handler runs
handler_addr:
; actual malicious code here — never disassembled without manual help
Anti-Debugging
API-Based Checks
IsDebuggerPresent() — reads the BeingDebugged byte at offset +2 in the PEB. Returns nonzero if a debugger is attached.
CheckRemoteDebuggerPresent() — same flag but for another process.
NtQueryInformationProcess(ProcessDebugPort) — returns the debug port number; nonzero means debugger attached. Used to avoid the hooked Windows API.
OutputDebugString trick: call SetLastError(0), then OutputDebugString. If a debugger is present, it receives the string and the last error stays 0. Without a debugger, an error occurs and the last error value changes. Check for the change.
Direct PEB Inspection
To avoid API calls (which AV might monitor), malware reads the PEB directly:
mov eax, fs:[0x30] ; load PEB address
movzx eax, byte [eax+2] ; BeingDebugged flag
test eax, eax
jnz debugger_detected
Other PEB checks:
PEB+0x18(ProcessHeap): the heap’s allocation flags differ when a debugger is presentPEB+0x68(NtGlobalFlag): set to0x70when a debugger allocates the heap
INT 3 Scanning
Software breakpoints replace an instruction byte with 0xCC (INT 3). Malware scans its own code for 0xCC bytes:
mov ecx, code_length
mov esi, code_start
scan:
lodsb
cmp al, 0xCC
je debugger_detected
loop scan
Bypass: use hardware breakpoints (DR0–DR3 registers) instead of software breakpoints. Hardware breakpoints do not modify code.
Counter-counter: malware can read the debug registers via GetThreadContext and check DR0–DR3 for nonzero values.
Timing Checks
Debuggers slow execution. Malware timestamps itself:
rdtsc ; read CPU timestamp counter into EDX:EAX
mov esi, eax ; save low 32 bits
; ... some work ...
rdtsc
sub eax, esi ; elapsed cycles
cmp eax, 0x1000 ; if more than 4096 cycles, assume debugger
jg exit_or_evade
Also implemented via GetTickCount, QueryPerformanceCounter.
Bypass: set a breakpoint after the check and modify the elapsed value directly, or use a debugger plugin that patches rdtsc.
A debugger does not trip a timing check merely by being attached — stopping inside the timed window is what trips it, and breakpoint placement alone defeats a surprising number of these. The in-class activity measures that distinction on a specimen you build yourself: the reference numbers are 224 cycles under GDB with no breakpoints, against 394,738 for a single stop inside the window and 11 billion for a three-second pause, all against a 50,000-cycle threshold.
Thread Local Storage (TLS) Callbacks
TLS callbacks execute before the PE entry point. A debugger that breaks at the entry point will miss TLS callback code entirely. Malware hides anti-analysis checks (or even payload execution) in TLS.
Fix: configure GDB/OllyDbg to pause before TLS callbacks run. In OllyDbg: Options → Debugging Options → Events → Break on new module (DLL) → also handles TLS.
Inserting INT 3 Floods
Insert hundreds of 0xCC bytes. Every time the analyst single-steps through them, the debugger catches each as a breakpoint. Tedious and disorienting.
Fix: NOP out the INT 3 flood, or use “run until cursor” to jump past the flood.
Crashing the Debugger
Some debuggers are less robust than the OS:
- Pass malformed PE headers (OllyDbg parses headers more strictly than Windows)
- Pass a format string vulnerability in
OutputDebugStringto crash OllyDbg v1.1 - Trigger processor instructions OllyDbg handles differently than the actual CPU
Anti-Virtual Machine
Registry and File Artifacts
VMware, VirtualBox, and Hyper-V leave traces. Malware checks for their presence:
// Check for VMware Tools registry key
RegOpenKeyEx(HKEY_LOCAL_MACHINE,
"SOFTWARE\\VMware, Inc.\\VMware Tools", 0, KEY_READ, &key);
Other checked artifacts:
C:\Program Files\VMware\VMware Tools\- Device names:
\\.\HGFS,\\.\vmci - Running processes:
vmtoolsd.exe,vmwaretray.exe,vboxservice.exe
Bypass: remove or rename the registry keys and files; uninstall VMware Tools.
MAC Address Prefixes
VMware NICs use MAC addresses beginning with 00:0C:29, 00:50:56, or 00:05:69. VirtualBox uses 08:00:27.
GetAdaptersInfo(...); // iterate adapters, check MAC prefix
Bypass: change the VM’s MAC address to a realistic vendor prefix.
Descriptor Table Checks (Red Pill / No Pill)
The x86 sidt, sgdt, and sldt instructions read hardware registers pointing to the IDT, GDT, and LDT. In a virtual machine, the guest OS has different table locations than it reports through the OS API.
sidt fword ptr [ebp-8] ; store IDTR to memory
mov al, [ebp-3] ; check byte 5 of IDTR
cmp al, 0xFF ; VMware sets this to 0xFF
je vm_detected
Bypass: NOP out the check. Or use VMware’s “hide from guest” options (performance cost).
I/O Port Communication
VMware virtualizes a special I/O port (0x5658, VX). Malware queries it:
mov eax, 0x564D5868 ; "VMXh" magic
mov dx, 0x5658
in eax, dx ; query VMware I/O port
cmp ebx, 0x564D5868 ; if VMware returns the magic, we're in a VM
je vm_detected
CPUID Hypervisor Bit
Bit 31 of ECX in CPUID leaf 1 is the hypervisor present bit. Set by most hypervisors.
mov eax, 1
cpuid
test ecx, 0x80000000 ; bit 31
jnz vm_detected
Selective Execution
Rather than crashing or exiting when analysis is detected, sophisticated malware only activates for specific targets:
- Username/hostname checks: only run if hostname matches a corporate naming convention
- Domain membership: check if the machine is joined to an Active Directory domain
- IP/MAC targeting: activate only for specific IP prefixes (ASUS supply chain attack, 2019)
- Language checks: Stuxnet required specific Siemens software; LODEINFO (2024) checks the OS language
- Mouse movement: Lumma Stealer (2023) analyzes mouse movement vectors to detect automated sandboxes (angles below 45° indicate real human movement)
Timing-Based Evasion
- Sleep before C2: SolarWinds slept 12–14 days after installation before phoning home
- Run-count gates: only execute after a specific number of reboots
- Time bombs: activate only after a fixed date
- Cryptographic time-lock: decrypt the payload using a key derived from the current time, making the sample undecryptable before/after a window
Analyst counter: hook all sleep/timing APIs and compress time. Some sandboxes do this. But a cryptographic time-lock defeats even this approach.
Packers and Unpacking
Packer Anatomy
A packed binary has:
- An unpacking stub (the only initially readable code)
- The packed original binary (compressed, encrypted, or both)
Execution:
- Stub saves CPU state (
PUSHAD,PUSHFD) - Stub decrypts/decompresses the payload into newly allocated memory
- Stub reconstructs the IAT (calls
LoadLibrary/GetProcAddressfor each import) - Stub transfers control to the Original Entry Point (OEP) — often a
jmp eaxorjmp [eax]far from normal control flow
Signs of packing: high-entropy sections, section names like UPX0/UPX1, very few imports (just VirtualAlloc, LoadLibrary), small amount of recognizable code.
Unpacking UPX
$ upx -d packed.exe -o unpacked.exe
Manual Dynamic Unpacking
For custom packers:
- Set a breakpoint on
VirtualAlloc(the stub allocates space for the payload) - Note the returned address
- Run — the stub decrypts the payload into that region
- Set a hardware breakpoint on the first byte of that region (execute)
- Run — the OEP transfer will hit the breakpoint
- Dump the process memory at this point using OllyDump or
process_vm_read
The dumped image is the unpacked payload, loadable in Ghidra or Cutter for static analysis.
Bypassing Checks: General Strategy
- Identify the check (from disassembly or behavioral observation)
- Understand what it detects (API, direct memory, timing, etc.)
- Choose a bypass:
- Patch the binary: NOP out the conditional branch or the check itself
- Debugger manipulation: set a register/flag to the expected “safe” value
- Environment modification: remove the artifact being checked for
- Verify the bypass worked: the malware should proceed past the check
In-Class Activity
The RDTSC Timing Check — twenty-five minutes, in
pairs, on one technique from this page. You download and compile the specimen
(two files, curled from that page), take a baseline measurement, locate the
rdtsc pair and the threshold constant in its disassembly, demonstrate which
breakpoint
placements trip the check and which do not, and then defeat it twice: once by
rewriting the measurement, once by overwriting the verdict on the way out of
the deciding function. The second bypass is the one that generalises, and it is
the technique to carry into the final project’s anti-analysis
section.
The lecture hands over to the activity with about half an hour of class left, so bring a laptop that can compile a C file.
Lab
From Practical Malware Analysis chapters 15–18:
- Find and bypass an
IsDebuggerPresentcheck using GDB register manipulation - Identify an anti-VM check (descriptor table or MAC address); NOP it out with a hex editor and verify the sample runs in the VM
- Unpack a UPX-packed binary and load the result in Ghidra or Cutter
- Identify a timing check using
rdtsc; explain what threshold it uses and why