courses

Week 7: Anti-Analysis Techniques

Overview

Anti-analysis techniques are the malware author’s counter-moves against the analyst’s tools. The goal is to slow down or prevent analysis — every extra hour of analyst time is an hour the malware can continue operating undetected. Hundreds of distinct techniques exist; this week covers the major categories.

The analyst’s response is always the same: identify the check, understand what it detects, and bypass it — either by patching the binary, manipulating the debugger, or configuring the analysis environment to look “real.”

Anti-Disassembly

Disassemblers (Ghidra, Cutter/Rizin/radare2, objdump) use heuristics to determine where code begins and ends. These heuristics can be exploited.

Linear vs. Flow-Oriented Disassembly

Fake Conditional Jumps

Two back-to-back conditional jumps with opposite conditions guarantee one always taken — effectively an unconditional jump, but the disassembler may decode the byte after the second jump as an instruction:

xor eax, eax       ; always sets ZF
jz  next           ; always taken
jnz next           ; never taken — but disassembler may decode byte after this
db  0xE8           ; bogus byte — disassembled as part of a "call" by linear disassembler
next:
; real code here

Fix: in Ghidra, press C on the bogus byte to clear it, then D at the real instruction boundary to disassemble. In Cutter, right-click → Set as Data, then right-click → Disassemble.

Inward Jumps (Impossible Disassembly)

A jump targets the middle of a two-byte instruction:

jmp  label+1       ; jumps to the second byte of the next instruction
db   0xFF          ; first byte of a two-byte sequence
label:
db   0xE0          ; jmp eax — the real instruction starts here

The disassembler decodes 0xFF 0xE0 as one instruction; the jump uses only the 0xE0 byte. The disassembler cannot represent both interpretations simultaneously.

Fix: NOP out the initial byte, or manually redefine the bytes.

call/pop Anti-Disassembly

A call to the next instruction pushes the return address and jumps — but if the target is just the next byte, the call is really just pushing the current EIP. If the “return” is never executed, the disassembler may treat the bytes after the call target as part of the called “function.”

call next
db   0x41          ; bogus byte the disassembler decodes as a "return address"
next:
pop eax            ; eax = address of the db byte

Control Flow Flattening (OLLVM)

All code paths are routed through a central switch dispatcher. A state variable encodes which basic block to execute next. The call graph becomes a single central dispatcher, and every block jumps back to it — making the structure unreadable.

Tools like Sophos’s emotet_unflatten_poc and the IDA plugin d-810 can deobfuscate flattened code. For Cutter/Rizin, the rz-pipe scripting interface allows writing custom deobfuscation scripts (use r2pipe for radare2).

Misusing SEH for Hidden Execution

An exception handler is pushed onto the stack, then a divide-by-zero (or other) exception is triggered deliberately. The malicious payload is the exception handler. The disassembler, seeing a call/ret pair around the triggering code, stops analysis there and never disassembles the handler.

push handler_addr         ; push SEH record
push fs:[0]
mov  fs:[0], esp          ; install exception handler
xor  eax, eax
div  eax                  ; divide by zero → exception → handler runs
handler_addr:
; actual malicious code here — never disassembled without manual help

Anti-Debugging

API-Based Checks

IsDebuggerPresent() — reads the BeingDebugged byte at offset +2 in the PEB. Returns nonzero if a debugger is attached.

CheckRemoteDebuggerPresent() — same flag but for another process.

NtQueryInformationProcess(ProcessDebugPort) — returns the debug port number; nonzero means debugger attached. Used to avoid the hooked Windows API.

OutputDebugString trick: call SetLastError(0), then OutputDebugString. If a debugger is present, it receives the string and the last error stays 0. Without a debugger, an error occurs and the last error value changes. Check for the change.

Direct PEB Inspection

To avoid API calls (which AV might monitor), malware reads the PEB directly:

mov eax, fs:[0x30]       ; load PEB address
movzx eax, byte [eax+2]  ; BeingDebugged flag
test eax, eax
jnz  debugger_detected

Other PEB checks:

INT 3 Scanning

Software breakpoints replace an instruction byte with 0xCC (INT 3). Malware scans its own code for 0xCC bytes:

mov  ecx, code_length
mov  esi, code_start
scan:
  lodsb
  cmp  al, 0xCC
  je   debugger_detected
  loop scan

Bypass: use hardware breakpoints (DR0–DR3 registers) instead of software breakpoints. Hardware breakpoints do not modify code.

Counter-counter: malware can read the debug registers via GetThreadContext and check DR0–DR3 for nonzero values.

Timing Checks

Debuggers slow execution. Malware timestamps itself:

rdtsc            ; read CPU timestamp counter into EDX:EAX
mov  esi, eax   ; save low 32 bits
; ... some work ...
rdtsc
sub  eax, esi   ; elapsed cycles
cmp  eax, 0x1000  ; if more than 4096 cycles, assume debugger
jg   exit_or_evade

Also implemented via GetTickCount, QueryPerformanceCounter.

Bypass: set a breakpoint after the check and modify the elapsed value directly, or use a debugger plugin that patches rdtsc.

A debugger does not trip a timing check merely by being attached — stopping inside the timed window is what trips it, and breakpoint placement alone defeats a surprising number of these. The in-class activity measures that distinction on a specimen you build yourself: the reference numbers are 224 cycles under GDB with no breakpoints, against 394,738 for a single stop inside the window and 11 billion for a three-second pause, all against a 50,000-cycle threshold.

Thread Local Storage (TLS) Callbacks

TLS callbacks execute before the PE entry point. A debugger that breaks at the entry point will miss TLS callback code entirely. Malware hides anti-analysis checks (or even payload execution) in TLS.

Fix: configure GDB/OllyDbg to pause before TLS callbacks run. In OllyDbg: Options → Debugging Options → Events → Break on new module (DLL) → also handles TLS.

Inserting INT 3 Floods

Insert hundreds of 0xCC bytes. Every time the analyst single-steps through them, the debugger catches each as a breakpoint. Tedious and disorienting.

Fix: NOP out the INT 3 flood, or use “run until cursor” to jump past the flood.

Crashing the Debugger

Some debuggers are less robust than the OS:

Anti-Virtual Machine

Registry and File Artifacts

VMware, VirtualBox, and Hyper-V leave traces. Malware checks for their presence:

// Check for VMware Tools registry key
RegOpenKeyEx(HKEY_LOCAL_MACHINE,
    "SOFTWARE\\VMware, Inc.\\VMware Tools", 0, KEY_READ, &key);

Other checked artifacts:

Bypass: remove or rename the registry keys and files; uninstall VMware Tools.

MAC Address Prefixes

VMware NICs use MAC addresses beginning with 00:0C:29, 00:50:56, or 00:05:69. VirtualBox uses 08:00:27.

GetAdaptersInfo(...);   // iterate adapters, check MAC prefix

Bypass: change the VM’s MAC address to a realistic vendor prefix.

Descriptor Table Checks (Red Pill / No Pill)

The x86 sidt, sgdt, and sldt instructions read hardware registers pointing to the IDT, GDT, and LDT. In a virtual machine, the guest OS has different table locations than it reports through the OS API.

sidt fword ptr [ebp-8]   ; store IDTR to memory
mov  al, [ebp-3]         ; check byte 5 of IDTR
cmp  al, 0xFF            ; VMware sets this to 0xFF
je   vm_detected

Bypass: NOP out the check. Or use VMware’s “hide from guest” options (performance cost).

I/O Port Communication

VMware virtualizes a special I/O port (0x5658, VX). Malware queries it:

mov  eax, 0x564D5868    ; "VMXh" magic
mov  dx,  0x5658
in   eax, dx             ; query VMware I/O port
cmp  ebx, 0x564D5868    ; if VMware returns the magic, we're in a VM
je   vm_detected

CPUID Hypervisor Bit

Bit 31 of ECX in CPUID leaf 1 is the hypervisor present bit. Set by most hypervisors.

mov eax, 1
cpuid
test ecx, 0x80000000     ; bit 31
jnz  vm_detected

Selective Execution

Rather than crashing or exiting when analysis is detected, sophisticated malware only activates for specific targets:

Timing-Based Evasion

Analyst counter: hook all sleep/timing APIs and compress time. Some sandboxes do this. But a cryptographic time-lock defeats even this approach.

Packers and Unpacking

Packer Anatomy

A packed binary has:

  1. An unpacking stub (the only initially readable code)
  2. The packed original binary (compressed, encrypted, or both)

Execution:

  1. Stub saves CPU state (PUSHAD, PUSHFD)
  2. Stub decrypts/decompresses the payload into newly allocated memory
  3. Stub reconstructs the IAT (calls LoadLibrary/GetProcAddress for each import)
  4. Stub transfers control to the Original Entry Point (OEP) — often a jmp eax or jmp [eax] far from normal control flow

Signs of packing: high-entropy sections, section names like UPX0/UPX1, very few imports (just VirtualAlloc, LoadLibrary), small amount of recognizable code.

Unpacking UPX

$ upx -d packed.exe -o unpacked.exe

Manual Dynamic Unpacking

For custom packers:

  1. Set a breakpoint on VirtualAlloc (the stub allocates space for the payload)
  2. Note the returned address
  3. Run — the stub decrypts the payload into that region
  4. Set a hardware breakpoint on the first byte of that region (execute)
  5. Run — the OEP transfer will hit the breakpoint
  6. Dump the process memory at this point using OllyDump or process_vm_read

The dumped image is the unpacked payload, loadable in Ghidra or Cutter for static analysis.

Bypassing Checks: General Strategy

  1. Identify the check (from disassembly or behavioral observation)
  2. Understand what it detects (API, direct memory, timing, etc.)
  3. Choose a bypass:
    • Patch the binary: NOP out the conditional branch or the check itself
    • Debugger manipulation: set a register/flag to the expected “safe” value
    • Environment modification: remove the artifact being checked for
  4. Verify the bypass worked: the malware should proceed past the check

In-Class Activity

The RDTSC Timing Check — twenty-five minutes, in pairs, on one technique from this page. You download and compile the specimen (two files, curled from that page), take a baseline measurement, locate the rdtsc pair and the threshold constant in its disassembly, demonstrate which breakpoint placements trip the check and which do not, and then defeat it twice: once by rewriting the measurement, once by overwriting the verdict on the way out of the deciding function. The second bypass is the one that generalises, and it is the technique to carry into the final project’s anti-analysis section.

The lecture hands over to the activity with about half an hour of class left, so bring a laptop that can compile a C file.

Lab

From Practical Malware Analysis chapters 15–18: