Anti-Analysis Techniques

• Anti-Analysis Techniques
  • Anti-Debugging
    • IsDebuggerPresent
    • CheckRemoteDebuggerPresent / NtQueryInformationProcess
    • Timing Checks
    • Hardware Breakpoint Detection
  • VM and Sandbox Detection
    • Registry and File Artifacts
    • CPUID Hypervisor Bit
    • User Activity Checks
  • Code Obfuscation
    • Packing
    • String Obfuscation
    • Control Flow Flattening
  • Anti-VM Techniques in Practice
  • Useful Resources

Modern malware rarely executes straightforwardly. Malware authors invest significant effort in detecting and evading analysis tools. Understanding these techniques — and how to defeat them — is a core analyst skill.

Anti-Debugging

Debuggers change the execution environment in detectable ways. Malware checks for these changes and alters behavior (or exits silently) when detected.

IsDebuggerPresent

The simplest check. Reads the BeingDebugged byte from the Process Environment Block (PEB):

call IsDebuggerPresent
test eax, eax
jne  exit_or_evade

Bypass: In a debugger, patch IsDebuggerPresent to always return 0, or manually clear the PEB flag:

(gdb) set *(char*)($rax + 2) = 0    # clear BeingDebugged in PEB

CheckRemoteDebuggerPresent / NtQueryInformationProcess

More robust API-based checks:

NtQueryInformationProcess(handle, ProcessDebugPort, &debugPort, ...);
if (debugPort != 0) { /* debugger detected */ }

Bypass: Hook or patch the API call to return a zero debug port.

Timing Checks

A debugger slows execution. Malware measures elapsed time between two points; if too slow, assume a debugger:

DWORD start = GetTickCount();
// some short operation
DWORD elapsed = GetTickCount() - start;
if (elapsed > 100) { exit(); }

Also seen with RDTSC (read timestamp counter):

rdtsc          ; read CPU cycle counter into edx:eax
; ... do something ...
rdtsc
sub eax, first_read
cmp eax, threshold
jg  debugger_detected

Bypass: Set a breakpoint after the check and manually set eax to a small value, or use a debugger plugin that patches RDTSC.

Hardware Breakpoint Detection

Hardware breakpoints use debug registers (DR0–DR3). Malware can read these with GetThreadContext:

CONTEXT ctx = { CONTEXT_DEBUG_REGISTERS };
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3) { evade(); }

Bypass: Use software breakpoints (INT3) instead of hardware breakpoints.

VM and Sandbox Detection

Analysis environments are often virtual machines or automated sandboxes. Malware detects these by looking for artifacts that wouldn’t exist on a real user’s system.

Registry and File Artifacts

VMware, VirtualBox, and Hyper-V leave traces in the registry and filesystem:

HKLM\SOFTWARE\VMware, Inc.\VMware Tools
HKLM\HARDWARE\DEVICEMAP\Scsi\Scsi Port 0\Scsi Bus 0\Target Id 0\Logical Unit Id 0\Identifier = "VMWARE"
C:\windows\system32\drivers\vmhgfs.sys

Bypass: Remove or rename these keys/files in the analysis VM. Tools like VBoxHardenedLoader automate this for VirtualBox.

CPUID Hypervisor Bit

When running in a VM, the CPUID instruction’s bit 31 of ECX (in leaf 1) is set by the hypervisor:

mov eax, 1
cpuid
test ecx, 0x80000000   ; bit 31 = hypervisor present
jnz  vm_detected

Bypass: Some hypervisors allow disabling this bit in VM settings.

User Activity Checks

Automated sandboxes often have no real user interaction: no recent mouse movement, no documents in recent files, no browser history. Malware checks:

• GetLastInputInfo — time since last user input
• Number of running processes (sandboxes have fewer)
• Screen resolution (sandboxes often use 800x600)
• Username/hostname against known sandbox names (sandbox, maltest, cuckoo, etc.)

Bypass: Use a VM configured to look like a real workstation; run browser activity scripts before analysis.

Code Obfuscation

Packing

Packers compress or encrypt the original code and embed a small stub that decompresses/decrypts it at runtime. The packed binary’s sections have very high entropy and few meaningful imports (just VirtualAlloc, LoadLibrary).

Common packers: UPX, MPRESS, Themida, ASPack.

Detection:

$ binwalk -E sample.exe    # entropy graph
$ pepack sample.exe        # packer detection heuristics

Bypass for UPX:

$ upx -d packed_sample.exe -o unpacked.exe

For custom packers: set a breakpoint at the OEP (Original Entry Point), let the stub run, then dump the unpacked image from memory.

String Obfuscation

Instead of storing strings in plaintext, the strings are built on the stack character by character or XOR-decoded at runtime. FLOSS detects many of these patterns.

Control Flow Flattening

The compiler (or a tool like OLLVM) restructures all control flow through a central dispatcher switch. Every basic block jumps back to the dispatcher, which decides the next block from a state variable. This makes the call graph useless and the decompiler output unreadable.

Bypass: Trace execution to determine the actual block sequence, or use tools like D-810 for IDA.

Anti-VM Techniques in Practice

When you encounter a sample that exits immediately or behaves differently in your VM:

1. Check VirusTotal for behavioral reports — does it sleep before doing anything?
2. Use x64dbg (Windows) or GDB to step through the early code path
3. Look for IsDebuggerPresent, GetTickCount, CPUID calls near the beginning
4. Patch the check: NOP out the conditional jump or set the comparison result directly
5. If it’s packed, unpack it first, then analyze the payload

Useful Resources

• The “Ultimate” Anti-Debugging Reference
• al-khaser — open-source collection of anti-analysis PoCs
• MITRE ATT&CK T1497 — Virtualization/Sandbox Evasion
• MITRE ATT&CK T1622 — Debugger Evasion