Advanced Static Analysis: x86 Assembly, IDA Pro, and C Code Constructs

• Advanced Static Analysis: x86 Assembly, IDA Pro, and C Code Constructs
  • Chapter 4: A Crash Course in x86 Disassembly
    • Levels of Abstraction
    • x86 Architecture and Memory
    • Registers
    • Instruction Set Overview
    • The Stack and Function Calls
    • Conditionals and Branching
    • Rep Instructions
    • C main and Pointer Offsets
  • Chapter 5: IDA Pro
    • Loading an Executable
    • The IDA Interface
    • Key Windows
    • Navigation
    • Searching
    • Cross-References (Xrefs)
    • Analyzing Functions
    • Graphing Options (WinGraph32)
    • Enhancing the Disassembly
    • Extending IDA with Scripts and Plug-ins
  • Chapter 6: Recognizing C Code Constructs in Assembly
    • Global vs. Local Variables
    • Arithmetic Operations
    • Recognizing if Statements
    • Recognizing Loops
    • Function Call Conventions
    • Analyzing switch Statements
    • Disassembling Arrays
    • Identifying Structs
    • Analyzing Linked List Traversal
  • Summary: Construct Recognition Quick Reference

This page summarizes the core concepts from Practical Malware Analysis (Sikorski & Honig), Chapters 4–6, which form the foundation for static disassembly-based malware analysis.

Chapter 4: A Crash Course in x86 Disassembly

Levels of Abstraction

Programs exist at six levels of abstraction: hardware → microcode → machine code → assembly → high-level language → interpreted language. Malware authors write at the high level; analysts work at the assembly level, using a disassembler to convert binary machine code back into human-readable mnemonics.

x86 Architecture and Memory

The x86 processor follows the Von Neumann model: CPU (registers, ALU, control unit), RAM, and I/O interconnected by a bus. The instruction pointer (EIP) fetches the next instruction from RAM.

Each running program sees four logical memory regions:

x86 is little-endian: the least significant byte occupies the lowest memory address. Network data is big-endian, so IP addresses appear byte-reversed when viewed in memory — important when analyzing network-related malware.

Registers

General-purpose registers (32-bit; 16-bit AX/BX/CX/DX and 8-bit AH/AL sub-registers accessible):

EFLAGS (status register) — individual bits set by arithmetic and logic instructions:

Segment registers (CS, SS, DS, ES, FS, GS) — mostly irrelevant to user-mode analysis, except FS, which points to the Thread Information Block (TIB) in Windows.

EIP — the instruction pointer; cannot be directly modified except by control-flow instructions (call, ret, jmp, jcc).

Instruction Set Overview

Intel syntax: mnemonic destination, source. Three operand types:

• Immediate — literal value embedded in the instruction (e.g., mov eax, 1)
• Register — one of the registers above (e.g., mov eax, ebx)
• Memory address — brackets dereference a pointer (e.g., mov eax, [ebx] reads the value at the address in EBX)

Key instructions:

xor eax, eax is the canonical zero-register idiom — smaller encoding than mov eax, 0 and commonly seen in compiler output.

The Stack and Function Calls

The stack is LIFO and grows downward (toward lower addresses). ESP always points to the current top. EBP anchors the frame for the current function, remaining fixed while the function executes.

Function call sequence:

1. Caller pushes arguments onto the stack (right to left in cdecl/stdcall)
2. call target — pushes the return address (next EIP) and jumps to the function
3. Prologue: push ebp / mov ebp, esp / sub esp, N — saves caller’s EBP, establishes new frame, allocates space for locals
4. Function body; local variables at negative EBP offsets [ebp-N], arguments at positive offsets [ebp+N]
5. Epilogue: mov esp, ebp / pop ebp / ret (equivalently, leave / ret) — restores caller’s frame and jumps to the return address

Return values are placed in EAX before ret.

Stack frame layout (addresses increase downward):

[ebp+8]   arg_0 (first argument)
[ebp+4]   return address
[ebp+0]   saved EBP  ← EBP points here
[ebp-4]   var_4 (first local variable)
[ebp-8]   var_8
...
          ← ESP points here (top of stack)

Conditionals and Branching

test op1, op2 — computes op1 AND op2, sets flags, discards result. test eax, eax is the standard NULL/zero check (sets ZF if EAX = 0).

cmp op1, op2 — computes op1 − op2, sets flags, discards result.

jmp — unconditional jump. Over 30 conditional jump variants examine EFLAGS after cmp/test:

Rep Instructions

The rep prefix repeats the following string instruction, decrementing ECX each iteration until ECX = 0 (or a flag condition halts it). ESI = source, EDI = destination.

Variant prefixes: repe/repz also stop when ZF=0; repne/repnz also stop when ZF=1.

Direction flag (DF): when DF=0, ESI/EDI increment after each step; when DF=1, they decrement (used for reverse-direction copies in shellcode).

C main and Pointer Offsets

int main(int argc, char **argv) — argc is the argument count (including program name); argv is an array of string pointers. On 32-bit systems each pointer is 4 bytes, so argv[1] is at [argv + 4], argv[2] at [argv + 8]. In assembly:

mov eax, [ebp+argv]   ; load argv base pointer
mov ecx, [eax+4]      ; load argv[1] (offset 4 = second pointer)
mov ecx, [eax+8]      ; load argv[2] (offset 8 = third pointer)

Chapter 5: IDA Pro

IDA Pro (Interactive Disassembler Professional) by Hex-Rays is the industry-standard disassembler for malware analysis. It performs automatic function discovery, stack frame analysis, local variable identification, and cross-referencing, then allows analysts to annotate, rename, and extend the disassembly interactively. Work is saved in a .idb database and can be resumed across sessions.

IDA’s FLIRT (Fast Library Identification and Recognition Technology) automatically identifies and labels standard library functions compiled into the binary, so analysts can identify and skip them quickly.

Loading an Executable

At load time, IDA detects the file format (PE, ELF, COFF, a.out) and processor architecture. Key loading options:

• Binary File — load raw bytes without parsing headers; required for shellcode or data appended after the PE structure that the OS loader would never map
• Manual Load — loads each PE section individually including the PE header itself; useful when malware hides code in normally-skipped sections
• Manual Load checkbox — specify a custom base address when analyzing a rebased DLL (DLLs frequently load at addresses different from their preferred base)

By default, IDA omits the PE header and resource sections from disassembly. Manual Load forces each section to be analyzed explicitly.

The IDA Interface

Graph mode (default): functions rendered as flowcharts with colored arrows:

• Red — conditional branch not taken
• Green — conditional branch taken
• Blue — unconditional jump
• Upward-pointing arrows — loop back-edge

Enable line prefixes and 6 opcode bytes via Options → General → Line prefixes and Number of Opcode Bytes = 6.

Text mode: traditional linear listing showing memory addresses, segment names (.text:00401050), and opcodes. Switch between modes with the spacebar. The left arrows window shows the program’s nonlinear flow; solid lines are unconditional, dashed lines are conditional.

Auto-comments can be enabled via Options → General → Auto comments.

Key Windows

Navigation

Link types in the disassembly:

• Sub links — jump to function entry points (e.g., printf, sub_4010A0)
• Loc links — jump to intermediate locations (e.g., loc_40107E)
• Offset links — reference a memory offset

Navigation band (color bar at the toolbar base): light blue = FLIRT library code, red = compiler-generated code, dark blue = user-written code. Perform focused analysis in the dark-blue region.

Searching

Cross-References (Xrefs)

A cross-reference (xref) records every location where a function is called or a data item is accessed. Press X on any name to open the Xrefs window.

• Code xref — another location calls or jumps to this address
• Data xref — another location reads or writes this data

The Xrefs window shows direction (up/down call), the calling address, and the calling function. This is the primary mechanism for understanding how functions relate to each other and where strings are used.

Analyzing Functions

IDA identifies EBP-based stack frames and labels variables:

• var_C = local variable at [ebp-0Ch]
• arg_4 = parameter at [ebp+4]

These dummy names should be renamed to meaningful identifiers as understanding develops (double-click the name, press N). IDA propagates the new name to every reference automatically.

If IDA fails to identify a function boundary, press P to manually create a function at the cursor. If the stack frame analysis is incorrect (e.g., showing [ebp-0Ch] instead of var_C), press ALT-P, enable BP Based Frame, and set 4 bytes for Saved Registers.

Graphing Options (WinGraph32)

Five graph types accessible from the toolbar:

These legacy graphs (WinGraph32) cannot be interactively edited but provide a quick structural overview.

Enhancing the Disassembly

Renaming locations: Replace sub_401200 with DNSrequest; IDA propagates the name everywhere it appears. This eliminates redundant re-analysis of the same function.

Comments:

• : (colon) — adds a local comment at the current address
• ; (semicolon) — adds a repeatable comment that echoes at every cross-reference to that address

Formatting operands: Right-click any immediate value to reformat as decimal, octal, binary, or ASCII character. Press O to toggle whether a value is treated as an address offset or a literal number.

Named constants: Right-click an operand and select Use Standard Symbolic Constants to replace raw integers with Windows API names (e.g., 80000000h → GENERIC_READ, 3 → OPEN_EXISTING). Load additional type libraries via View → Open Subviews → Type Libraries when needed (e.g., ntapi for the Windows NT Native API, gnuunx for GNU/Linux binaries).

Redefining code and data:

• U — undefine (reduces to raw bytes)
• C — redefine as code (forces disassembly at that location)
• D — redefine as data
• A — redefine as ASCII string

This is necessary when malware appends shellcode after the PE structure — IDA loads only the mapped sections, so the shellcode appears as undefined bytes until manually disassembled.

Extending IDA with Scripts and Plug-ins

IDC — IDA’s built-in scripting language; all functions are declared static; local variables use auto. Load via File → Script File or enter single commands via File → IDC Command. The idc.idc file (in the IDA installation) documents the built-in functions.

IDAPython — Python integration exposing IDA’s full SDK through three modules:

• idaapi — core IDA API
• idc — IDC function wrappers
• idautils — utility functions (e.g., Heads() to iterate instructions, Functions() to iterate functions)

IDAPython uses effective addresses (EAs) as the primary reference type; most calls accept either an EA or a name string. Load via File → Python Command.

Commercial plug-ins:

• Hex-Rays Decompiler — converts disassembly to C-like pseudocode; dramatically reduces analysis time
• zynamics BinDiff — compares two IDA databases to identify differences between malware variants; provides similarity scores useful for attribution

Chapter 6: Recognizing C Code Constructs in Assembly

Effective reverse engineering requires recognizing groups of instructions as high-level constructs — loops, conditionals, function calls — rather than reading every instruction in isolation. Since most malware is written in C, the following patterns appear constantly.

Global vs. Local Variables

The distinction matters because a write to a global modifies persistent state, while a local disappears when the function returns.

Arithmetic Operations

Most arithmetic maps directly. Notable cases:

• idiv ecx — divides edx:eax by ECX; quotient → EAX, remainder → EDX (the modulo result)
• cdq — sign-extends EAX into EDX:EAX before signed division
• Compilers often emit sub/add instead of dec/inc for optimization reasons

Recognizing if Statements

Simple if-else assembly pattern:

cmp   eax, [ebp+var_4]     ; compare
jnz   short loc_40102B     ; if NOT equal, jump to else-body
push  offset aXEqualsY     ; true-body
call  printf
jmp   short loc_401038     ; skip else-body
loc_40102B:
push  offset aXIsNotEqual  ; else-body
call  printf
loc_401038:
...                        ; code after the if/else

Key: a conditional jump skips the true body, followed by an unconditional jump skipping the else body. In IDA graph mode, the branch node has false and true edges.

Nested if statements add additional cmp+jcc pairs within a branch. Three nested levels produce three conditional jumps in sequence.

Recognizing Loops

for loop — four identifiable components:

mov   [ebp+var_4], 0    ; (1) initialization
jmp   short loc_401016  ; (2) jump to comparison
loc_40100D:
mov   eax, [ebp+var_4]  ; (4) body
add   eax, 1            ;     increment
mov   [ebp+var_4], eax
loc_401016:
cmp   [ebp+var_4], 64h  ; (3) comparison
jge   short loc_40102F  ;     exit if done
... body ...
jmp   short loc_40100D  ; back to increment

In IDA graph mode, the upward-pointing arrow after the increment block is the definitive loop indicator.

while loop — similar structure but without a separate initialization or explicit increment section. A conditional jump at the top exits when the condition fails; an unconditional jmp at the bottom returns to the top. Frequently seen as an infinite receive-and-process loop:

while(status == 0) {
    result = performAction();
    status = checkResult(result);
}

Function Call Conventions

Calling conventions govern argument order, stack cleanup responsibility, and register use:

Push vs. Move: Visual Studio typically uses push to pass arguments. GCC may instead use sub esp, N followed by mov [esp+offset], value. The Visual Studio version contains a stack-pointer restore instruction that has no GCC equivalent. An analyst must recognize both patterns.

Analyzing switch Statements

If-style (few or non-contiguous cases): a chain of cmp+jz pairs, each case body ending with an unconditional jmp to the function end. Visually identical to nested if statements.

Jump table (many contiguous cases): the compiler generates a single indirect jump through a lookup table:

sub   ecx, 1                    ; adjust to 0-based index
mov   [ebp+var_8], ecx
cmp   [ebp+var_8], 3            ; bounds check
ja    short loc_401082          ; default case if out of range
mov   edx, [ebp+var_8]
jmp   ds:off_401088[edx*4]      ; indirect jump through table

off_401088  dd offset loc_40102C  ; case 1
            dd offset loc_401042  ; case 2
            dd offset loc_401058  ; case 3
            dd offset loc_40106E  ; case 4

Each entry in the table is a 4-byte address, so the index is multiplied by 4. This is significantly more efficient than chained comparisons and is a distinctive pattern in disassembly. The table itself appears as dd offset entries in the data section immediately after the function.

Adding even one case to a dense switch can trigger the compiler to switch from if-style to jump table — the two look entirely different in assembly.

Disassembling Arrays

Arrays are accessed as base address + (index × element size):

; Local array a[i] where elements are 4-byte ints
mov   ecx, [ebp+var_18]          ; load index i
mov   [ebp+ecx*4+var_14], edx    ; a[i] = edx

; Global array b[i]
mov   ecx, [ebp+var_18]          ; load index i
mov   dword_40A000[ecx*4], eax   ; b[i] = eax

The multiplier on the index register reveals the element size: ×1 = byte, ×2 = short, ×4 = int/pointer, ×8 = double. When no multiplier is present, the element is 1 byte or the index has been prescaled.

Identifying Structs

Structures are accessed via a base pointer + field offset:

mov   eax, [ebp+arg_0]       ; load struct pointer (q)
mov   byte ptr [eax+14h], 61h; q->y = 'a' (field at offset 0x14)
fld   ds:dbl_40B120          ; load double constant
fstp  qword ptr [ecx+18h]    ; q->z = 15.6 (field at offset 0x18)

Floating-point fields are identifiable by fld/fstp instructions at a particular offset. The sequence of offsets accessed reveals the field layout. In IDA, press T on a memory reference to apply a structure definition, transforming [eax+14h] into [eax + my_structure.y]. This requires first defining the structure in IDA’s Structures window.

Windows API functions frequently require passing populated structs; identifying the base pointer and field offsets is essential for understanding API call parameters.

Analyzing Linked List Traversal

A linked list consists of structs where each struct contains a pointer to the next struct of the same type (self-referential). The key assembly signature:

mov   eax, [ebp+var_4]    ; load current node pointer (curr)
mov   eax, [eax+4]        ; load next field (curr->next, 4 bytes in)
mov   [ebp+var_4], eax    ; curr = curr->next

The self-referential assignment — var_4 is assigned from [eax+4] where EAX previously held var_4 — is the definitive indicator. Whatever struct var_4 points to must contain a pointer 4 bytes into it, which points to another struct of the same type. Malware uses linked lists for managing active connections, task queues, and plugin registrations.

Summary: Construct Recognition Quick Reference