courses

Shellcode Analysis

Shellcode is raw executable machine code injected into a running process by an exploit or loader. Unlike a normal binary, shellcode has no PE/ELF header, no loader to fix up relocations, and no guaranteed memory address. These constraints drive the techniques described here.

Position-Independent Code

Shellcode cannot use hard-coded addresses because it may be placed anywhere in the address space of the target process. All references to embedded data must be computed relative to the shellcode’s current location.

Control-flow instructions (jmp, call, jnz, etc.) already use relative offsets and work anywhere. The problem is data access — x86-32 has no instruction for loading the current instruction pointer into a register.

call/pop

The most common technique to get the current EIP into a register:

call get_eip        ; push address of *next* instruction onto stack
get_eip:
pop ebx             ; pop that address into ebx — now ebx == get_eip

Everything embedded after the call target is now addressable as [ebx + offset]. This pattern also appears as an anti-disassembly trick (Chapter 15 of Practical Malware Analysis) because the call looks like a function call but never “returns” in the normal sense.

fnstenv

The x87 FPU maintains its own instruction pointer pointing to the last FPU instruction executed. fnstenv stores a 28-byte FPU state structure to memory, and offset 12 of that structure holds the FPU’s last instruction pointer — which gives us an EIP anchor with no call:

fldz                ; any FPU instruction to set FPU IP
fnstenv [esp-0x1c]  ; store FPU state structure below current esp
pop ebx             ; offset 12 (fpu_instruction_pointer) now in ebx

This avoids the call/pop pattern and evades signatures that scan for it.

Manual Symbol Resolution

A normal binary relies on the Windows loader to populate the Import Address Table with function addresses. Shellcode has no loader. It must find the functions it needs by walking in-memory data structures.

Finding kernel32.dll

The Thread Environment Block (TEB) is always at fs:[0x18]. From the TEB, offset 0x30 points to the Process Environment Block (PEB). The PEB’s Ldr field (offset 0x0C) points to PEB_LDR_DATA, which contains three doubly-linked lists of loaded modules. The second list (InMemoryOrderModuleList) typically has ntdll.dll first and kernel32.dll second.

mov eax, fs:[0x30]        ; PEB
mov eax, [eax + 0x0C]     ; Ldr
mov eax, [eax + 0x14]     ; InMemoryOrderModuleList.Flink (ntdll)
mov eax, [eax]            ; .Flink (kernel32)
mov eax, [eax + 0x10]     ; DllBase of kernel32

Parsing the PE Export Table

Once the base address of kernel32.dll is known, shellcode parses its PE export table to locate LoadLibraryA and GetProcAddress. From those two functions, any other library and function can be located.

The IMAGE_EXPORT_DIRECTORY contains three parallel arrays:

Algorithm:

  1. Iterate over AddressOfNames until the desired name matches
  2. Use that index in AddressOfNameOrdinals to get the ordinal
  3. Use the ordinal to index AddressOfFunctions and get the RVA
  4. Add the RVA to the DLL base to get the absolute address

API Hashing

Including plaintext function name strings in shellcode makes it trivially detectable. Instead, shellcode pre-computes a hash of each desired function name, then iterates over the export table computing the same hash for each name until it matches.

Common hash algorithms used: ror13 (rotate-right 13, add character), custom polynomial hashes. To analyze: identify the hash function, extract the hash constants from the binary, brute-force the function name offline.

def ror13(s):
    h = 0
    for c in s + '\x00':
        h = ((h >> 13) | (h << 19)) & 0xFFFFFFFF
        h = (h + ord(c)) & 0xFFFFFFFF
    return h

Shellcode Encodings

Some exploit delivery channels corrupt certain byte values. The most common restriction: NULL bytes (0x00) terminate C strings, so strcpy-based overflows are truncated at the first NULL.

Avoiding NULLs:

mov eax, 0          ; BAD  — encodes as B8 00 00 00 00
xor eax, eax        ; GOOD — no NULL bytes

Similarly avoid 0x0A (newline) if the vulnerability is in line-based input, 0x20 (space) for word-split vulnerabilities, etc.

ASCII armor / alphanumeric shellcode: Some exploit mitigations force all injected bytes to fall in the printable ASCII range. Specialized encoders (Metasploit’s x86/alpha_mixed) transform arbitrary shellcode into a self-decoding sequence using only printable characters.

NOP Sleds

When exploiting a memory corruption bug, the exact address of the shellcode in memory may be imprecise. A NOP sled is a long sequence of 0x90 (nop) instructions placed before the shellcode. Any jump that lands anywhere in the sled slides forward into the shellcode.

[padding] [NOP NOP NOP ... NOP] [shellcode] [return address]
                  ^--- any landing here works

Modern mitigations (ASLR, DEP/NX) largely defeat NOP sleds in conventional stack overflows, but they remain relevant in ROP chains and heap sprays.

Finding Shellcode in the Wild

In JavaScript

Browser exploits commonly encode shellcode as Unicode escape sequences or as arrays of integers. Look for patterns like:

var sc = unescape("%u9090%u9090%ucce8%u0000...");

%uXXYY encodes a 16-bit little-endian value: %u1122 = bytes 22 11.

In Documents

Office macros, PDF /JavaScript actions, and Flash ActionScript all serve as shellcode delivery vectors. Tools:

In Memory

If you suspect a running process is injected, dump its memory and search for shellcode heuristics:

$ volatility -f memory.dmp --profile=Win7SP1x64 malfind

Analysis Workflow

  1. Identify that you’re looking at shellcode (no PE/ELF header, raw bytes, position-independent patterns)
  2. Load in a disassembler — in Cutter: File > Open, select Raw format and set architecture manually; in IDA: File > Open, choose the processor type when prompted
  3. Find the entry point — look for call/pop or fnstenv early on
  4. Identify the symbol resolution loop — the API hashing or name-matching loop over the export table
  5. Decode the hash constants to determine which functions are called
  6. Follow the resolved calls to understand what the shellcode does (download, inject, spawn shell, etc.)

Useful Resources