Week 2: Malware Triage and File Identification

• Week 2: Malware Triage and File Identification
  • Goals of Triage
  • File Identification
    • The file Command
    • Common Magic Bytes
    • xxd / hexyl
  • Hashing
  • String Extraction
    • strings
    • FLOSS
  • VirusTotal
  • Packing Detection
    • Entropy Analysis
    • Section Names
    • pev (PE tools)
    • Known Packers
  • Triage Checklist

Goals of Triage

Triage is the first step in any malware investigation. The goal is to quickly determine what you are dealing with — enough to decide how to prioritize deeper analysis — without yet executing anything.

A good triage answers:

• What kind of file is this? (PE, ELF, script, document, archive)
• Has it been seen before? (VirusTotal lookup)
• Is it packed or obfuscated?
• What strings are visible in plaintext?
• Does it connect to any known infrastructure?

Triage should take minutes, not hours. If it takes longer, you have already crossed into static analysis.

File Identification

The file Command

file reads the first few bytes of a file (the “magic bytes”) to determine its type, regardless of extension. Malware commonly uses misleading extensions.

$ file sample.exe
sample.exe: PE32 executable (GUI) Intel 80386, for MS Windows

$ file sample
sample: ELF 64-bit LSB executable, x86-64, dynamically linked, stripped

$ file document.pdf
document.pdf: PDF document, version 1.7

Key things to look for:

• Architecture (32-bit vs 64-bit)
• Linkage (statically vs dynamically linked)
• Stripped (no symbol table) vs unstripped
• GUI vs console subsystem (Windows PE)

Common Magic Bytes

xxd / hexyl

Quickly inspect raw bytes at any offset:

$ xxd sample | head -4
$ hexyl sample | head      # color hex viewer

Hashing

Always record hashes before analysis. The hash is the unique fingerprint for the sample.

$ sha256sum sample.exe        # primary — use for IOC reporting
$ md5sum sample.exe           # legacy — still common in threat intel
$ ssdeep sample.exe           # fuzzy hash — detects similar variants

Why ssdeep? Regular hashes change completely with one byte difference. Fuzzy hashing computes a similarity score — two samples that are 90% identical will show ~90% similarity. Useful for finding related variants of the same malware family.

String Extraction

strings

Extracts sequences of printable characters above a minimum length. The default minimum is 4; use 6 or higher to reduce noise.

$ strings -n 6 sample.exe          # ASCII strings
$ strings -n 6 -el sample.exe      # 16-bit little-endian (common in Windows PE)
$ strings -n 6 sample | grep -i "http\|ftp\|cmd\|powershell\|reg"

What to look for:

• URLs, domain names, IP addresses (C2 indicators)
• File paths (C:\Windows\, %APPDATA%, /tmp/)
• Registry keys (HKEY_, SOFTWARE\)
• API function names (if not using IAT obfuscation)
• Error messages or debug strings (often reveal program logic)
• Base64-encoded blobs
• Encoded/encrypted data (high entropy, no readable content)

FLOSS

Standard strings misses string-building techniques common in malware:

• Stack strings: characters pushed onto the stack one at a time
• Tight loops: characters concatenated in a loop
• Encoded strings: XOR or other simple encoding decoded at runtime

FLOSS (FireEye Labs Obfuscated String Solver) recovers these:

$ floss sample.exe

FLOSS output is larger but contains strings that would otherwise be invisible to static analysis.

VirusTotal

Search by hash to check if the sample is known:

$ sha256sum sample.exe | awk '{print $1}'
# paste into virustotal.com or use the API

VirusTotal shows:

• Detection ratio (X/72 engines detect it)
• Family names assigned by different engines
• Behavioral sandbox reports
• Network indicators from dynamic analysis
• Relationships to other files, URLs, domains, IPs

A low detection ratio does not mean the file is safe — it may be a new or targeted sample. A high ratio with consistent family names (e.g., 45/72 flag it as “Ryuk”) gives strong classification confidence.

Packing Detection

Packed binaries have their real code compressed or encrypted. Triage should determine if a sample is packed before investing time in disassembly.

Entropy Analysis

High entropy sections indicate compressed or encrypted data. Normal code has entropy around 5–6 bits/byte; packed or encrypted sections approach 8.

$ binwalk -E sample.exe    # plot entropy by offset

A .text section with entropy above 7.0 is a strong packing indicator.

Section Names

Packers often replace standard section names (.text, .data, .rdata) with their own:

$ readelf -S sample          # ELF section headers
$ objdump -p sample.exe      # PE section headers (via MinGW readelf on Windows)

Suspicious section names: UPX0, UPX1, .packed, .armadillo, blank names, or names with high-entropy content.

pev (PE tools)

$ pepack sample.exe          # packer identification heuristics
$ pesec sample.exe           # security features: ASLR, DEP, SafeSEH
$ pehash sample.exe          # multiple hash types

Known Packers

UPX is by far the most common. It can be detected and unpacked automatically:

$ upx -d packed.exe -o unpacked.exe

For custom packers, note the packer signatures and proceed to dynamic unpacking (run the unpack stub in a debugger, dump memory at the Original Entry Point — covered in Week 7).

Triage Checklist