File Carving

File Carving

File recovery (undelete) relies on intact file system metadata to locate file content. File carving is what you do when the metadata are gone — when the MFT record has been overwritten or corrupted, but the data clusters may still be on disk. Carving pulls raw bytes from the media, entirely ignoring the file system. You do not need to know what file system was used; you only need to know what type of file you are looking for.

File carving is more necessary on modern systems (Windows overwrites deleted file records quickly) and also more viable (lower fragmentation, larger disks with mostly unmodified media files).

Basic File Carving Techniques

The simplest case is when the file’s content clusters are intact and contiguous (unfragmented). Three main approaches:

Header-Footer Carving

Most file formats embed standard magic bytes at the start and end of the file. To carve a JPEG:

Search unallocated space for the start-of-image marker: 0xFF 0xD8
Read forward until the end-of-image marker: 0xFF 0xD9
Extract the bytes in between — that is the image

This works reliably when the file is unfragmented and the header/footer pair is unique enough on the disk.

Common problems:

Some formats lack a footer — carver must use the file length encoded in the header (if present) or fall back to copying up to the maximum known file size
Header bytes that are also common data sequences cause false positives (e.g., the MP3 header is just “mp”, which appears everywhere)
Embedded files — a JPEG inside a Word document will generate spurious hits
Truncated or partially overwritten files

Tools: Scalpel, Foremost, EnCase file finder.

A reference for file format signatures: Gary Kessler’s File Signatures Table.

File-Structure-Based Carving

Rather than relying solely on a header and footer, structure-based carving reads the internal layout of a file format to determine its size, validate its structure, and reduce false positives.

JPEGs encode their own size and may contain embedded thumbnails — a thumbnail visible inside a candidate cluster is strong evidence that the cluster belongs to that JPEG
PDFs and Word documents have predictable internal structure (object tables, section boundaries) that can be verified
Knowing the cluster size on the disk allows the carver to align its search to cluster boundaries rather than arbitrary byte offsets, improving both accuracy and performance

Tools: Foremost, PhotoRec.

Content-Based Carving

Content-based carving applies machine learning or statistical models to classify clusters by file type, rather than searching for explicit signatures.

Useful features:

Information entropy: plaintext has low entropy (highly orderly); compressed or encrypted data has high entropy; images fall in between. A low-entropy cluster is likely a text file even without a header.
Character count and language recognition: natural-language text has distinctive character frequency distributions
Structured data signatures: HTML and XML have recognizable patterns
Statistical resemblance to known file types: train a model on known-good JPEGs, then score candidate clusters

Content-based carving is especially useful for recovering plaintext data, which has no standard header or footer but is easily recognized by its statistical properties.

Carving Fragmented Files

A fragmented file has its content stored in multiple non-contiguous clusters on disk. If the MFT record is gone, there is no pointer linking those clusters together. They could be gigabytes apart. This is the hardest problem in file carving.

Fragmentation is less common on modern large disks (better allocation algorithms, mostly static media files), but must still be handled — failure to attempt recovery of fragmented files can be attacked in court as a failure of due diligence.

Bifragment Gap Carving (BGC)

Developed by Simson Garfinkel (2007). Works when a file has been split into exactly two fragments — which is the most common case.

Key insight: the first fragment contains the header; the second contains the footer. The “gap” between them is unrelated data.

Algorithm:

Find all header clusters and all footer clusters in unallocated space
For each header-footer pair, iterate over possible gap sizes g = 1, 2, 3, ...
For each gap size, try removing every possible set of g contiguous clusters between the header and footer
Validate the remaining data as a candidate file
Stop when validation succeeds

Runtime: O(nd²) in the worst case, where d is the distance between header and footer clusters and n is the number of files being carved. Missing or corrupt clusters push toward worst-case runtime.

Requirements for success: all clusters intact, header fragment before footer fragment, reliable validation mechanism.

SmartCarving

Developed by Anandabrata Pal and Nasir Memon (2009). Works for files fragmented into more than two parts (over 99.9% accuracy). Does not assume anything about how or where the fragments are located.

Three phases:

Preprocessing: decompress and decrypt compressed/encrypted clusters; remove all allocated clusters. This dramatically reduces the candidate set — only unallocated clusters of the right type need further analysis.
Collation: classify remaining clusters by file type using entropy, keyword searches, ASCII detection, and file signatures. A JPEG cluster looks different from a PDF cluster. Breaking the problem into per-type subsets reduces the search space.
Reassembly: apply the SHT-PUP (sequential hypothesis parallel unique path) algorithm — a graph-theoretic approach. Build a graph where nodes are candidate clusters and edges (with weights) connect clusters that are likely to be adjacent based on content similarity. For JPEG clusters, adjacent clusters in an image typically have similar color palettes; the algorithm exploits this to find the most probable ordering. Find the maximal weighted paths starting from each header cluster.

SmartCarving is analogous to DNA fragment assembly in bioinformatics and represents the current state of the art. It handles up to (and in principle, more than) four fragments per file, provided no fragments are missing.

Carving Beyond Hard Drives

File carving can be applied to any binary stream — not just disk images.

Recovering Data from Slack Space

When a file is deleted and its clusters are reallocated to a smaller new file, the old file’s data can persist in the slack space — the unused bytes at the end of the last cluster of the new file.

Slack space cannot yield a complete file, only fragments
But a fragment containing a name, username, or account number can still be compelling evidence

How to find it:

If a file record survives but the content has been reallocated, manually examine the slack space of the clusters to which it points
Perform a keyword search in unallocated space (e.g., search for a suspect’s name or account number)
Look for directory index entries that still exist (unallocated but not overwritten) — they will point to a file record that has been reused, but the parent directory’s B-tree rebalancing sometimes leaves old entries in place, and those entries may point to clusters whose slack space still holds the original data

For short files whose entire content fit in a resident $DATA attribute in the file record, the directory index entry pointing to that record may survive even after the record is reused. The content was embedded in the record itself, so reading the record the entry points to may yield the original content.

Memory Carving

RAM can be treated exactly like a disk image: carve it for file headers, footers, and content.

Why capture RAM before unplugging:

Encryption keys for full-disk encryption (BitLocker, VeraCrypt) are loaded into RAM while the system runs. Once power is cut, the key is gone.
Malware is often packed, obfuscated, or encrypted on disk but runs in a more readable form in memory
Chat logs, recently viewed files, and browser session data may reside in RAM

Sources of memory images:

Live acquisition: dd /dev/mem on Linux, or tools like WinPmem on Windows
Crash dumps (kernel panic, BSOD): contain a full RAM image
Sleep/hibernate images: when a laptop sleeps, RAM is written to disk (e.g., hiberfil.sys on Windows, /private/var/vm/sleepimage on macOS). Even on an encrypted drive, this may contain the key needed to decrypt it.

Tooling: Volatility — extract running processes (including hidden rootkit processes), DLLs, network connections, registry hives, and file content from memory images.

Challenge: RAM is highly fragmented; advanced carving techniques are required.

Carving Network Traffic

A packet capture (PCAP) is a binary stream of network frames. Strip the Ethernet, IP, and TCP/UDP headers from each packet; what remains is application-layer payload. That payload, reassembled in sequence order, is the binary file that the application received.

File carving against the reassembled payload can recover files transferred over the network:

A JPEG sent over Yahoo Messenger, IRC, or any other protocol can be carved out without reverse-engineering the application protocol
Tools like Wireshark have file carving capabilities built in (File → Export Objects)

Advantage: no need to understand the application-layer protocol. The file signature will appear somewhere in the data stream regardless of the enclosing protocol.

Common Carving Pitfalls

RIFF container ambiguity — RIFF at offset 0 is shared by WAV, AVI, and WebP. You need to check bytes 8–11 (WAVE, AVI , WEBP) to disambiguate. The by_magic() helper will return all three, so add a secondary check.
ZIP masquerade — DOCX, XLSX, PPTX, ODF, JAR, APK, and XPS all start with PK\x03\x04. To identify the specific format, open the ZIP and inspect the contained filenames (word/, xl/, AndroidManifest.xml, etc.).
Non-zero offset formats — TAR (ustar at offset 257), ISO 9660 (CD001 at offset 32769), MP4/M4A (ftyp at offset 4), and Prefetch (SCCA at offset 4) require seeking before matching. A naive buf.find(magic) on the raw image will find them wherever they appear, not just at the start of a file.
CA FE BA BE collision — Java .class files and macOS Mach-O fat binaries share the same magic. Disambiguate by checking bytes 4–7: a Mach-O fat header has a CPU architecture count there; a .class file has the major/minor version numbers.
Footer reliability — Many formats (BMP, ELF, MKV, etc.) have no reliable footer. For those, the file size is encoded in the header itself, so a real carver reads the length field and extracts exactly that many bytes.

Scalpel

Scalpel is the standard command-line header/footer carver. It reads a configuration file that maps file types to their magic bytes and optionally their footers, then scans the image and writes each hit to its own numbered file in a subdirectory.

Installation

sudo apt install scalpel

The default configuration file is at /etc/scalpel/scalpel.conf. Every file type is commented out by default — you must uncomment the types you want.

Configuration file format

Each active line has five fields:

extension   case-sensitive   max-size   header   [footer]

Field	Meaning
`extension`	Output file extension and subdirectory name
`case-sensitive`	`y` or `n` — almost always `n`
`max-size`	Maximum carved file size in bytes (not M/G — raw bytes)
`header`	Hex sequence or quoted string: `\xff\xd8\xff\xe0`
`footer`	Same format, optional — omit when the format encodes its own length

Hex bytes are written as \xNN. Literal strings are quoted. Wildcards are not supported in the header or footer fields.

Example: carving JPEG, PDF, and ZIP

Create a local config file my.conf:

# ext   case  max-size   header                 footer
jpg     n     5000000    \xff\xd8\xff\xe0        \xff\xd9
jpg     n     5000000    \xff\xd8\xff\xe1        \xff\xd9
pdf     n     5000000    %PDF                    %EOF
zip     n     10000000   PK\x03\x04

Notes:

JPEG has two common header variants (\xe0 = JFIF, \xe1 = EXIF) — add one line for each.
PDF uses the printable string %PDF as the header and %EOF as the footer.
ZIP has no reliable footer; Scalpel will carve up to max-size bytes.

Running Scalpel

# Use your custom config, write output to carved/
scalpel -c my.conf -o carved/ disk.img

# Use the system config (after editing /etc/scalpel/scalpel.conf)
scalpel -o carved/ disk.img

# Quiet mode (no per-file progress output)
scalpel -q -c my.conf -o carved/ disk.img

Scalpel creates one subdirectory per configured type under the output directory, and a audit.txt file that records every carved file, its byte offset in the image, and its size.

Reading the output

carved/
├── audit.txt            ← carving log with offsets and sizes
├── jpg-0-0/
│   ├── 00000000.jpg     ← carved at offset 0
│   └── 00004096.jpg     ← carved at offset 4096
└── pdf-0-0/
    └── 00008192.pdf

The filename encodes the byte offset in decimal. audit.txt contains a table with the exact offset, size, and source file for each carved artifact.

Limitations

No header-based length extraction — formats without a footer (BMP, ELF, ZIP) are carved to max-size; set that value conservatively
No ZIP-family disambiguation — DOCX/XLSX/JAR all share PK\x03\x04 and must each be added as separate lines if you want separate output directories
No validation — Scalpel does not verify internal file structure; a false positive is only detected visually or by opening the file