courses

ZFS Forensics

ZFS (Zettabyte File System) was developed by Sun Microsystems and released as open source in 2005. It is now widely deployed on FreeBSD, illumos-based systems, Linux (via OpenZFS), and NAS appliances (FreeNAS/TrueNAS, Synology). Its architecture differs fundamentally from traditional file systems like NTFS, ext4, and HFS+. A forensic examiner who approaches a ZFS pool with NTFS assumptions will draw incorrect conclusions.

ZFS is simultaneously a volume manager and a file system. It eliminates the boundary between the two, which has significant implications for evidence acquisition and analysis.

Core Design Principles

Copy-on-Write (CoW)

ZFS never overwrites existing data. When any block — file content or metadata — is modified, ZFS writes the new version to a new location on disk. The old version remains in place until its space is explicitly reclaimed. This behavior is fundamental to nearly every other ZFS feature and to its forensic profile.

Forensic implication: on a live ZFS system, previous versions of modified files and metadata may be physically present on disk even without snapshots, for as long as the free space has not been consumed. The pool’s transaction log preserves the history of which block was the “current” version at each point in time.

Transactions and Transaction Groups

All writes in ZFS are grouped into transaction groups (TXGs). A TXG is a set of changes that are committed atomically to the pool. Every ~5 seconds (by default), the current open TXG is flushed to disk. Because of CoW, the previous TXG’s data remains on disk until it is overwritten by new allocations.

The uberblock — the pool’s root pointer, written to every TXG — records the transaction group ID and the root of the current valid pool state. ZFS maintains a ring of 128 uberblock slots and writes them in rotation, so several previous uberblocks (and thus several previous complete pool states) may be simultaneously present on disk.

Forensic implication: on a recently seized or powered-down system, it may be possible to access previous TXG states by identifying earlier valid uberblocks in the ring. This can recover the state of the pool as it existed several transaction groups ago — effectively a limited form of “time travel” without snapshots.

End-to-End Checksumming

Every block in ZFS — data blocks, metadata blocks, even the checksum blocks themselves — carries a checksum (default: Fletcher-4 for metadata, SHA-256 optional for data). The checksum is stored in the parent block that references the current block, not in the block itself. This prevents an attacker who modifies a block from simply updating that block’s checksum.

Forensic implication: checksum mismatches detected during pool import or zpool scrub indicate physical corruption or deliberate modification of on-disk data. A pool with widespread checksum errors on a suspect’s drive may indicate a tamper attempt that failed to update all the parent pointers correctly.

Pool Structure

vdevs

A ZFS pool (zpool) is built from one or more virtual devices (vdevs). Each vdev is a logical storage unit:

vdev Type Description
Disk / file A single raw device
mirror Two or more disks mirrored
RAID-Z1/2/3 RAID-5/6/7 equivalent; striped with parity
spare Hot spare
cache (L2ARC) Read cache (SSD)
log (ZIL) Write-ahead log device

Forensic implication: to image a ZFS pool correctly, you must image every leaf device in the pool. A pool spread across three disks requires images of all three. Imaging only one disk of a mirror yields a complete (if redundant) copy; imaging only one disk of a RAID-Z stripe yields an incomplete and uninterpretable image.

Labels

Each device in a ZFS pool contains four labels (two at the start, two at the end of the device). Each label holds:

The uberblock with the highest valid transaction group ID is the current pool state. Older uberblocks in the ring represent previous states.

Forensic implication: labels are redundant and spread across all devices. If a suspect attempts to destroy a pool by wiping the beginning of one device, the labels at the end of that device and on other devices may survive. The label array can be examined with zdb -l <device> even on a non-imported pool.

Dataset Model

Within a pool, ZFS organizes data into datasets:

Dataset Type Description
Filesystem A mountable POSIX file system
Volume (zvol) A block device backed by ZFS (e.g., for iSCSI or swap)
Snapshot A read-only, point-in-time copy of a filesystem or volume
Clone A writable copy derived from a snapshot
Bookmark A lightweight pointer to a snapshot’s transaction group (no data retained)

Datasets exist in a hierarchy (e.g., pool/home, pool/home/user1) and inherit properties from their parent unless overridden.

Dataset On-Disk Representation

Each dataset is represented by a DSL (Dataset and Snapshot Layer) directory object and a DSL dataset object pointing to the root of the file system’s object set. The file system’s objects (files, directories, attributes) are stored in an object set — an array of dnodes (similar in concept to inodes).

Object Layer

DMU (Data Management Unit)

The DMU is ZFS’s object abstraction layer. Every piece of data in ZFS — whether a file, a directory, a symlink, or an internal metadata structure — is stored as a DMU object. Each object is identified by an object number within its object set.

Dnodes

A dnode is ZFS’s equivalent of an inode. Each dnode contains:

Field Purpose
Object type File, directory, symlink, ZFS attribute, etc.
Data size Logical size of the object
Block pointers Up to three direct block pointers (or an indirect block tree)
Bonus buffer Small embedded space for type-specific metadata (e.g., file timestamps, link target for short symlinks)

Forensic note: dnodes do not have a direct equivalent to ext’s i_dtime deletion timestamp. ZFS’s CoW design means deleted file data remains on disk, but there is no per-dnode record of when deletion occurred. Deletion time must be inferred from transaction group IDs and pool import history.

Block Pointers

A ZFS block pointer (blkptr_t) is a 128-byte structure that describes a single block of data. It contains:

Forensic note: the birth transaction group field in every block pointer is a timestamp of when that block was written, expressed as a TXG number. TXG numbers increase monotonically. Given the pool’s uberblock history, it is possible to bound the wall-clock time at which any block was created. This provides a forensic timeline that is resistant to timestamp manipulation — an attacker can change file timestamps (stored in the dnode bonus buffer), but cannot retroactively alter the birth TXG of the underlying block without rewriting it (which creates a new block with a new birth TXG).

ZFS Attribute Layer

File metadata (timestamps, permissions, ownership) is stored in a System Attribute (SA) layout within each dnode’s bonus buffer or in a separate SA attribute file. Timestamps stored here can be manipulated by a user with appropriate permissions, just as on any POSIX file system.

Forensic note: because ZFS stores birth TXGs on all blocks, a file whose POSIX timestamps have been forged can often be detected: the birth TXG of the inode’s data block post-dates the forged mtime. The discrepancy between the forged POSIX timestamp and the CoW-derived birth timestamp is hard to eliminate without re-creating the entire file.

Snapshots

A ZFS snapshot is a read-only, instantaneous copy of a dataset. Because of CoW, creating a snapshot is nearly instantaneous — it simply marks all current blocks as referenced by both the live dataset and the snapshot. No data is copied.

When blocks in the live dataset are later modified (or the live dataset is deleted), the blocks still referenced by the snapshot are retained on disk. The snapshot’s data persists until the snapshot is explicitly destroyed.

Forensic note: snapshots are one of the most valuable forensic artifacts in a ZFS environment. They preserve the exact state of the file system at a specific point in time, including files that have since been deleted or modified. On a production server with automatic snapshotting (e.g., via zfs-auto-snapshot or Time Machine on macOS with APFS — a ZFS descendant), a complete history of file system changes may be available:

zfs list -t snapshot -r poolname

Accessing a snapshot:

ls poolname/filesystem/.zfs/snapshot/

The .zfs/snapshot/ directory is hidden by default but accessible. Each snapshot is mountable as a read-only file system, making evidence acquisition from snapshots straightforward and non-destructive.

Snapshot retention: many environments automate snapshot creation (hourly, daily, weekly) and retention. An automated snapshot from the day before a deletion event may contain the deleted file. The snapshot creation time is stored in the DSL dataset object and reflects when zfs snapshot was run.

ZIL: ZFS Intent Log

For applications requiring synchronous writes (databases, NFS servers), ZFS maintains a ZIL (ZFS Intent Log) — a write-ahead log where synchronous write requests are recorded before being acknowledged to the application. Once the next TXG commits the data to its permanent location, the ZIL records are no longer needed.

The ZIL exists in one of two forms:

Forensic implication: if a system crashed while writes were in the ZIL, those uncommitted writes remain in the ZIL and will be replayed on next pool import. A ZIL on a separate device that is seized independently of the pool may contain recent write activity that never made it into the main pool’s permanent storage. zdb can be used to examine ZIL contents.

Free Space and Data Recovery

How Blocks Are Freed

When a file is deleted, ZFS schedules its blocks for freeing by appending them to the dataset’s dead list. The blocks are not immediately reclaimed; they are freed when the TXG containing the deletion commits. If a snapshot exists that references those blocks, they are not freed until the snapshot is destroyed.

Forensic implication: on a pool with no snapshots and ample free space, deleted files may still be physically present on disk (because CoW leaves old blocks in place until the space is reused). However, without metadata linking deleted blocks to their original file structure, recovery requires carving. There is no inode bitmap equivalent in ZFS — identifying deleted content requires either snapshot access, uberblock archaeology, or raw block carving.

Identifying Recoverable Data

Approaches for recovering deleted content from ZFS:

  1. Snapshots: the most reliable method. If a snapshot predates the deletion, mount it and access the file directly.

  2. Uberblock archaeology: identify older valid uberblocks in the label’s uberblock ring. Import the pool at an older TXG using zpool import -T <txg> (on OpenZFS). This effectively rolls back the pool to a previous state. Destructive — use only on a forensic copy.

  3. zdb examination: zdb (the ZFS debugger) can traverse the pool’s internal metadata structures, walk object sets, and dump block contents. It can reveal live and recently freed objects.

  4. Raw block carving: if the above are unavailable, carve the raw device images for known file signatures. ZFS compression (LZ4 is the default) complicates carving — compressed blocks will not match uncompressed file signatures.

Compression Consideration

ZFS supports transparent compression (LZ4, LZJB, gzip, zstd) per dataset. When compression is enabled:

Forensic note: evidence that was on a compressed dataset is most reliably extracted by mounting the pool (or a snapshot) and reading files through the normal interface, rather than trying to carve the raw device.

Encryption

ZFS native encryption (available in OpenZFS 0.8+) encrypts dataset contents at the block level. Each dataset has its own encryption key derived from a user-supplied passphrase (via PBKDF2) or a raw key.

What is not encrypted:

Forensic note: even on an encrypted ZFS pool, an examiner without the key can still determine: the pool name and structure, which datasets exist, the names of snapshots, when snapshots were created, and dataset property settings. This structural metadata may reveal organizational information (dataset names often reflect content: tank/home/users, backup/finance, vm/suspects-laptop) even when the content itself is inaccessible.

Anti-Forensics Considerations

zfs destroy on Snapshots

Destroying a snapshot is irreversible: once a snapshot is destroyed and the blocks it exclusively referenced are freed and overwritten, that file system state is gone. An attacker who destroys all snapshots before or during an investigation eliminates the most powerful recovery vector.

Forensic note: snapshot creation and destruction events are logged in the ZFS pool’s history:

zpool history poolname

The pool history log records every administrative command run against the pool (with timestamps and the user who ran it). It is stored within the pool itself and survives reboots. Evidence of snapshot destruction in the history log is itself forensically significant.

Secure Erasure

Because of CoW, simply deleting a file does not overwrite its data. A suspect who wants to securely erase data must either:

Standard tools like shred and wipe, which rely on overwriting specific file blocks, do not work as expected on ZFS — the file system will write new blocks for the new data, leaving the original blocks in place until they are naturally reclaimed. This is an important difference from ext or NTFS.

Forensic Workflow Summary

Step Action Tool
1 Image all pool member devices dd, dc3dd
2 Identify pool structure and labels zdb -l <device>
3 List available uberblocks and TXGs zdb -u <device>
4 Import pool read-only zpool import -o readonly=on <pool>
5 List all datasets and snapshots zfs list -t all -r <pool>
6 Mount and examine snapshots ls <mountpoint>/.zfs/snapshot/
7 Review pool command history zpool history <pool>
8 Examine object internals zdb -ddddd <pool>/<dataset> <object>
9 Carve free space (if no snapshots) File carving tools on raw device images

Always import a forensic copy read-only (readonly=on) or with the altroot option to prevent modifying the evidence pool. Importing a pool normally will write a new TXG and update the uberblock, modifying the evidence.

Further Reading