ZFS Forensics
- ZFS Forensics
ZFS (Zettabyte File System) was developed by Sun Microsystems and released as open source in 2005. It is now widely deployed on FreeBSD, illumos-based systems, Linux (via OpenZFS), and NAS appliances (FreeNAS/TrueNAS, Synology). Its architecture differs fundamentally from traditional file systems like NTFS, ext4, and HFS+. A forensic examiner who approaches a ZFS pool with NTFS assumptions will draw incorrect conclusions.
ZFS is simultaneously a volume manager and a file system. It eliminates the boundary between the two, which has significant implications for evidence acquisition and analysis.
Core Design Principles
Copy-on-Write (CoW)
ZFS never overwrites existing data. When any block — file content or metadata — is modified, ZFS writes the new version to a new location on disk. The old version remains in place until its space is explicitly reclaimed. This behavior is fundamental to nearly every other ZFS feature and to its forensic profile.
Forensic implication: on a live ZFS system, previous versions of modified files and metadata may be physically present on disk even without snapshots, for as long as the free space has not been consumed. The pool’s transaction log preserves the history of which block was the “current” version at each point in time.
Transactions and Transaction Groups
All writes in ZFS are grouped into transaction groups (TXGs). A TXG is a set of changes that are committed atomically to the pool. Every ~5 seconds (by default), the current open TXG is flushed to disk. Because of CoW, the previous TXG’s data remains on disk until it is overwritten by new allocations.
The uberblock — the pool’s root pointer, written to every TXG — records the transaction group ID and the root of the current valid pool state. ZFS maintains a ring of 128 uberblock slots and writes them in rotation, so several previous uberblocks (and thus several previous complete pool states) may be simultaneously present on disk.
Forensic implication: on a recently seized or powered-down system, it may be possible to access previous TXG states by identifying earlier valid uberblocks in the ring. This can recover the state of the pool as it existed several transaction groups ago — effectively a limited form of “time travel” without snapshots.
End-to-End Checksumming
Every block in ZFS — data blocks, metadata blocks, even the checksum blocks themselves — carries a checksum (default: Fletcher-4 for metadata, SHA-256 optional for data). The checksum is stored in the parent block that references the current block, not in the block itself. This prevents an attacker who modifies a block from simply updating that block’s checksum.
Forensic implication: checksum mismatches detected during pool import or zpool scrub indicate physical corruption or deliberate modification of on-disk data. A pool with widespread checksum errors on a suspect’s drive may indicate a tamper attempt that failed to update all the parent pointers correctly.
Pool Structure
vdevs
A ZFS pool (zpool) is built from one or more virtual devices (vdevs). Each vdev is a logical storage unit:
| vdev Type | Description |
|---|---|
| Disk / file | A single raw device |
| mirror | Two or more disks mirrored |
| RAID-Z1/2/3 | RAID-5/6/7 equivalent; striped with parity |
| spare | Hot spare |
| cache (L2ARC) | Read cache (SSD) |
| log (ZIL) | Write-ahead log device |
Forensic implication: to image a ZFS pool correctly, you must image every leaf device in the pool. A pool spread across three disks requires images of all three. Imaging only one disk of a mirror yields a complete (if redundant) copy; imaging only one disk of a RAID-Z stripe yields an incomplete and uninterpretable image.
Labels
Each device in a ZFS pool contains four labels (two at the start, two at the end of the device). Each label holds:
- Pool configuration (name, GUID, vdev tree topology)
- The uberblock array (128 uberblock slots in a ring)
The uberblock with the highest valid transaction group ID is the current pool state. Older uberblocks in the ring represent previous states.
Forensic implication: labels are redundant and spread across all devices. If a suspect attempts to destroy a pool by wiping the beginning of one device, the labels at the end of that device and on other devices may survive. The label array can be examined with zdb -l <device> even on a non-imported pool.
Dataset Model
Within a pool, ZFS organizes data into datasets:
| Dataset Type | Description |
|---|---|
| Filesystem | A mountable POSIX file system |
| Volume (zvol) | A block device backed by ZFS (e.g., for iSCSI or swap) |
| Snapshot | A read-only, point-in-time copy of a filesystem or volume |
| Clone | A writable copy derived from a snapshot |
| Bookmark | A lightweight pointer to a snapshot’s transaction group (no data retained) |
Datasets exist in a hierarchy (e.g., pool/home, pool/home/user1) and inherit properties from their parent unless overridden.
Dataset On-Disk Representation
Each dataset is represented by a DSL (Dataset and Snapshot Layer) directory object and a DSL dataset object pointing to the root of the file system’s object set. The file system’s objects (files, directories, attributes) are stored in an object set — an array of dnodes (similar in concept to inodes).
Object Layer
DMU (Data Management Unit)
The DMU is ZFS’s object abstraction layer. Every piece of data in ZFS — whether a file, a directory, a symlink, or an internal metadata structure — is stored as a DMU object. Each object is identified by an object number within its object set.
Dnodes
A dnode is ZFS’s equivalent of an inode. Each dnode contains:
| Field | Purpose |
|---|---|
| Object type | File, directory, symlink, ZFS attribute, etc. |
| Data size | Logical size of the object |
| Block pointers | Up to three direct block pointers (or an indirect block tree) |
| Bonus buffer | Small embedded space for type-specific metadata (e.g., file timestamps, link target for short symlinks) |
Forensic note: dnodes do not have a direct equivalent to ext’s i_dtime deletion timestamp. ZFS’s CoW design means deleted file data remains on disk, but there is no per-dnode record of when deletion occurred. Deletion time must be inferred from transaction group IDs and pool import history.
Block Pointers
A ZFS block pointer (blkptr_t) is a 128-byte structure that describes a single block of data. It contains:
- Up to three DVAs (Data Virtual Addresses) — the same block may be stored in up to three locations for redundancy
- The logical and physical sizes of the block
- The checksum type and checksum value for the block
- A birth transaction group (the TXG in which the block was written)
- Compression type and gang block flag
- A “dead list” bit indicating the block is scheduled for freeing
Forensic note: the birth transaction group field in every block pointer is a timestamp of when that block was written, expressed as a TXG number. TXG numbers increase monotonically. Given the pool’s uberblock history, it is possible to bound the wall-clock time at which any block was created. This provides a forensic timeline that is resistant to timestamp manipulation — an attacker can change file timestamps (stored in the dnode bonus buffer), but cannot retroactively alter the birth TXG of the underlying block without rewriting it (which creates a new block with a new birth TXG).
ZFS Attribute Layer
File metadata (timestamps, permissions, ownership) is stored in a System Attribute (SA) layout within each dnode’s bonus buffer or in a separate SA attribute file. Timestamps stored here can be manipulated by a user with appropriate permissions, just as on any POSIX file system.
Forensic note: because ZFS stores birth TXGs on all blocks, a file whose POSIX timestamps have been forged can often be detected: the birth TXG of the inode’s data block post-dates the forged mtime. The discrepancy between the forged POSIX timestamp and the CoW-derived birth timestamp is hard to eliminate without re-creating the entire file.
Snapshots
A ZFS snapshot is a read-only, instantaneous copy of a dataset. Because of CoW, creating a snapshot is nearly instantaneous — it simply marks all current blocks as referenced by both the live dataset and the snapshot. No data is copied.
When blocks in the live dataset are later modified (or the live dataset is deleted), the blocks still referenced by the snapshot are retained on disk. The snapshot’s data persists until the snapshot is explicitly destroyed.
Forensic note: snapshots are one of the most valuable forensic artifacts in a ZFS environment. They preserve the exact state of the file system at a specific point in time, including files that have since been deleted or modified. On a production server with automatic snapshotting (e.g., via zfs-auto-snapshot or Time Machine on macOS with APFS — a ZFS descendant), a complete history of file system changes may be available:
zfs list -t snapshot -r poolname
Accessing a snapshot:
ls poolname/filesystem/.zfs/snapshot/
The .zfs/snapshot/ directory is hidden by default but accessible. Each snapshot is mountable as a read-only file system, making evidence acquisition from snapshots straightforward and non-destructive.
Snapshot retention: many environments automate snapshot creation (hourly, daily, weekly) and retention. An automated snapshot from the day before a deletion event may contain the deleted file. The snapshot creation time is stored in the DSL dataset object and reflects when zfs snapshot was run.
ZIL: ZFS Intent Log
For applications requiring synchronous writes (databases, NFS servers), ZFS maintains a ZIL (ZFS Intent Log) — a write-ahead log where synchronous write requests are recorded before being acknowledged to the application. Once the next TXG commits the data to its permanent location, the ZIL records are no longer needed.
The ZIL exists in one of two forms:
- Embedded in the pool (stored in regular pool space)
- On a separate log device (SLOG): a fast SSD or NVMe device
Forensic implication: if a system crashed while writes were in the ZIL, those uncommitted writes remain in the ZIL and will be replayed on next pool import. A ZIL on a separate device that is seized independently of the pool may contain recent write activity that never made it into the main pool’s permanent storage. zdb can be used to examine ZIL contents.
Free Space and Data Recovery
How Blocks Are Freed
When a file is deleted, ZFS schedules its blocks for freeing by appending them to the dataset’s dead list. The blocks are not immediately reclaimed; they are freed when the TXG containing the deletion commits. If a snapshot exists that references those blocks, they are not freed until the snapshot is destroyed.
Forensic implication: on a pool with no snapshots and ample free space, deleted files may still be physically present on disk (because CoW leaves old blocks in place until the space is reused). However, without metadata linking deleted blocks to their original file structure, recovery requires carving. There is no inode bitmap equivalent in ZFS — identifying deleted content requires either snapshot access, uberblock archaeology, or raw block carving.
Identifying Recoverable Data
Approaches for recovering deleted content from ZFS:
-
Snapshots: the most reliable method. If a snapshot predates the deletion, mount it and access the file directly.
-
Uberblock archaeology: identify older valid uberblocks in the label’s uberblock ring. Import the pool at an older TXG using
zpool import -T <txg>(on OpenZFS). This effectively rolls back the pool to a previous state. Destructive — use only on a forensic copy. -
zdbexamination:zdb(the ZFS debugger) can traverse the pool’s internal metadata structures, walk object sets, and dump block contents. It can reveal live and recently freed objects. -
Raw block carving: if the above are unavailable, carve the raw device images for known file signatures. ZFS compression (LZ4 is the default) complicates carving — compressed blocks will not match uncompressed file signatures.
Compression Consideration
ZFS supports transparent compression (LZ4, LZJB, gzip, zstd) per dataset. When compression is enabled:
- Blocks are stored compressed on disk
- File carving tools that search for uncompressed signatures will fail
- The compressed data must be read through the ZFS stack (which decompresses on the fly) to be interpretable
Forensic note: evidence that was on a compressed dataset is most reliably extracted by mounting the pool (or a snapshot) and reading files through the normal interface, rather than trying to carve the raw device.
Encryption
ZFS native encryption (available in OpenZFS 0.8+) encrypts dataset contents at the block level. Each dataset has its own encryption key derived from a user-supplied passphrase (via PBKDF2) or a raw key.
What is not encrypted:
- Dataset metadata: dataset names, properties, dataset tree structure, snapshot names
- Pool-level metadata: pool name, vdev topology, labels
Forensic note: even on an encrypted ZFS pool, an examiner without the key can still determine: the pool name and structure, which datasets exist, the names of snapshots, when snapshots were created, and dataset property settings. This structural metadata may reveal organizational information (dataset names often reflect content: tank/home/users, backup/finance, vm/suspects-laptop) even when the content itself is inaccessible.
Anti-Forensics Considerations
zfs destroy on Snapshots
Destroying a snapshot is irreversible: once a snapshot is destroyed and the blocks it exclusively referenced are freed and overwritten, that file system state is gone. An attacker who destroys all snapshots before or during an investigation eliminates the most powerful recovery vector.
Forensic note: snapshot creation and destruction events are logged in the ZFS pool’s history:
zpool history poolname
The pool history log records every administrative command run against the pool (with timestamps and the user who ran it). It is stored within the pool itself and survives reboots. Evidence of snapshot destruction in the history log is itself forensically significant.
Secure Erasure
Because of CoW, simply deleting a file does not overwrite its data. A suspect who wants to securely erase data must either:
- Use
zfs destroyon an encrypted dataset (the encryption key is discarded, making data cryptographically inaccessible) - Physically destroy the storage devices
Standard tools like shred and wipe, which rely on overwriting specific file blocks, do not work as expected on ZFS — the file system will write new blocks for the new data, leaving the original blocks in place until they are naturally reclaimed. This is an important difference from ext or NTFS.
Forensic Workflow Summary
| Step | Action | Tool |
|---|---|---|
| 1 | Image all pool member devices | dd, dc3dd |
| 2 | Identify pool structure and labels | zdb -l <device> |
| 3 | List available uberblocks and TXGs | zdb -u <device> |
| 4 | Import pool read-only | zpool import -o readonly=on <pool> |
| 5 | List all datasets and snapshots | zfs list -t all -r <pool> |
| 6 | Mount and examine snapshots | ls <mountpoint>/.zfs/snapshot/ |
| 7 | Review pool command history | zpool history <pool> |
| 8 | Examine object internals | zdb -ddddd <pool>/<dataset> <object> |
| 9 | Carve free space (if no snapshots) | File carving tools on raw device images |
Always import a forensic copy read-only (readonly=on) or with the altroot option to prevent modifying the evidence pool. Importing a pool normally will write a new TXG and update the uberblock, modifying the evidence.
Further Reading
- OpenZFS Documentation
zdb(8)man page — the primary tool for low-level ZFS forensic examinationzpool(8)andzfs(8)man pages- A. Watanabe. (2011). “ZFS Forensics.” Digital Forensics Magazine, Issue 9.
- B. Carrier. (2005). File System Forensic Analysis. Addison-Wesley. (General file system forensics foundations applicable to ZFS analysis.)