Digital Archiving and Digital Forensics

• Digital Archiving and Digital Forensics
  • Overview
  • Where forensics and archiving converge
  • Capturing the bits: forensic acquisition for archives
    • Imaging and write protection
    • Fixity: the archival word for “hash verification”
    • Old physical media: flux-level imaging
  • Abandoned and obsolete file formats
    • Two preservation strategies
    • Identifying an unknown format
  • Binary file formats
    • Anatomy of a binary format
    • Documenting a format with Kaitai Struct
  • Old software and emulation
    • Migration vs. emulation, revisited
    • Reading old data through old software
    • Carving when the filesystem is gone
  • Authenticity, provenance, and chain of custody
  • A combined archival-forensics workflow
  • References

Overview

Digital archiving (more precisely, digital preservation) is the practice of keeping digital information usable far beyond the lifespan of the hardware, software, and formats it was created with. Keeping the bits alive is only half the problem — a perfectly intact file is worthless if nothing can interpret it. Preservation therefore fights three distinct kinds of decay:

Digital forensics is, at its core, the same problem approached from the opposite motive. The forensic examiner recovers data from media that was never meant to be archived — a seized laptop, a wiped drive, a corrupted card — and must interpret unknown or hostile formats with no cooperation from the creator. The two fields are mirror images: the archivist preserves so the future can read; the examiner reconstructs because the past did not preserve. They use the same tools and the same evidentiary discipline, which is why a forensics course is the right place to study archiving.

Why this matters to you: the techniques below — bit-exact imaging, write-blocking, hashing for fixity, file carving, format identification, emulation — are the shared core of both disciplines. Learn them once and you can work an incident response case or rescue a dead researcher’s 1980s data.

Where forensics and archiving converge

The clearest proof that these are one skill set is BitCurator — an Ubuntu-derived environment, funded by the digital-preservation community, that is built almost entirely from digital-forensics software. Archivists adopted the forensic toolchain wholesale because it already solved their hardest problems.

The vocabulary differs; the practice is identical. A disk image is a disk image whether it backs a court exhibit or a library accession.

Capturing the bits: forensic acquisition for archives

Before you can worry about formats you must get the data off the carrier exactly once, without changing it. This is the forensic principle of acquisition, and archives apply it verbatim.

Imaging and write protection

Always read original media through a write blocker (a hardware device, or a software/read-only mount) so the act of reading cannot alter the source — this preserves both evidentiary value and the original timestamps. Capture a bit-for-bit image, not a file copy:

# Raw image with a hash computed during acquisition
sudo dd if=/dev/sdb of=disk.img bs=4M conv=noerror,sync status=progress
sha256sum disk.img > disk.img.sha256

# Or use the forensic EWF/E01 container (stores hashes + metadata internally)
ewfacquire -t evidence -f encase6 /dev/sdb

The conv=noerror,sync flags keep dd going past read errors (common on aged media) and pad the bad sectors so offsets stay aligned — vital when you carve the image later.

Fixity: the archival word for “hash verification”

A preservation system re-hashes its holdings on a schedule and compares against the value recorded at ingest. A mismatch means silent corruption. This is the same hash you’d use to prove an exhibit is unchanged:

sha256sum -c disk.img.sha256        # re-verify months later

Old physical media: flux-level imaging

This is where archiving pushes beyond ordinary forensics. A standard floppy controller decodes magnetic transitions into bytes — and discards everything it can’t decode, including weak bits, non-standard sectors, and copy protection. For degraded or unusual media that loss is fatal.

Dedicated hardware — Greaseweazle (open-source) and KryoFlux (from the Software Preservation Society) — instead captures the raw flux transitions: a high-resolution magnetic scan of the surface, format-agnostic, preserving even protection schemes for later analysis or emulation.

# Preservation master: raw flux, one stream per track (decode later, never re-read)
gw read flux_master.scp

# Logical image of a standard PC disk, decoded to a mountable .img
gw read --format ibm.1440 disk.img

Keep the flux file as the master and derive logical images from it — you can re-decode flux with better algorithms in the future, but you can never go back to the physical disk once it crumbles.

Abandoned and obsolete file formats

A file format is abandoned when its creator stops supporting it and, often, no current software reads it. The bytes are fine; the meaning is lost. This is the central problem of digital preservation and a routine problem in forensics, where you constantly meet formats no longer in common use.

Two preservation strategies

Deciding between them requires knowing a format’s significant properties — which characteristics must survive (the text? the pagination? the embedded formulas? the interactivity?). A spreadsheet migrated to CSV keeps the numbers but loses the formulas; for some collections that is acceptable, for others it destroys the record.

Identifying an unknown format

You cannot migrate or emulate what you cannot name. Identification relies on format registries that map byte signatures to format identifiers:

• PRONOM — the UK National Archives’ registry; assigns each format a PUID (e.g. fmt/40 for Word 97–2003 .doc).
• DROID — the National Archives’ Java identifier that matches files against PRONOM signatures.
• siegfried — a faster, modern PRONOM-based identifier (command sf).
• Brunnhilde — wraps siegfried to produce aggregate reports across a whole disk image or directory (counts by format, flags obsolete ones, runs ClamAV).

Worked example — identify and report on a directory of unknowns:

# Quick guess from libmagic
file mystery.dat
#   mystery.dat: Lotus 1-2-3 wk1 spreadsheet

# Authoritative identification with a PRONOM PUID
sf mystery.dat
#   ...
#   matches:
#    - ns      : pronom
#      id      : x-fmt/114        # Lotus 1-2-3 WK1
#      format  : Lotus 1-2-3 Worksheet
#      version : 2.x

# Whole-collection appraisal: what formats are here, which are obsolete?
brunnhilde.py -z source_dir/ reports/ accession_2024_017

When libmagic and PRONOM disagree or come up empty, fall back to TrID, which ranks candidate formats by statistical match against a large signature database — useful for formats too rare for the main registries:

trid mystery.dat
#   45.7% (.WK1) Lotus 1-2-3 spreadsheet
#   12.1% (.BIN) Generic binary

Binary file formats

Most obsolete formats are binary: structured byte layouts (headers, records, offsets, length fields) rather than human-readable text. When documentation is gone — abandoned, proprietary, or never published — recovering the data means reverse-engineering the format itself. This is identical to the forensic task of parsing an unknown application artifact.

Anatomy of a binary format

Most binary formats begin with a magic number — a fixed signature in the first bytes that identifies the format. This is exactly what file, DROID, and siegfried key on:

Beyond the magic, you reason about the structure by inspection.

Worked example — inspect and triage an unknown binary:

# Look at the header bytes directly
hexdump -C mystery.bin | head
#   00000000  4d 4d 00 2a 00 00 00 08  ...   |MM.*....|   ← "MM" = big-endian TIFF

# Find embedded files / known signatures inside a blob (firmware, containers)
binwalk mystery.bin
#   DECIMAL    HEX        DESCRIPTION
#   0          0x0        TIFF image data, big-endian
#   65536      0x10000    Zlib compressed data

binwalk walks a file looking for the magic numbers of embedded content — the standard first move on firmware dumps, disk images, and proprietary container formats that bundle several payloads.

Documenting a format with Kaitai Struct

Once you understand a layout, capture it formally so it is reusable and self-documenting. Kaitai Struct lets you describe a binary format declaratively in a .ksy (YAML) file, then compile a parser for Python, C++, Java, and more — turning ad-hoc reverse engineering into a durable preservation artifact.

# simple_record.ksy — a made-up legacy record format
meta:
  id: simple_record
  endian: le
  file-extension: rec
seq:
  - id: magic
    contents: [0x52, 0x45, 0x43, 0x31]   # "REC1"
  - id: record_count
    type: u4
  - id: records
    type: entry
    repeat: expr
    repeat-expr: record_count
types:
  entry:
    seq:
      - id: id
        type: u4
      - id: name_len
        type: u2
      - id: name
        type: str
        size: name_len
        encoding: UTF-8

# Compile to a Python parser and read a file
kaitai-struct-compiler -t python simple_record.ksy
python3 -c "from simple_record import SimpleRecord; \
  d = SimpleRecord.from_file('old.rec'); \
  print([(r.id, r.name) for r in d.records])"

A .ksy specification is itself a preservation deliverable: it documents the format so the data remains readable even if your one-off parsing script is lost. For interactive work, hex editors with templating (e.g. 010 Editor binary templates, ImHex patterns) serve the same role.

Old software and emulation

Formats and the software that reads them are coupled: an abandoned format is often readable only by an abandoned program, which in turn runs only on an abandoned operating system and CPU. Preserving the file can require preserving an entire computing environment.

Migration vs. emulation, revisited

When migration would destroy a format’s significant properties — interactive works, software-dependent art, formats with no modern converter — the answer is to keep the original bytes and recreate the environment that runs them:

• Emulators reproduce historic hardware in software: MAME (arcade and microcomputers), DOSBox (DOS), 86Box/PCem (vintage PCs), Mini vMac / Basilisk II (classic Macs), QEMU (broad architectures).
• Emulation-as-a-Service Infrastructure (EaaSI), led by Yale University Library, packages curated emulated environments behind a browser so an archive can offer “open this 1997 file in its original Windows 95
  • Office” without distributing the software itself.
• Software Heritage archives source code at scale, preserving the programs (and the ability to rebuild them) rather than only the data they produced.

Reading old data through old software

The forensic move — image first, then analyze in a controlled environment — applies directly. You carve or extract the obsolete files, then open them inside an emulator running the original application:

# 1. Make a logical image of a recovered disk (via flux master, above)
gw read --format ibm.1440 wp51_disk.img

# 2. Boot a period environment with the disk attached, in QEMU
qemu-system-i386 -fda wp51_disk.img -boot a   # e.g. a DOS + WordPerfect 5.1 setup

Carving when the filesystem is gone

Aged media frequently arrive with intact data sectors but a destroyed directory structure. File carving ignores filesystem metadata and reconstructs files purely from their magic numbers and structure — the same technique used to recover deleted evidence:

# Recover files from a damaged image by signature, no filesystem required
photorec disk.img

PhotoRec, Scalpel, and foremost all work this way; the format signatures that make carving possible are the very magic numbers described in Anatomy of a binary format, tying the whole workflow together.

Authenticity, provenance, and chain of custody

Both disciplines must answer “is this really what it claims to be, and has it changed?” — a legal question for the examiner, a scholarly one for the archivist, with the same technical answers:

• Fixity — store a hash (SHA-256) at ingest; re-verify on a schedule. Any drift signals silent corruption or tampering.
• Write-once handling — capture through write blockers; store masters on write-protected or append-only media so the authoritative copy cannot be altered.
• Provenance metadata — record where the bits came from, every tool and transformation applied, and by whom. In forensics this is the chain of custody; in archives it is the accession/processing record. Both are contemporaneous, auditable logs that let someone else trust your copy.
• Preserve the original, derive copies — never migrate or normalize in place. Keep the untouched master (the flux image, the E01, the original bytes) and generate access derivatives from it, exactly as an examiner works from a copy and never the seized original.

A combined archival-forensics workflow

Putting the pieces together, recovering and preserving an obsolete dataset looks like this:

1. Acquire through a write blocker; for old magnetic media, capture a flux master (Greaseweazle/KryoFlux), then decode a logical image from it.
2. Hash the image immediately (sha256sum) and record the value as the fixity baseline.
3. Carve if the filesystem is damaged (PhotoRec/Scalpel); otherwise mount read-only.
4. Identify every format with siegfried/DROID against PRONOM; aggregate with Brunnhilde and flag obsolete formats.
5. Reverse-engineer any undocumented binary formats (hexdump, binwalk) and capture the layout as a Kaitai Struct .ksy specification.
6. Decide migration vs. emulation per format, guided by its significant properties.
7. Provide access — migrate to a current format and/or stand up an emulated environment (EaaSI/QEMU) running the original software.
8. Record provenance for every step and re-verify fixity on a schedule.

The examiner and the archivist part ways only at step 6: one is building a case, the other is building a collection. Everything before that is the same craft.

References

• BitCurator — digital forensics environment for archives
• BitCurator documentation and walkthrough
• PRONOM file format registry (UK National Archives)
• DROID — file format identification tool
• siegfried — PRONOM-based format identification
• siegfried, a PRONOM-based file format identification tool (Open Preservation Foundation)
• Brunnhilde — aggregate format reporting
• TrID — file identifier by statistical signature
• Greaseweazle — open-source floppy flux imaging
• KryoFlux — flux-level floppy preservation (Software Preservation Society)
• Kaitai Struct — declarative binary format parsing
• The Sleuth Kit — filesystem and disk analysis
• PhotoRec — file carving / data recovery
• bulk_extractor — scanning images for PII and features
• Emulation-as-a-Service Infrastructure (EaaSI)
• Software Heritage — archiving the world’s source code
• Library of Congress — Sustainability of Digital Formats
• Digital Preservation Coalition — Digital Preservation Handbook