courses

FreeBSD Netgraph

Networking as a graph you can rewire at runtime

Every networking stack you have used so far — Linux included — is essentially a fixed pipeline: packets come in a NIC, pass through a predetermined sequence of layers (driver → bridge → netfilter hooks → routing → socket), and the most you can do is configure the stages, not rearrange them. FreeBSD’s netgraph(4) takes a different approach: kernel networking functions are packaged as small modules called nodes, and you — at runtime, from the shell — wire them together into an arbitrary directed graph that packets flow through. Need a bridge that feeds a NetFlow probe that feeds a Bluetooth stack that feeds a userland socket? Connect the nodes. Need to tap a link mid-stream to watch traffic? Splice in a tee node without dropping a packet.

For a network security course this matters twice over. Defensively, netgraph is how FreeBSD-based appliances build flow export, tunnels, and traffic capture into the kernel path. Offensively and analytically, it is a packet-plumbing toolkit: you can divert, duplicate, rewrite, and inject raw frames from userland with a couple of commands — the same capabilities you get from scapy or a capture setup on Linux, but implemented as kernel graph surgery.

ℹ️ Netgraph was designed in 1996 by Julian Elischer and Archie Cobbs at Whistle Communications for the InterJet, a FreeBSD-based office router that had to speak a zoo of WAN protocols (frame relay, ISDN, PPP variants) that a fixed stack handled poorly. It shipped in the main FreeBSD tree in 3.4 (December 1999) and remains FreeBSD-only — there is no Linux port, which is exactly why it makes such a good lens for comparing the two systems’ design philosophies.

Theoretical foundations

Nodes, hooks, and edges

Netgraph is a literal graph:

Concept What it is
Node An instance of a node type (a C module implementing a fixed set of methods). Each node has a unique hex ID and, optionally, a global ASCII name. Types are loadable kernel modules (ng_ether, ng_bridge, ng_tee, …).
Hook A named connection point on a node. Hooks are always connected in pairs — a hook cannot dangle. The hook name usually carries meaning to the node (e.g., ng_ether’s lower hook is the raw device side, upper is the protocol-stack side).
Edge A connected pair of hooks. Data flows along edges in both directions.

A node type declares which hooks it will accept and what it does with data arriving on each. The graph is fully dynamic: nodes are created, named, connected, and destroyed at runtime, and the kernel refuses only connections that a node type’s own rules disallow.

Two kinds of traffic: data and control

Everything moving through a netgraph is one of two things:

This separation is the key design idea. The data plane is a graph of connected hooks; the control plane is an addressing scheme layered over it. You reconfigure a live graph by sending control messages while data keeps flowing.

Addressing: paths and names

Control messages address nodes by path:

The relative form is worth staring at: fxp0:lower means “whatever node is currently connected to fxp0’s lower hook” — the address follows the wiring, so scripts keep working when the graph is assembled in different orders.

The userland boundary

Userland reaches the graph through the PF_NETGRAPH socket family. Opening a netgraph socket creates an ng_socket node inside the kernel; your process then is a node in the graph, able to send/receive both data and control messages. Three interfaces build on this:

This is the “prototype in userland” property netgraph was built for: a new protocol can be developed as a normal process connected into the live kernel data path, then reimplemented as a kernel node type once it stabilizes — same graph, same hooks, no API change to its neighbors.

The node-type toolbox

A stock FreeBSD system ships dozens of node types (see ls /boot/kernel/ng_* or man -k netgraph). The ones you will actually reach for:

Node type Role
ng_ether Auto-created per Ethernet interface; hooks lower (raw device), upper (protocol stack), orphans (frames the stack would discard)
ng_bridge Learning Ethernet bridge with loop detection; hooks link0…linkN
ng_tee Four-hook splice: passes left↔right traffic through while copying it to left2right/right2left — a live wiretap
ng_hub Dumb repeater — every frame in one hook goes out all others (build your own hub to sniff on)
ng_bpf Berkeley Packet Filter as a graph node — classify/steer packets by BPF program
ng_netflow NetFlow v5/v9 flow accounting and export (pairs with the collectors on the flow analysis page)
ng_nat In-kernel NAT (libalias-based)
ng_ipfw Bidirectional gateway between ipfw rules and the graph — a firewall rule can shunt matching packets into netgraph
ng_ksocket Wraps a kernel socket as a node — the graph can originate/terminate real UDP/TCP traffic
ng_eiface A fake Ethernet interface backed by a hook — the graph appears to the OS as a NIC
ng_pipe Delay/loss/bandwidth emulator for testing
ng_pppoe, ng_l2tp, ng_gre, ng_vlan Encapsulation and tunneling protocols as composable pieces

The heaviest real-world user is mpd5, FreeBSD’s multi-link PPP daemon (the PPPoE/L2TP/PPTP server behind many ISPs’ FreeBSD gear): the daemon does only signaling in userland, then assembles netgraph nodes so established sessions’ data never leaves the kernel.

Practical usage

Everything below runs on a stock FreeBSD system as root. Node-type modules autoload on first use in most cases; kldload ng_bridge (etc.) loads one explicitly, and kldstat | grep ng_ shows what’s resident.

First contact: ngctl

# What node types does this kernel know about?
ngctl types

# What nodes exist right now?
ngctl list
There are 2 total nodes:
  Name: em0             Type: ether           ID: 00000001   Num hooks: 0
  Name: ngctl4242       Type: socket          ID: 00000005   Num hooks: 0

Two things to notice: every Ethernet NIC already has an ng_ether node (created at boot, zero hooks until you use it), and ngctl itself shows up as a socket node — the tool is a node, per the userland-boundary design above.

The core ngctl verbs:

Command Effect
mkpeer <path> <type> <hook> <peerhook> Create a new node of <type> and connect it: <hook> on the existing node ↔ <peerhook> on the new one
name <path> <name> Give a node a global name
connect <path> <relpath> <hook> <peerhook> Wire two existing nodes together
msg <path> <cmd> [args] Send a control message (human-readable form)
show <path> Show one node: type, ID, and per-hook peers
rmhook <path> <hook> Break one edge
shutdown <path> Destroy a node (breaking all its edges)
dot Emit the whole graph as GraphViz — render it and see your plumbing

Worked example 1: a two-port bridge from raw parts

The canonical demo (FreeBSD ships a fuller version as /usr/share/examples/netgraph/ether.bridge): bridge two NICs entirely in netgraph.

# 1. Create a bridge node hanging off em0's lower (raw device) hook
ngctl mkpeer em0: bridge lower link0

# 2. It has no name yet — address it via the path "em0:lower" and name it
ngctl name em0:lower br0

# 3. Hook em0's own protocol stack back in, so the host keeps its IP on em0
ngctl connect em0: br0: upper link1

# 4. Attach the second NIC's raw side as another bridge port
ngctl connect em1: br0: lower link2

# 5. Both NICs must run promiscuous, without rewriting source MACs
ngctl msg em0: setpromisc 1
ngctl msg em0: setautosrc 0
ngctl msg em1: setpromisc 1
ngctl msg em1: setautosrc 0

# Inspect the result
ngctl show br0:
  Name: br0             Type: bridge          ID: 00000009   Num hooks: 3
  Local hook      Peer name       Peer type    Peer ID         Peer hook
  ----------      ---------       ---------    -------         ---------
  link2           em1             ether        00000002        lower
  link1           em0             ether        00000001        upper
  link0           em0             ether        00000001        lower

Read step 1 carefully, because mkpeer syntax trips everyone: “on node em0:, create a peer of type bridge; connect em0:’s hook lower to the new node’s hook link0.” The existing node’s hook comes third, the new node’s hook fourth.

What you built: frames arriving on em0’s wire go to the bridge (not the host stack); the bridge learns MACs and forwards between link0 (em0’s wire), link2 (em1’s wire), and link1 (the host’s own stack on em0). ng_bridge also does loop detection — a MAC seen on one link and then another within a short window marks the second link looped and mutes it (default 60 s) rather than melting down in a broadcast storm.

Tear it all down with one command — shutting down the bridge breaks every edge, and the ng_ether nodes revert to normal stack behavior:

ngctl shutdown br0:

Worked example 2: kernel NetFlow export

This is ng_netflow’s manpage example, and it composes three node types — ether, netflow, ksocket — into a flow sensor with no userland process in the data path. Compare with the collector-side tooling on the network flow analysis page.

ngctl -f- <<'EOF'
mkpeer em0: netflow lower iface0
name em0:lower netflow
connect em0: netflow: upper out0
mkpeer netflow: ksocket export inet/dgram/udp
msg netflow:export connect inet/10.0.0.1:4444
EOF

Line by line: inbound frames from em0’s wire enter the netflow node’s iface0 hook and are accounted, then passed through out0 back to the host stack (upper), so connectivity is unaffected. Expired flow records leave the export hook into an ng_ksocket node — a UDP socket living in the kernel — aimed at the collector at 10.0.0.1:4444. Point nfdump/nfcapd at that port and you have a line-rate NetFlow v5 probe built from three shell commands.

Worked example 3: nghook as a raw-frame pipe

nghook connects a shell pipeline to any unconnected hook. Attached to an ng_ether node’s orphans hook, it hands you every frame the kernel would have thrown away (unknown EtherTypes, weird protocols) — a zero-setup way to see what’s lurking on a segment:

# -a: print received data as ASCII+hex dump
nghook -a em0: orphans

Writing to stdin injects raw frames out the hook — which on lower means transmitting arbitrary Ethernet frames you compose yourself. The security implication cuts both ways: it is a protocol-development tool and a frame-injection primitive, so treat a box where untrusted users can reach netgraph sockets as one where they can forge traffic.

⚠️ Reconnecting an ng_ether node’s lower hook diverts all inbound traffic away from the host stack — do this on the NIC you’re SSH’d in over and you have cut your own connection. Lab rule: experiment on a second interface, or from the console. (This is the netgraph rite of passage; see also the identical warning culture around firewall rules.)

How netgraph differs from Linux networking

Linux has no single netgraph equivalent. Its answer is a collection of fixed-position mechanisms, each excellent at its slot in the pipeline:

Capability Netgraph (FreeBSD) Linux equivalent
Overall model One dynamic graph of composable nodes Fixed pipeline with configurable stages
Packet filtering ng_bpf, ng_ipfw nodes placed anywhere in the graph netfilter/nftables at five predefined hook points (prerouting, input, forward, output, postrouting)
Bridging ng_bridge node, wired by hand bridge device + ip link set ... master br0
Tunnels/encap ng_gre, ng_l2tp, ng_pppoe nodes you compose Per-protocol virtual devices (gre0, l2tp, pppoe via kernel + userspace daemons)
Traffic shaping/emulation ng_pipe, ng_car nodes in-line tc qdiscs attached at device ingress/egress
Custom in-kernel logic Write a node type (a kernel module), or prototype as a userland node eBPF programs at defined attach points (XDP, tc, socket)
Userland packet access ng_socket/nghook — connect anywhere in the graph AF_PACKET, TUN/TAP, NFQUEUE — each at its own fixed location
Flow export ng_netflow node Out-of-tree modules, conntrack-based exporters, or userland sniffers

The philosophical difference, in three points:

  1. Composition vs. configuration. Netfilter, tc, and bridges are stages with knobs: you configure what each does, but a packet’s route through them is fixed by the kernel’s design. Netgraph gives you the topology itself. The netflow example above — accounting spliced between the device and the protocol stack — has no direct Linux expression; you would approximate it with a tap or an eBPF program at a predefined attach point.
  2. eBPF is Linux’s convergent answer — from the opposite direction. eBPF also makes the kernel data path programmable, but its model is safe programs at fixed attach points (verified, JIT-compiled, sandboxed), where netgraph’s is trusted modules in arbitrary topology. eBPF can’t rewire the pipeline; netgraph nodes run as full kernel code with no verifier protecting you. One prioritizes safety and dynamism of logic; the other, dynamism of structure. It is a genuinely instructive contrast in OS design.
  3. Discoverability. A Linux box’s network config is scattered across ip, nft, tc, bridge, and eBPF object files. A netgraph configuration is one queryable object: ngctl list, or ngctl dot for a picture of the whole data plane.

Where you’ll meet each in the wild: netgraph inside FreeBSD-based appliances and ISP access gear (mpd5 PPPoE/L2TP concentrators, pfSense/OPNsense internals, NetFlow probes); the Linux stack everywhere else. Neither model “won” — Linux’s XDP/eBPF took the programmable-data-plane crown for performance, while netgraph remains the cleanest compositional design, 30 years on.

Key takeaways

References

Related course pages: Networking Fundamentals · Defensive Measures · Network Flow Analysis · Network Traffic Capture

🛠️ Maintenance note: node-type inventories and ngctl output formats drift across FreeBSD releases — re-check the worked examples against the current RELEASE manpages each term (interface names in examples use em0/em1; students on VMs will see vtnet0 or similar). The Klara Systems article link and the AsiaBSDCon tutorial PDF are third-party hosts worth a link-check. If Linux upstreams anything netgraph-like (or eBPF gains topology-level rewiring), the comparison section needs revisiting.