courses

Lua in Network Security

Lua is a lightweight, embeddable scripting language that has become a standard extension point for network security tools. Its small runtime, clean C API, and easy-to-learn syntax make it the natural choice when tool authors want to give users programmable customization without depending on a heavy scripting runtime. In network security you encounter Lua in three major contexts:

Understanding Lua in these tools lets you move from running pre-built scripts to writing your own, which is essential for analyzing non-standard protocols, detecting novel threats, or automating repetitive tasks.


Lua Basics

Lua is dynamically typed, garbage-collected, and uses 1-based array indexing. A few things to know before reading the examples below:

-- This is a comment

-- Variables are global by default; use local to scope them
local x = 42
local s = "hello"

-- Tables are the universal data structure (array, map, object)
local t = { "a", "b", "c" }        -- array-style (index 1, 2, 3)
local m = { host = "10.0.0.1", port = 80 }  -- map-style
print(m.host)   -- "10.0.0.1"
print(m["port"]) -- 80

-- Functions are first-class values
local function greet(name)
    return "hello, " .. name       -- .. is string concatenation
end

-- String methods
local line = "GET /index.html HTTP/1.1"
if line:find("^GET") then          -- Lua pattern, not regex
    print("it's a GET")
end

-- String patterns use % as the escape character (not \)
-- %d = digit, %s = whitespace, %a = letter, %p = punctuation
-- . * + ? work as in regex; ^ and $ anchor to start/end

Lua patterns are similar to regular expressions but not identical. The most common difference: use %d instead of \d, %s instead of \s, and %. to match a literal dot.


Nmap Scripting Engine (NSE)

Overview

NSE is what makes Nmap more than a port scanner. When you run nmap -sC or nmap --script=vuln, you are executing Lua scripts from /usr/share/nmap/scripts/. NSE scripts can:

There are roughly 600 scripts in the default Nmap installation, organized into categories:

Category Description
default Run with -sC; safe and generally useful
safe Non-intrusive, unlikely to harm the target
discovery Enumerate information about hosts and services
version Enhance version detection
auth Authentication credential testing
brute Brute-force login attacks
vuln Vulnerability checks
exploit Actively exploit vulnerabilities (use with authorization)
intrusive Likely to affect the target’s state or logs
malware Detect signs of malware or backdoors
fuzzer Send unexpected data to find crashes
dos Denial of service tests (destructive; use with care)
external Consult external resources (DNS, whois, etc.)
broadcast Discover hosts via broadcast/multicast

Running Scripts

# Run default scripts
nmap -sC 10.0.0.1

# Run a specific script
nmap --script=http-title 10.0.0.1

# Run all scripts in a category
nmap --script=vuln 10.0.0.1

# Run multiple scripts
nmap --script=http-title,http-headers 10.0.0.1 -p 80,443

# Run scripts matching a pattern
nmap --script="http-*" 10.0.0.1 -p 80

# Pass arguments to a script
nmap --script=http-brute --script-args http-brute.path=/login 10.0.0.1

# List all scripts in a category
nmap --script-help vuln

# Update the script database after adding new scripts
nmap --script-updatedb

Script Structure

Every NSE script is a Lua file with three sections:

-- ============================================================
-- HEAD: metadata
-- ============================================================
description = [[
Checks whether a web server is running behind a known CDN provider
by examining the Server header and IP geolocation.
]]

categories = {"discovery", "safe"}

author = "Your Name"
license = "Same as Nmap -- See https://nmap.org/book/man-legal.html"

-- NSE library imports
local http = require "http"
local stdnse = require "stdnse"
local shortport = require "shortport"

-- ============================================================
-- RULE: when should this script run?
-- ============================================================
-- portrule: run once per open port
portrule = shortport.http   -- shorthand: run on any open HTTP port

-- hostrule:  run once per host (use instead of portrule for host-level checks)
-- prerule:   run once before scanning begins
-- postrule:  run once after all hosts scanned

-- ============================================================
-- ACTION: the script's logic
-- ============================================================
action = function(host, port)
    local response = http.get(host, port, "/")
    if not response or not response.header then
        return nil
    end
    local server = response.header["server"] or "unknown"
    return stdnse.format_output(true, "Server: " .. server)
end

The action function receives host and port table objects with fields like host.ip, host.name, port.number, port.protocol, port.state. It returns a string (shown in nmap output) or nil to produce no output.

Key NSE Libraries

nmap — core API

local nmap = require "nmap"

-- Create a socket
local socket = nmap.new_socket()
socket:set_timeout(5000)  -- milliseconds

-- Exception-safe wrapper: on error, calls the cleanup function and re-raises
local try = nmap.new_try(function() socket:close() end)

try(socket:connect(host.ip, port.number))
try(socket:send("HELLO\r\n"))
local response = try(socket:receive())
socket:close()

-- Check port state
local p = nmap.get_port_state(host, {number=22, protocol="tcp"})
if p and p.state == "open" then ... end

-- Set a port's state/version info
nmap.set_port_version(host, port)

shortport — concise portrule helpers

local shortport = require "shortport"

-- True if the port is in a given list (open or open|filtered)
portrule = shortport.port_or_service(80, "http")

-- True for any HTTP-like port
portrule = shortport.http

-- True for SSL/TLS ports
portrule = shortport.ssl

-- True for a port number
portrule = shortport.portnumber(22)

-- True for a named service
portrule = shortport.service("ftp")

stdnse — standard Nmap script utilities

local stdnse = require "stdnse"

-- Formatted output (shown indented under host in nmap output)
return stdnse.format_output(true, {"line one", "line two"})

-- Debug printing (shown with -d)
stdnse.debug(1, "Got response: %s", response)

-- Get a script argument (from --script-args)
local path = stdnse.get_script_args("myscript.path") or "/"

-- Sleep (seconds as float)
stdnse.sleep(0.5)

http — HTTP requests

local http = require "http"

local response = http.get(host, port, "/admin/")
-- response.status   (integer, e.g. 200)
-- response.body     (string)
-- response.header   (table, lowercased keys)
-- response.rawheader (table)

local resp2 = http.post(host, port, "/login", nil, nil,
    "username=admin&password=admin")

-- Pipeline multiple requests
local pipeline = http.pipeline_new()
http.pipeline_add("/", nil, pipeline)
http.pipeline_add("/robots.txt", nil, pipeline)
local results = http.pipeline_go(host, port, pipeline)

vulns — standardized vulnerability reporting

local vulns   = require "vulns"
local vuln_db = vulns.DB:new()

local vuln = {
    title     = "CVE-2021-12345: Remote Code Execution in Foo Service",
    state     = vulns.STATE.VULN,   -- or NOT_VULN, LIKELY_VULN, UNKNOWN
    IDS       = { CVE = "CVE-2021-12345" },
    risk_factor = "High",
    description = "An unauthenticated attacker can execute arbitrary code.",
    dates     = { disclosure = { year='2021', month='03', day='15' } },
    references = { "https://nvd.nist.gov/vuln/detail/CVE-2021-12345" }
}
return vuln_db:add(vuln)

Writing a Simple Script from Scratch

A script that checks for an open Telnet banner and flags it as insecure:

description = [[
Detects open Telnet services and retrieves the login banner.
Telnet transmits credentials in plaintext — flag for remediation.
]]

categories = {"discovery", "safe"}
author = "Your Name"
license = "Same as Nmap -- See https://nmap.org/book/man-legal.html"

local nmap      = require "nmap"
local shortport = require "shortport"
local stdnse    = require "stdnse"

portrule = shortport.port_or_service(23, "telnet")

action = function(host, port)
    local socket = nmap.new_socket()
    socket:set_timeout(3000)

    local status, err = socket:connect(host.ip, port.number)
    if not status then
        return nil
    end

    -- Read the banner (Telnet sends it immediately)
    local ok, banner = socket:receive()
    socket:close()

    if not ok then
        return "Open Telnet port (no banner received)"
    end

    -- Strip non-printable bytes from the banner
    banner = banner:gsub("[%c]", ".")
    return stdnse.format_output(true, {
        "WARNING: Telnet transmits credentials in plaintext",
        "Banner: " .. banner
    })
end

Save to /usr/share/nmap/scripts/telnet-insecure.nse, run nmap --script-updatedb, then:

nmap --script=telnet-insecure -p 23 10.0.0.0/24

Wireshark Lua Scripting

Overview

Wireshark’s Lua API lets you extend the application without recompiling it. The primary uses in network security are:

Scripts are loaded from:

Core Objects

Object Description
Proto Defines a new protocol; has a name, description, and fields
ProtoField A typed field that appears in the packet detail tree
Tvb A “TV Buffer” — a reference to a packet’s raw bytes; supports slicing
Pinfo Packet information: source/destination addresses, ports, timestamps
TreeItem A node in the packet detail tree; used to add children and labels
Dissector Reference to an existing dissector; used to hand off to sub-dissectors
DissectorTable A table mapping protocol values to dissectors (e.g., udp.port)
Listener A tap listener invoked for each packet matching a filter

Writing a Protocol Dissector

A dissector lets Wireshark understand a custom protocol and display its fields. The example below parses a simple 4-byte header: 1 byte type, 1 byte flags, 2 bytes length.

-- myproto.lua — simple 4-byte header dissector
-- Load with: tshark -X lua_script:myproto.lua -r capture.pcap

-- 1. Create the protocol object
local myproto = Proto("myproto", "My Custom Protocol")

-- 2. Define fields (name, display label, type, optional base/values)
local f_type  = ProtoField.uint8 ("myproto.type",  "Message Type",  base.HEX,
    { [0x01]="HELLO", [0x02]="DATA", [0x03]="BYE" })
local f_flags = ProtoField.uint8 ("myproto.flags", "Flags",         base.HEX)
local f_len   = ProtoField.uint16("myproto.length","Payload Length", base.DEC)
local f_data  = ProtoField.bytes ("myproto.data",  "Payload")

myproto.fields = { f_type, f_flags, f_len, f_data }

-- 3. Implement the dissector function
function myproto.dissector(buf, pkt, tree)
    -- Minimum packet length check
    if buf:len() < 4 then return end

    -- Set the Info column
    pkt.cols.protocol = "MYPROTO"

    -- Add protocol subtree to the packet detail pane
    local subtree = tree:add(myproto, buf(0, 4))

    -- Add each field: buf(offset, length) slices the buffer
    subtree:add(f_type,  buf(0, 1))
    subtree:add(f_flags, buf(1, 1))
    subtree:add(f_len,   buf(2, 2))

    local payload_len = buf(2, 2):uint()
    if buf:len() >= 4 + payload_len and payload_len > 0 then
        subtree:add(f_data, buf(4, payload_len))
    end
end

-- 4. Register the dissector for UDP port 9999
local udp_table = DissectorTable.get("udp.port")
udp_table:add(9999, myproto)

Run it against a capture:

tshark -X lua_script:myproto.lua -r capture.pcap -Y myproto

Or in Wireshark, reload plugins via Tools → Lua → Reload Lua Plugins, or place the script in the plugins directory and restart Wireshark.

ProtoField Types

Method C equivalent Common options
ProtoField.uint8(name, label, base) uint8_t base.DEC, base.HEX, base.OCT
ProtoField.uint16(name, label, base) uint16_t
ProtoField.uint32(name, label, base) uint32_t
ProtoField.int8/16/32(...) signed int
ProtoField.bool(name, label, size) boolean size in bits
ProtoField.ipv4(name, label) IPv4 address
ProtoField.ipv6(name, label) IPv6 address
ProtoField.ether(name, label) MAC address
ProtoField.string(name, label) C string
ProtoField.bytes(name, label) byte array

Tap Listeners

A Listener (tap) is invoked for every packet matching a display filter. It is useful for extracting data across a whole capture — generating reports, counting events, or building tables of activity.

-- dns_tap.lua — print all DNS query names to stdout
-- Usage: tshark -X lua_script:dns_tap.lua -r capture.pcap -q

local tap = Listener.new("dns", "dns.flags.response == 0")

-- Called once per matching packet
function tap.packet(pinfo, tvb, tapdata)
    local qname = tostring(pinfo.private["dns.qry.name"] or "")
    if qname ~= "" then
        print(tostring(pinfo.number) .. "\t" .. qname)
    end
end

-- Called after all packets are processed
function tap.draw()
    print("--- DNS query extraction complete ---")
end

function tap.reset()
    -- Called when the capture is restarted; reset any counters here
end

A more useful pattern collects data and prints a summary:

-- beacon_counter.lua — count packets per destination IP
-- Useful for spotting beaconing malware (regular-interval connections)
-- Usage: tshark -X lua_script:beacon_counter.lua -r capture.pcap -q

local counts = {}

local tap = Listener.new("ip")

function tap.packet(pinfo, tvb)
    local dst = tostring(pinfo.dst)
    counts[dst] = (counts[dst] or 0) + 1
end

function tap.draw()
    print("\nDestination IP packet counts:")
    print(string.format("%-20s %s", "IP Address", "Packets"))
    print(string.rep("-", 35))
    -- Collect and sort
    local sorted = {}
    for ip, count in pairs(counts) do
        table.insert(sorted, {ip=ip, count=count})
    end
    table.sort(sorted, function(a,b) return a.count > b.count end)
    for _, entry in ipairs(sorted) do
        print(string.format("%-20s %d", entry.ip, entry.count))
    end
end

Run it in tshark with -q to suppress per-packet output:

tshark -X lua_script:beacon_counter.lua -r capture.pcap -q

Suricata Lua Detection Scripts

Overview

Suricata’s signature language is powerful but has limits: it cannot express logic like “alert if the same host makes more than 10 requests in 60 seconds” or “alert only if the User-Agent matches one of 200 known-bad strings.” Lua scripts fill this gap.

The lua keyword in a Suricata rule invokes a Lua script to decide whether the rule matches. The rest of the rule (source/destination IPs, ports, protocol, other keywords) still applies as normal — the Lua script is an additional condition.

The lua Keyword

alert http any any -> any any (
    msg:"Suspicious PHP POST via HTTP/1.0";
    flow:established,to_server;
    lua:detect_php_post.lua;
    classtype:web-application-attack;
    sid:9000001; rev:1;
)

The script file is resolved relative to the Suricata rules directory. If Lua compilation is disabled at build time, the lua keyword silently fails to match — check suricata --build-info | grep lua.

Script Structure

Every Lua detection script must define exactly two functions: init and match.

-- detect_php_post.lua
-- Matches HTTP/1.0 POST requests to .php files

-- init: declare which buffer(s) to inspect
function init(args)
    local needs = {}
    needs["http.request_line"] = tostring(true)
    return needs
end

-- match: inspect the buffer and return 1 (match) or 0 (no match)
function match(args)
    local request_line = tostring(args["http.request_line"])

    -- Lua pattern: matches "POST /anything.php HTTP/1.0"
    if request_line:find("^POST%s+.+%.php%s+HTTP/1%.0$") then
        return 1
    end
    return 0
end

The init function registers buffers by name in the returned table. Only one HTTP buffer can be registered per script. The match function receives those buffers as string values in args, inspects them, and returns 1 to signal a match.

Available Inspection Buffers

Buffer Contents
packet Full packet including headers (non-stream)
payload Packet payload only (non-stream)
http.uri Normalized HTTP URI
http.uri.raw Raw (un-normalized) HTTP URI
http.request_line Full HTTP request line (METHOD URI VERSION)
http.request_headers HTTP request headers, normalized
http.request_headers.raw HTTP request headers, raw
http.request_body HTTP request body (POST data, etc.)
http.request_cookie Cookie header value
http.request_user_agent User-Agent header value
http.response_headers HTTP response headers
http.response_body HTTP response body
http.response_cookie Set-Cookie value
dns.query DNS query name
tls.sni TLS SNI (Server Name Indication)

Multi-Pattern Matching Example

Lua detection becomes most powerful when matching against a large set of patterns that would be impractical as multiple Suricata signatures:

-- detect_malicious_ua.lua
-- Alert when User-Agent matches any known-malicious string

local bad_agents = {
    "python%-requests",     -- common in automated attacks
    "zgrab",                -- internet scanner
    "masscan",              -- mass scanner
    "libwww%-perl",         -- old automated client often abused
    "Go%-http%-client/1%.1",-- common in Go-based scanners
}

function init(args)
    local needs = {}
    needs["http.request_user_agent"] = tostring(true)
    return needs
end

function match(args)
    local ua = tostring(args["http.request_user_agent"])
    if #ua == 0 then return 0 end
    ua = ua:lower()
    for _, pattern in ipairs(bad_agents) do
        if ua:find(pattern) then
            return 1
        end
    end
    return 0
end

The corresponding rule:

alert http any any -> $HTTP_SERVERS any (
    msg:"Known malicious scanner User-Agent";
    flow:established,to_server;
    lua:detect_malicious_ua.lua;
    classtype:web-application-attack;
    sid:9000002; rev:1;
)

DNS Exfiltration Detection Example

Detecting DNS-based data exfiltration using Lua — high-entropy subdomain labels are a hallmark of DNS tunneling:

-- detect_dns_tunnel.lua
-- Flag DNS queries with high-entropy subdomains (likely base32/base64 encoded data)

-- Shannon entropy calculation
local function entropy(s)
    local freq = {}
    for i = 1, #s do
        local c = s:sub(i,i)
        freq[c] = (freq[c] or 0) + 1
    end
    local h = 0
    for _, v in pairs(freq) do
        local p = v / #s
        h = h - p * math.log(p, 2)
    end
    return h
end

function init(args)
    local needs = {}
    needs["dns.query"] = tostring(true)
    return needs
end

function match(args)
    local query = tostring(args["dns.query"])
    if #query == 0 then return 0 end

    -- Extract the leftmost label (before first dot)
    local label = query:match("^([^%.]+)")
    if not label or #label < 12 then return 0 end

    -- Entropy above ~3.8 bits/character is suspicious for a hostname label
    if entropy(label) > 3.8 then
        return 1
    end
    return 0
end

Deploying Lua Scripts in Suricata

  1. Place the .lua file in your rules directory (e.g., /etc/suricata/rules/).
  2. Reference it in the rule with lua:filename.lua;.
  3. Add the rule to your .rules file.
  4. Reload rules without restarting: suricatasc -c reload-rules or kill -USR2 $(pidof suricata).
  5. Test in offline mode first:
suricata -r capture.pcap -S /etc/suricata/rules/custom.rules -l /tmp/suricata-test/

Comparison and When to Use Each

  Nmap NSE Wireshark Lua Suricata Lua
Primary purpose Active scanning and enumeration Packet analysis and visualization Inline detection and alerting
Runs on Live scan targets Captured traffic (pcap) or live capture Live traffic (inline or IDS)
Script trigger Port/host rule; before/after scan Every packet, or tap filter Matching flow + other rule conditions
Network access Full (sockets, raw packets) Read-only (packet bytes via Tvb) Read-only (buffer strings from dissectors)
Typical use Vuln checks, banner grabs, brute force Custom protocol display, statistics Complex pattern matching, entropy checks
Script location /usr/share/nmap/scripts/ Plugins dir or -X lua_script: Rules dir, referenced in rule
Key API nmap, http, shortport, vulns Proto, ProtoField, Listener, Tvb init(args) / match(args)

Things to Try Yourself

  1. NSE: Write a script that connects to a service on port 80 and checks whether the Server: response header reveals a version number. Output a warning if it does.

  2. NSE: Write a postrule script (runs after all hosts are scanned) that prints a summary table of all hosts where port 22 was open and the service banner contained “OpenSSH”.

  3. Wireshark: Write a tap listener that extracts every unique TLS SNI from a capture file and prints a sorted list. Compare the output to tshark -T fields -e tls.handshake.extensions_server_name.

  4. Wireshark: Write a dissector for a simple 8-byte header protocol you define yourself (e.g., 2-byte magic, 2-byte type, 4-byte sequence number). Generate test traffic with scapy and confirm your dissector parses it correctly.

  5. Suricata: Write a Lua detection script that flags HTTP requests where the URI contains a base64-encoded string (look for sequences of [A-Za-z0-9+/=] longer than 20 characters in the path). Test it against a pcap containing benign and suspicious traffic.


Further Reading