courses

Unix Text Processing: grep, sed, and awk

Why grep, sed, and awk

Security work is, to a surprising degree, text work: every log file, packet-capture export, configuration file, and command output is a stream of lines waiting to be filtered, reshaped, and summarized. Across these courses you will pull failed-login attempts out of auth.log, tally HTTP status codes from a web log, extract fields from a tshark export, and rewrite config files in automation — all faster at the shell than by hand or in a throwaway script.

These three tools are the workhorses, and they layer naturally: grep finds the lines you care about, sed edits them, and awk computes over their fields. All operate line by line, so they stay efficient on files of any size (they never load the whole file into memory). grep filters by regular expression; sed (stream editor) is optimized for substitution and deletion; awk is a full programming language with field splitting, arithmetic, and associative arrays. Together they handle the vast majority of text transformation that would otherwise require Python or Perl, alongside the other classic pipeline tools (cut, sort, uniq) covered toward the end of this page.

grep

grep prints the lines of its input that match a pattern. It is usually the first stage of a pipeline — narrow a huge log down to the handful of relevant lines, then hand them to sed/awk for reshaping. The form is grep [flags] PATTERN [file...]; with no file it reads stdin.

grep "Failed password" /var/log/auth.log   # lines containing the literal text
some_command | grep error                  # filter another command's output

Essential flags

Flag Effect
-i Case-insensitive match
-v Invert — print lines that do not match
-n Prefix each match with its line number
-c Print only a count of matching lines
-o Print only the matched text, not the whole line
-w Match whole words only
-r / -R Recurse into directories (-R follows symlinks)
-l / -L List filenames with matches / without matches
-A n / -B n / -C n Show n lines of context after / before / around each match
-E Extended regex (ERE) — + ? | ( ) without backslashes
-F Fixed strings — treat the pattern literally, no regex
-P Perl-compatible regex (PCRE), e.g. \d, \b, lookarounds
-q Quiet — exit 0 on first match, print nothing (for scripts)

ℹ️ egrep and fgrep are deprecated — current GNU grep prints a warning. Use grep -E and grep -F instead.

Regular expressions in grep

By default grep uses POSIX basic regex (BRE), where +, ?, |, (, and ) are literal unless backslash-escaped; -E switches to extended regex (ERE) where they are operators. The building blocks:

Pattern Matches
^ / $ Start / end of line
. Any single character
[abc] / [^abc] Any one of / none of the listed characters
[0-9] / [[:digit:]] A digit (range or POSIX class)
* / \+ / \? Zero-or-more / one-or-more / optional (the last two need \ in BRE, or use -E)
\{m,n\} Between m and n repetitions
foo\|bar foo or bar (use -E to drop the backslash)
\< / \> / \b Word boundaries 
grep -E "^(GET|POST) " access.log         # lines starting with GET or POST
grep -Eo "[0-9]{1,3}(\.[0-9]{1,3}){3}" f  # extract IPv4-looking addresses
grep -P "\bpassword\b" config.txt         # the whole word "password" (PCRE)

Worked examples

# Count failed SSH logins and rank the source IPs
grep "Failed password" /var/log/auth.log \
  | grep -oE "from [0-9.]+" | awk '{print $2}' \
  | sort | uniq -c | sort -rn | head

# Recursively hunt for hardcoded secrets in a codebase (names with matches only)
grep -rinE "(password|api[_-]?key|secret)\s*=" --include="*.py" src/

# Show a config setting with surrounding context, ignoring commented-out lines
grep -n -A2 -B2 "PermitRootLogin" /etc/ssh/sshd_config | grep -v "^[0-9]*[:-]#"

# Use grep's exit status in a script (no output, just success/failure)
if grep -q "^PasswordAuthentication no" /etc/ssh/sshd_config; then
    echo "password auth is disabled"
fi

sed

sed (stream editor) reads input line by line, applies commands, and writes to stdout. It never modifies input in place unless you use -i.

Basic Substitution

sed 's/pattern/replacement/' file

The s command replaces the first match on each line. Flags modify behavior:

Flag Effect
g Replace all matches on each line (not just the first)
i Case-insensitive match (GNU sed)
2 Replace only the 2nd match on each line
p Print the line if a substitution was made (use with -n) 
# Replace first occurrence per line
sed 's/foo/bar/' file

# Replace all occurrences per line
sed 's/foo/bar/g' file

# Case-insensitive replacement
sed 's/error/ERROR/gi' logfile

# Print only changed lines
sed -n 's/old/new/p' file

The delimiter does not have to be / — any character works. This avoids escaping slashes in paths:

sed 's|/usr/local|/opt|g' config
sed 's,http://,https://,g' urls.txt

Addresses

Commands can be restricted to specific lines using addresses before the command:

# Line 5 only
sed '5s/foo/bar/' file

# Lines 3 through 7
sed '3,7s/foo/bar/' file

# Lines matching a regex
sed '/^#/d' file          # delete comment lines
sed '/ERROR/s/old/new/g' logfile  # substitute only on lines containing ERROR

# From line 5 to end of file
sed '5,$s/foo/bar/' file

# From first match of /start/ to first match of /end/
sed '/start/,/end/s/foo/bar/' file

Common Commands

Command Effect
s/pat/rep/ Substitute
d Delete line
p Print line
q Quit after this line
a\text Append text after this line
i\text Insert text before this line
= Print line number
y/abc/ABC/ Transliterate characters (like tr) 
# Delete blank lines
sed '/^[[:space:]]*$/d' file

# Delete lines 1-5 (e.g., skip a file header)
sed '1,5d' file

# Print lines 10-20
sed -n '10,20p' file

# Add a blank line after every line matching a pattern
sed '/^Section/a\\' file

# Print only lines containing a pattern (like grep)
sed -n '/error/p' logfile

# Delete everything from a pattern to end of file
sed '/^END/,$d' file

Multiple Expressions

Use -e to chain multiple commands, or separate commands with semicolons:

sed -e 's/foo/bar/g' -e 's/baz/qux/g' file

# Equivalent
sed 's/foo/bar/g; s/baz/qux/g' file

In-Place Editing

-i edits the file in place instead of writing to stdout.

# Edit in place (GNU sed — Linux)
sed -i 's/old/new/g' file

# Edit in place with a backup (file.bak created)
sed -i.bak 's/old/new/g' file

macOS ships BSD sed, which requires a backup suffix even when you do not want one:

# macOS
sed -i '' 's/old/new/g' file

To write portable scripts:

# Works on both GNU and BSD
sed -i.bak 's/old/new/g' file && rm file.bak

Apply to multiple files:

sed -i 's/localhost/10.0.0.5/g' config/*.conf

Worked Examples

Strip trailing whitespace from every line:

sed 's/[[:space:]]*$//' file

Remove HTML tags from output:

curl -s https://example.com | sed 's/<[^>]*>//g'

Extract lines between two markers:

sed -n '/^BEGIN/,/^END/p' file

Uncomment a specific line in a config file:

sed -i 's/^#\(net.ipv4.ip_forward\)/\1/' /etc/sysctl.conf

Double-space a file (insert blank line after each):

sed 'G' file

awk

awk splits each input line into fields and runs a program of pattern { action } rules. It is a full language: variables, arithmetic, arrays, functions, and I/O.

Structure

awk 'pattern { action }' file

Both pattern and action are optional:

# action only — runs on every line
awk '{ print $1 }' file

# pattern only — prints matching lines (like grep)
awk '/error/' file

# Both — action runs only when pattern matches
awk '/error/ { print $2, $3 }' file

# BEGIN and END run once before/after all input
awk 'BEGIN { print "start" } { print } END { print "done" }' file

Fields and Field Separator

awk splits each line into fields $1, $2, … $NF. $0 is the whole line. NF is the field count, NR is the line number.

# Print the third field of every line
awk '{ print $3 }' file

# Print last field
awk '{ print $NF }' file

# Print second-to-last field
awk '{ print $(NF-1) }' file

# Custom delimiter: colon-separated (like /etc/passwd)
awk -F: '{ print $1, $3 }' /etc/passwd

# Tab-separated
awk -F'\t' '{ print $2 }' data.tsv

# Multiple delimiters (regex): colon or comma
awk -F'[,:]' '{ print $1 }' file

Built-in Variables

Variable Meaning
NR Current line number (across all files)
FNR Line number within the current file
NF Number of fields on the current line
FS Input field separator (default: whitespace)
OFS Output field separator (default: space)
RS Input record separator (default: newline)
ORS Output record separator (default: newline)
FILENAME Name of the current input file 
# Print line numbers alongside content
awk '{ print NR": "$0 }' file

# Change output separator
awk -F: 'BEGIN { OFS="," } { print $1, $3, $7 }' /etc/passwd

# Process two files, track which file you're in
awk 'FNR==1 { print "--- " FILENAME " ---" } { print }' a.txt b.txt

Conditionals and Arithmetic

# Print lines where field 3 is greater than 100
awk '$3 > 100 { print }' file

# Print lines 10 through 20
awk 'NR >= 10 && NR <= 20' file

# Sum a column
awk '{ sum += $2 } END { print sum }' file

# Count matching lines
awk '/error/ { n++ } END { print n " errors" }' logfile

# Average of a column
awk '{ sum += $1; n++ } END { print sum/n }' numbers.txt

Associative Arrays

awk arrays are key-value stores with string keys. They require no declaration.

# Count occurrences of each value in field 1
awk '{ count[$1]++ } END { for (k in count) print k, count[k] }' file

# Sum bytes per IP from an access log (field 1 = IP, field 10 = bytes)
awk '{ bytes[$1] += $10 } END { for (ip in bytes) print ip, bytes[ip] }' access.log \
  | sort -k2 -rn | head 20

# Deduplicate field 1, keeping only first occurrence of each value
awk '!seen[$1]++' file

String Functions

Function Effect
length(s) Length of string
substr(s, start, len) Substring
split(s, arr, sep) Split string into array
gsub(pat, rep, s) Global substitute in s (modifies in place)
sub(pat, rep, s) First-match substitute
match(s, pat) Find regex in string; sets RSTART, RLENGTH
sprintf(fmt, ...) Format string
tolower(s) / toupper(s) Case conversion
index(s, t) Position of t in s (0 if not found) 
# Extract substring: characters 1-8 of field 1
awk '{ print substr($1, 1, 8) }' file

# Replace all commas with tabs in field 3
awk '{ gsub(/,/, "\t", $3); print }' file

# Print fields that are longer than 10 chars
awk '{ for (i=1; i<=NF; i++) if (length($i) > 10) print $i }' file

Worked Examples

Parse /etc/passwd into a readable table:

awk -F: 'BEGIN { OFS="\t" } { print $1, $3, $7 }' /etc/passwd \
  | column -t

Print duplicate lines only:

awk 'seen[$0]++' file

Summarize HTTP status codes from an nginx access log:

awk '{ print $9 }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

Extract key=value pairs and reformat:

# Input:  name=alice age=30 city=portland
# Output: alice is 30 from portland
awk '{
    split($0, pairs, " ")
    for (i in pairs) {
        split(pairs[i], kv, "=")
        data[kv[1]] = kv[2]
    }
    print data["name"], "is", data["age"], "from", data["city"]
}' file

Top 10 source IPs from a packet log:

tshark -r capture.pcapng -T fields -e ip.src \
  | awk '{ count[$1]++ } END { for (ip in count) print count[ip], ip }' \
  | sort -rn | head 10

Validation and Conditional Replacement

A common scripting task: ensure a setting is present in a config file, adding it if missing and updating it if wrong.

Check and append if absent

# Add a line if it does not already exist
grep -qF 'net.ipv4.ip_forward = 1' /etc/sysctl.conf \
  || echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf

-q suppresses output; -F treats the pattern as a literal string, not a regex.

Update if present, append if absent

# If the key exists (commented or not), replace the whole line.
# If it does not exist, append it.
KEY="net.ipv4.ip_forward"
VALUE="1"
FILE="/etc/sysctl.conf"

if grep -qE "^#?${KEY}" "$FILE"; then
    sed -i "s|^#\?${KEY}.*|${KEY} = ${VALUE}|" "$FILE"
else
    echo "${KEY} = ${VALUE}" >> "$FILE"
fi

Ensure a block is present

# Append an entire block if a marker line is absent
if ! grep -qF '# managed by automation' /etc/ssh/sshd_config; then
    cat >> /etc/ssh/sshd_config <<'EOF'

# managed by automation
PasswordAuthentication no
PermitRootLogin no
EOF
fi

Validate format and report

Use awk to validate that every line in a file matches an expected format and report violations:

# Validate that each line of an IP list is a valid IPv4 address
awk '!/^([0-9]{1,3}\.){3}[0-9]{1,3}$/ { print NR": invalid: "$0; bad++ }
     END { if (!bad) print "all lines valid"; else print bad" invalid lines" }' ips.txt

Replace in multiple files, only if the pattern exists

# Only touch files that actually contain the old string
grep -rl 'old_hostname' /etc/apache2/ \
  | xargs sed -i 's/old_hostname/new_hostname/g'

Pipeline Patterns

Classic pipeline tools

Tool Role in pipeline
grep Filter lines by pattern
cut Extract fixed columns by delimiter
sort Sort lines
uniq Deduplicate or count adjacent duplicates
tr Translate or delete individual characters
head / tail Limit output
wc Count lines, words, or bytes
paste Merge files side by side
tee Split output to a file and stdout simultaneously

sed and awk in pipelines

Both read stdin when given no file argument:

# Remove blank lines from a command's output
some_command | sed '/^$/d'

# Extract the third column from whitespace-delimited output
ps aux | awk '{ print $3 }'

# Chain sed and awk
cat /var/log/auth.log \
  | grep 'Failed password' \
  | sed 's/.*from \([^ ]*\).*/\1/' \
  | sort | uniq -c | sort -rn | head 20

Process substitution

Process substitution <(command) lets you use a command’s output as if it were a file:

# diff two command outputs without temporary files
diff <(sort file1) <(sort file2)

# Join two command outputs on a common field
join <(sort -k1 a.txt) <(sort -k1 b.txt)

# awk processing two generated streams
awk 'NR==FNR { a[$1]=$2; next } { print $1, a[$1] }' \
    <(cut -d: -f1,3 /etc/passwd) \
    <(cut -d: -f1,7 /etc/passwd)

Multi-stage worked examples

Find the top 5 users by failed SSH login attempts:

grep 'Failed password' /var/log/auth.log \
  | awk '{ print $(NF-5) }' \
  | sort | uniq -c | sort -rn | head 5

Reformat a CSV into an INSERT statement:

# Input:  alice,30,portland
# Output: INSERT INTO users VALUES ('alice', 30, 'portland');
awk -F, '{
    printf "INSERT INTO users VALUES (\047%s\047, %s, \047%s\047);\n",
           $1, $2, $3
}' data.csv

(\047 is the octal for a single quote, avoiding quoting nightmares.)

Remove duplicate blank lines (squeeze to one):

cat -s file
# or with sed (D restarts the cycle with the remaining content,
# preserving the first blank line; lowercase d would remove all blanks):
sed '/^$/{N;/^\n$/D}' file

Count bytes transferred per hour from a web server log:

# Log format: IP - - [DD/Mon/YYYY:HH:MM:SS +0000] "..." STATUS BYTES
awk '{ match($4, /:[0-9]{2}:/, m); hour=substr(m[0],2,2); bytes[hour]+=$NF }
     END { for (h in bytes) print h":00", bytes[h] }' access.log \
  | sort

Watch a log file and alert on new errors:

tail -F /var/log/syslog \
  | awk '/error|critical|failed/ { print strftime("%T"), $0; fflush() }'

fflush() flushes awk’s output buffer so each alert prints immediately rather than waiting for awk’s internal buffer to fill.

Quick Reference

grep

Task Command
Match a pattern grep "pat" file
Case-insensitive grep -i "pat" file
Invert (non-matching) grep -v "pat" file
Count matches grep -c "pat" file
Recurse a tree, list files grep -rl "pat" dir/
Extended regex grep -E "a|b" file
Only the matched text grep -oE "[0-9]+" file
Lines of context grep -C2 "pat" file

sed

Task Command
Substitute first match per line sed 's/old/new/'
Substitute all matches sed 's/old/new/g'
Delete matching lines sed '/pattern/d'
Delete blank lines sed '/^$/d'
Edit file in place sed -i 's/old/new/g' file
Edit in place with backup sed -i.bak 's/old/new/g' file
Print lines 10–20 sed -n '10,20p'
Delete first 5 lines sed '1,5d'
Strip trailing whitespace sed 's/[[:space:]]*$//'

awk

Task Command
Print field N awk '{ print $N }'
Custom field separator awk -F: '{ print $1 }'
Sum a column awk '{ s+=$1 } END { print s }'
Count matching lines awk '/pat/ { n++ } END { print n }'
Deduplicate lines awk '!seen[$0]++'
Count values in field 1 awk '{ c[$1]++ } END { for (k in c) print k, c[k] }'
Print lines where $3 > N awk '$3 > N'
Print lines 10–20 awk 'NR>=10 && NR<=20'

Key takeaways

References

Related course pages: Technical Writing · Introduction to Networking · Capturing Packets with tcpdump · Wireshark · Software Configuration

🛠️ Maintenance note: grep/sed/awk syntax is POSIX-stable, but portability gotchas recur — GNU vs BSD/macOS sed -i (backup-suffix handling), gawk-only extensions like the three-argument match() and gensub(), and grep’s regex flavors (BRE by default, -E for ERE, GNU-only -P/PCRE). Verify any one-liner on the target platform before scripting it.