CS 492/592: Malware Reverse Engineering
Location: FAB 48 (TR 09:00-10:40)
Instructor: D. Kevin McGrath
- PDX username: dmcgrath
- Office hours:
- Location: FAB 120-15
- Times:
- Whenever you see me in my office with the door open
- Code Party!
TA: Chris Lu
- PDX username: chlu
- Office hours:
- Location: FAB 140
- Times:
- TBD
- or Code Party, or by email
Code Party:
- Tuesdays of even weeks 18:00 - 22:00 in FAB 86-01
- Tuesdays/Thursdays of odd weeks 11:00 - 13:00 in TBD
- CS 492/592: Malware Reverse Engineering
- Schedule (SUBJECT TO CHANGE – CHECK REGULARLY)
- Pages
- Lecture content
- Week 1 — Introduction and Lab Environment
- Week 2 — Malware Triage and File Identification
- Week 3 — Static Analysis: PE/ELF Structure and x86 Assembly
- Week 4 — Reverse Engineering Platforms
- Week 5 — Dynamic Analysis
- Week 6 — Malware Network Behavior
- Week 7 — Anti-Analysis Techniques
- Week 8 — Malware Families
- Week 9 — YARA Rules and Shellcode Analysis
- Week 10 — Fuzzing and Symbolic Execution
- Other stuff
- Analysis Environment
- Useful links for learning
- Lecture content
- Homework
- Previous Offerings
Schedule (SUBJECT TO CHANGE – CHECK REGULARLY)
Instruction runs 28 September – 6 December 2026, and finals week is 7–11 December. The university is closed on Thursday 26 November (Thanksgiving), so Week 9 has a single meeting. Week numbers link to that week’s pages below.
| Week | Meetings (TR) | Topic | Learning Activities | Due (23:59:59) |
|---|---|---|---|---|
| 1 | Sep 29, Oct 1 | Introduction, lab environment, ethics/legal | ||
| 2 | Oct 6, Oct 8 | Malware triage and file identification | HW1 | |
| 3 | Oct 13, Oct 15 | Static analysis: PE/ELF structure | HW1 — Mon Oct 12 | |
| 4 | Oct 20, Oct 22 | Static analysis: IDA Classroom and Cutter | HW2 | |
| 5 | Oct 27, Oct 29 | Dynamic analysis: sandboxes and tracing | HW3 | |
| 6 | Nov 3, Nov 5 | Dynamic analysis: network behavior | HW2 — Mon Nov 2 | |
| 7 | Nov 10, Nov 12 | Anti-analysis techniques | HW4 | HW3 — Mon Nov 9 |
| 8 | Nov 17, Nov 19 | Malware families: ransomware, RATs, rootkits | HW4 — Mon Nov 16 | |
| 9 | Nov 24 only (Nov 26 — Thanksgiving, university closed) |
YARA rules; shellcode analysis | ||
| 10 | Dec 1, Dec 3 | Fuzzing (AFL++, libFuzzer), C vulnerabilities, and symbolic execution (angr) | HW5 | HW5 — Fri Dec 4 |
| Finals | Dec 7–11 no class meeting |
No lecture; no office hours | Final project | Final project — Fri Dec 11 |
*Explorations are ungraded learning activities that typically involve a hands-on activity related to the current topic.
Pages
Lecture content
- Using Wine – running Windows malware on Linux
- Dynamic Analysis of Windows Malware from Linux – the three ways to execute a PE on Linux, and how each one lies to you
Week 1 — Introduction and Lab Environment
- REMnux Installation
- Running x86-64 Linux on Apple Silicon – emulation vs Rosetta translation for M-series Macs, with measured performance
- AI-Assisted Analysis: REMnux and Kali MCP Servers
- Using the REMnux MCP Server with Claude Code
Week 2 — Malware Triage and File Identification
Week 3 — Static Analysis: PE/ELF Structure and x86 Assembly
- Static Analysis
- Advanced Static Analysis: x86, IDA Pro, and C Constructs – summary of PMA Part 2
- Reverse Engineering
- Unix Text Processing: sed and awk – substitution, in-place editing, validation, and pipeline patterns
Week 4 — Reverse Engineering Platforms
- IDA Classroom: Installation and Licensing
- IDA Pro Cheat Sheet
- Ghidra Cheat Sheet
- Reverse Engineering Platforms: IDA, Ghidra, and Cutter – comprehensive intro to the three main RE platforms
Week 5 — Dynamic Analysis
- Dynamic Analysis
- Dynamic Analysis of Windows Malware from Linux – Wine relay tracing, Speakeasy/Qiling emulation, KVM detonation and out-of-guest capture
- Advanced Dynamic Analysis – debugger scripting, runtime unpacking, Frida, eBPF/ETW, injection detection
Week 6 — Malware Network Behavior
- Malware Network Behavior Analysis – downloaders, DGA, C2 protocols, credential stealers, exfiltration, FakeNet-NG/INetSim
- Lab: Malware Network Behavior and Network Signatures – Ch. 11 & 14 labs: downloaders, reverse shells, beacon analysis, Snort rules
- Suricata: Network Signatures for Malware Detection – rule syntax, application-layer keywords, HTTP/DNS/TLS detection, EVE JSON
Week 7 — Anti-Analysis Techniques
Week 8 — Malware Families
- Malware Families
- Adversary Models: MITRE ATT&CK and Related Frameworks – ATT&CK, Kill Chain, Diamond Model, D3FEND, ENGAGE, UKC
- APT Groups – definitions, naming conventions (Mandiant/CrowdStrike/Microsoft/MITRE), active groups by nation-state, common TTPs
Week 9 — YARA Rules and Shellcode Analysis
- Yara – YARA and YARA-X
- Shellcode Analysis
Week 10 — Fuzzing and Symbolic Execution
- Fuzzing and C Vulnerabilities – AFL++, libFuzzer, ASan, C vulnerability classes
- Symbolic Execution with angr – recovering passwords, triggers, and checksums; bypassing anti-analysis; managing path explosion
Other stuff
- Setup
- Running x86-64 Linux on Apple Silicon – for M-series Mac users
- Working with Wine
- Hyper-V setup
- Poster-style overviews of binary types (PE, ELF, mach-O, etc.)
Analysis Environment
- REMnux install
- REMnux — Linux distribution for malware analysis (our primary analysis environment)
- REMnux documentation — installation, tools, and usage guides
Useful links for learning
- Malware Traffic Analysis
- Any.run — interactive malware sandbox
- VirusTotal — file/URL scanning
- Hybrid Analysis — free malware analysis
- MalwareBazaar — malware sample repository
- IDA Classroom — free educational edition of IDA Pro (industry standard)
- Cutter — GUI frontend for Rizin
- Rizin — maintained fork of radare2 with cleaner API
- radare2 — open source reverse engineering framework
- Ghidra — open source reverse engineering suite developed by the US National Security Agency (NSA)
- YARA — malware identification/classification
- AFL++ — coverage-guided fuzzer
- angr — binary analysis and symbolic execution framework
- angr CTF — guided angr exercises
- Exploit-DB shellcodes — shellcode database
Homework
All homework is submitted via a private GitLab repo on gitlab.cecs.pdx.edu. Add dmcgrath and chlu as developer or higher. Each assignment should be in its own folder (e.g., hw1/) with a hw1.md file and any supporting files.
Previous Offerings
- None yet