CS 492/592: Malware Analysis
Catalog Description
Introduction to the analysis of malicious software. Static and dynamic analysis techniques, reverse engineering, shellcode analysis, malware classification, fuzzing, and symbolic execution. Students will analyze real-world malware samples in a controlled environment and apply automated vulnerability discovery tools.
Prerequisites: CS 201, CS 346 or equivalent.
Course Overview
Malware Triage
Before diving into deep analysis, the analyst must perform quick triage to determine what kind of sample they are dealing with and prioritize analysis effort. We cover file identification, hashing, packing detection, and initial behavioral indicators. Tools such as file, strings, FLOSS, Detect-It-Easy, and VirusTotal are used here.
Static Analysis
Static analysis involves examining a binary without executing it. We begin with manual static analysis using tools such as objdump, readelf, nm, and strings, then progress to disassembly and decompilation with IDA Classroom (the industry-standard disassembler) and Cutter (a GUI for Rizin, a maintained fork of radare2). Students will learn to read assembly, identify common patterns (encryption loops, shellcode stubs, anti-analysis checks), and reconstruct high-level behavior from raw bytes.
Dynamic Analysis
Dynamic analysis involves executing a sample in a controlled environment and observing its behavior. We cover sandbox environments, process monitoring, network traffic capture, system call tracing, and registry/filesystem monitoring. Tools include strace, ltrace, Wireshark, and automated sandboxes such as Any.run and Cuckoo.
Anti-Analysis Techniques
Modern malware routinely attempts to detect and evade analysis. We examine common techniques: anti-debugging checks, VM/sandbox detection, code obfuscation, packing/encryption, and timing-based evasion. Students learn to identify and bypass these techniques.
Malware Classification
We survey the major categories of malware — ransomware, rootkits, botnets, RATs, infostealers, droppers/loaders — and examine representative samples from each family. Students develop YARA rules to classify new samples based on patterns identified during analysis.
Shellcode Analysis
Shellcode is raw, position-independent machine code injected by an exploit or loader. We cover the techniques shellcode uses to operate without a normal binary header: the call/pop and fnstenv tricks for locating itself in memory, manual PE export table traversal and API hashing for resolving library functions, NULL-byte avoidance, NOP sleds, and how to identify and analyze shellcode extracted from documents, network captures, or injected processes.
Fuzzing and Symbolic Execution
Fuzzing and symbolic execution are automated techniques for discovering bugs and reverse-engineering program behavior. We cover coverage-guided fuzzing with AFL++ and libFuzzer, common C vulnerability classes (buffer overflow, format string, use-after-free), and symbolic execution with angr. Students use these tools to find crash-inducing inputs and to automatically solve binaries that would resist purely manual analysis.
Communication
Please post all course-related questions through zulip so that the whole class may benefit from our conversation. Please contact me privately via university supplied email for matters of a personal nature or if you are uncomfortable posting where the whole class can see/comment. I strive to reply to course-related questions within 24 hours. I will strive to return your assignments and grades for course activities to you within five business days of the due date. No other communication channels are used in this course.
Student Learning Outcomes
Upon successful completion of this course, students will be able to:
- Perform triage on unknown binary samples to guide analysis
- Apply static analysis techniques to identify malware behavior without execution
- Apply dynamic analysis techniques to observe malware behavior at runtime
- Identify and bypass common anti-analysis techniques
- Classify malware samples by family using YARA rules and behavioral signatures
- Analyze shellcode to determine its capabilities and delivery mechanism
- Apply coverage-guided fuzzing to discover vulnerabilities in binary targets
- Apply symbolic execution to automatically recover program inputs and reverse-engineer binary behavior
Evaluation of Student Performance
This course uses a weighted average of assignments. All assignments will be graded out of 100 points, with the following weights towards the final grade:
- Final Project: 40%
- Homework Assignments: 60%
- While attendance will not be explicitly graded, it is expected that you will be involved in the classes.
Letter Grade
Letter grades will be assigned based on standard ranges with (optionally) +/- steps.
| Grade | Percent Range |
|---|---|
| A | 90-100 |
| B | 80-90 |
| C | 70-80 |
| D | 60-70 |
| F | <60 |
Course Content
| Week | Topic | Learning Activities |
|---|---|---|
| 1 | Introduction, lab environment, ethics/legal | Module summary, explorations* |
| 2 | Malware triage and file identification | Module summary, explorations*, HW1 |
| 3 | Static analysis: PE/ELF structure | Module summary, explorations* |
| 4 | Static analysis: IDA Classroom and Cutter | HW2, explorations*, module summary |
| 5 | Dynamic analysis: sandboxes and tracing | HW3, explorations*, module summary |
| 6 | Dynamic analysis: network behavior | explorations*, module summary |
| 7 | Anti-analysis techniques | HW4, explorations*, module summary |
| 8 | Malware families: ransomware, RATs, rootkits | explorations*, module summary |
| 9 | YARA rules; shellcode analysis | explorations*, module summary |
| 10 | Fuzzing (AFL++, libFuzzer) and C vulnerabilities | HW5, explorations*, module summary |
| 11 | Symbolic execution (angr) | explorations*, module summary |
| Finals | Final project |
*Explorations are ungraded learning activities that typically involve a hands-on activity related to the current topic.
Course Policies
Late Work Policy
No late work will be accepted without prior discussion. I understand that life happens, but request for late hand-in must be submitted prior to the due date. Permission will be granted dependent upon reasons, current state of completion, etc.
Incompletes
Incomplete (I) grades will be granted only in emergency cases (usually only for a death in the family, major illness or injury, or birth/adoption of a child), and if the student has turned in 90% of the points possible AT THE TIME OF REQUEST. In other words, if you have been keeping up, but a major life event occurs, let me know as soon as possible. If you are having any difficulty that might prevent you completing the coursework, please don’t wait until the end of the term; let me know right away.
Academic or Student Support Services
Accommodations
Accommodations for students with disabilities are determined and approved by Disability Resource Center (DRC). If you, as a student, believe you are eligible for accommodations but have not obtained approval please contact DRC immediately at 503-725-4150, drc@pdx.edu, or https://www.pdx.edu/disability-resource-center. DRC notifies students and faculty members of approved academic accommodations and coordinates implementation of those accommodations. If you have accommodations through DRC and wish to take the Midterm or Final Exam in the testing center, I strongly recommend that you schedule it before the end of week 1. If you are not registered with the DRC, you cannot register to take an exam in the testing center.
I want to make this class an open and welcoming environment for all. Your success is my goal.
Religious Observance
Portland State University strives to respect all religious practices. If you have religious holidays that conflict with any of the requirements of this class, please see me immediately so that we can make alternative arrangements.
Reach Out for Success
The PSU Center for Student Health and Counseling (SHAC) is staffed with folks who care and can help with a wide range of personal challenges. Here at PSU, there is never a need to tough things out alone.
As a student you may experience a range of issues that can cause barriers to learning, such as strained relationships, increased anxiety, alcohol/drug problems, feeling down, difficulty concentrating and/or lack of motivation. These mental health concerns or stressful events may lead to diminished academic performance or reduce a student’s ability to participate in daily activities. PSU is committed to advancing the mental health and well-being of its students. If you or someone you know is feeling overwhelmed, depressed, and/or in need of support, services are available. You can learn more about the broad range of confidential mental health services available on campus via SHAC https://www.pdx.edu/health-counseling/.
SHAC also has resources for physical health, including flu shots. You can check out their COVID-19 resources page here: https://www.pdx.edu/health-counseling/covid-19-resources (including testing).
Get Food Now Here at PSU, there is never a need to tough things out alone. Those who can, give, so those who need, have.
Housing / financial crisis help Here at PSU, there is never a need to tough things out alone. Emergency Housing, etc.
Title IX
As an instructor, students frequently come to me for assistance in matters that are not related to the course material. Please be aware that PSU’s policies require instructors to report any instance of sexual harassment, sexual and relationship violence and/or other forms of prohibited discrimination to University Officials, who keep the information private. If you would rather share information about these experiences with a PSU staff member who does not have these reporting responsibilities and can keep the information confidential, please contact one of the following campus resources.
- Confidential Advocates: 503.894.7982, or by scheduling on-line (for matters regarding sexual harassment and sexual and relationship violence)
- Center for Student Health and Counseling (SHAC): 1880 SW 6th Ave, 503.725.2800
- Student Legal Services: 1825 SW Broadway, (SMSU) M343, 503.725.4556
PSU Sexual Misconduct Response website gives you comprehensive information about how to support and/or report an incident.
Please complete the required student module Understanding Sexual Misconduct and Resources in Canvas, which provides information about PSU policy and resources.
You may also report sexual and relationship violence to law enforcement on campus with Campus Public Safety Office (CPSO)
Or you may file an anonymous report with Campus Public Safety Office or a Bias Incident report with the Bias Review Team (BRT). PSU does not typically investigate the reports that are made through these two avenues. These reports help PSU understand what students and employees are experiencing on and around campus and provide support where needed.