EE 516, Winter 2023

Mathematical Foundations of Machine Learning

This is the course website for EE 516, Mathematical Foundations of Machine Learning, Winter 2023 quarter.

Meeting time: Tue/Thu 6:40-8:30PM, FAB 150
Office hours: Wed 12:30-1:30PM Online via Zoom, Thu 8:30-9:30PM FAB 150 (or by appointment)

Course Description

The goal of this course is to move from familiarity to fluency with the use of linear algebra to solve problems in machine learning and signal processing. Through a combination of theory, modeling, and applications, students will gain intuition into the fundamentals of matrix methods and optimization. Topics covered include least squares, the singular value decomposition, eigenvalue decomposition, subspace methods, and optimization methods such as stochastic gradient descent, ADMM, and iteratively reweighted least squares. Applications will include principal components analysis, image compression and denoising, low rank matrix completion, kernel ridge regression, and spectral clustering.

While the course emphasizes mathematical analysis, there is a significant programming component that may be completed using either MATLAB or Python 3.

Textbook: This course does not require a textbook. Lecture notes will be provided via the course Slack workspace. Reference textbooks can be found in the Resources section below.

Communication: All written questions should be posted to the appropriate channel on the Slack workspace (see Homework 0).

Course Schedule

This course will practice a flipped classroom. You are responsible for reading the listed lecture notes before class on the specified day. Please post any questions you have to the course slack channel. I will spend the first portion of class giving a mini-lecture on the most important/confusing topics, and you will work problems in groups during the rest of class.

Date Lecture Sections Notes Exercises Solutions
1/10 1 0.1 - 1.20 Syllabus & Introduction, Introduction to Matrices Exercises 1 Solutions 1
1/12 2 1.21 - 1.55 Introduction to Matrices Exercises 2 Solutions 2
1/17 3 2.1 - 2.24 Matrix Factorizations & Decompositions — —
1/19 4 2.25 - 2.29, 3.1 - 3.15 Matrix Factorizations & Decompositions, Subspaces & Rank Exercises 3 Solutions 3
1/24 5 3.16 - 3.46 Subspaces & Rank Exercises 4 Solutions 4
1/26 6 — Demo 1: Spectral Clustering Demo 1 Starter Code Demo 1 Solution Code
1/31 7 4.1 - 4.27 Least Squares Exercises 5 Solutions 5
2/2 8 4.28 - 4.52 Least Squares    
2/7 9 — Exam 1    
2/9 10 6.1 - 6.30 Low-Rank Approximation Exercises 6 Solutions 6
2/14 11 — Demo 2: Nearest-Subspace Classifier Demo 2 Starter Code](./demos/demo-02-digits-class.zip) Demo 2 Solution Code
2/16 12 6.31 - 6.37, 7.1 - 7.10 Low-Rank Approximation, [Optimization Basics Exercises 7 Solutions 7
2/21 13 — Demo 3: Kernel Ridge Regression Demo 3 Starter Code Demo 3 Solution Code
2/23 14 7.11 - 7.16, 8.1 - 8.3 Optimization Basics, Sparse Regression Exercises 8 Solutions 8
2/28 15 — ADMM, IRLS    
3/2 16 — Exam 2    
3/7 17 — Kernel Ridge Regression    
3/9 18 — Demo 4: Robust PCA Demo 4 Starter Code Demo 4 Solution Code
3/14 19 — Low-Rank Matrix Completion, Accelerated First-Order Methods    
3/16 20 — Demo 5: Low-Rank Matrix Completion Demo 5 Starter Code Demo 5 Solution Code

Homework

All assignments must be submitted via gradescope to obtain credit. See Homework 0 below for information on how to set up an account.

I provide the \(\LaTeX\) file used to generate each homework below. You must use this as a template to receive extra credit.

Optional Weekly Readings

Below are some completely optional, less technical readings that you may find interesting.

  1. The State of Being Stuck: This is great to keep in mind as you work problems for this class.

  2. How to solve hard problem sets: I found this helpful when I started graduate school.
    • A somewhat related article on how walking around is useful when solving problems.
  3. Does one have to be a genius to do maths?: Written by Terence Tao, a Fields Medal winner who has published on basically everything, including signal processing.
    • I also recommend his whole section on career advice, and his whole blog in general.

Resources