\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}
\usepackage{caption,subcaption}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 516: Mathematical Foundations of Machine Learning
            \hfill Winter \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{9}{March 19, \the\year}{\X}
% PUT YOUR NAME BELOW WHERE I HAVE \X

\section{\normalfont{(5 pts each)}}

In Homework 8 we focused on solving sparse regression problems by incorporating an $\ell_{1}$-norm regularizer. If we want to encourage a different type of behavior, we can consider different regularizers. Moreover, there are structures in signals besides sparsity that still allow recovery in the case of highly under-determined systems. One popular regularizer is called the \emph{total variation} (TV), which has applications in image denoising and inpainting. For a signal $w \in \bR^{D}$, the TV is defined as
\begin{equation*}
    \text{TV}(w) = \sum_{i = 1}^{D-1} \abs{w_{i+1} - w_{i}},
\end{equation*}
where $w_{i}$ denotes the $i$th element of $w$. In this problem, you will implement TV-regularized regression, which is useful for estimating signals that are known to be piecewise constant, as shown in Fig. \ref{fig:wPW} below.

\begin{figure}[h!]
    \centering
    %\includegraphics[width=3in]{wPW.eps}
    \caption{Example piecewise constant function for total variation-penalized regression.}
    \label{fig:wPW}
\end{figure}

\begin{enumerate}[(a)]
    \item Define a matrix $C \in \bR^{D-1 \times D}$ such that
        \begin{equation*}
            \text{TV}(w) = \sum_{i = 1}^{D-1} \abs{w_{i+1} - w_{i}} = \norm{Cw}_{1}.
        \end{equation*}

    \newpage
    \item The TV-regularized regression problem can be written as
        \begin{equation}
            \min_{w \in \bR^{D}} \frac{1}{2} \norm{Xw - y}_{2}^{2} + \lambda \norm{Cw}_{1}
            \label{eq:tvADMM}
        \end{equation}
        or in summation form as
        \begin{equation*}
            \min_{w \in \bR^{D}} \frac{1}{2} \sum_{i = 1}^{N} \left( x_{i}^{T}w - y_{i} \right)^{2} + \lambda \sum_{i = 1}^{D-1} \abs{w_{i+1} - w_{i}}.
        \end{equation*}
        Determine the ADMM updates to solve Eq.~\eqref{eq:tvADMM}. You may need to apply the splitting technique and introduce a new optimization variable.

    \item Implement your devised ADMM algorithm by completing \texttt{tvADMM} in the files for download and test your algorithm on the script \texttt{prob1}. \textbf{Turn in:}
        \begin{itemize}
            \item Your \texttt{tvADMM} code
            \item A plot of the true and estimated functions
            \item A plot of the relative error between iterations as a function of the iteration number, where the error is defined as
                \begin{equation*}
                    \text{err}_{k} = \frac{\norm{w^{(k)} - w^{(k-1)}}}{\norm{w^{(k)}}},
                \end{equation*}
                where $w^{(k)} \in \bR^{D}$ denotes the estimate of $w$ at the $k$th iteration.
        \end{itemize}

\end{enumerate}

\section{\normalfont{(5 pts)}}

In Demo 3, we used the matrix inversion lemma to make ridge regression amenable to the kernel trick. Go through the description of how Eq. (1) in Demo 2 is derived and write out the steps explicitly here.

\section{\normalfont{(5 pts, 5 pts)}}

On previous assignments, you developed classifiers based on ridge regression/least squares as well as through minimization of both the hinge and logistic loss. One drawback to all of these approaches is that they result in \emph{linear} decision boundaries. However, many datasets are not linearly separable, and hence our goal in this problem is to develop a classifier that allows for nonlinear decision boundaries. To do this, you will use kernel ridge regression (KRR, see demo code) to form a
kernelized least squares classifier, which you will test on the MNIST dataset.

\begin{enumerate}[(a)]
    \item Use either my \texttt{KRR} function or your own to perform classification on MNIST digits 1 and 2 after reducing the dimensionality to 2 using PCA by completing the \texttt{prob3} script, taking $\lambda = 0.01$ in \texttt{KRR}. \textbf{Turn in} plots showing the separator and the traning data for $\sigma \in \set{0.1, 1, 5}$. Provide the training and test error for each combination and comment on each plot.
    \item Describe in words how the provided code plots the decision boundary formed by \texttt{KRR}.
\end{enumerate}

\section{\normalfont{(5 pts)}}

Find one article, podcast, or talk on the subject of AI ethics that you think would be beneficial to anyone working in this area. Share your resource in the \texttt{interesting-reading} channel along with one or two sentences describing why you chose it.

\section{\normalfont{(3 pts + 50 Lipor pts)}}

You should have received one or more messages requesting that you complete a course evaluation. Please complete this evaluation, and type ``I have completed the course evaluation'' once you've done so. \textbf{Note:} To maintain anonymity, I will not review which students answer affirmatively to this question.

\end{document}
