\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}
\usepackage{caption,subcaption}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 516: Mathematical Foundations of Machine Learning
            \hfill Winter \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{6}{February 24, \the\year}{\X}
% PUT YOUR NAME BELOW WHERE I HAVE \X

\section{\normalfont{(2 pts, 4 pts)}}

Recall that a vector space is a general term, where ``vectors'' may be other elements (e.g., functions or matrices). Consider the vector space of $n \times n$ real-valued matrices. In this case, we may define an inner product between two ``vectors'' as
\begin{equation*}
    \ip{A}{B} = \tr(B^{T}A),
\end{equation*}
where $A, B \in \bR^{n \times n}$.

\begin{enumerate}[(a)]
    \item Verify that the norm corresponding to this vector space is the Frobenius norm.
    \item Verify the following ``orthogonality principle'' for low-rank approximation.
        \begin{equation*}
            (A - \hat{B}_{K}) \perp \hat{B}_{K},
        \end{equation*}
        where $\hat{B}_{K} = \argmin_{B: \text{rank}(B) \leq K} \norm{A - B}_{F}$. The above states that the error resulting from low-rank approximation is orthogonal to the estimate itself.
\end{enumerate}

\section{\normalfont{(4 pts)}}

In the setting of slide 6.15 of the Low-Rank Approximation notes, verify the statement
\begin{equation*}
    \sum_{k = 1}^{K+1} \sigma_{k}^{2} \norm{v_{k}^{T}x}_{2}^{2} = \norm{Ax}_{2}^{2}.
\end{equation*}

\section{\normalfont{(5 pts, 5 pts, 4 pts)}}

The classic least-squares problem seeks to minimize the $\ell_{2}$-norm of the residual $r = Ax - b$. It is well-known that this norm is not robust to outliers, and hence we may wish to consider a more robust formulation involving the $\ell_{1}$-norm known as the mean absolute error (MAE) or least absolute deviations (LAD)  minimization
\begin{equation}
    J(x) = \norm{Ax - b}_{1} = \sum_{i = 1}^{m} \abs{a_{i}^{T}x - b_{i}},
    \label{eq:maeCost}
\end{equation}
where $b_{i} \in \bR$ denotes the $i$th element of the vector $b$ and $a_{i} \in \bR^{n}$ is the (column version of the) $i$th row of $A$. Unlike least-squares, there is no closed-form solution to this problem. Further, $J(x)$ is not even differentiable, since the derivative of $\abs{\cdot}$ does not exist at the origin. However, the cost function above is convex, so we do have tools to minimize it.

One popular approach to solving \eqref{eq:maeCost} is to apply the \href{https://en.wikipedia.org/wiki/MM_algorithm}{majorize-minimize} (MM) algorithm, of which the popular \href{https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm}{expectation-maximization} (EM) algorithm is a special case. Given an initialization $x_{0}$, the MM algorithm alternates between the following steps.
\begin{enumerate}
    \item Find a function $\bar{J}_{k}(x)$ such that \emph{majorizes} $J(x)$, i.e., satisfies
        \begin{eqnarray*}
            J(x_{k}) = \bar{J}_{k}(x_{k}) \\
            J(x) \leq \bar{J}_{k}(x) \;\; \forall x.
        \end{eqnarray*}
    \item Solve $x_{k+1} = \argmin_{x} \bar{J}_{k}(x)$.
\end{enumerate}
The goal is to choose $\bar{J}$ such that the minimization in step 2 can be carried out efficiently.

\begin{enumerate}[(a)]
    \item Let
        \begin{equation*}
            r_{i}(x) = a_{i}^{T}x - b_{i}.
        \end{equation*}
        Verify that
        \begin{equation*}
            \bar{J}_{k}(x) = \sum_{i = 1}^{m} \sqrt{r_{i}^{2}(x_{k})} + \frac{1}{2} \frac{r_{i}^{2}(x) - r_{i}^{2}(x_{k})}{\sqrt{r_{i}^{2}(x_{k})}}
        \end{equation*}
        majorizes $J(x)$. Note that if $r_{i}^{2}(x_{k}) = 0$, we set that term in the sum to zero, but this event is extremely unlikely.
        \\
        Hint: Use the fact that for a convex function $g: \bR^{n} \rightarrow \bR$
        \begin{equation*}
            g(v) \geq g(u) + \ip{\nabla g(u)}{v-u}.
        \end{equation*}
    \item Find and define the matrix $W \in \bR^{m \times m}$ such that the minimization in step 2 is reduced to solving
        \begin{equation*}
            \hat{x} = \argmin_{x} \norm{W(Ax-b)}_{2}^{2}.
        \end{equation*}
    \item The resulting algorithm for minimizing \eqref{eq:maeCost} is known as \emph{iteratively reweighted least squares}. Implement this algorithm in \texttt{mae\_irls} and test your implementation on the included \texttt{prob3}. 
        \\
        \textbf{Turn in} your \texttt{mae\_irls} code and the resulting plot.
\end{enumerate}

\section{\normalfont{(4 pts each)}}

In this problem, you will implement the (classical) Multidimensional Scaling (MDS) algorithm for embedding points into a low-dimensional space given only a notion of distance between points. We derived this algorithm beginning on Pg. 6.31 of the notes. In that case, we considered only Euclidean distances, but in practice we could use any notion of dissimilarity. In this problem, we'll use the \emph{cosine similarity}, i.e., inner products, to get a notion of distance between points on a union of
subspaces. To convert this to a distance, we define
\begin{equation}
    d(x_{i},x_{j}) = 1 - \ip{\frac{x_{i}}{\norm{x_{i}}}}{\frac{x_{j}}{\norm{x_{j}}}},
    \label{eq:cosDist}
\end{equation}
which is between 0 and 1 due to the normalization of points.

\begin{enumerate}[(a)]
    \item Implement the MDS algorithm derived in class by completing the \texttt{MDS} function included. To test your algorithm, we'll consider again the MNIST digits used in the class demo. Run your \texttt{MDS} on the \texttt{mnistSubset.mat} data using \emph{only digits 1 and 2}. \textbf{Turn in} a plot of the data embedded in three-dimensional space. Plot the points from each class in a different color (easily done using \texttt{hold all} in \matlab).
    \item Now we'll compare to the embedding we get by using PCA for dimensionality reduction. Let $X$ be the matrix whose columns are the vectors from classes 1 and 2 only. Using PCA as described in Section 6.2 of the notes, embed these points into three-dimensional space. \textbf{Turn in} a plot of the embedded data, again using a different color for each of the two classes. How does this compare to what you get from MDS? Note that PCA operates on the data, while MDS operates only on
        the pairwise distances.
\end{enumerate}

\section{\normalfont{(5 pts each)}}

In this problem, you'll use least squares to perform binary classification, an algorithm known as the \emph{least-squares classifier} (see BV, Ch. 14). Suppose you are given training data consisting of $N$ feature-label pairs $\set{(x_{i}, y_{i})}_{i=1}^{N}$, where $x_{i} \in \bR^{D}$ are the MNIST images and $y_{i} \in \set{-1,1}$ are the corresponding labels. We can use least-squares to find the vector $w \in \bR^{D}$ such that the sign of $w^{T}x_{i}$ is a good predictor of $y_{i}$. Define the
data matrix and vector of labels as
\begin{equation*}
    X = \begin{bmatrix} x_{1} & x_{2} & \dots & x_{N} \end{bmatrix} 
    \qquad
    y = \begin{bmatrix} y_{1} & y_{2} & \dots & y_{N} \end{bmatrix}^{T},
\end{equation*}
respectively. Then we can learn a separator by solving the ridge regression problem
\begin{equation*}
    \hat{w} = \argmin_{w \in \bR^{D}} \norm{X^{T}w - y}_{2}^{2} + \lambda \norm{w}_{2}^{2}.
\end{equation*}
The label of any point $x \in \bR^{D}$ can then be predicted as
\begin{equation*}
    \hat{y} = \sign(\hat{w}^{T}x).
\end{equation*}

\noindent \textbf{Practical note:} We're likely to obtain a better classifier if we add a bias and regularizer to the above. However, this classifier isn't the most practical choice anyway, so consider this problem as being only for instructional purposes of how classification works.

% mention bias, regularizer, add train/test split to code
\begin{enumerate}[(a)]
    \item Implement the above least-squares classifier on digits 1 and 2 from the MNIST subset provided using the \texttt{prob5} script. Train your classifier using the variables $\texttt{Xtrain, ytrain}$. \textbf{Turn in} the resulting error on both the training and test datasets.
    \item Now use PCA on the training data to project your data into two dimensions and re-train your LS classifier on the reduced-dimension training data. \textbf{Turn in} your training and test error and a plot of the reduced-dimension data with the learned LS separator (I've given you code for plotting the separator).
\end{enumerate}

\section{\normalfont{(5 pts)}}

For the remainder of the quarter, we will spend some time thinking about the ethics of machine learning. For this week, either watch the presentation by Timnit Gebru \href{https://www.youtube.com/watch?v=23MxOh99N54&feature=youtu.be}{here} or listen to the podcast \href{https://www.radicalai.org/e12-timnit-gebru}{here}. After watching/listening, \textbf{share one thing you learned and one question you have to the \texttt{interesting-reading} channel on Slack}.

\end{document}
