\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}
\usepackage{caption,subcaption}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 516: Mathematical Foundations of Machine Learning
            \hfill Winter \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{7}{March 5, 2023}{\X}
% PUT YOUR NAME BELOW WHERE I HAVE \X

\section{\normalfont{(4 pts, 2 pts, 2 pts)}}

Define the unit $\ell_{p}$-norm ball as
\begin{equation*}
    B_{p} = \set{x \in \bR^{D}: \norm{x}_{p} \leq 1}.
\end{equation*}
It is often useful to visualize the unit ball for different norms, as we saw in the lecture on sparse regression. To do this, we instead visualize the unit sphere for each norm, which is
\begin{equation*}
    \bS_{p} = \set{x \in \bR^{D}: \norm{x}_{p} = 1}.
\end{equation*}
One way to visualize this sphere is to generate a large number of random points and normalize these points so that they have $\norm{x}_{p} = 1$.

\begin{enumerate}[(a)]
    \item Complete the script \texttt{normalize} that takes in a matrix $X = [x_{1} \; x_{2} \; \dots \; x_{N}] \in \bR^{D \times N}$ and normalizes it so that each \emph{column} of $X$ has unit $\ell_{p}$-norm. \textbf{Do not} use any loops or built in \texttt{norm} or \texttt{norms} functions. \textbf{Turn in} your code and a brief explanation of how it works.
    \item Test your implementation by running \texttt{prob1}. \textbf{Turn in} the resulting plot.
    \item Intuitively, a set is convex if the line connecting any two points in a set is contained within that set. Based on this definition and your plot, for which norms (i.e., which values of $p$) is the unit ball convex?
\end{enumerate}

\newpage
\section{\normalfont{(5 pts, 4 pts, 4 pts)}}

In this problem, you will learn a linear classifier using empirical risk minimization (ERM) with the \emph{logistic loss}. In this case, for a linear function parameterized by the weight vector $w$, ERM seeks to minimize
\begin{equation}
    J(w) = \frac{1}{N} \sum_{i = 1}^{N} L(y_{i},x_{i}^{T}w) + \frac{\lambda}{2} \norm{w}_{2}^{2},
    \label{eq:erm}
\end{equation}
where 
\begin{equation}
    L(y,t) = \log(1 + e^{-yt}).
    \label{eq:logisticLoss}
\end{equation}
The above loss is convex and differentiable. Using the above loss and assuming there are two class labels $y_{i} \in \set{-1,1}$, solving \eqref{eq:erm} yields the optimal \emph{logistic regression} classifier. In this problem, you will find this classifier using gradient descent and stochastic gradient descent.

\begin{enumerate}[(a)]
    \item First, rewrite \eqref{eq:erm} as a summation
        \begin{equation}
            J(w) = \sum_{i = 1}^{N} J_{i}(w)
            \label{eq:lrSeparated}
        \end{equation}
        for an appropriately defined $J_{i}(w)$.
        Using the loss function defined in \eqref{eq:logisticLoss}, differentiate \eqref{eq:lrSeparated} for a single $i$ with respect to the vector $w \in \bR^{D}$ to obtain the gradient.
    \item Solve \eqref{eq:erm} with logistic loss using gradient descent by computing the full gradient vector, which is the sum of the above ``stochastic'' gradients. Complete the \texttt{lrgd} function using the learning rate $\mu = \frac{100}{kk}$, where $kk$ is the iteration number. Test your code on digits 1 and 2 from the MNIST dataset \emph{after reducing the dimensionality to 2 using PCA} by running \texttt{prob2}. \textbf{Turn in} your code. Plots will be formed in part (c).
    \item Solve \eqref{eq:erm} using \emph{stochastic} gradient descent by taking 20 full passes through the data in random order. Complete \texttt{lrsgd} using the learning rate $\mu = \frac{100}{kk}$, where $kk$ is the iteration number. \textbf{Turn in} the following:
        \begin{itemize}
            \item A plot of the resulting cost as a function of the number of iterations
            \item The resulting plot of the two separators and the reduced-dimension data (code provided in \texttt{prob2}) 
            \item A sentence or two describing how the two methods compare.
        \end{itemize}
        \textbf{Note:} For fair comparison, you should compute the cost for SGD after each of the 20 cycles, not after each individual update.
\end{enumerate}

\newpage
\section{\normalfont{(5 pts, 4 pts, 4 pts, 4 pts)}}

In this problem, you will learn a linear classifier using empirical risk minimization (ERM) with the \emph{hinge loss}. You will again minimize \eqref{eq:erm}, but this time take the loss defined by
\begin{equation}
    L(y,t) = \max(0,1 - yt).
    \label{eq:hingeLoss}
\end{equation}
The above loss is convex but not differentiable at $0$, so we will make use of subgradients (see notes on sparse regression). Using the above loss and assuming there are two class labels $y_{i} \in \set{-1,1}$, solving \eqref{eq:erm} yields the optimal \emph{soft margin hyperplane}, which is the subspace of dimension $D-1$ that best separates data from two classes (where $D$ is the dimension of the features/examples/data points). In this problem, you will find this classifier using gradient descent and stochastic gradient descent.

\begin{enumerate}[(a)]
    \item First, rewrite \eqref{eq:erm} as a summation
        \begin{equation}
            J(w) = \sum_{i = 1}^{N} J_{i}(w)
            \label{eq:hingeSeparated}
        \end{equation}
        for an appropriately defined $J_{i}(w)$.
        Using the loss function defined in \eqref{eq:hingeLoss}, differentiate \eqref{eq:hingeSeparated} for a single $i$ with respect to the vector $w \in \bR^{D}$ to obtain a subgradient. Be careful at the point $yt = 1$.
    \item Solve \eqref{eq:erm} with hinge loss using gradient descent by computing the full gradient vector, which is the sum of the above ``stochastic'' gradients. Complete the \texttt{osmgd} function using the learning rate $\mu = \frac{100}{kk}$, where $kk$ is the iteration number. Test your code on digits 1 and 2 from the MNIST dataset \emph{after reducing the dimensionality to 2 using PCA} by running \texttt{prob3}. \textbf{Turn in} your code. Plots will be formed in part (c).
        \\
        \textbf{Hint:} It may be easier to create a separate \texttt{subg} function to compute the subgradient, which you can use in both parts (b) and (c).
    \item Solve \eqref{eq:erm} using \emph{stochastic} gradient descent by taking 20 full passes through the data in random order. Complete \texttt{osmsgd} using the learning rate $\mu = \frac{100}{kk}$, where $kk$ is the iteration number. \textbf{Turn in} the following:
        \begin{itemize}
            \item A plot of the resulting cost as a function of the number of iterations
            \item The resulting plot of the two separators (code provided in \texttt{prob3}) 
            \item A sentence or two describing how the two methods compare.
        \end{itemize}
        \textbf{Note:} For fair comparison, you should compute the cost for SGD after each of the 20 cycles, not after each individual update.
    \item Finally, plot the reduced-dimension data with the classifiers obtained via (1) logistic loss + SGD, (2) hinge loss + SGD, and (3) LS classifer from Homework 6. \textbf{Turn in} the resulting plot and a table displaying the training and test errors in the form below.
\end{enumerate}

\begin{table*}[h!]
    \centering
    \begin{tabular}{ | c || c | c | c | }
        \hline 
        & Logistic Loss & Hinge Loss & Least Squares \\
        \hline 
        Training Error (\%) & & & \\
        \hline 
        Test Error (\%) & & & \\
        \hline 
    \end{tabular}
    \caption{Comparison of classification performance on MNIST digits 1 and 2 after reducing dimension to two using PCA.}
    \label{tab:classComp}
\end{table*}

\end{document}
