\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 518: Machine Learning Theory \& Algorithms
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{2}{April 19, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section{Multiclass Logistic Regression Classifier \normalfont{(5 pts, 5 pts, 5 pts, 2 pts)}}

In this problem, you will derive and implement a multiclass linear classifier based on \emph{logistic regression} (LR, see Section 9.3 of UML). For binary classification, LR has two interpretations. The first, which is given in UML, is that it performs ERM with the logistic loss, i.e., using
\begin{equation*}
    \ell(h_{w}, (x,y)) = \log\left( 1 + e^{-y (w^{T}x)} \right),
\end{equation*}
where $x, w \in \bR^{d}$, $y \in \set{-1,1}$, and
\begin{equation*}
    h_{w}(x) = \sign\left( w^{T}x \right).
\end{equation*}
This results in the ERM objective function
\begin{equation}
    \label{eq:lrERM}
    J_{1}(w) = \frac{1}{m} \sum_{i=1}^{m} \log\left( 1 + e^{-y_{i} (w^{T}x_{i})} \right).
\end{equation}
The second interpretation of (binary) LR is that it models/learns the conditional distribution $\bP[Y = y \given x]$ and then uses this model to estimate the Bayes classifier. In particular, LR assumes that 
\begin{equation*}
    \bP[Y = y \given x, w] \approx \sigma(w^{T}x)  
\end{equation*}
where
\begin{equation*}
    \sigma(t) = \frac{1}{1 + e^{-t}}
\end{equation*}
is called the \emph{sigmoid} or \emph{logistic} function. You can check that $\sigma(w^{T}x) \in [0,1]$ and goes to 1 when $w^{T}x$ is large and 0 when $w^{T}x$ is small. The vector $w$ can be learned using maximum likelihood estimation, which results in minimizing the negative log-likelihood
\begin{equation}
    \label{eq:lrML}
    J_{2}(w) = -\frac{1}{m} \sum_{i=1}^{m} y_{i} \log\left( \sigma(w^{T}x_{i}) \right) + (1-y_{i}) \log\left(1 - \sigma(w^{T}x_{i}) \right),
\end{equation}
where now we set $y_{i} \in \set{0,1}$. The learned $w$ is then used in $\sigma(t)$ above to estimate the Bayes classifier.

The other benefit of the probabilistic interpretation of LR is that it easily extends to the multiclass case. In this case, the conditional probability is modeled as
\begin{equation*}
    \bP[Y = k \given x, w_{1},\dots,w_{K}] \approx \text{softmax}(k, w_{1}^{T}x,\dots,w_{K}^{T}x),
\end{equation*}
where
\begin{equation*}
    \text{softmax}(k,t_{1},\dots,t_{K}) = \frac{e^{t_{k}}}{\sum_{j=1}^{K} e^{t_{j}}}
\end{equation*}
is the \emph{softmax} function, which is the generalization of the sigmoid to multiple classes.
We learn $K$ weight vectors (one for each class) by minimizing the cost function
\begin{equation}
    \label{eq:mlr}
    J(w_{1},\dots,w_{K}) = - \sum_{i=1}^{m} \sum_{k=1}^{K} \ind{y_{i} = k} \log\left( \text{softmax}(k, w_{1}^{T}x_{i},\dots,w_{K}^{T}x_{i}) \right).
\end{equation}
The above is minimized using (stochastic) gradient descent and is known as \href{https://en.wikipedia.org/wiki/Multinomial_logistic_regression}{multiclass LR/multinomial LR/softmax regression}.

\begin{enumerate}[(a)]
    \item Verify that \eqref{eq:lrERM} and \eqref{eq:lrML} are equivalent cost functions. \emph{Hint:} Start with \eqref{eq:lrML} and use algebraic manipulations to show that it is equivalent to \eqref{eq:lrERM}.
    \item Verify that the gradient of \eqref{eq:mlr} with respect to one weight vector is
        \begin{equation*}
            \nabla_{w_{k}} J(w_{1},\dots,w_{K}) = \sum_{i=1}^{m} x_{i} \left( \text{softmax}(k, w_{1}^{T}x,\dots,w_{K}^{T}x) - \ind{y_{i}=k} \right).
        \end{equation*}
    \item Complete the script \texttt{prob1.py} by minimizing \eqref{eq:mlr} using stochastic gradient descent and then using the learned weight vectors to predict on the MNIST dataset (use the data from Homework 1). Use a learning rate of $\mu = 10^{-2}$ and run ten full passes through the data (you can play with these if you want). \textbf{Turn in} your code, as well as the classification error on the training and test sets. \emph{Hint:} I've given you code to create a smaller dataset for debugging that selects only three digits. Your final training and test errors should both be below 10\%.
    \item What do you notice about the differences between LR and the multiclass ridge regression classifier from Homework 1?
\end{enumerate}

\section{DSS: Visualizing Data \normalfont{(3 pts, 5 pts, 5 pts)}}

(\emph{DSS rules apply.}) Another important tool for any data scientist is knowing how to visualize results. Two popular approaches to visualizing high-dimensional datasets are \emph{t-distributed stochastic neighbor embedding} (t-SNE) and \emph{Uniform Manifold Approximation and Projection for Dimension Reduction} (UMAP). In this problem, you will use UMAP to embed the MNIST \textbf{test} dataset and create an interactive plot showing which digits were misclassified. To complete this
problem, you will hack the tutorial \href{https://umap-learn.readthedocs.io/en/latest/basic_usage.html#}{here}, which requires the following packages: \texttt{pandas, seaborn, bokeh, umap}. 

\noindent \emph{Hint:} Start by grabbing a subset of the digits to reduce computation time while debugging.

\begin{enumerate}[(a)]
    \item Use UMAP to create a scatter plot of the MNIST test dataset embedded into two dimensions, with the color of each point in your plot corresponding to the true class of the image. Note that the `Digits' dataset is not the same as the MNIST dataset. \textbf{Turn in} your plot.
    \item Hack the code in the linked tutorial to create an interactive plot that allows you to view the image and true class of each point in the MNIST test dataset when you hover over it. \textbf{Turn in} a screenshot of your plot with your mouse hovering over a point to show the tooltip.
    \item An important method for troubleshooting and displaying results is understanding what types of examples are misclassified by your predictor. First, classify the MNIST test dataset using \texttt{LogisticRegression} from \texttt{sklearn}. Next, hack the code from part (b) so that the points are colored based on whether they are classified correctly. \textbf{Turn in} your code, a screenshot of your plot with your mouse hovering over a point to show the tooltip, and some observations of what you learned about which types of
        points are incorrectly classified.
\end{enumerate}
\section{SLT \normalfont{(10 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 3, Exercise 7. State how long you worked on the problem before looking at the solution.

\section{SLT \normalfont{(10 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 4, Exercise 2. State how long you worked on the problem before looking at the solution.

\end{document}
