\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 518: Machine Learning Theory \& Algorithms
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{1}{April 12, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section{Multiclass Ridge Regression Classifier \normalfont{(5 pts each)}}

In this problem, you will derive and implement a multiclass linear classifier based on \href{https://en.wikipedia.org/wiki/Tikhonov_regularization}{ridge regression}. Suppose you are given $m$ training examples $\set{(x_{i},y_{i})}$, from $K$ classes, where $x_{i} \in \bR^{d}$ and $y_{i} \in \set{e_{1},\dots,e_{K}}$\footnote{The vector $e_{k}$ is the $k$th standard basis vector, taking zero everywhere except in the $k$th element, where it takes the value one.} is a
\href{https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/}{one-hot vector}. One way to perform classification is to find the matrix $W \in \bR^{d \times K}$ that minimizes the least squares cost function given below. You can then estimate the \emph{integer-valued} class labels as
\begin{equation*}
    \hat{y}_{i} = \argmax_{k \in [K]} W^{T}x_{i}.
\end{equation*}

\begin{enumerate}[(a)]
    \item Derive a closed-form solution to the problem
        \begin{equation*}
            \widehat{W} = \argmin_{W \in \bR^{d \times K}} \sum_{i=1}^{m} \norm{W^{T}x_{i} - y_{i}}_{2}^{2} + \lambda \norm{W}_{F}^{2},
        \end{equation*}
        where $\lambda > 0$ is a regularization parameter.
    \item Implement your trained classifier with $\lambda = 10^{-4}$ on the full MNIST dataset included in the homework files. You may not use \texttt{sklearn} or other high-level libraries to perform the one-hot encoding, but you may use \texttt{numpy} and \texttt{scipy}. \textbf{Turn in} your code, as well as the classification error on the training and test sets.
\end{enumerate}

\section{DSS: Using \texttt{scikit-learn} \normalfont{(10 pts)}}

(\emph{DSS rules apply.}) One important part of data science is fluency with popular libraries. One library that is useful for a variety of machine learning tasks (especially those not related to deep learning) is \texttt{scikit-learn}, a.k.a., \texttt{sklearn}. Use \texttt{sklearn} to perform multiclass ridge regression classification on the full MNIST dataset included in the homework files, taking special care to make sure the regularization parameter $\lambda$ and the use of an offset are the same. \textbf{Turn in} your code, an explanation of what
functions and options you used to perform the
multiclass classification, as well as the classification error on the training and test sets.

\section{SLT \normalfont{(5 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 2, Exercise 2. State how long you worked on the problem before looking at the solution.

\section{SLT \normalfont{(5 pts each)}}

(\emph{SLT rules apply.}) UML, Ch. 2, Exercise 3.1-3.2. State how long you worked on the problem before looking at the solution.

\end{document}
