\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 518: Machine Learning Theory \& Algorithms
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{5}{May 24, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section{Feature Representations \normalfont{(10 pts)}}

In our study of model selection, we discussed several options of what to do when learning fails. Perhaps the most opaque of these was to ``change the feature representation of the data.'' In our upcoming study of kernels, we will see how to implicitly work with different feature representations. In this problem, we will explicitly work with a new feature representation known as \emph{random Fourier features}, described in the seminal work
\href{https://people.eecs.berkeley.edu/~brecht/papers/07.rah.rec.nips.pdf}{here}.

You will work with the MNIST dataset using your multiclass ridge regression implementation from Homework 1. Transform each vector $x_{i} \in \bR^{d}$ via the equation
\begin{equation*}
    \bar{x}_{i} = \cos\left( Gx_{i} + b \right),
\end{equation*}
where $G \in \bR^{p \times d}$ is a Gaussian random matrix whose elements are drawn iid from the normal distribution with mean zero and variance $\sigma^{2} = 0.1$, and the elements of $b \in \bR^{p}$ are chosen iid from the uniform distribution on $[0,2\pi]$.

To select the new feature dimension $p$, perform a training/validation split of 80\%/20\% on your training data and use the validation set to perform model selection. \textbf{Turn in} the test error for your selected value of $p$, as well as a plot showing your training and validation error as a function of $p$. 

\noindent \textit{Notes:} 
\begin{enumerate}
    \item You should use the same $G$ to transform the entire dataset.
    \item You may run into memory issues for $p > 6000$.
    \item You may use $\lambda = 10^{-4}$ as in Homework 1, but feel free to perform model selection on $\lambda$ and/or $\sigma^{2}$ as well.
\end{enumerate}

\section{Strong Convexity \normalfont{(10 pts)}}

Prove that $f(w) = \lambda \norm{w}^{2}$ is $2\lambda$-strongly convex.

\section{SLT \normalfont{(10 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 12, Exercise 3. State how long you worked on the problem before looking at the solution.

\section{SLT \normalfont{(5 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 13, Exercise 1, first bullet only. State how long you worked on the problem before looking at the solution.

\end{document}
