\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 518: Machine Learning Theory \& Algorithms
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{6}{May 31, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section{Kernel SVM \normalfont{(15 pts)}}

Your task in this problem is to implement the \texttt{SGD for Solving Soft-SVM with Kernels} algorithm given on pg. 223 of UML and test this algorithm on \texttt{prob1.py}. While the algorithm appears to require actually mapping the feature vectors to a high-dimensional space, this is actually not necessary for prediction. Note that prediction is performed via the equation
\begin{equation*}
    \hat{y}_{i} = \sign\left( \bar{w}^{T} \psi\left( x_{i} \right) \right) = \sign\left( \sum_{j=1}^{m} \bar{\alpha}_{j} \psi\left( x_{j} \right)^{T}\psi\left( x_{i} \right) \right),
\end{equation*}
and hence the predictions can also be kernelized.
Use the Gaussian kernel with your choice of values for $\sigma$ and $\lambda$, and use $T = 10,000$ iterations. \textbf{Do not} use any loops when computing your kernel matrices or your predictions. \textbf{Turn in} your code, your selected values of $\sigma$ and $\lambda$, as well as the training and test errors. \emph{Note:} Your final test error should be below 5\%.

\section{DSS: Regression Models \normalfont{(10 pts, 5 pts, 5 pts)}}

(\emph{DSS rules apply.}) Our focus in this class has been largely on classification problems. However, as you saw with (not-so-mini) Mini Project 2, the models we have studied frequently have variants that can be used for regression as well. In this problem, you will utilize three of the models we have studied to perform regression on the \href{https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html#sklearn.datasets.fetch_california_housing}{California housing dataset}.

\begin{enumerate}[(a)]
    \item Use the \texttt{sklearn} support vector regression implementation to predict on the housing dataset as loaded in \texttt{prob2.py}. There are several kernels available, as well as a number of parameters to select for each kernel. You should use model selection to decide which kernel and parameters are most appropriate. Note that you may choose to either use a validation set or cross validation, but you should justify your choice. Further, the performance of SVR will be greatly improved by performing data preprocessing, for which you can make use of the \texttt{sklearn.preprocessing}
        library. You should also decide on the appropriate preprocessing using \emph{only the training data}, then apply your chosen preprocessing to the entire dataset (training and test) simultaneously. \textbf{Turn in} a description of your approach to model selection, your approach to preprocessing, your selected SVR parameters, your training and test errors, and your code.
    \item Compare your SVR errors to ridge regression and $k$-nearest neighbors (you may use the \texttt{sklearn} implementation of these). Use the same data preprocessing for all three algorithms, but be sure to tune parameters for each algorithm individually. \textbf{Turn in} your training and test errors for ridge regression and $k$-nearest neighbors.
    \item Provide a brief (half page maximum) description of how SVR works. Assume your audience is familiar with SVMs for classification and cite any sources you used to gain your understanding.
\end{enumerate}

\section{Positive Definite Kernels \normalfont{(10 pts)}}

Lemma 16.2 states that a function $K: \sX \times \sX \to \bR$ is a valid kernel function if and only if the resulting Gram matrix is positive semidefinite. In the proof the authors claim that it is ``trivial to see that if $K$ implements an inner product in some Hilbert space then the Gram matrix is positive semidefinite.'' As we all know, one textbook author's trivial is often a student's homework problem. Prove this ``trivial'' claim.

\section{SLT \normalfont{(5 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 15, Exercise 1. State how long you worked on the problem before looking at the solution.

\end{document}
