\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 518: Machine Learning Theory \& Algorithms
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{4}{May 10, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section{Multiclass AdaBoost \normalfont{(5 pts, 3 pts, 3 pts)}}

The AdaBoost algorithm implemented last week is only capable of performing binary classification. In this problem, you will see how a binary classifier can be used to perform multiclass classification. For cleaner code, you may wish to package your AdaBoost classifier into its own class, as with the decision stump. 

\begin{enumerate}[(a)]
    \item Extend your AdaBoost implementation from Homework 3 to multiclass classification by following the One-versus-All approach described in Section 17.1 of UML. Test your code on the synthetic data generated in \texttt{prob1a.py} using $T = 500$ rounds of boosting. \textbf{Turn in} your code and a plot of the training and test error versus number of boosting rounds. A logarithmic y-axis may be helpful to see the resulting behavior.
    \item Generate the same plot as for part (a) using XGBoost with \texttt{max\_depth} set to 1. What similarities or differences do you notice?
    \item Apply your multiclass AdaBoost to the MNIST dataset. \textbf{Turn in} your training and test error, as well as the number of boosting rounds you performed and why you chose this value.
\end{enumerate}

\section{DSS: Visualizing with Random Forests \normalfont{(5 pts each)}}

(\emph{DSS rules apply.}) Aside from their use in classification and regression, decision trees can also be used to understand the similarity between datapoints, even when the features have different scales and types. This is important, since a good notion of similarity can be used to produce meaningful embeddings via algorithms such as UMAP. First, read the articles \href{https://gdmarmerola.github.io/forest-embeddings/}{here} and
\href{https://gdmarmerola.github.io/umap-supervised-embeddings/}{here}.

\begin{enumerate}[(a)]
    \item Apply the technique described in the articles to perform supervised dimensionality reduction/visualization with UMAP on the MNIST \textbf{test} dataset (you may choose to do the full 70,000-digit dataset if time allows). \textbf{Turn in} your plot of data embedded into two dimensions, as well as a plot of the UMAP embedding on the raw data (\emph{without} random forest features, as in Homework 2, Problem 2).
    \item Apply the technique described in the articles to perform supervised dimensionality reduction/visualization with UMAP on the Titanic dataset from MP1. Note that you may wish to use the raw data after filling in missing entries instead of the feature-engineered data you used in your project. \textbf{Turn in} your plot of data embedded into two dimensions, as well as a plot of the UMAP embedding on the raw data.
    \item Describe the procedure you used to generate these embeddings \textit{in your own words} (i.e., do not copy text from the articles). Your description should be such that you could have understood it prior to taking this course.
\end{enumerate}

\section{Information Gain \normalfont{(5 pts)}}

UML mentions that concave measures of information gain for decision trees are preferred. Show that if $C(a)$ is concave, then the \texttt{Gain} function is nonnegative. This indicates that splitting a node will not ever decrease the measures of gain defined in the text.

\section{SLT \normalfont{(10 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 10, Exercise 4.2. Note that the boook has a typo, and the upper bound should be
\begin{equation*}
    \text{VCdim}(B_{d}) \leq 16 + 2 \log_{2} (d).
\end{equation*}
State how long you worked on the problem before looking at the solution.

\section{SLT \normalfont{(10 pts)}}

(\emph{SLT rules apply.}) UML, Ch. 18, Exercise 1. State how long you worked on the problem before looking at the solution.

\end{document}
