\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 518: Machine Learning Theory \& Algorithms
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{3}{May 3, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section{AdaBoost with Decision Stumps \normalfont{(10 pts, 5 pts, 5 pts)}}

Your task in this problem will be to implement the AdaBoost algorithm using decision stumps (DS) as your base classifier. You will test your implementation on both synthetic data and the Titanic dataset.

\begin{enumerate}[(a)]
    \item Implement ERM for decision stumps by completing the \texttt{DecisionStump} class. Note that the description in the textbook only accounts for the case where $b = +1$ but you need to account for $b = -1$ as well. \textbf{Turn in} your code, your training error on \textbf{both} datasets generated in \texttt{prob1a.py}, and a scatter plot of each dataset with color corresponding to your predicted label.
    \item Implement AdaBoost with DS as the base classifier. \textbf{Turn in} your code, training error, number of boosting rounds needed to get this error, and scatter plots for \textbf{both} datasets generated in \texttt{prob1b.py}.
    \item Apply your implementation of AdaBoost to your processed data from MP1, selecting the parameters however you would like. Report the error for both your best linear classifier and AdaBoost.
\end{enumerate}

\section{DSS: XGBoost \normalfont{(5 pts each)}}

(\emph{DSS rules apply.}) Aside from \texttt{sklearn}, the other go-to package for algorithms that don't involve neural networks is \texttt{XGBoost}. Note that in this case the ``boost'' stands for \href{https://en.wikipedia.org/wiki/Gradient_boosting}{gradient boosting}, which is another approach to ensemble learning with decision trees. XGBoost is extremely popular, both because it achieves excellent empirical performance and very low computational complexity. It is fair to say that
XGBoost is a good first choice of nonlinear method for classification or regression. In this problem, you will apply XGBoost to the MNIST dataset.

\begin{enumerate}[(a)]
    \item Apply XGBoost to the MNIST dataset using a multiclass softmax loss. For an initial validation, use 10 rounds of boosting and a max depth of 2. \textbf{Turn in} your training and test error and which API you used.
    \item Your above error should be worse than what you achieved with the multiclass LR from Homework 2. Vary the parameters to try to get an improvement over LR. \textbf{Turn in} your ultimate training and test error, as well as a description of each parameter that you changed and why you think this setting resulted in an improvement.
    \item Apply XGBoost to your processed data from MP1, selecting the parameters however you would like. Report the error for both your best linear classifier and XGBoost.
\end{enumerate}

\section{VC-Dimension of Decision Stumps \normalfont{(5 pts)}}

Prove that the VC-dimension of decision stumps on $\sX = \bR$ is 2. DSS rules do not apply.

\section{AdaBoost Step Size \normalfont{(10 pts)}}

AdaBoost and XGBoost can both be viewed through the more general framework of functional gradient descent. In this case, the loss we aim to minimize is
\begin{equation*}
    F(w) = \sum_{i=1}^{m} e^{-y_{i} \sum_{t = 1}^{T} w_{t} h_{t}(x_{i})},
\end{equation*}
where $w \in \bR^{T}$ is a weight vector. Letting $f_{T} = \frac{1}{m} \sum_{t = 1}^{T} w_{t} h_{t}$, we see that this is exactly the term that we upper bounded in the proof of Thm. 10.2. Since we proceed one coordinate of $w$ at a time, our goal at round $t+1$ is to minimize
\begin{equation}
    \label{eq:coordDescent}
    F(w_{t+1}) = \sum_{i=1}^{m} e^{-y_{i} \left( f_{t}(x_{i}) + w_{t+1} h_{t+1}(x_{i}) \right)}.
\end{equation}
Prove that the minimum of \eqref{eq:coordDescent} yields the AdaBoost weight
\begin{equation*}
    w_{t+1} = \frac{1}{2} \log\left( \frac{1}{\varepsilon_{t+1}} - 1 \right).
\end{equation*}

\noindent \textit{Hint:} Several steps from the proof of Thm. 10.2 are useful in simplifying the derivative of the above.

\end{document}
