\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}
\usepackage{caption,subcaption}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 516: Mathematical Foundations of Machine Learning
            \hfill Winter \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{8}{March 12, 2023}{\X}
% PUT YOUR NAME BELOW WHERE I HAVE \X

\section{\normalfont{(5 pts, 2 pts)}}

In Homework 6, problem 3, you implemented a majorize-minimize algorithm for solving the mean absolute error minimization problem. The resulting algorithm fit the iteratively reweighted least squares (IRLS) formulation. In this problem, you will implement the IRLS algorithm directly for minimizing the $p$-norm of the residuals, i.e., you will attempt to solve the problem
\begin{equation*}
    \min_{x \in \bR^{D}} \norm{Ax-b}_{p}^{p}.
\end{equation*}

\begin{enumerate}[(a)]
    \item Implement the IRLS algorithm for $\ell_{p}$ regression described in Algorithm 1 of the IRLS notes by completing the \texttt{lpIRLS} function. Note the stopping criterion (for yourself). \textbf{Turn in} your code.
    \item Test your algorithm by running \texttt{lpIRLS} for $p \in \set{0.1, 0.5, 0.7, 1}$ in \texttt{prob1}. \textbf{Turn in} the resulting plot and state which value of $p$ performed best. Why do you think this is the case?
\end{enumerate}

\section{\normalfont{(5 pts, 3 pts)}}

In this problem, you will implement an IRLS algorithm for solving sparse regression problems. Recall that in the IRLS lecture notes we developed a method of (approximately) solving
\begin{eqnarray*}
    \min_{x \in \bR^{D}} && \norm{x}_{p} \\
    \text{subject to} && Ax = b,
\end{eqnarray*}
where we typically consider highly under-determined systems so that the equality constraint can be satisfied. When $p = 1$, this optimization problem is referred to as \emph{basis pursuit}.

\begin{enumerate}[(a)]
    \item Implement the IRLS algorithm for sparse regression described in Algorithm 2 of the IRLS notes by completing the \texttt{bpIRLS} function. Note the stopping criterion (for yourself). \textbf{Turn in} your code.
    \item One important way of visualizing algorithm performance is through the use of a \emph{heatmap}, which shows the error as a function of two algorithm parameters in the form of an image. The error of interest in this case is
        \begin{equation*}
            \frac{\norm{x_{\rm true} - \hat{x}}_{2}}{\norm{x_{\rm true}}_{2}},
        \end{equation*}
        where $\hat{x}$ is the output of your \texttt{bpIRLS} algorithm. The parameters of interest are the \emph{sparsity} (number of nonzero elements in $x_{\rm true}$) and the \emph{number of measurements} $N$ (where $A \in \bR^{N \times D})$. To test your sparse regression algorithm, display a heatmap by completing the \texttt{srHeatmap} script. Let the horizontal axis be sparsity$/D$ and the vertical axis be $N/D$. \textbf{Turn in} heatmaps for $p \in \set{0.1, 1, 2}$ and a description of when the
        recovery error is low (in terms of sparsity and $N$) for each value of $p$.
            \\
            \textbf{Note} that it is common to divide both sparsity and number of measurements by the problem size $D$, which is a form of normalization. If time permits (optional), change the value of $D$ and inspect the resulting plots. You will notice that they look exactly the same.
\end{enumerate}

\section{\normalfont{(5 pts, 3 pts)}}

In this problem, you will implement an ADMM algorithm for solving the Lasso, which is an alternative formulation of the sparse regression problem. Recall that in the ADMM lecture notes we developed a method of solving
\begin{equation*}
    \min_{x \in \bR^{D}} \frac{1}{2} \norm{Ax - b}_{2}^{2} + \lambda \norm{x}_{1}.
\end{equation*}
Using the \emph{variable splitting} technique, we derived the update equations for ADMM to solve this problem.

\begin{enumerate}[(a)]
    \item Implement the ADMM algorithm for solving the Lasso described in the ADMM notes by completing the \texttt{lassoADMM} function. Note the stopping criterion (for yourself). \textbf{Turn in} your code.
    \item An important part of using algorithms like ADMM is choosing the tuning parameters. For ADMM, it is common to choose $\rho$ as a function of $\lambda$ as I have done in the given file. Use the script \texttt{srHeatmap} to try several values of $\lambda$ and note any trends that you see. \textbf{Turn in:} 
        \begin{itemize}
            \item the heatmap for the best value of $\lambda$ that you found with your choice of $\lambda$ specified in either the title or the figure caption
            \item a sentence or two describing how the results change when $\lambda$ is both smaller and larger than your selected value
            \item a sentence comparing the performance of ADMM and IRLS for sparse regression.
        \end{itemize}
\end{enumerate}

\section{\normalfont{(10 pts, 5 pts, 2 pts)}}

As mentioned in class, sparse regression can be used as a tool for \href{https://en.wikipedia.org/wiki/Feature_selection}{feature selection} in machine learning. The idea is to start with a large number of features, some of which may not be useful for prediction, and automatically determine which are most valuable for predicting the given labels. Follow the \texttt{prob4} script to load the
\href{https://archive.ics.uci.edu/ml/datasets/Residential+Building+Data+Set}{Residential Building dataset} from the UCI Machine Learning Repository. This dataset includes a variety of features that can be used to predict the sale price of the corresponding building. You can open the xlsx file to view the actual features. You will use your \texttt{lassoADMM} function to perform feature selection on this dataset.

\begin{enumerate}[(a)]
    \item Split the data into training, validation, and test sets. Use the first 250 examples for training, the next 50 for validation, and the last 72 for testing. Run your \texttt{lassoADMM} script for 20 values of $\lambda$ spaced linearly between 1,000-10,000 and plot the squared error of your learned regression function $\hat{w}$ on both the training and validation sets. The validation error can be used to select the ``best'' choice for $\lambda$ in terms of generalizing to unseen data.
        \textbf{Turn in} your plots, the best $\lambda$ value, and a comment on the what you notice about the training and validation error plots.
    \item Now create a stem plot of your learned regressor $\hat{w}$ for the choice of $\lambda$ you found above. Manually examine the features corresponding to the elements of $\hat{w}$ with largest magnitude and list the top 5 here.
    \item Report the error on the test dataset for your chosen value of $\lambda$.
\end{enumerate}

\section{\normalfont{(5 pts)}}

Read the article, ``Who Should Stop Unethical A.I.?'' linked \href{https://www.newyorker.com/tech/annals-of-technology/who-should-stop-unethical-ai}{here} and also available on the \texttt{interesting-reading} channel in pdf format (in case you're paywalled). After reading, share one major ethical issue/concern you learned about from the article and one idea (whether your own or from the article) for mitigating unethical A.I. in the \texttt{interesting-reading} channel on Slack.

\end{document}
