\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}
\newcommand{\todo}[1]{\textcolor{red}{#1}}

% header command
\newcommand{\project}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 519: Deep Learning Theory \& Fundamentals
            \hfill Spring \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Mini Project #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Mini Project #1}{Mini Project #1}
   \vspace*{4mm}
}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\project{1}{April 28, \the\year}{\X}
% PUT YOUR NAME ABOVE WHERE I HAVE \X

\section*{Introduction}

Most homework problems encountered in machine learning courses are well-formulated problems on nicely curated datasets. However, real world problems are never this nice. First, the actual task at hand is typically poorly defined (e.g., ``here's my data, tell me something from it''). Second, even when a task is clearly defined, datasets contain features of different types (binary, real-valued, categorical, location), missing data, and outliers. One aim of these mini projects is to acquaint
you with the task of dealing with real problems.

As stated in class, a problem of interest (at least to the professor) is that of using historical geophysical data to predict geothermal energy favorability, i.e., which locations in the U.S. are amenable to producing electricity by using the earth's heat \cite{eere2022geothermal}. 
For this mini project, you will work with data from the United States Geological Survey (USGS) and the INGENIOUS project \cite{great2022ingenious} to predict a surrogate for geothermal energy favorability. While the geothermal favorability problem is naturally modeled as a classification problem, there are very few known high-favorability locations in the Western U.S., and the resulting imbalance makes it very difficult to train reliable classifiers (see Dr. Lipor's work
\cite{mordensky2023when, mordensky2022what}). For this project, we will instead consider predicting a surrogate for geothermal favorability referred to as the \textit{heat flow residual}. One key indicator of geothermal favorability is when the measured heat flow at a location is much higher than the expected value. Since heat flow is relatively easy to measure, high-quality heat flow estimates have been produced in recent years. As a result, we can compare the actual well measurements to the
predicted heat flow to obtain the ``label'' at these sites corresponding to the difference between the measured and estimated heat flow. We refer to this quantity as the heat flow residual.

You are provided with the heat flow residuals at each point in the training set. While the problem may be viewed as one of pure regression, in practice, we wish to categorize heat flow residuals into four categories. For a residual $r_{i}$, we define the categories
\begin{enumerate}
    \item low: $r_{i} \leq 25$
    \item transition: $r_{i} \in (25, 50]$
    \item high: $r_{i} \in (50, 200]$
    \item very high: $r_{i} > 200$
\end{enumerate}
The resulting labels 1-4 are \textit{ordinal labels}. In contrast to pure classification, for ordinal data, there is a notion of distance between classes; e.g., predicting ``very high'' for a true class ``low'' is worse than predicting ``very high'' for a true class ``high.'' Due to the ordinal nature of our problem, your final predictor will be evaluated using the following loss
\begin{equation}
    \ell(y, \hat{y}) = \abs{y - \hat{y}},
    \label{eq:ordinalLoss}
\end{equation}
where $y, \hat{y} \in \set{1,\cdots,4}$.

Your task is to train an accurate neural network-based predictor of heat flow residual that can be used to predict other locations likely to have a high residual, and therefore likely to have high geothermal favorability. You are provided with a training dataset consisting of 3112 examples, each with 28 features, as well as the corresponding heat flow residual. You will also be provided with the features only of a test dataset. As part of your project, you will
submit your predictions on the test set, and I will return your score evaluted using \eqref{eq:ordinalLoss}.\footnote{The top scoring project will receive 500 Lipor points.}

%\vspace{1em}
%\noindent \textbf{Notes:} 
\subsection*{Notes}
\begin{enumerate}
    \item Your network must be coded and trained using PyTorch and D2L (not Keras or other similar packages).
    \item You may make full use of StackExchange, Medium, or any other online resource you can find.
    \item You may copy code from the internet, but you must \textbf{understand} what the code is doing and \textbf{cite} the source you copied it from.
    \item You may \textbf{not} copy code from other students in this class.
\end{enumerate}

\subsection*{The Dataset}

The training dataset is included in \texttt{heatflow\_resid\_train.csv} and the test dataset is in \\\texttt{heatflow\_resid\_test.csv}. The target (label) that we are trying to predict is \texttt{hfqc\_resid}.

The file \texttt{featureNames.xlsx} contains a brief description of each feature used.
Of the 53 features included, it is very likely that many are not useful for prediction. In our work on this problem, we have utilized the following 16 features:

\texttt{['hf22\_tgwt','cond\_sur','cond\_lcr','cond\_man','cond\_mcr','cond\_uma','geop\_mag', \\ 
        \phantom{x} \quad 'geop\_grv','geop\_dtb','eqi200n5','eqd200n5','geod\_shr', 'geod\_dil','geod\_2nd',\\
        \phantom{x} \quad 'faultAl','vent\_ing']}

You may choose to utilize only these features, but your EDA (see below) should explore all available features. Selecting which features to drop based on EDA and preliminary analysis will result in a higher score for the ``EDA'' and ``Approach'' categories (see Grading below).

\section*{Requirements}

You must create a report in either \LaTeX or an approved type setting language (\textbf{not} Microsoft Word) that contains the following sections:

\begin{enumerate}
    \item Problem description 
        \begin{itemize}
            \item Explain what data you have to work with, what algorithms you will use, what your goal is, and why anyone should care.
        \end{itemize}
    \item Exploratory data analysis (EDA)
        \begin{itemize}
            \item This dataset likely contains many features that will hurt prediction. Further, the data types are heterogeneous, and so some sort of preprocessing will likely be required. The first evaluation of a dataset is known as EDA and is an essential part of a data science pipeline. You must perform an EDA on this dataset and explain your decisions.
            \item There are many excellent tutorials on EDA online. Cite whichever ones you found helpful.
            \item Be sure to make any significant findings stand out and try to keep this concise.
        \end{itemize}
    \clearpage
    \item Challenges
        \begin{itemize}
            \item What were the challenges you encountered when applying machine learning to this dataset?
            \item Did these challenges mainly result from the data? From results? Installing libraries to perform preprocessing?
        \end{itemize}
    \item Approach
        \begin{itemize}
            \item Provide a thorough but concise description of your approach, including (but not limited to) your approach to wrangling, preprocessing, and improving the performance of your predictor.
            \item To avoid overfitting, you must use some form of validation (either a single validation set or cross validation) to tune your hyperparameters. Describe how you went about this and why.
            \item A summary paragraph at the beginning of this section could be helpful.
        \end{itemize}
    \item Evaluation and summary
        \begin{itemize}
            \item Make use of any meaningful metrics (e.g., the loss \eqref{eq:ordinalLoss}, regression MSE, classification error), but only include metrics that tell you something different.
            \item Consider diving into the data to see if you can determine any trends regarding which points were misclassified.
            \item Likewise consider if there were any features that were particularly important or unimportant.
            \item Summarize your solution, describing what worked, what didn't, what the main limitations are (e.g., if nonlinear classifiers would improve results), and any general conclusions about the dataset.
        \end{itemize}
    \item What I learned
        \begin{itemize}
            \item Describe the main skills/tools that you learned and used for this project and how you learned them.
        \end{itemize}
\end{enumerate}

\section*{Grading}

The grading breakdown is below. Each section will be graded according to technical correctness, effort, and creativity. Note that clarity of writing is a major component. You should put yourself in the place of writing for a boss or senior in a workplace. If your writing is terrible, they will assume your work is terrible.
\begin{table}[h!]
\centering
\normalsize % The size of the table text can be changed depending on content. Remove if desired.
\begin{tabular}{ | c | c | }
\hline
\textbf{Item} & \textbf{Percentage} \\
\hline
Description & 5\% \\
\hline
EDA & 20\% \\
\hline
Challenges & 10\% \\
\hline
Approach & 30\% \\
\hline
Evaluation & 20\% \\
\hline
What I learned & 5\% \\
\hline
Clarity/conciseness of written communication & 10\% \\
\hline
\end{tabular}
\end{table}

\clearpage
\section*{Advice}

\begin{itemize}
    \item To start, you may approach this problem as either a pure regression or pure classification problem. This will simplify the process, allowing you to think more carefully about minimizing the ordinal loss later on.
    \item In our work on this problem, we found the simple approach of performing regression using XGBoost worked well, but only when we trained the predictor using residuals in the ``low'' and ``transition'' categories. You may consider a network architecture that has shared layers, followed by separate feature extraction and output layers for categories 1-2 and categories 3-4. More on this in class.
    \item You may make use of linear dimensionality reduction methods such as PCA or dropping features when classifying.
    \item If helpful, you may make use of nonlinear embedding/dimensionality reduction methods such as t-SNE or UMAP for visualization.
    \item You may find articles on ``data wrangling'' and ``exploratory data analysis'' useful. Two examples are \cite{jakki2022simple,ray2016comprehensive}. Please feel free to share these in the \texttt{\#interesting-reading} channel.
    \item A great example of really thinking through the results of an algorithm is given in \cite{ullah2019explainability}.
\end{itemize}

\bibliographystyle{IEEEtran}
\bibliography{bibliography}

\end{document}
