\documentclass{article}[12pt]

% useful packages
\usepackage{titlesec}
\usepackage{fullpage}
\usepackage{amsmath,amssymb,amsthm,amsfonts}
\usepackage{graphicx}
\usepackage{enumerate}
\usepackage{algorithm,algorithmic}
\usepackage{xcolor}
\usepackage{bbm}
\usepackage{url}
\usepackage{hyperref}
\usepackage{caption,subcaption}

% theorem type environments
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{lemma}{Lemma}
\newtheorem{cor}{Corollary}
\newtheorem{defn}{Definition}
\newtheorem{assump}{Assumption}
\newtheorem{example}{Example}
\newtheorem{conjecture}{Conjecture}

% frequently used symbols
\newcommand{\bE}{\mathbb{E}}
\newcommand{\bP}{\mathbb{P}}
\newcommand{\bQ}{\mathbb{Q}}
\newcommand{\bR}{\mathbb{R}}
\newcommand{\bS}{\mathbb{S}}
\newcommand{\bN}{\mathbb{N}}
\newcommand{\bZ}{\mathbb{Z}}
\newcommand{\sC}{{\mathcal C}} 
\newcommand{\sD}{{\mathcal D}} 
\newcommand{\sE}{{\mathcal E}} 
\newcommand{\sF}{{\mathcal F}} 
\newcommand{\sL}{{\mathcal L}} 
\newcommand{\sH}{{\mathcal H}} 
\newcommand{\sN}{{\mathcal N}} 
\newcommand{\sO}{{\mathcal O}} 
\newcommand{\sP}{{\mathcal P}} 
\newcommand{\sR}{{\mathcal R}} 
\newcommand{\sS}{{\mathcal S}}
\newcommand{\sU}{{\mathcal U}} 
\newcommand{\sX}{{\mathcal X}} 
\newcommand{\sY}{{\mathcal Y}} 
\newcommand{\sZ}{{\mathcal Z}}

% operators
\newcommand{\sign}{\mathop{\mathrm{sign}}}
\newcommand{\supp}{\mathop{\mathrm{supp}}} % support
\newcommand{\argmin}{\operatornamewithlimits{arg\ min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\ max}}
\newcommand{\dist}{\operatorname{dist}}
\newcommand{\tr}{\text{tr}}
\newcommand{\vecop}{\text{vec}}
\newcommand{\st}{\operatorname{s.t.}}
\newcommand{\cut}{\setminus}
\newcommand{\ind}[1]{\mathbbm{1}\left\{#1\right\}} 
\newcommand{\given}{\ | \ }

% grouping operators
\newcommand{\brac}[1]{\left[#1\right]}
\newcommand{\set}[1]{\left\{#1\right\}}
\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}
\newcommand{\paren}[1]{\left(#1\right)}
\newcommand{\norm}[1]{\left\|#1\right\|}
\newcommand{\ip}[2]{\left\langle #1,#2 \right\rangle}

% code commands
\newcommand{\matlab}{\textsc{Matlab }}
\newcommand{\python}{\textsc{Python }}

% header command
\newcommand{\homework}[3]{
    \pagestyle{myheadings}
    \thispagestyle{plain}
    \newpage
    \setcounter{page}{1}
    \setlength{\headsep}{10mm}
    \noindent
    \begin{center}
    \framebox{
        \vbox{\vspace{2mm}
            \hbox to 6.28in { {\bf EE 516: Mathematical Foundations of Machine Learning
            \hfill Winter \the\year} }
        \vspace{4mm}
        \hbox to 6.28in { {\Large \hfill Homework #1 \hfill} }
        \vspace{2mm}
        \hbox to 6.28in { \Large \hfill Due: #2, 11:59PM PT \hfill }
        \vspace{2mm}
        \hbox to 6.28in { {\it Student Name: #3} \hfill {\it Instructor Name: John Lipor}}
        \vspace{2mm}}
   }
   \end{center}
   \markboth{Homework #1}{Homework #1}
   \vspace*{4mm}
}

% For problem titles
\titleformat{\section}{\normalfont\bf}{Problem \thesection}{1em}{}

\begin{document}

% PUT YOUR NAME BELOW WHERE I HAVE \X
\homework{2}{January 27, 2023}{\X}
% PUT YOUR NAME BELOW WHERE I HAVE \X

\section{\normalfont{(5 pts)}}

Describe how the eigenvalues and eigenvectors of $B = A - 10I$ are related to the eigenvalues and eigenvectors of $A$.

\section{\normalfont{(6 pts)}}

Let $v_{1},v_{2},\dots,v_{n}$ be orthonormal vectors in $\bR^{n}$. Show that $Av_{1},Av_{2},\dots,Av_{n}$ are also orthonormal if and only if $A \in \bR^{n \times n}$ is an orthogonal matrix.

\emph{Note:} Proving a statement of the form ``A if and only if B'' is equivalent to showing that B implies A (if) \emph{and} A implies B (only if).

\section{\normalfont{(5 pts)}}

Let $A \in \bR^{n \times n} = U\Sigma V^{T}$ with rank $r = n$. Express $A^{-1}$ in terms of the SVD of $A$.

\section{\normalfont{(8 pts)}}

Let $B = T^{-1}AT$ for an invertible matrix $T$. Determine the relationship between the eigenvalues and eigenvectors of $B$ and those of $A$. Explain. The matrices $A$ and $B$, when thus related, are said to be ``similar''.

\emph{Note:} Be sure to follow the convention that eigenvectors are normalized to have unit norm.

\section{\normalfont{(5 pts)}}

Let $X \in \bR^{m \times n}$. Using the SVD only, show that if $X^{T}X = 0$, then $X = 0$.

\section{\normalfont{(8 pts)}}

The Frobenius norm of $A \in \bR^{m \times n}$ is defined as $\norm{A}_{F} = \sqrt{\sum_{i,j} \abs{a_{ij}}^{2}}$. Express the Frobenius norm of $A$ in terms of its singular values. Hint: How can $\norm{A}_{F}$ be computed from the matrix $A^{T}A$?


\newpage
\section{\normalfont{(6 pts, 2 pts, 2 pts)}}

Let $A \in \bR^{m \times n}$ with singular values $\sigma_{1} \geq \sigma_{2} \geq \dots \geq \sigma_{\min(m,n)}$. Recall that the ``operator norm'' of a matrix $\norm{A}_{2} = \sigma_{1}$. There are many other interesting and useful matrix norms, and we will continue to see these throughout the course. Bounding these norms in terms of each other is an important problem.

\begin{enumerate}[(a)]
    \item Show that
        \begin{equation*}
            \sigma_{1} \leq \norm{A}_{F} \leq \sqrt{\min(m,n)} \sigma_{1}.
        \end{equation*}
    \item If the Frobenius norm is small, what does this tell us about the operator norm?
    \item If the operator norm is small, what does this tell us about the Frobenius norm?
\end{enumerate}

\newpage
\section{\normalfont{(6 pts, 2 pts, 2 pts)}}

This problem develops a tool that will be used in later homework assignments for an application called \emph{photometric stereo}. To approximate the derivatives of a function $f(x)$ that is sampled on a grid $x_{1},\dots,x_{n}$ where $x_{i+1} = x_{i} + \delta$, a typical \emph{finite difference} approach is:
\begin{equation*}
    \frac{\partial f(x)}{\partial x}\Bigr|_{x = x_{i}} \approx \frac{f(x_{i+1}) - f(x_{i})}{\delta}.
\end{equation*}
When the sample spacing is $\delta = 1$, this approximation simplifies to
\begin{equation*}
    f'(x_{i}) := \frac{\partial f(x)}{\partial x}\Bigr|_{x = x_{i}} \approx f(x_{i+1}) - f(x_{i}).
\end{equation*}
We can express this relation for all $x_{i}$ samples via the matrix-vector product
\begin{equation*}
    \begin{bmatrix} f'(x_{1}) \\ f'(x_{2}) \\ \vdots \\ f'(x_{n}) \end{bmatrix} \approx D_{n} \begin{bmatrix} f(x_{1}) \\ f(x_{2}) \\ \vdots \\ f(x_{n}) \end{bmatrix},
\end{equation*}
where $D_{n}$ is the so-called \emph{first-difference} matrix
\begin{equation*}
    D_{n} = \begin{bmatrix} -1 & 1 & & & \\ 
                            & -1 & 1 \\
                            & & \ddots & \ddots \\
                            & & & -1 & 1 \\
                            1 & & & & -1
                        \end{bmatrix}.
\end{equation*}
Here we choose to set $D_{n}(n,1) = 1$, which corresponds to the (perhaps unexpected) approximation $f'(x_{n}) \approx f(x_{1}) - f(x_{n})$. This choice is called a periodic boundary condition because essentially we are assuming that the domain wraps around. We make this assumption because the resulting $D_{n}$ is a circulant matrix, so its eigenvectors can be computed in closed form!

The goal of this problem is for you to derive and implement the analog of $D_{n}$ for two-dimensional differentiation. Let $f(x,y)$ be a function of two variables. We can approximate its partial derivatives using finite differences as follows:
\begin{equation}
    \frac{\partial f(x,y)}{\partial x} \approx \frac{f(x+1,y) - f(x,y)}{(x+1)-x} = f(x+1,y) - f(x,y)
    \label{eq:diffx}
\end{equation}
\begin{equation}
    \frac{\partial f(x,y)}{\partial y} \approx \frac{f(x,y+1) - f(x,y)}{(y+1)-y} = f(x,y+1) - f(x,y)
    \label{eq:diffy}
\end{equation}
To simplify notation, define the $m \times n$ matrices \texttt{FXY}, \texttt{DFDX}, and \texttt{DFDY} having elements as follows:
\begin{equation*}
    \texttt{FXY}(i,j) = f(i,j)
\end{equation*}
\begin{equation*}
    \texttt{DFDX}(i,j) = \frac{\partial f(i,j)}{\partial x}
\end{equation*}
\begin{equation*}
    \texttt{DFDY}(i,j) = \frac{\partial f(i,j)}{\partial y}.
\end{equation*}
The $x$ coordinate is along the column of \texttt{FXY} and the $y$ coordinate is along the row of \texttt{FXY}, so we can think in terms of $\texttt{FXY}(x,y)$. Define the corresponding $mn \times 1$ vectors \texttt{fxy}, \texttt{dfdx}, and \texttt{dfdy} to be the vectorized versions obtianed using the $\vecop(\cdot)$ operation (HW1, Problem 9). With this notation, we can succinctly express equations \eqref{eq:diffx} and \eqref{eq:diffy} as
\begin{equation*}
    \begin{bmatrix} \texttt{dfdx} \\ \texttt{dfdy} \end{bmatrix} = A \; \texttt{fxy},
\end{equation*}
where $A$ is a $2mn \times mn$ matrix.
\begin{enumerate}[(a)]
    \item Find an expression for $A$ in terms of the first difference matrices $D_{n}$ and $D_{m}$, appropriately sized identity matrices, and appropriate Kronecker products of these matrices. Use periodic boundary conditions.
    \item Once you have determined $A$, write the function \texttt{first\_diffs\_2d\_matrix} that takes as input the dimensions $m$ and $n$ of \texttt{FXY} and returns the appropriate $A$ matrix, stored in \textbf{sparse} format, since the resulting matrix will be prohibively large to store as a full double-precision matrix. \python users should make use of the \texttt{scipy.sparse} library. 
    \item In this problem your function is designed to compute finite-difference approximations to derivatives along $x$ and $y$. If you create an $m \times n$ array \texttt{X} that is a picture of a disk, then the finite derivatives will be mostly zero except near the edges of the disk. The code for generating such a disk is on the course website. Test your code using your function and turn in plots of the resulting images, which should match those below. Note that the \python \texttt{reshape} requires setting \texttt{order='F'}.
\end{enumerate}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{./fxy.eps}
    \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{./dfdx.eps}
    \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=\textwidth]{./dfdy.eps}
    \end{subfigure}
\end{figure}

\section{Reflection \normalfont{(5 pts)}}

Read Chapter 1 of \textit{Solving Mathematical Problems} by Terence Tao, available \href{http://www.math.ucla.edu/~tao/preprints/problem.ps}{here}, and answer the following questions. 
%If you took EE 520 with me last quarter, you must provide new answers for the first two bullets but may reuse your answer for the third bullet.
\begin{itemize}
    \item Among the problem solving strategies discussed, have you used any (whether implicitly or explicitly) in the course so far?
    \item Among the problem solving strategies discussed, which do you think would be helpful to use going forward?
    \item Comment on whether you think this article will be helpful to further problem solving in the course and beyond.
\end{itemize}


\end{document}
