The Exponential Growth Trend of Artificial Intelligence (AI) brings with it a number of key challenges related to the tremendous scale of computation required to drive large language models. High Performance Computing today achieves performance in part by using heterogeneous platforms comprising a combination of CPUs and specialized accelerators. This approach has been widely adopted because it can yield a very high rate of operations per watt of power. The most common accelerator is the Graphics Processing Unit (GPU). By 2024, one company alone, Meta, operated over 340,000 NVIDIA GPUs at its data centers worldwide. Accelerators offer the potential for performance gains and energy savings, but it is challenging to achieve all or most of that potential.
In this class we will cover the basics of GPU computing, including a look at the underlying architecture, runtime system, performance, and programming approaches. Our deeper dive will focus specifically on the use of NVIDIA Graphics Processing Units (GPUs) and our hands on programming will use C/CUDA running on NVIDIA GPUs. Our scope will include a range of scientific computing and AI applications.
Ph.D. students are welcome, please email the instructor before the first class to discuss your additional requirements.
Required Textbook
Programming Massively Parallel Processors: A Hands-on Approach 5th Edition by Wen-mei W. Hwu, David B. Kirk and Izzat El Hajj
ISBN-13: 978-0443439001
ISBN-10: 0443439001
Additional required readings will be from freely available notes, papers and articles.
Workload: Reading assignments, lectures, practice exercises, homeworks, group project
Assessment (Grading): Homeworks (30%) Project (30%) Exams (40%)