Skip to content

CMU 15-442/642: Machine Learning Systems

Course Overview

  • University: CMU
  • Prerequisites: Intro to Computer Systems (e.g., 15-213), Intro to Deep Learning, Basic Math
  • Programming Languages: Python, C++, CUDA (PTX), TIRx
  • Difficulty: 🌟🌟🌟🌟
  • Estimated Hours: 100+ hours

With the explosive growth of Large Language Models (LLMs), efficiently training and deploying these massive models has become one of the core challenges in the AI field today. If you are not satisfied with merely being a "simple API caller" who just pieces together PyTorch functions, but are extremely curious about the underlying mechanisms of cutting-edge inference frameworks like vLLM and SGLang, and want to personally write low-level kernels that squeeze every drop of compute out of modern GPUs, then this course taught by Prof. Tianqi Chen and Prof. Zhihao Jia is an excellent starting point.

The course adopts a holistic, full-stack perspective of "Model, Data, Systems, and Hardware," building a complete knowledge map from bare-metal hardware to high-level distributed frameworks. It starts with the foundational elements of deep learning frameworks like automatic differentiation, then dives deep into hardware acceleration and native-level programming on modern GPUs (such as the latest Blackwell B200 architecture). After unlocking extreme single-card performance, the course extends to distributed training for large models, detailing mechanisms like the ZeRO redundancy optimizer, as well as tensor and pipeline model parallelism. Finally, the course focuses on current LLM inference serving, providing in-depth analyses of core optimization strategies such as Continuous Batching, PagedAttention, and Speculative Decoding, while offering a forward-looking exploration of modern machine learning compiler abstractions and Mega-Kernel systems.

The biggest highlight of this course lies in its highly challenging and industry-aligned programming assignments. There are three assignments in total: 1. Implement an automatic differentiation framework for computational graphs. 2. In the distributed training assignment, you will use MPI and NumPy to implement the communication logic for ZeRO Stage 3 parameter sharding and tensor model parallelism. 3. In the most hardcore GPU kernel assignment, you will use the TIRx DSL targeting the latest NVIDIA Blackwell (B200) architecture to manually write and optimize an FP16 GEMM (Matrix Multiplication) kernel from scratch. You will not only need to handle the 128B Swizzle memory layout to eliminate bank conflicts, but also manually dispatch TMA (Tensor Memory Accelerator) asynchronous load instructions, design Warp Specialization software pipelines, and implement 2-CTA cluster scheduling. Ultimately, you will skyrocket the performance from an initial 0.02 TFLOP/s to 1300+ TFLOP/s, rivaling the highly-optimized official cuBLAS library.

Course Resources

Resource Aggregation

All resources and assignment implementations used by @RisingUppercut in this course are consolidated in RisingUppercut/CMU_15442_2026Spring - GitHub