Skip to content

MIT6.5940: TinyML and Efficient Deep Learning Computing

Descriptions

  • Offered by: MIT
  • Prerequisites: Computer architecture, Deep Learning
  • Programming Languages: Python
  • Difficulty: 🌟🌟🌟🌟
  • Class Hour: 50h

This course, taught by MIT Professor Song Han, focuses on efficient machine learning techniques. Students are expected to have a pre-requisite of deep learning basics.

The course is divided into three main sections. The first section covers various key techniques for lightweight neural networks, such as pruning, quantization, distillation, and neural architecture search (NAS). Building on these foundations, the second section introduces efficient optimization techniques tailored to specific application scenarios. These include cutting-edge topics in deep learning, such as inference for large language models, long-context support, post-training acceleration, multimodal large language models, GANs, diffusion models, and so on. The third section focuses on efficient training techniques, such as large-scale distributed parallelism, automatic parallel optimization, gradient compression, and on-device training. Professor Song Han’s lectures are clear and insightful, covering a wide range of topics, with a strong focus on trending areas. Those interested in gaining a foundational understanding of large language models may particularly benefit from the second and third sections.

The course materials and resources are available on the course website. Official lecture videos can be found on YouTube, and both raw and subtitled versions are available on Bilibili. There are five assignments in total: the first three focus on quantization, pruning, and NAS, while the last two involve compression and efficient deployment of large language models. The overall difficulty is relatively manageable, making the assignments an excellent way to solidify core knowledge.

Course Resources

Personal Resources

All the resources and assignments used by @PKUFlyingPig in this course are maintained in PKUFlyingPig/MIT6.5940_TinyML- GitHub.