A Short Course on Optimization for Data Science and Machine Learning Problems

Abstract

Optimization lies at the core of modern data science, providing scalable and principled solutions to high-dimensional problems in statistics, machine learning, and deep learning. The first part of the course introduces the fundamentals of gradient-based optimization and progresses to advanced optimization techniques, emphasizing their theoretical underpinnings and practical implementation. These algorithms will be illustrated through applications in high-dimensional statistics and machine learning, including sparse regression, matrix completion, graphical models, and feed-forward neural networks.

The second part of the course explores recent advances in optimization motivated by emerging challenges in large-scale and distributed learning systems. Topics include: (i) Federated and distributed learning, where decentralized optimization enables efficient model training across multiple devices while preserving data privacy; (ii) Minimax optimization, a powerful framework underlying adversarial learning, robust statistics, and generative modeling; and (iii) Bilevel optimization, which has gained prominence in recent years through its applications in hyperparameter tuning, meta-learning, and reinforcement learning.

Throughout the course, emphasis will be placed on developing both conceptual understanding and technical depth, offering an accessible yet rigorous perspective on the theory and practice of modern optimization methods driving today’s statistical and machine learning research.

Download