Mathematical Topics in Machine Learning

COMP34312 Mathematical Topics in Machine Learning 10 Semester 2 Undergraduate Level 3 Gavin Brown Unit coordinator Department of Computer Science 1 FHEQ level (Framework for Higher Education Qualifications) ' Last part of a Bachelors ' European Credit Transfer & Accumulation System Rating : 5.0 Topic 1: Empirical risk minimization, regularisation; bias/variance theory and the relation to overfitting; probabilistic view: likelihood vs loss, introducing exponential families. Topic 2: Information theory: KL-divergence vs. cross-entropy, mutual information; the view of ML as compression. Topic 3: Optimization theory (calculus). Why GD?  What are convex and non-convex functions? How do gradients inform how we optimize a function? How can we use second order properties? How can we prove whether a method will converge? Topic 4: Dimensionality reduction (matrix algebra). “refine, denoise, and visualise your data”.  Data usually has limited degrees of interest, living on a low-dimensional manifold within a high-dimensional space. This topic will introduce students to matrix-algebra-intensive methods used to learn feature dimensions that can aid in your model fitting process. Examples include PCA, spectral embedding, Fisher discriminant analysis, etc. These allow visualisation, denoising and enhancing the separation of data for classification. Topic 1: Empirical risk minimization, regularisation; bias/variance theory and the relation to overfitting; probabilistic view: likelihood vs loss, introducing exponential families. Topic 2: Information theory: KL-divergence vs. cross-entropy, mutual information; the view of ML as compression. Topic 3: Optimization theory (calculus). Why GD?  What are convex and non-convex functions? How do gradients inform how we optimize a function? How can we use second order properties? How can we prove whether a method will converge? Topic 4: Dimensionality reduction (matrix algebra). “refine, denoise, and visualise your data”.  Data usually has limited degrees of interest, living on a low-dimensional manifold within a high-dimensional space. This topic will introduce students to matrix-algebra-intensive methods used to learn feature dimensions that can aid in your model fitting process. Examples include PCA, spectral embedding, Fisher discriminant analysis, etc. These allow visualisation, denoising and enhancing the separation of data for classification. This course introduces the theoretical foundations of supervised machine learning. We aim at students who want to move beyond just using models, to understanding why they work . This will be a rigorous mathematical and conceptual treatment that may require developing your knowledge of probability theory, matrix algebra, and calculus. You'll learn how we formalize learning problems, interpret performance through the lens of probability theory, and reason about why some models generalize while others do not. We focus on foundational ideas—risk, generalization, capacity, and trade-offs—that underlie all learning systems, regardless of the algorithm used. This could be seen as preparing you to engage with modern research literature, perhaps as an enabler toward a research career, in industry or academia.  ILO 1: Define supervised learning problems using probabilistic assumptions and notation, identifying appropriate loss functions and models for a given scenario.ILO 2: Describe how Rademacher complexity provides a framework for data-dependent generalisation bounds, and compute the bounds for learning scenarios with linear models.ILO 3: Critically assess the limitations of classical learning theory in the context of modern machine learning models (e.g., overparameterized models).ILO 4: Evaluate/critique how model combination affects generalization error, and describe learning algorithms that support this.ILO 5: Describe and analyze the behavior of gradient descent algorithms using formal mathematical properties, such as convexity and Lipschitzness.ILO 6: Derive and compare theoretical frameworks for understanding generalisation error, including: bias–variance, approximation–estimation, and PAC bounds. Week 1 – The Foundations of Supervised Learning  This week lays the groundwork by formalizing supervised learning. We ask: what are we optimizing, over what space, and why? By the end, you will understand learning as the problem of finding a function that performs well—not just on the training data—but on the unknown probability distribution that generated it.Week 2 – The Probabilistic View of Learning  This week shows how common loss functions arise naturally from probabilistic modeling assumptions. By deriving squared and cross-entropy losses from Gaussian and Bernoulli likelihoods, we see why these choices are principled. The connection between negative log-likelihood and KL divergence clarifies what it means for a model to “fit” the data.Week 3 – The Problem with Learning a Model  Here the focus turns to generalization failure and overfitting. The bias–variance decomposition is derived to separate sources of error. Regularization and the double-descent phenomenon illustrate how model capacity influences performance.Week 4 – Combining Models  We explore how ensembling multiple hypotheses can reduce variance and improve generalization. The lecture introduces Bagging, majority-vote theory, and the bias–variance–diversity decomposition. These ideas provide both a theoretical and intuitive foundation for why ensembles work.Week 5 – Understanding the Sources of Risk  This week examines how hypothesis-class complexity drives the trade-off between approximation, estimation, and optimization error. The formal decomposition clarifies why larger models can both help and hinder generalization. Examples link these abstract ideas back to practical model families.Week 6 – How to be Probably Approximately Correct  The course introduces Hoeffding’s inequality and PAC learning as a framework for probabilistic generalization guarantees. We learn to bound the gap between empirical and true risk. The session closes with a discussion on the realism and limits of PAC assumptions in modern ML.Week 7 – Optimization and the Learning Algorithm  Attention shifts from theory about models to the algorithms that find them. We examine empirical risk minimization as an optimization problem and how algorithmic choices affect generalization. Gradient descent is introduced conceptually as the foundation for modern learning.Week 8 – A Formal Introduction to Gradient Descent  This week develops the mathematical underpinnings of gradient descent. We study convergence under convexity, step-size selection, and Lipschitz continuity. The aim is to understand when and why gradient-based optimization actually finds a good solution.Week 9 – Gradient Descent in Overparameterized Linear Regression  The analysis focuses on linear regression with more parameters than data points. We see that gradient descent implicitly selects the minimum-norm solution, introducing the idea of algorithmic bias. This provides a bridge between optimization dynamics and generalization behaviour.Week 10 – A Measure of Capacity: Rademacher Complexity  Here we meet Rademacher complexity as a formal measure of a function class’s ability to fit random noise. They learn how this quantity bounds generalization error in data-dependent ways. The material extends PAC ideas to richer hypothesis spaces.Week 11 – Bounding Rademacher Complexity in Practice  The final week applies the Rademacher framework to simple hypothesis classes such as linear models. Worked examples illustrate how to compute and interpret capacity bounds. The course concludes by connecting these ideas back to the central question of why learning works.  Unit will consist of 4 major topics, delivered in 2-week blocks before Easter.  Each topic will consist of videos to watch, and readings to cover, before the interactive sessions. The interactive sessions will act to reinforce the videos/readings.  Weekly MCQs in class will be used as formative and summative assessments. After Easter, a series of carefully selected classic research papers will be read week by week in groups, introducing students to the methods in how to read/interpret research results.  Presentations of the papers will consolidate the depth of understanding. 0 Other 20% 1 Written exam 80% Poster Presentation Correct answers discussed the following week COMP24112 Machine Learning Pre-Requisite Compulsory COMP24112 is a pre-requisite.COMP24112 is a pre-requisite. Y Selected chapters: Machine Learning, A Probabilistic Perspective by Kevin Murphy;Selected chapters: Probability in Data Science, by Sidney Chan Assessment written exam 2 Lectures 11 Practical classes & workshops 11 0 76