Reinforcement Learning

COMP44202 Reinforcement Learning 15 Semester 2 Undergraduate Level 6 Michele Caprio Unit coordinator Department of Computer Science 1 FHEQ level (Framework for Higher Education Qualifications) ' Masters/Integrated Masters P4 ' European Credit Transfer & Accumulation System Rating : 7.5 Reinforcement Learning (COMP64202) is an advanced artificial intelligence course that explores how machines can learn to make sequential decisions through trial and error interaction with their environment, similar to how humans learn. The course covers both theoretical foundations and cutting-edge applications in areas such as game playing, robotics, and resource management. Reinforcement Learning (RL) is revolutionizing how machines learn to solve complex problems, from mastering games like Go to controlling robots and managing data centers. Unlike traditional machine learning approaches, RL enables systems to learn through trial and error, much like humans do.This course explores the key concepts and methods that make RL powerful, starting with fundamental tabular methods for Markov Decision Processes, before diving into how deep learning helps RL scale to real-world problems. We'll then cover advanced approaches including policy gradients and model-based methods, culminating in the challenges of partial observability and multi-agent systems. Throughout, we'll balance theoretical understanding with practical insights into how these methods are applied in real-world settings. The unit aims to provide an explanation of the key ideas and algorithms of reinforcement learning (RL). This course is meant to provide both an introductory and advanced treatment of reinforcement learning, emphasizing foundations and ideas as well as the latest developments and mathematical proofs. This course aims to make the work accessible to the broadest possible audience in artificial intelligence, control engineering, operations research, psychology and neuroscience.  Knowledge and understanding:1. List the strengths and limitations of modern deep RL approaches.2. Explain the underlying concepts of RL methods and how they differ from each other.3. Derive the objectives and constraints of selected algorithms.  Intellectual skills: 4. Analyze new tasks to decide which algorithms/architectures to apply. 5. Compare and evaluate different RL approaches for specific problem contexts.  Practical skills: 6. Implement selected RL algorithms and architectures. 7. Apply RL methods to solve practical problems.  Transferable skills and personal qualities: 8. Break down complex problems into manageable components. 9. Evaluate technical approaches against practical constraints. 10. Communicate technical concepts and results effectively.   Analytical skills Group/team working Innovation/creativity Project management Problem solving Research Written communication The course covers the following topics:<ul><li>Fundamentals of Markov Decision Processes (MDPs)</li><li>Dynamic Programming methods</li><li>Monte Carlo methods</li><li>Temporal-Difference learning</li><li>Function approximation and Deep Reinforcement Learning</li><li>Policy gradient methods</li><li>Actor-critic algorithms</li><li>Model-based Reinforcement Learning</li><li>Partial observability and multi-agent systems</li><li>Real-world applications and case studies</li></ul> The learning and teaching in this course combines three complementary components. Lectures provide the theoretical foundations and key concepts of reinforcement learning. Tutorial sessions focus on solving theoretical exercises to deepen understanding and develop problem-solving skills. Computer lab sessions offer hands-on experience implementing and experimenting with reinforcement learning algorithms, allowing students to see the practical challenges and opportunities of these methods. 1 Written exam 80% 8 Practical skills assessment 20% Feedback is provided through:<ul><li>Cohort and individual feedback for the practical assignment when marks are returned</li><li>Discussion during tutorial sessions</li><li>Peer feedback during group activities</li><li>Office hours with teaching staff</li></ul> Y An Introduction to Reinforcement Learning, R.S. Sutton and A.G. Barto. MIT Press, 2018.Algorithms for Reinforcement Learning, C. Szepesvari, Morgan and Claypool, 2010. Lectures 20 Practical classes & workshops 25 Tutorials 5 100 This course unit is at Level 6 with 15 credits (7.5 ECTS). The unit is taught by Dr. Wei Pan from the School of Engineering.