Data Engineering Concepts

COMP63301 Data Engineering Concepts 15 Semester 1 Postgraduate Taught Level 6 Sandra Sampaio Unit coordinator Department of Computer Science 1 FHEQ level (Framework for Higher Education Qualifications) ' Masters/Integrated Masters P4 ' European Credit Transfer & Accumulation System Rating : 7.5 Data Engineering plays a crucial role in enabling organisations to leverage big data for insightful analytics, driving business strategies and innovations. The field has evolved significantly from its early days of simple database management to encompass advanced technologies for large-scale data processing and analytics. This evolution reflects the growing complexity and volume of data, as well as the need for robust data infrastructures to support AI systems. As AI continues to dominate various industries, the relevance of data engineering in the job market has surged. Data engineers are essential for designing, building, and maintaining the data pipelines that AI systems depend on. Consequently, there is increased demand for professionals with analytical thinking, innovation, and problem-solving skills in data engineering, which represent the objectives of this course unit. Data engineering tends to involve a lifecycle, in which typical phases include data acquisition, profiling, cleaning, integration, modelling, and usage. This unit introduces the student to relevant stages of the data engineering lifecycle and related concepts, tasks and techniques. It deepens selected aspects of this lifecycle, e.g., transformation, modelling and visualisation, and addresses cross-cutting topics such as security, trust, and robustness. We will investigate pain points, trade-offs, limitations and evaluation criteria that can inform the development of data engineering pipelines in practice. This unit aims to equip students with a comprehensive understanding of the foundational concepts that underpin data engineering, alongside practical experience in applying these concepts to real-world scenarios. Data engineering encompasses the processes and mechanisms that enable value extraction from data, and this unit explores the lifecycle that governs these activities, from data integration and cleaning to transformation and visualisation. Through a combination of theoretical exploration and hands-on practice, students will gain insight into the challenges, trade-offs, and technologies that define the field. ILO 1: Discuss trade-offs between various design options.ILO 2: Critically analyse data engineering technologies.ILO 3: Apply selected DE techniques for data integration, cleaning, transformation and visualisation, ensuring data quality for the purpose of data analysis.ILO 4: Identify relevant data properties, understanding the shape of data and its representation of the world.ILO 5: Explain the Data Engineering (DE) lifecycle, related concepts, challenges and research questions. Analytical skills Innovation/creativity Problem solving Research In the Data Engineering Concepts course, you will explore the foundational principles and practical techniques that underpin modern data engineering. Key topics include:<ul><li>Data Profiling and Quality: Learn to assess and improve data quality using industry-standard dimensions and metrics.</li><li>Data Preparation and Transformation: Gain hands-on experience in cleaning, structuring, and transforming raw data for analysis and modelling.</li><li>Data Integration and Storage: Understand how to model, store, and query data using relational and non-relational databases.</li><li>Data Analysis and Visualisation: Discover how to extract insights and communicate findings effectively through visualisation techniques.</li><li>Big Data and Streaming: Tackle the challenges of working with large-scale and real-time data, including orchestration and secure data movement.</li></ul>Tools and TechnologiesYou’ll work with a range of technologies that are essential to today’s data engineering landscape, including:<ul><li>Python for data manipulation and transformation</li><li>SQL for querying and managing structured data</li><li>MongoDB MQL for handling semi-structured data</li><li>Data visualisation tools to present insights clearly and effectively</li></ul>By the end of the course, you’ll be equipped with both the theoretical understanding and practical skills needed to design and implement robust data engineering solutions. Whether you are preparing data for machine learning models or building pipelines for real-time analytics, this course will empower you to turn raw data into actionable intelligence. Asynchronous learning material will be made available in the form of videos and directed reading, as well as formative and normative exercises delivered via the VLE. Synchronous activities include in-person workshops, focusing on discussion of examples, clarifications and Q&A. Labs allow for exploration of coursework, what is expected and how to go about doing it, and for receiving feedback on formative coursework. 1 Written exam 70% 9 Set exercise 30% Cohort level feedback after marking.Individual feedback on request in lab.Individual feedback via rubric.Cohort feedback in workshops.Auto-graded quizzes providing immediate feedback. Data Management, including traditional (e.g., CSV, relational, etc.) and non-traditional (JSON, text, noSQL, etc.) data types and associated data management technologies.Programming in Python.SQL and the Relational Algebra.Data Analytics. Y Fundamentals of Data Engineering: Plan and Build Robust Data Systems    Joe Reis,Matt Housley    O'Reilly Media    2022    ISBN: <a href="https://www.librarysearch.manchester.ac.uk/discovery/search?query=isbn,contains,978&search_scope=MyInst_and_CI&sortby=rank&vid=44MAN_INST:MU_NUI&lang=en&mode=advanced&offset=0" target="_blank">978</a> Eleven quick tips for data cleaning and feature engineering    Chicco, Davide ; Oneto, Luca ; Tavazzi, Erica    PLoS computational biology    2022    DOI: <a href="https://doi.org/10.1371/journal.pcbi.1010718" target="_blank">10.1371/journal.pcbi.1010718</a> Python for Data Analysis     Wes McKinney   —    2022    ISBN: <a href="https://www.librarysearch.manchester.ac.uk/discovery/search?query=isbn,contains,9781098104009&search_scope=MyInst_and_CI&sortby=rank&vid=44MAN_INST:MU_NUI&lang=en&mode=advanced&offset=0" target="_blank">9781098104009</a> Python for data analysis : data wrangling with Pandas, NumPy, and Jupyter     McKinney, Wes,    O'Reilly    2022    ISBN: <a href="https://www.librarysearch.manchester.ac.uk/discovery/search?query=isbn,contains,9781098104030&search_scope=MyInst_and_CI&sortby=rank&vid=44MAN_INST:MU_NUI&lang=en&mode=advanced&offset=0" target="_blank">9781098104030</a> The Turing Way    The Turing Way Community        Snakemake | Snakemake 9.8.1 documentation        Web site: <a href="https://snakemake.readthedocs.io/en/stable/" target="_blank">snakemake.readthedocs.io </a>   Sustainable data analysis with Snakemake [version 2; peer review: 2 approved]    Mölder, Felix ; Jablonski, Kim Philipp ; Letcher, Brice ; Hall, Michael B ; Tomkins-Tinch, Christopher H ; Sochat, Vanessa ; Forster, Jan ; Lee, Soohyun ; Twardziok, Sven O ; Kanitz, Alexander ; Wilm, Andreas ; Holtgrewe, Manuel ; Rahmann, Sven ; Nahnsen, Sven ; Köster, Johannes    F1000 research    2021    DOI: <a href="https://doi.org/10.12688/f1000research.29032.2" target="_blank">10.12688/f1000research.29032.2</a> Modern data engineering with Apache Spark : a hands-on guide for building mission-critical streaming applications     Haines, Scott,    Apress    2022    ISBN: <a href="https://www.librarysearch.manchester.ac.uk/discovery/search?query=isbn,contains,9781484274521&search_scope=MyInst_and_CI&sortby=rank&vid=44MAN_INST:MU_NUI&lang=en&mode=advanced&offset=0" target="_blank">9781484274521</a> Assessment written exam 2 Practical classes & workshops 40 Work based learning 40 0 68 Videos (~5 Hours)Formative Quizzes (~2 Hours)Formative Coursework (~10 Hours)Assessed Coursework (~10 Hours)Assessed Quizzes (~3 Hours)