COMP63301 15 Semester 1 Postgraduate Taught Level 6 Sandra Sampaio Unit coordinator 1 FHEQ level (Framework for Higher Education Qualifications) ' Masters/Integrated Masters P4 ' European Credit Transfer & Accumulation System Rating : 7.5 <p>Data Engineering plays a crucial role in enabling organisations to leverage big data for insightful analytics, driving business strategies and innovations. The field has evolved significantly from its early days of simple database management to encompass advanced technologies for large-scale data processing and analytics. This evolution reflects the growing complexity and volume of data, as well as the need for robust data infrastructures to support AI systems. As AI continues to dominate various industries, the relevance of data engineering in the job market has surged. Data engineers are essential for designing, building, and maintaining the data pipelines that AI systems depend on. Consequently, there is increased demand for professionals with analytical thinking, innovation, and problem-solving skills in data engineering, which represent the objectives of this course unit.</p> <p>Data engineering tends to involve a lifecycle, in which typical phases include data acquisition, profiling, cleaning, integration, modelling, and usage.</p><p><br>This unit introduces the student to relevant stages of the data engineering lifecycle and related concepts, tasks and techniques. It deepens selected aspects of this lifecycle, e.g., transformation, modelling and visualisation, and addresses cross-cutting topics such as security, trust, and robustness. We will investigate pain points, trade-offs, limitations and evaluation criteria that can inform the development of data engineering pipelines in practice.</p> <p>The unit aims to provide students with an understanding of the concepts that underpin data engineering and the experience of applying those concepts. In turn, data engineering provides processes and mechanisms that enable value to be obtained from data. These processes and mechanisms can be considered to give rise to a data engineering lifecycle, and this unit explores the concepts that underpin the different stages in such a lifecycle, which include data transformation and visualisation.</p> <p>1. Explain the Data Engineering (DE) lifecycle, related concepts, challenges and research questions.</p><p>2. Identify relevant data properties, understanding the shape of data and its representation of the world.</p><p>3. Apply selected DE techniques for data integration, cleaning, transformation and visualisation, ensuring data quality for the purpose of data analysis.</p><p>4. Critically analyse data engineering technologies.</p><p>5. Discuss trade-offs between various design options.</p> Analytical skills Innovation/creativity Problem solving Research <ul><li>Data Acquisition and Reduction</li><li>Understanding the shape of data</li><li>Data Modelling and Storage Considering Traditional and Non-Traditional Data Types</li><li>Data Integration</li><li>Data Profiling, Quality and Cleaning</li><li>Data Dissemination and Security</li><li>Data Querying</li><li>Data Analytics through Machine and Deep Learning</li><li>Data Visualisation and Serving</li><li>Data Mutability/Volatility, Robustness and Trust</li></ul> <p><strong>Asynchronous </strong>learning material will be made available in the form of videos and directed reading, as well as formative and normative exercises delivered via the VLE.</p><p><br><strong>Synchronous</strong> activities include in-person workshops, focusing on discussion of examples, clarifications and Q&amp;A. Labs allow for exploration of coursework, what is expected and how to go about doing it, and for receiving feedback on formative coursework.</p> 1 Written exam 70% 8 Practical skills assessment 15% 9 Set exercise 15% <p>Cohort level feedback after marking.</p><p>Individual feedback on request in lab.</p><p>Individual feedback via rubric.</p><p>Cohort feedback in workshops.</p><p>Auto-graded quizzes providing immediate feedback.</p> <p>Data Management, including traditional (e.g., CSV, relational, etc.) and non-traditional (JSON, text, noSQL, etc.) data types and associated data management technologies.</p><p>Programming in Python.</p><p>SQL and the Relational Algebra.</p><p>Data Analytics.</p> Y <p>Joe Reis, Matt Housley (2022): Fundamentals of Data Engineering. O'Reilly Media. ISBN 9781098108304.</p> Assessment written exam 2 Practical classes & workshops 20 Work based learning 10 0 118 <p>Videos (~5 Hours)</p><p>Formative Quizzes (~2 Hours)</p><p>Formative Coursework (~10 Hours)</p><p>Assessed Coursework (~10 Hours)</p><p>Assessed Quizzes (~3 Hours)</p>