Data Engineering Technologies

COMP63502 Data Engineering Technologies 15 Semester 2 Postgraduate Taught Level 6 Jiaoyan Chen Unit coordinator 1 FHEQ level (Framework for Higher Education Qualifications) ' Masters/Integrated Masters P4 ' European Credit Transfer & Accumulation System Rating : 7.5 In the world of data analytics, preparing and managing data of often the most time-consuming task --- estimated to take up to 80% of a whole workload by many reports and surveys. This unit focuses on the essential data engineering techniques that make large-scale data processing and analysis possible and efficient. Students will explore the foundational concepts and tools used in modern data engineering, including scalable data storage systems, advanced querying methods, parallel and distributed data processing, data interpretation, and effective data retrieval strategies. Emphasis is placed not just on theory, but on hands-on, practical skills that prepare students to work with real-world data. Data engineering can be considered to follow a lifecycle that captures activities such as data ingestion, storage, transformation and dissemination. Data engineering is a long standing, substantial and evolving discipline, and many technologies have been developed that can underpin different steps in the data engineering lifecycle, addressing challenges such as data heterogeneity and scale. The judicious selection and effective deployment of suitable specialised technologies can have a significant impact of the success of data engineering projects. In this unit, we explore some of these technologies, including those that support scalability, data curation and data analysis. Scalability is a cross-cutting concern of relevance to multiple stages in a data engineering pipeline. In relation to scalability, we investigate techniques for storage and indexing, algorithmic strategies for large data sets, query processing and big data platforms. Data curation and analysis applies technologies for managing data complexity, including inconsistent representations. In relation to data curation and analysis, we investigate techniques for data representation, data integration and data retrieval. This unit aims to provide students with exposure to and experience of specialised technologies that support data storage, access, integration and use at scale. Data engineering relates to the processes, tools and techniques required to maximise the value that can be obtained from the data resources an individual or organisation has access to. Many of the challenges faced by data engineers have been prominent for a considerable period, and have benefited from research and development that has given rise to specialised techniques for obtaining value from data. This unit aims to provide potential data engineers with the ability to select, evaluate and apply data engineering technologies to problems that involve complex data at scale. 1. Describe technologies that underpin scalability in data intensive systems and their properties.2. Describe and discuss data integration and data retrieval techniques.3. Compare and contrast approaches to the development of data intensive applications.4. Analyse how different algorithms and data structures affect data intensive system performance.5. Construct and apply different data representations that support data curation and analysis.6. Design experiments for comparing and analysing different data engineering techniques.7. Write reports that analyse properties of data engineering techniques. Analytical skills Innovation/creativity Oral communication Problem solving Research Written communication Part I: Techniques for Scalability Week 1: Storage: Storing Datasets for Scalability  •    File Systems •    Storage structures •    Indexes on disk and in memory Week 2: Algorithms •    Algorithmic strategies •    Modelling algorithm behaviour Week 3: Queries •    Query processing  •    Modelling query properties Week 4: Parallelism/Distribution •    Architectures •    Paradigms Week 5: Platforms •    Batch •    Interative •    Streaming Week 6:  •    Complete laboratory work. Part I: Data Curation and Analysis Week 7: Graph-based Data Analysis •    Graph database •    Graph query Week 8: Table Representation •    Models and learning methods •    Discussion and applications Week 9: Semantic Table Interpretation •    Entity annotation •    Type annotation •    Attribute and relation annotation •    Table to graph transformation Week 10: Data Integration •    Schema inference •    Entity alignment Week 11: Advanced Topics and Recent Development •    Question answering •    Retrieval augmented generation Week 12:  •    Complete laboratory work <ul> </ul> The unit will adopt a blended learning approach, with videos and quizzes for students to engage with asynchronously, in addition to synchronous activities in the form of: (i) workshops that include both presentation of new material and problem solving; (ii) laboratory sessions that explore specific techniques in more detail and apply them in practice. 1 Written exam 50% 2 Written assignment (inc essay) 50% Summative lab-based coursework: individual rubric-based feedback after marking. Formative weekly quizzes: Autograded quizzes providing immediate feedback. Exam: cohort level feedback after marking.   COMP63301 Data Engineering Concepts Pre-Requisite Recommended Prior knowledge of machine learning is needed N Martin Kleppmann, Designing Data-Intensive Applications, O’Reilly, 2017.Jure Leskovec, Anand Rajaraman, Jeff Ullman, Mining of Massive Datasets, 3rd Edition, Cambridge University Press, 2020.Joe Reis and Matt Housley, Fundamentals of Data Engineering, O’Reilly, 2022.  Assessment written exam 1.5 Lectures 20 Practical classes & workshops 12 0 116.5