 |
 |
 |
|
|
Data management involves a variety of processes and practices to collect, organize, store, and deliver high-quality data for data science, business intelligence, performance management, and business operations. Data engineering is an essential discipline that is responsible to design, build, and deploy data management capabilities. The data engineering discipline encompasses three distinct roles: database engineer, data pipeline engineer, and data product engineer.
|
|
This course focuses on the role of data pipeline engineer – the technical professional who designs, builds, deploys, and support the processes to acquire, ingest, integrate, transform, and prepare data, to load data into databases, and to deliver data to consumers. The data pipeline engineering skill set includes requirements analysis, data modeling, data pipeline design, data consolidation, data transformation, data integration, and pipeline implementation and operations across a broad spectrum of technologies. Data pipeline technologies include ETL, data stream processing, data virtualization, change data capture (CDC), database replication, pipeline orchestration, data observability, and more. Pipeline infrastructure spans on-premises, cloud, multi-cloud, and edge platforms to manage and move data across transactional, data warehouse, data mart, and data lake systems.
On completion of this course you will have a solid foundation of knowledge and skills needed for data pipeline engineering.
You will learn:
- What is Data Pipeline Engineering?
- Components of Pipelines: Origin, Destination, Dataflow, Workflow, Storage, Processing, Monitoring, and Technology
- Data Pipeline Design Patterns: Bulk Loading, ETL and variations, Virtualization, CDC, Stream Processing, etc.
- Analytic Data Provisioning Patterns (what do algorithms work with?): List, Array, Stack, Queue, Graph, etc.
- The Data Pipeline Design Process
- Data Pipeline Deployment, Orchestration, and Operation
This course is geared towards:
- Data Engineers
- Data Architects
- Data Scientists
- Data Analysts
- DevOps Engineers
- IT Professionals
- Data Warehousing Specialists
- ETL Developers
- Big Data Engineers
Module 0: About the Course (3 min)
Module 1: Introduction to Data Pipelines (53 min)
- Module Overview
- Data Pipelines Defined
- Data Consumption
- Data Pipeline Engineering
- Types of Data Pipelines
- Module Summary
Module 2. Data Pipeline Design Patterns (34 min)
- Anatomy of a Data Pipeline
- Module Overview
- Bulk Loading
- ELT
- CDC
- Data Stream Processing
- Module Summary
Module 3. Anatomy of a Data Pipeline (50 min)
- Module Overview
- Data Pipeline Architecture
- Components
- Sourcing and Processing Components
- Post-Processing Components
- Module Summary
Module 4. Data Pipeline Development Process (33 min)
- Module Objectives
- Requirements Analysis
- Data Analysis
- Design and Build
- Testing and Development
- Module Summary
Module 5. Data Pipeline Operations (16 min)
- Module Objectives
- Orchestration, Monitoring & Observability
- Service Levels
- Module Summary
Module 6. Data Pipelines Best Practices (15 min)
- Module Objectives
- Best Practices in Design
- Best Practices in Operations
- Module Summary
- Course Summary
Click
–here- to download a more detailed outline of this course.
|
This exam tests knowledge and understanding of basic concepts, principles, and terminology of analytics.
|
You will be tested in these areas:
- Functions and purpose of data pipelines
- Data consumption and data pipelines
- The role of Data Pipeline Engineer and overlap with database & product engineering
- Kinds of data pipelines
- Bulk loading, ETL, CDC, and stream processing pipelines
- Components of a data pipeline
- Data pipeline requirements analysis
- Data analysis for data pipeline engineering
- Designing and building data pipelines
- Pipeline orchestration and monitoring
- Data pipeline service level management
- Best practices for data pipeline design and operations
|
Additional Information
Number of Questions: 23
Time Limit: 46 minutes
Passing Score: 70%
|
|
Once you pass the exam, you will receive a Certificate of Education
documenting that you have demonstrated mastery of the topic. Course
exams count towards eLC certification programs. Visit our
Certification page for more information about our various programs.
We recommend that you take detailed notes and review the course material multiple times before taking this exam.
|
|
|
|
 |
|
 |
|