
Program
Data Engineer
Build the pipelines analysts and BI developers depend on
Learn SQL, ETL patterns, and pipeline design for moving and transforming data reliably.
8
Weeks
40
Chapters
4
Practice formats
1
Portfolio artifact
8 weeks — one module per week (8 modules, 40 chapters), each week ending with a live class every Saturday, plus a module exam before the next module opens.
Curriculum
40 chapters across 8 modules — each with guided notes, a live example, a quiz, an exercise, a challenge, and class practice.
Module 1 · DATA ENGINEERING FOUNDATIONS
- 1Introduction to Data EngineeringWhat data engineers do, the data lifecycle, ETL vs ELT, batch vs streaming, and the modern data stack.Learn →
- 2Data ModelingOLTP vs OLAP, normalization, star schemas with facts and dimensions, keys and grain, and slowly changing dimensions.Learn →
- 3SQL for Data EngineeringDDL and constraints, data types, indexes, inserting and updating data, transactions, and integrity checks.Learn →
- 4Advanced SQL for PipelinesCTEs, window functions for deduplication and latest records, incremental loads, upserts, and set operations.Learn →
- 5OLTP vs. OLAP: From Application Databases to AnalyticsHow transactional (OLTP) and analytical (OLAP) systems differ, normalised vs. denormalised designs, why analytics needs its own copy of data, and flattening joins for reporting.Learn →
Module 2 · BUILDING PIPELINES WITH PYTHON
- 6Python for Data EngineeringWorking with files and formats, functions and modules, error handling, logging, and configuration for reliable pipeline code.Learn →
- 7Extracting DataSources and connectors, databases, files, and REST APIs, pagination and rate limits, and incremental extraction.Learn →
- 8Transforming DataCleaning and standardizing, typing, joining and enriching, deduplication, business rules, and idempotent transformations.Learn →
- 9Loading Data & WarehousingLoad strategies (full, append, upsert, partition overwrite), bulk loading, staging tables, and validating loads.Learn →
- 10Ingesting from APIs & JSONREST APIs, authentication, pagination, rate limits and retries, incremental extraction with timestamps, flattening nested JSON, and landing raw data safely.Learn →
Module 3 · PLATFORMS, ORCHESTRATION & STREAMING
- 11Warehouses, Lakes & LakehousesData warehouses, data lakes, and lakehouses, file formats, Delta tables, and choosing an architecture.Learn →
- 12Microsoft Fabric for Data EngineeringOneLake, lakehouses and warehouses, Data Factory pipelines and dataflows, notebooks, and serving data to Power BI.Learn →
- 13Orchestration & SchedulingDAGs and dependencies, schedulers like Airflow and Fabric pipelines, retries, backfills, and parameterized runs.Learn →
- 14Streaming DataEvents and streams, Kafka and event hubs, windows and aggregations, delivery guarantees, and when to stream.Learn →
- 15Cloud Data Services: AWS, Azure & Google CloudMapping storage, compute, warehouse, orchestration and streaming services across the three major clouds, managed vs. self-managed choices, and IAM basics for data workloads.Learn →
Module 4 · PRODUCTION DATA ENGINEERING
- 16Data Quality & TestingData quality dimensions, automated checks and tests, data contracts, and handling failures in pipelines.Learn →
- 17Monitoring & ReliabilityObservability for pipelines, SLAs and freshness, alerting without noise, lineage, incident handling, and cost awareness.Learn →
- 18Security & GovernanceAccess control and least privilege, protecting personal data, masking and pseudonymization, compliance, and data catalogs.Learn →
- 19CI/CD & Environments for Data PipelinesVersion control workflows, code review, automated tests in CI, dev/test/prod environments, deploying pipeline code and SQL models safely, and rollback.Learn →
- 20Data SLAs, Freshness & Pipeline HealthDefining SLAs/SLOs for data, freshness and completeness indicators, success rates, error budgets, and a pipeline health scorecard.Learn →
Module 5 · ON THE JOB: BUILDING THE ANALYTICS WAREHOUSE
- 21On the Job: Designing a Sales Data MartTicket DE-142: "Finance and Sales need one trusted model for revenue." Gather requirements, choose the grain, design facts and dimensions, handle history with SCD Type 2, and document the model.Learn →
- 22On the Job: Incremental Loads & Change Data CaptureThe nightly full reload now takes 5 hours. Switch to incremental loading — watermarks, upserts/MERGE, deletes, late-arriving data, idempotency, and backfills.Learn →
- 23On the Job: Transformation Layers, dbt & TestingTurn a tangle of SQL scripts into maintainable layers — staging, intermediate and marts, modular SQL models, built-in tests, documentation, and code review.Learn →
- 24On the Job: Slowly Changing DimensionsType 1 vs. Type 2 dimensions, validity ranges and current flags, point-in-time joins, and how history changes the answers reports give.Learn →
- 25On the Job: Aggregate Tables & the Semantic LayerPre-aggregated summary tables for fast dashboards, choosing grain, additive vs. non-additive measures, incremental refresh, and semantic layers that define metrics once.Learn →
Module 6 · ON THE JOB: BIG DATA & CLOUD PIPELINES
- 26On the Job: Processing Big Data with SparkThe clickstream is 2 TB a day and pandas can't cope. Learn how Spark distributes work — DataFrames, lazy evaluation, partitions, shuffles, skew — and write efficient jobs.Learn →
- 27On the Job: File Formats, Partitioning & Table FormatsQueries on the data lake are slow and expensive. Choose the right file format, partition and cluster data, fix the small-files problem, and use Delta/Iceberg tables for reliability.Learn →
- 28On the Job: Cloud Pipelines, Serverless & Cost ControlMove the on-prem ETL to the cloud and keep the bill under control — managed ingestion and orchestration services, serverless vs clusters, secrets, environments, and FinOps for data.Learn →
- 29On the Job: Batch vs. Streaming ArchitecturesLatency requirements, micro-batch vs. true streaming, Lambda and Kappa architectures, exactly-once and late data, and choosing the simplest design that meets the need.Learn →
- 30On the Job: Capacity & Cost EstimationEstimating data growth, storage and scan costs, compounding growth, the impact of partitioning and compression, and presenting cost forecasts.Learn →
Module 7 · ON THE JOB: OPERATING DATA PLATFORMS
- 31On the Job: The 6 a.m. Pipeline FailureYou're on call and the overnight load failed. Triage, communicate, fix and backfill — then run a blameless post-incident review and prevent it happening again.Learn →
- 32On the Job: Data Contracts & Schema ChangesA product team renamed a column and broke five pipelines. Agree data contracts with producers, evolve schemas safely, version breaking changes, and detect drift automatically.Learn →
- 33On the Job: Warehouse Performance & Query TuningDashboards time out every Monday morning. Find the slow queries, read query plans, fix joins and filters, add clustering and aggregates, and manage concurrency.Learn →
- 34On the Job: Backfills & ReprocessingFinding missing or bad partitions, planning backfills safely, idempotent reruns, controlling load on sources, and communicating reprocessing to users.Learn →
- 35On the Job: PII Masking & Access ControlClassifying personal data, masking and pseudonymisation techniques, secure views, role-based and row-level access, and auditing who sees what.Learn →
Module 8 · CAPSTONE & CAREER LAUNCH
- 36Capstone: End-to-End PipelineBuild and document an end-to-end ETL pipeline that loads real data into a queryable warehouse table, with tests and monitoring.Learn →
- 37Portfolio & Interview Preparation for Data EngineersShow you can build reliable pipelines — portfolio projects, the DE CV, SQL and Python coding rounds, data modelling and system-design interviews.Learn →
- 38Job Search Strategy for Data EngineersEntry routes into data engineering, reading job ads by stack, building a GitHub portfolio, tailoring applications, and networking in data communities.Learn →
- 39System Design Interviews for Data EngineersHow data system design interviews run, a step-by-step framework, estimating scale, trade-offs to discuss, and common prompts with model approaches.Learn →
- 40Your First 90 Days as a Data EngineerStart well on a data platform team — learn the architecture and on-call, ship small safe changes, build trust with consumers, and own a pipeline.Learn →
Delivery
Blended — self-paced lessons plus live lab sessions
Portfolio outcome
An end-to-end ETL pipeline moving and transforming real data into a queryable warehouse table.
Certificate
Certificate of completion, verifiable online with its credential code.
By the end of this program, you'll be job-ready to…
- Design an ETL/ELT pipeline from source to warehouse
- Write production-grade SQL, including window functions and CTEs
- Build in data quality checks and monitoring, not just happy-path code
Salary expectations
Estimated U.S. national average annual salary by experience level.
$85,000
Entry-level
$115,000
Mid-level
$155,000
Senior-level
Estimated figures for planning purposes — actual pay varies by location, employer, and experience.
Tools you'll master
Practical training, every chapter
- A live "try it yourself" example in every chapter you can run immediately
- A hands-on exercise per chapter, with hints and a reveal-able answer
- An open-ended challenge per chapter, graded against real success criteria
- Class practice prompts for pairs or cohorts, not just solo homework
Why take this program
Interview- and job-ready
Built to the same bar Aicloud actually hires against — not a generic syllabus.
Tier 1 company partners
Program partnerships with leading employers are in progress — see Affiliation for status.
Job connection access
Graduates get access to Aicloud's own hiring pipeline and employer referral network.
Professional networking
Cohort discussion boards and live sessions connect you with peers and mentors, not just content.
A real portfolio
You leave with a concrete, explainable artifact — proof of knowledge, not just a certificate.