Data Engineering

What data engineers do, and how AI changed the job

Understand what a data engineer builds, how the role differs from analysts and data scientists, and what AI tools have and have not changed. Start here before the technical lessons.

Advanced SQL: window functions, CTEs and query plans

Learn the SQL that data engineering interviews test: ranking, running totals and LAG with window functions, readable CTEs, and how to read a query plan to find out why a query is slow. Runs on your laptop with Python and SQLite.

Data modelling: normalisation, star schemas and SCDs

Learn how to design tables for apps (normalisation) and for analysis (star schemas with fact and dimension tables), and how slowly changing dimensions keep history correct. A staple of data engineering interviews.

OLTP vs OLAP: databases, warehouses, lakes and lakehouses

Understand why app databases and analytics systems are built differently, why columnar storage is fast for analysis, and when to choose a warehouse, a data lake or a lakehouse. Includes the bronze-silver-gold layout.

ETL vs ELT and pipeline design

Learn the difference between ETL and ELT, batch and streaming, and the habits that make pipelines reliable: incremental loads, idempotency, backfills, late data and failing loudly. Core knowledge for interviews and real jobs.

Batch pipelines with Python

Build a complete batch pipeline in plain Python: read a dirty CSV, validate rows, save rejects with reasons, and load into a database idempotently. You will run it, re-run it safely, test it and learn how to make it production-ready.

Apache Spark and PySpark basics

Learn how Spark processes big data in parallel, then build a PySpark job that reads CSV files, joins and aggregates them with the DataFrame API and SQL, and writes partitioned Parquet. You will also learn lazy evaluation and shuffles.

Orchestration with Apache Airflow

Learn what an orchestrator does, then build a daily Apache Airflow DAG with the TaskFlow API: tasks, dependencies, retries, XCom and idempotent loads. Run it locally and read its logs in the web UI.

Transformations with dbt

Build a small dbt project on DuckDB: staging and mart models joined with ref(), tests for bad data, generated docs and lineage. Learn why dbt became the standard way to do the T in ELT, and when to use views, tables or incremental models.

File and table formats: Parquet, Delta Lake and Iceberg

Understand why columnar Parquet beats CSV for analytics, why plain files are not enough, and how Delta Lake and Apache Iceberg add transactions, updates and time travel on top of Parquet. Includes runnable Parquet and Delta examples.

Streaming with Apache Kafka

Run Kafka locally in Docker and learn topics, partitions, keys, offsets and consumer groups by building a Python producer and consumer for Brew & Bean order events. Understand ordering and at-least-once delivery.

Stream processing: Flink and Spark Structured Streaming

Understand how stream processors work: event time versus processing time, windows, watermarks, state and checkpoints. Compare Apache Flink and Spark Structured Streaming with examples of the same windowed count in both.

Cloud data platforms: BigQuery, Snowflake and Databricks

Understand how BigQuery, Snowflake and Databricks work, how each is billed and how to choose between them. Learn the shared idea of separate storage and compute and the habits that keep cloud data costs under control.

Data quality, testing and observability

Build a small data quality checker in Python and DuckDB, run it against data with planted problems and learn where to place checks. See how dbt tests and observability apply the same ideas, so bad data is caught before users see it.

Governance, privacy and the DPDP Act

Learn the pieces of data governance (ownership, catalogs, access, lineage, retention) and what India's DPDP Act, 2023 means for pipelines. Includes masking, pseudonymisation and a deletion-request process. An engineering view, not legal advice.

Data engineering for AI: RAG pipelines and feature stores

See how data engineers feed AI. Build the stages of a RAG ingestion pipeline (chunking, hashes, metadata, permissions), keep the index fresh, and learn how feature stores and point-in-time joins give ML models consistent inputs.

Project: build an end-to-end data pipeline

Build a Brew & Bean batch pipeline in Python and DuckDB: ingest raw JSON orders, clean and reject bad rows, run quality checks, publish a sales table and report. It is re-runnable, tested and ready to show in interviews.

Data engineering roadmap and certifications

A practical roadmap into data engineering: the roles, what to learn in order, projects that prove it, the main certifications (with how to check current details) and what interviews test. Includes a 90-day plan.