Total
78
Done
0
Progress
0%
Left
78
Est. time
~160 h
Storage internals
storage 0/6 ▾Storage engines — B-trees vs LSM-trees
Write amplification, compaction, read paths. Why Bigtable, Cassandra and RocksDB are LSM.
DDIA, ch. 3
Column stores vs row stores
Late materialization, compression, vectorized scans. Why analytics moved to columns.
Nested data in columns — Dremel and Parquet
Repetition and definition levels: how Parquet stores nested records column by column.
Compression and encoding trade-offs
Dictionary, RLE, delta encoding; Snappy vs Zstd vs LZ4 for CPU, size and scan speed.
Object storage at scale
S3 strong consistency, request rates, listing costs, and why table formats exist on top of it.
Distributed file systems — GFS and HDFS
Single master, chunk replication, append-heavy workloads. The design every data lake inherited.
Distributed processing
compute 0/7 ▾MapReduce
Map, shuffle, reduce, stragglers and re-execution. The model under every batch engine.
Spark execution — RDDs, stages, shuffle
Lineage, narrow vs wide dependencies, stages, tasks, and what the shuffle costs.
Joins at scale and data skew
Broadcast, shuffle hash and sort-merge joins. Finding a hot key and fixing it with salting or AQE.
Query optimizers — Catalyst and AQE
Logical and physical plans, rule-based vs cost-based, re-optimizing at runtime.
Vectorized execution — Photon and Velox
Columnar batches, SIMD, why engines are being rewritten in C++.
MPP SQL engines — Presto and Trino
Coordinator and workers, pipelined execution, connectors, federated queries.
Cloud warehouses — BigQuery and Snowflake
Separated storage and compute, micro-partitions, slots and credits.
Table formats & lakehouse
lakehouse 0/6 ▾The lakehouse architecture
Warehouse guarantees on open files in object storage. What it fixes and what it costs.
Delta Lake — the transaction log
ACID on S3 with a JSON log and checkpoints. Optimistic concurrency, time travel.
Iceberg — snapshots, manifests, hidden partitioning
Metadata trees instead of directory listings; partition evolution without rewrites.
Incremental processing — Hudi at Uber
Upserts and incremental pulls on a lake instead of full rewrites.
Table maintenance — compaction and small files
Small-file problem, compaction, snapshot expiry, orphan files. What breaks at scale if nobody runs them.
Schema evolution and compatibility
Backward, forward and full compatibility; adding, renaming and dropping columns safely.
Streaming at scale
streaming 0/8 ▾The log as the unifying abstraction
Why LinkedIn built Kafka: a log as the source of truth for databases, caches and pipelines.
Kafka internals — partitions, replication, ISR
Leaders and followers, in-sync replicas, acks, consumer groups and rebalancing.
Exactly-once semantics
Idempotent producers, transactions, and why end-to-end exactly-once needs an idempotent sink.
The Dataflow model — event time, windows, triggers
Correctness vs latency vs cost on unbounded, out-of-order data.
Stateful processing and checkpoints — Flink
Keyed state, state backends, asynchronous snapshots, recovery and rescaling.
Lambda vs Kappa
One code path for batch and streaming, and when reprocessing from the log is enough.
Change data capture
Reading the database log instead of polling tables. Snapshots plus streams without locking.
Real-time pipelines in production
Kafka, Flink and Pinot for billing-grade numbers: how Uber made ad events exactly-once.
Distributed systems
distributed 0/6 ▾Replication and consistency models
Leader-based, multi-leader, leaderless. Linearizable vs causal vs eventual.
DDIA, ch. 5 and 9
Partitioning and consistent hashing
Hash vs range partitioning, hot spots, rebalancing, virtual nodes.
DDIA, ch. 6
Consensus — Raft
Leader election and log replication. How Kafka (KRaft) and etcd agree on state.
Time, clocks and ordering
Lamport clocks, clock skew, and how Spanner uses TrueTime for global transactions.
DDIA, ch. 8
CAP and its limits
Why "CP or AP" is the wrong question, and what PACELC adds.
Transactions and isolation
Isolation anomalies, snapshot isolation, serializability, MVCC.
DDIA, ch. 7
Data modeling at scale
modeling 0/5 ▾Dimensional modeling — grain first
Declaring the grain, facts vs dimensions, conformed dimensions across teams.
Metrics layer and metric consistency
One definition per metric, served everywhere. How Airbnb stopped teams from reporting different numbers.
One Big Table vs star schema
Denormalized tables on columnar engines: when they win and what they cost.
Data contracts
Schemas, owners and SLAs between producers and consumers, enforced in CI.
Data mesh
Domain ownership, data as a product, self-serve platform, federated governance.
Reliability, quality & governance
reliability 0/8 ▾Functional data engineering
Idempotent, deterministic tasks over immutable partitions. The base of safe re-runs.
Backfills and reprocessing
Re-running months of data without double counting or starving production.
Data quality at scale
Declarative checks, anomaly detection on metrics, blocking vs alerting.
SLOs for data pipelines
Freshness, completeness and correctness as SLOs, with error budgets and on-call.
Lineage
Column-level lineage to answer "what breaks if this column changes" before you change it.
Privacy at scale — deletion and PII
Right-to-be-forgotten on immutable files, tokenization, access by column and row.
Online data migrations
Dual writes, backfill, verify, switch reads, clean up — without downtime.
Cost engineering
Cost per pipeline and per query, storage tiers, spot capacity, killing unused tables.
Big-tech interview problems
interview 0/18 ▾SQL — trips and users
Cancellation rate with filtered joins and dates. A classic hard screen.
SQL — top N per group with ties
DENSE_RANK per partition, and explaining why not ROW_NUMBER.
SQL — gaps and islands
Consecutive streaks with the row-number difference trick.
SQL — retention and cohorts
Day-1 retention from an events table: self-joins vs window functions.
SQL — rolling metrics
7-day moving averages with frames, and what missing days do to them.
Coding — top-K heavy hitters
Most frequent keys in a huge stream: heaps, and Count-Min sketch when memory runs out.
Coding — merge sorted streams
k-way merge with a heap: how sorted runs are merged in external sort and compaction.
Coding — versioned key-value lookup
Value of a key at a timestamp: the point-in-time join behind SCD2 and feature stores.
Coding — median of a stream
Two heaps for an exact running median; when to accept an approximate percentile instead.
Coding — dependency order of a DAG
Topological sort and cycle detection: what an orchestrator does before it runs anything.
Data modeling — design a ride-sharing warehouse
Trips, drivers, riders, pricing changes. Pick the grain, the facts and the SCDs, then answer five product questions with SQL.
Data modeling — model product events
Sessions, funnels and attribution from raw click events, with late and duplicated events.
System design — clickstream pipeline
Billions of events a day: ingestion, dedup, late data, the lake layout and the serving layer. Defend each trade-off.
System design — store trillions of messages
Partition keys, hot partitions, compaction and migrations under live traffic.
System design — exactly-once billing events
Ad clicks that charge money: idempotency keys, transactional sinks, reconciliation.
System design — metrics platform
One source of truth for metrics across dashboards, experiments and finance.
System design — backfill a petabyte table
Re-process a year of history after a logic change, with production still running and a budget.
Pipeline exercise — incremental load, dedup and upsert
Load only what changed since the last run, keep the latest record per key, and make a re-run produce the same table. The daily bread of a big-tech data team.
// bibliography
The books, courses and practice sites behind this roadmap.
// books
Designing Data-Intensive Applications ↗
Storage, replication, partitioning and consistency.
The Data Warehouse Toolkit ↗
Dimensional modeling: facts, dimensions, slowly changing dimensions.
// other resources
Courses for the basics this roadmap doesn't cover.
Data Engineering Zoomcamp ↗
Docker, Terraform, orchestration, a warehouse, dbt, Spark and Kafka, end to end.
The Missing Semester of Your CS Education ↗
Shell, scripting, Vim, data wrangling, Git and debugging.
Python for Everybody ↗
Python from zero: files, web data, databases.
Select Star SQL ↗
SQL by doing: joins, aggregation, subqueries.
SQL tutorial ↗
From basic queries to window functions and performance.
Pro Git ↗
Branching, rebasing and Git internals.
Docker — get started ↗
Images, containers, Compose.
AWS Certified Data Engineer — Associate ↗
S3, Glue, Athena, EMR, Redshift, Kinesis, IAM: the AWS data stack.
Azure Data Fundamentals (DP-900) ↗
Core data concepts and the Azure data services.
dbt Learn ↗
Models, tests, sources, snapshots.
Astronomer Academy ↗
Airflow DAGs, operators, scheduling.
MongoDB University ↗
Document modeling, indexes, aggregation.
NeetCode practice ↗
Arrays, hashing, heaps, graphs: the coding screen baseline.
Practice
practice 0/8 ▾Open a Parquet file and read its metadata
Write the same table with different row-group sizes, encodings and codecs. Read the footer: row groups, min/max stats, dictionary pages. Check which filters skip which row groups.
Break a Spark join and fix it
Generate a table where one key holds 30% of the rows, join it, find the straggler task in the Spark UI, then fix it with a broadcast, salting and AQE. Compare the three.
Iceberg on local object storage
Create a table on MinIO or Garage, evolve its schema and partitioning, time travel, then generate small files and compact them.
Kill a Kafka broker
Run three brokers, produce with acks=all and min.insync.replicas=2, stop one broker, then two. Watch the ISR, the leader election and what the producer sees.
Late events in a streaming job
Send out-of-order events with Flink or Spark Structured Streaming, count per 5-minute window by event time, and move the watermark to see what gets dropped.
Provoke isolation anomalies
Reproduce lost updates and write skew in Postgres with two sessions, then fix them with the right isolation level or locks.
CDC from Postgres to Kafka
Stream a table's changes with Debezium, then rebuild the table downstream from the change log and compare.
Backfill without double counting
Run a daily job for a year of data, change its logic, backfill three months, and prove with checks that nothing was counted twice.
Building
building 0/6 ▾Set up a home lab
One machine at home instead of a cloud bill. A used mini PC or workstation with 8+ cores, 64 GB RAM and a 1–2 TB NVMe runs Kafka (3 brokers), Spark, Trino, Flink, MinIO and Postgres at once; 32 GB is enough if you run one stack at a time. No GPU needed. Run the stacks with Docker Compose or k3s, reach the lab from anywhere with Tailscale, and expose a lab to others with a Cloudflare Tunnel.
Build a lakehouse on the home lab
NYC taxi trips into Iceberg on MinIO, transformed with Spark or dbt, queried with Trino. Partition it, compact it, and measure query times before and after.
Build a real-time pipeline
Events into Kafka, aggregated by event time in Flink, served from ClickHouse to a dashboard. Make the counts survive a restart without duplicates.
Build CDC into the lakehouse
Postgres changes through Debezium and Kafka into an Iceberg table with upserts, then answer "what did this row look like last Tuesday".
Build a metrics layer with quality and lineage
dbt models with one definition per metric, quality checks that block bad loads, and lineage you can browse.
Build your own storage engine
Write a small LSM-tree or a SQLite clone. Nothing explains compaction, indexes and write amplification like building them.