// 01 — data engineering

Data Engineering — Top 1% Roadmap

For working at the companies that run data at the largest scale. Each topic links to how they built it: papers, engineering blogs and their interview problems. Practice and building projects come at the end, with a home lab to run them.
Total
78
Done
0
Progress
0%
Left
78
Est. time
~160 h

Storage internals

storage 0/6 ▾

Storage engines — B-trees vs LSM-trees

Write amplification, compaction, read paths. Why Bigtable, Cassandra and RocksDB are LSM.
DDIA, ch. 3

Column stores vs row stores

Late materialization, compression, vectorized scans. Why analytics moved to columns.

Nested data in columns — Dremel and Parquet

Repetition and definition levels: how Parquet stores nested records column by column.

Compression and encoding trade-offs

Dictionary, RLE, delta encoding; Snappy vs Zstd vs LZ4 for CPU, size and scan speed.

Object storage at scale

S3 strong consistency, request rates, listing costs, and why table formats exist on top of it.

Distributed processing

compute 0/7 ▾

MapReduce

Map, shuffle, reduce, stragglers and re-execution. The model under every batch engine.

Joins at scale and data skew

Broadcast, shuffle hash and sort-merge joins. Finding a hot key and fixing it with salting or AQE.

Vectorized execution — Photon and Velox

Columnar batches, SIMD, why engines are being rewritten in C++.

Table formats & lakehouse

lakehouse 0/6 ▾

The lakehouse architecture

Warehouse guarantees on open files in object storage. What it fixes and what it costs.

Incremental processing — Hudi at Uber

Upserts and incremental pulls on a lake instead of full rewrites.

Table maintenance — compaction and small files

Small-file problem, compaction, snapshot expiry, orphan files. What breaks at scale if nobody runs them.

Schema evolution and compatibility

Backward, forward and full compatibility; adding, renaming and dropping columns safely.

Streaming at scale

streaming 0/8 ▾

The log as the unifying abstraction

Why LinkedIn built Kafka: a log as the source of truth for databases, caches and pipelines.

Kafka internals — partitions, replication, ISR

Leaders and followers, in-sync replicas, acks, consumer groups and rebalancing.

Exactly-once semantics

Idempotent producers, transactions, and why end-to-end exactly-once needs an idempotent sink.

The Dataflow model — event time, windows, triggers

Correctness vs latency vs cost on unbounded, out-of-order data.

Stateful processing and checkpoints — Flink

Keyed state, state backends, asynchronous snapshots, recovery and rescaling.

Lambda vs Kappa

One code path for batch and streaming, and when reprocessing from the log is enough.

Change data capture

Reading the database log instead of polling tables. Snapshots plus streams without locking.

Real-time pipelines in production

Kafka, Flink and Pinot for billing-grade numbers: how Uber made ad events exactly-once.

Distributed systems

distributed 0/6 ▾

Replication and consistency models

Leader-based, multi-leader, leaderless. Linearizable vs causal vs eventual.
DDIA, ch. 5 and 9

Consensus — Raft

Leader election and log replication. How Kafka (KRaft) and etcd agree on state.

Time, clocks and ordering

Lamport clocks, clock skew, and how Spanner uses TrueTime for global transactions.
DDIA, ch. 8

CAP and its limits

Why "CP or AP" is the wrong question, and what PACELC adds.

Transactions and isolation

Isolation anomalies, snapshot isolation, serializability, MVCC.
DDIA, ch. 7

Data modeling at scale

modeling 0/5 ▾

Metrics layer and metric consistency

One definition per metric, served everywhere. How Airbnb stopped teams from reporting different numbers.

Data contracts

Schemas, owners and SLAs between producers and consumers, enforced in CI.

Data mesh

Domain ownership, data as a product, self-serve platform, federated governance.

Reliability, quality & governance

reliability 0/8 ▾

Functional data engineering

Idempotent, deterministic tasks over immutable partitions. The base of safe re-runs.

Backfills and reprocessing

Re-running months of data without double counting or starving production.

SLOs for data pipelines

Freshness, completeness and correctness as SLOs, with error budgets and on-call.

Lineage

Column-level lineage to answer "what breaks if this column changes" before you change it.

Privacy at scale — deletion and PII

Right-to-be-forgotten on immutable files, tokenization, access by column and row.

Online data migrations

Dual writes, backfill, verify, switch reads, clean up — without downtime.

Cost engineering

Cost per pipeline and per query, storage tiers, spot capacity, killing unused tables.

Big-tech interview problems

interview 0/18 ▾

SQL — trips and users

Cancellation rate with filtered joins and dates. A classic hard screen.

SQL — top N per group with ties

DENSE_RANK per partition, and explaining why not ROW_NUMBER.

SQL — gaps and islands

Consecutive streaks with the row-number difference trick.

SQL — retention and cohorts

Day-1 retention from an events table: self-joins vs window functions.

SQL — rolling metrics

7-day moving averages with frames, and what missing days do to them.

Coding — top-K heavy hitters

Most frequent keys in a huge stream: heaps, and Count-Min sketch when memory runs out.

Coding — merge sorted streams

k-way merge with a heap: how sorted runs are merged in external sort and compaction.

Coding — versioned key-value lookup

Value of a key at a timestamp: the point-in-time join behind SCD2 and feature stores.

Coding — median of a stream

Two heaps for an exact running median; when to accept an approximate percentile instead.

Coding — dependency order of a DAG

Topological sort and cycle detection: what an orchestrator does before it runs anything.

Data modeling — design a ride-sharing warehouse

Trips, drivers, riders, pricing changes. Pick the grain, the facts and the SCDs, then answer five product questions with SQL.

Data modeling — model product events

Sessions, funnels and attribution from raw click events, with late and duplicated events.

System design — clickstream pipeline

Billions of events a day: ingestion, dedup, late data, the lake layout and the serving layer. Defend each trade-off.

System design — store trillions of messages

Partition keys, hot partitions, compaction and migrations under live traffic.

System design — exactly-once billing events

Ad clicks that charge money: idempotency keys, transactional sinks, reconciliation.

System design — metrics platform

One source of truth for metrics across dashboards, experiments and finance.

System design — backfill a petabyte table

Re-process a year of history after a logic change, with production still running and a budget.

Pipeline exercise — incremental load, dedup and upsert

Load only what changed since the last run, keep the latest record per key, and make a re-run produce the same table. The daily bread of a big-tech data team.

// bibliography

The books, courses and practice sites behind this roadmap.

// books
Designing Data-Intensive Applications ↗
Martin Kleppmann · cited in 5 topics
Storage, replication, partitioning and consistency.
The Data Warehouse Toolkit ↗
Ralph Kimball & Margy Ross · cited in 4 topics
Dimensional modeling: facts, dimensions, slowly changing dimensions.

// other resources

Courses for the basics this roadmap doesn't cover.

Data Engineering Zoomcamp ↗
DataTalksClub · free
Docker, Terraform, orchestration, a warehouse, dbt, Spark and Kafka, end to end.
The Missing Semester of Your CS Education ↗
MIT · free
Shell, scripting, Vim, data wrangling, Git and debugging.
Python for Everybody ↗
Charles Severance · free
Python from zero: files, web data, databases.
Select Star SQL ↗
Zi Chong Kao · free
SQL by doing: joins, aggregation, subqueries.
SQL tutorial ↗
Mode / ThoughtSpot · free
From basic queries to window functions and performance.
Pro Git ↗
Scott Chacon & Ben Straub · free
Branching, rebasing and Git internals.
Docker — get started ↗
Docker · free
Images, containers, Compose.
AWS Certified Data Engineer — Associate ↗
AWS
S3, Glue, Athena, EMR, Redshift, Kinesis, IAM: the AWS data stack.
Azure Data Fundamentals (DP-900) ↗
Microsoft Learn · free
Core data concepts and the Azure data services.
dbt Learn ↗
dbt Labs · free
Models, tests, sources, snapshots.
Astronomer Academy ↗
Astronomer · free
Airflow DAGs, operators, scheduling.
MongoDB University ↗
MongoDB · free
Document modeling, indexes, aggregation.
NeetCode practice ↗
NeetCode · free
Arrays, hashing, heaps, graphs: the coding screen baseline.

Practice

practice 0/8 ▾

Open a Parquet file and read its metadata

Write the same table with different row-group sizes, encodings and codecs. Read the footer: row groups, min/max stats, dictionary pages. Check which filters skip which row groups.

Break a Spark join and fix it

Generate a table where one key holds 30% of the rows, join it, find the straggler task in the Spark UI, then fix it with a broadcast, salting and AQE. Compare the three.

Kill a Kafka broker

Run three brokers, produce with acks=all and min.insync.replicas=2, stop one broker, then two. Watch the ISR, the leader election and what the producer sees.

Late events in a streaming job

Send out-of-order events with Flink or Spark Structured Streaming, count per 5-minute window by event time, and move the watermark to see what gets dropped.

Provoke isolation anomalies

Reproduce lost updates and write skew in Postgres with two sessions, then fix them with the right isolation level or locks.

CDC from Postgres to Kafka

Stream a table's changes with Debezium, then rebuild the table downstream from the change log and compare.

Backfill without double counting

Run a daily job for a year of data, change its logic, backfill three months, and prove with checks that nothing was counted twice.

Building

building 0/6 ▾

Set up a home lab

One machine at home instead of a cloud bill. A used mini PC or workstation with 8+ cores, 64 GB RAM and a 1–2 TB NVMe runs Kafka (3 brokers), Spark, Trino, Flink, MinIO and Postgres at once; 32 GB is enough if you run one stack at a time. No GPU needed. Run the stacks with Docker Compose or k3s, reach the lab from anywhere with Tailscale, and expose a lab to others with a Cloudflare Tunnel.

Build a lakehouse on the home lab

NYC taxi trips into Iceberg on MinIO, transformed with Spark or dbt, queried with Trino. Partition it, compact it, and measure query times before and after.

Build a real-time pipeline

Events into Kafka, aggregated by event time in Flink, served from ClickHouse to a dashboard. Make the counts survive a restart without duplicates.

Build CDC into the lakehouse

Postgres changes through Debezium and Kafka into an Iceberg table with upserts, then answer "what did this row look like last Tuesday".

Build a metrics layer with quality and lineage

dbt models with one definition per metric, quality checks that block bad loads, and lineage you can browse.

Build your own storage engine

Write a small LSM-tree or a SQLite clone. Nothing explains compaction, indexes and write amplification like building them.