System Design for Data Pipelines

Batch and streaming, queues, backfills, data quality and feature stores.

Free PDF · sent by email the day it’s ready

Who it’s for

For data and ML engineers asked to design the pipelines that feed models, dashboards and search indexes.

Expect questions on late data, reprocessing and how you know the data is right.

What it covers

  • Batch vs streaming
  • Queues
  • Data quality
  • Feature stores

What the guide will include

  • A framework for the round

    Requirements, architecture, tradeoffs, evaluation and failure modes, in the order you say them.

  • Worked designs

    Prompts worked end to end the way you would answer them at the whiteboard, with the diagram.

  • Tradeoffs to name

    The choices interviewers push on, and what each option costs in latency, money and risk.

  • Failure modes and evals

    How the system breaks in production, how you would notice, and what you would measure.

  • A PDF you can print

    Read it on your phone before the call, or print the sections you want to drill.

Planned contents

The guide isn’t out yet, so chapters and their order may still change.

  1. 1. How the design round runs
  2. 2. Batch vs streaming
  3. 3. Queues
  4. 4. Data quality
  5. 5. Feature stores
  6. 6. Practice prompts

Free PDF

Get the Data Pipelines PDF free

Batch and streaming, queues, backfills, data quality and feature stores.

Data Pipelines System Design: The PDF isn’t out yet. We’ll email it to you the day it is. No spam; unsubscribe with one click.

Questions about the guide

Is the PDF really free?

Yes. Leave your email and we send the PDF the day it is ready. No payment details, and every email has a one-click unsubscribe link.

Is it based on a specific company’s interviews?

No. It is written from the concepts the role or system depends on. We don’t name companies or claim to know what any one company asks.

When will it be ready?

It isn’t out yet. Leave your email and we send it the day it is; nothing else arrives before then.

Related guides

System design · Free PDF

System Design for Databases

Choosing a database, indexing, replication, sharding, consistency and vector stores.

Email me the PDF

System design · Free PDF

System Design for ML Systems

Recommendations, ranking, search and fraud detection, from data to serving.

Email me the PDF

Role guide · Free PDF

MLOps Engineer

Deploying and serving models, pipelines, monitoring, drift and rollbacks.

Email me the PDF

All guides →