A system design prompt is intentionally incomplete. “Design notifications,” “design a feed,” or “design file storage” does not tell you which users matter, which operations dominate, what failure is unacceptable, or how much scale the system must handle.

The solution is not memorizing one architecture per product. It is using a repeatable process that turns ambiguity into explicit requirements, numbers, contracts, data access patterns, and defensible trade-offs.

By the end of this lesson, you will be able to:

  • clarify a vague prompt without spending the whole discussion asking questions;
  • turn desired qualities into measurable requirements;
  • use estimates to identify likely bottlenecks;
  • define APIs, events, and data models around invariants and access patterns;
  • draw a simple end-to-end architecture before scaling it;
  • organize deep dives around risk rather than random technology;
  • present alternatives, failure behavior, and evolution clearly.

The complete framework#

Use these eight stages:

1. Clarify product and scope
2. Define measurable requirements
3. Estimate load and growth
4. Design APIs and event contracts
5. Design data around access patterns and invariants
6. Draw the simplest end-to-end flow
7. Deep-dive bottlenecks, failures, and operations
8. Compare trade-offs and state the evolution plan
Architecture canvasThe repeatable system design flowAmbiguity is reduced into requirements, contracts, an end-to-end design, and explicit decisions.
The repeatable system design flowAmbiguity is reduced into requirements, contracts, an end-to-end design, and explicit decisions.shape designClarifyRequirementsEstimateContracts + dataSimple flowDeep divesDecisions
Diagram key and text version
  • Entry
  • Process
  • Data
  • Outcome
  1. Clarify → Requirements
  2. Requirements → Estimate
  3. Estimate → Contracts + data
  4. Contracts + data → Simple flow: shape design
  5. Simple flow → Deep dives
  6. Deep dives → Decisions

The stages are ordered but iterative. If estimation reveals petabytes of media, revisit retention and product scope. If the data model reveals a money invariant, strengthen the consistency requirement and API idempotency contract.

Step 1: clarify product and scope#

Ask only questions whose answers change the design. Group them so the discussion remains efficient.

Users and workflows

  • Who produces data, and who consumes it?
  • What are the two or three critical workflows?
  • Is the workload interactive, background, streaming, or mixed?
  • Does one action fan out to many recipients?

Boundaries

  • Which features are required now?
  • Which attractive features are explicitly out of scope?
  • Are authentication, billing, moderation, or analytics external systems?
  • Is this a new system or a migration with compatibility constraints?

Data and correctness

  • What data must never be lost?
  • Which operations require ordering, uniqueness, or immediate visibility?
  • Can conflicts occur, and who resolves them?
  • How long is data retained or deletable?

Environment

  • What regions, devices, protocols, or dependencies exist?
  • Are there privacy, residency, compliance, or tenant-isolation constraints?
  • What team will build and operate it?

Do not ask “what is the expected scale?” and stop. If no number is supplied, make a reasonable assumption, label it, and proceed. Design discussions test reasoning under uncertainty.

Try it yourself

Clarify “design a notification system”

Write five high-impact clarification questions. For each, explain one architecture decision the answer could change.

Reveal solution

Useful questions include:

  1. Which channels—push, email, SMS, in-app? This changes providers, cost, payloads, and delivery feedback.
  2. Transactional, marketing, or both? This changes latency, consent, priority, and retry policies.
  3. What delivery guarantee is required? This changes persistence, deduplication, and reconciliation.
  4. How quickly must a transactional notification arrive? This shapes queue priority, regional placement, and SLOs.
  5. Can users configure quiet hours and channel preferences? This introduces preference reads, scheduling, time zones, and compliance behavior.

Step 2: define measurable requirements#

List functional requirements first, then attach the important quality requirements to specific operations.

Weak:

The system should be fast, scalable, and reliable.

Useful:

Accept 99% of high-priority notification requests within 200 ms at peak load.
Do not lose an accepted transactional notification during one worker failure.
Deliver 99% of eligible push notifications to the provider within 60 seconds.
Prevent a user opt-out from being bypassed by retries.

The third statement deliberately measures delivery to the provider, not to the human. Mobile devices, email servers, and telecom networks are outside the service’s full control. Define the boundary of every objective.

Prioritize instead of collecting wishes

Classify requirements:

PriorityMeaning
MustDesign is unacceptable without it
ShouldImportant, but degradable during constrained operation
CouldUseful extension after the core path works
OutExplicitly excluded from this design

This prevents an unbounded feature list and creates an honest degraded mode.

Step 3: estimate load, storage, and skew#

Estimate enough to choose the architecture’s shape:

users × actions/user/day → actions/day
actions/day ÷ 86,400 → average requests/second
average × explicit peak factor → peak requests/second
writes/day × bytes/write × retention → logical storage
responses/day × bytes/response → transfer volume

Then ask what averages hide:

  • one tenant sending a campaign to 20 million recipients;
  • a breaking-news event creating a synchronized peak;
  • a celebrity producing a hot fan-out key;
  • a retry storm multiplying traffic after a provider slows down;
  • attachments dominating bytes while metadata dominates operations.

Suppose the notification system receives 500 million notification requests daily:

average ≈ 500,000,000 / 86,400 ≈ 5,787 requests/s
assumed 5× peak ≈ 29,000 requests/s

If one request fans out to an average of three channels, downstream work is closer to 1.5 billion deliveries per day. The product request rate is not the worker operation rate.

The dedicated next lesson will calculate capacity in depth. Here the objective is to connect numbers to components rather than decorate the design with estimates afterward.

Demand estimates are not capacity plans

The arithmetic above estimates demand. It does not tell you how many service instances, partitions, or workers to provision. That requires a measured safe throughput per unit at an explicit latency and error objective:

required units ≈ peak operation demand / measured safe throughput per unit
                 × redundancy and uncertainty headroom

“Safe throughput” comes from load tests and production service curves, not a vendor maximum. The capacity plan must also survive an unavailable zone or replica, deployments, uneven traffic, downstream quotas, and the delay before new capacity becomes useful.

Keep three artifacts separate:

  1. an assumption sheet that converts product behavior into operation demand;
  2. benchmarks or service curves that show how one unit behaves under that demand;
  3. a capacity plan that adds redundancy, headroom, and scaling lead time.

This separation makes uncertainty visible. If a benchmark changes, the product assumptions do not need to be rewritten; if user behavior changes, the hardware measurement remains evidence rather than guesswork.

Step 4: define APIs and events as contracts#

APIs expose the product’s invariants and failure behavior. Define:

  • method and resource semantics;
  • request and response shape;
  • authentication and authorization boundary;
  • idempotency and retry behavior;
  • pagination and filtering;
  • error representation and rate limits;
  • sync versus async acknowledgement.

Example command:

POST /v1/notifications
Authorization: Bearer <token>
Idempotency-Key: order-481-receipt-v1
Content-Type: application/json

{
  "recipientId": "user-73",
  "template": "order-receipt",
  "channels": ["email", "push"],
  "data": {"orderId": "481"},
  "priority": "transactional"
}
HTTP/1.1 202 Accepted
Content-Type: application/json

{"notificationId":"n-901","status":"accepted"}

202 Accepted means processing has begun or been queued, not that delivery succeeded. Expose a status resource or event if callers need the outcome.

Events need contracts too

An event such as NotificationRequested should include a stable event ID, schema version, occurrence time, producer, tenant, trace context, and deduplication identity when applicable. Define whether consumers can receive duplicates and how ordering is scoped.

Step 5: design data from access patterns and invariants#

Start with questions the system must answer:

  • fetch notification status by ID;
  • list recent notifications for one user;
  • find pending deliveries ready for retry;
  • read channel preferences for a user;
  • prove whether a user had opted out at decision time;
  • aggregate provider success and latency by channel.

Then separate authoritative and derived data:

Notification(id, tenant_id, recipient_id, template, priority, state, created_at)
Delivery(id, notification_id, channel, attempt, state, next_attempt_at, provider_id)
Preference(recipient_id, channel, allowed, quiet_hours, version, updated_at)

Indexes follow access patterns, for example (recipient_id, created_at) for a user history and (state, next_attempt_at) for retry scheduling. Analytics aggregates belong in a derived analytical store rather than the transactional write path.

State the invariants:

  • an idempotency key creates at most one logical notification for a tenant;
  • opt-out is checked before each marketing delivery attempt;
  • state transitions cannot move from terminal success back to pending;
  • provider callbacks are deduplicated by provider event identity.

Database choice comes after these requirements. “Use NoSQL for scale” is not a data model.

Step 6: draw the simplest end-to-end flow#

Begin with the critical path and one source of truth:

Architecture canvasAsynchronous notification deliveryThe request is durably accepted before channel workers contact external providers and record outcomes.
Asynchronous notification deliveryThe request is durably accepted before channel workers contact external providers and record outcomes.requestpersistpublish workattemptoutcomeProducerNotification APISource of truthDurable queueChannel workerProviderDelivery status
Diagram key and text version
  • Entry
  • Process
  • Data
  • Async
  • External
  1. Producer → Notification API: request
  2. Notification API → Source of truth: persist
  3. Source of truth → Durable queue: publish work
  4. Durable queue → Channel worker
  5. Channel worker → Provider: attempt
  6. Provider → Delivery status: outcome

Explain one request:

  1. API authenticates the producer and validates the request.
  2. It applies the idempotency contract and persists accepted work.
  3. Durable work becomes available to channel workers.
  4. A worker checks current preferences and attempts the provider call.
  5. The outcome updates delivery state and metrics.
  6. Retriable failure receives bounded backoff; terminal failure does not loop forever.

Only now add components for named problems: priority queues for latency separation, scheduler storage for quiet hours, provider-specific circuit breakers, or regional routing for latency and residency.

Step 7: deep-dive the highest risks#

Choose two or three deep dives based on the design, not a standard list.

Throughput and backpressure

What happens when producers create 30,000 requests per second but an email provider accepts 10,000? The queue absorbs a bounded burst, not infinite overload. Monitor queue age, apply per-tenant quotas, shed optional work, and define maximum retention.

Retry safety

Classify provider errors, use exponential backoff with jitter, cap attempts, and send terminal cases to a review/dead-letter path. A worker crash after a provider accepted a request but before local acknowledgement can create a duplicate. Provider idempotency keys or reconciliation reduce that risk.

Preference correctness

Caching preferences improves throughput but risks delivering after opt-out. Marketing communication may require stronger freshness than a low-value in-app recommendation. Define invalidation, maximum staleness, and audit evidence.

Hot tenants and fairness

One campaign must not starve every transactional receipt. Use priority and tenant-aware scheduling, quotas, and isolated provider budgets where necessary.

Observability and recovery

Track acceptance latency, queue age by priority, attempt rate, provider latency/error codes, retry counts, duplicate suppression, preference rejection, and end-to-end delivery latency. Write runbooks for a slow provider, poison payload, growing retry queue, and regional loss.

Step 8: compare trade-offs and evolution#

Use a compact decision record:

Decision: acknowledge after durable persistence, then deliver asynchronously.
Reason: provider latency and availability must not block producer requests.
Cost: callers receive acceptance rather than final delivery; status becomes a state machine.
Alternative: synchronous provider call.
Choose alternative when: volume is tiny and the caller truly requires immediate provider outcome.
Failure behavior: queue growth triggers quotas and shedding for optional traffic while transactional work keeps priority.
Evolution trigger: sustained queue age or throughput approaches the tested operating limit.

Avoid declaring one option universally best. A trade-off includes the conditions under which the alternative becomes preferable.

The reusable design worksheet#

Copy this structure into every design note:

# Design: <system>

## 1. Scope
Users / critical workflows / in scope / out of scope

## 2. Requirements
Functional / latency / availability / durability / consistency
security / privacy / observability / cost / degraded mode

## 3. Estimates
DAU / actions / average and peak QPS / reads:writes
item sizes / retention / storage / bandwidth / skew

## 4. Contracts
APIs / events / auth / idempotency / pagination / errors

## 5. Data
entities / invariants / access patterns / indexes
partition keys / retention / authoritative vs derived

## 6. High-level flow
one request or event from entry to durable result

## 7. Deep dives
top bottlenecks / failures / recovery / operations / security

## 8. Decisions
choice / reason / cost / alternative / evolution trigger

Try it yourself

Apply the worksheet to file upload

Sketch a file-upload design using only the worksheet headings. Include one API, one data invariant, a rough scale assumption, the simplest request flow, and two deep dives.

Reveal solution

A strong outline separates file bytes from metadata. The client requests an upload session, then uploads directly to object storage using a scoped URL. Metadata records the owner, expected size/checksum, upload state, and final object key. A completion step verifies the object before marking it available. One invariant is that an object is downloadable only after verified completion. Deep dives should cover resumable multipart uploads and orphan cleanup, plus download bandwidth/CDN behavior. If files average 10 MB and 2 million arrive daily, logical growth is roughly 20 TB/day before replication and derived variants—enough to justify object storage and lifecycle policy immediately.

Common framework failures#

  • Requirements without targets: “fast” cannot guide capacity or testing.
  • Numbers without consequences: estimates are useless when no decision changes.
  • Technology before access patterns: storage choice becomes fashion rather than fit.
  • Only the happy path: retries, slow dependencies, duplicates, and recovery remain undefined.
  • Deep-diving everything: the main risks disappear inside a component encyclopedia.
  • No authoritative state: cache, queue, search, and database all appear to be truth.
  • Trade-offs as slogans: “consistency versus availability” replaces actual failure behavior.
  • No evolution trigger: the design is either prematurely huge or unable to grow.
What should determine the first high-level architecture?

Why return 202 Accepted for asynchronous notification creation?

Study optional flashcards

Invariant

A condition that must remain true across state transitions.

Access pattern

A query or write shape the data model must support.

Authoritative data

The source whose state defines truth for a decision.

Derived data

Rebuildable or asynchronously maintained data created from authoritative state.

Idempotency

Repeating a logical request does not repeat its intended effect.

Backpressure

Mechanisms that prevent producers from overwhelming constrained consumers.

Evolution trigger

Measured evidence that justifies changing the current architecture.

Memory chart#

Clarify → Requirements → Estimate → Contracts
        → Data → Simple flow → Deep dives → Decisions

Ask questions that change the design.
Attach targets to important operations.
Make assumptions and calculate consequences.
Model invariants and access before choosing storage.
Explain one request end to end.
Deep-dive the largest risks.
State cost, alternative, failure behavior, and evolution trigger.

The next lesson turns scale assumptions into concrete calculations for QPS, peaks, storage, bandwidth, cache working sets, and hot-key risk.

References#