CHAPTER 14 · MASTERY
System Design & Interviews
System design isn't reciting model answers; it's a structured way of thinking: start from requirements and orders of magnitude, derive an architecture, and state clearly what each choice costs. Interviews test this skill, and day to day it decides whether you can review the designs agents propose.
- Run a system design with a five-step framework and control the pace of the discussion.
- Estimate orders of magnitude — especially token costs of AI features — and find the design pressure from them.
- Design LinkNote for a million users and dive into dedupe, queues, rate limiting, search and realtime updates.
- Answer common TS full-stack and AI interview questions with outline frameworks.
Opening: design LinkNote for a million users
The interviewer says: "Design a bookmarking service with AI summaries for one million registered users." You have 45 minutes. What should your first sentence be? And what will turn out to be the biggest design pressure in this system?
Where the answer lies: your first sentence should be a question, not a box diagram: what are the core features? What's the read/write ratio? How fast must summaries appear? Then do a rough estimate. The estimate reveals a counter-intuitive conclusion: in this system, the biggest pressure isn't the database; it's LLM cost — and the most effective fix is deduplicating by URL (many people save the same popular articles). The whole design unfolds around that discovery.
A five-step framework
- Clarify requirements (5 min): functional (what it does) + non-functional (scale, latency, availability, consistency, cost, security). Say what's out of scope.
- Estimate (5 min): users → requests (QPS) → storage → bandwidth → AI cost. Orders of magnitude only.
- High-level architecture (10 min): draw the main components and data flow; get the whole thing working first.
- Deep dives (15 min): pick 2–3 of the hardest parts: data model, bottlenecks, failure handling.
- Trade-offs and evolution (5 min): state the main trade-offs and what you'd change first at 10× scale.
The same framework works for reviewing an agent's design: did it clarify requirements? Did it estimate? Are its "key points" really the key points?
Step 1: clarify requirements
Functional
• Save links; auto-generate summaries and tags
• List, filter by tag, full-text and semantic search
• "Ask my bookmarks" chat
• Out of scope: social sharing, collaboration, browser extension (this time)
Non-functional
• 1M registered users, ~10% daily active
• Save returns within 1s; summary within 30s is fine
• List page P95 < 300ms
• Strict per-user data isolation; brief summary delay acceptable (eventual consistency)
Step 2: estimate Deep
| Metric | Calculation | Result |
|---|---|---|
| Daily active users | 1M × 10% | 100k |
| Writes (saving links) | 100k × 3/day = 300k/day ÷ 86,400 | ~3.5/s average, ~35/s at a 10× peak |
| Reads (list, search) | 100k × 20/day = 2M/day | ~23/s average, ~230/s peak |
| Storage | ~2KB per row (with summary) × 300k/day × 365 | ~220 GB/year (excluding page snapshots) |
| LLM input | 300k × ~3,000 tokens (cleaned text) | ~900M tokens/day |
| LLM output | 300k × ~300 tokens | ~90M tokens/day |
At an example mid-tier price ($2 per million input tokens, $10 per million output): input ≈ $1,800/day, output ≈ $900/day — about $2,700 a day, roughly $80k a month. By comparison, a few dozen database reads and writes per second are no pressure for any decent Postgres.
Estimates needn't be precise; their job is to find where the design pressure is. Here the conclusion is: the database is an "ordinary" problem; LLM cost is the "core" one. That decides the deep-dive focus — summary dedupe, input cleaning, model choice, batching. Without the estimate, you might spend 15 minutes on database sharding, which isn't the bottleneck at all.
Step 3: high-level architecture
This diagram maps onto any of Chapter 09's deployment paths: on Cloudflare, the app service is a Worker, the queue is Queues, the cache is KV, rate limiting can use Durable Objects; on Vercel, the app is Functions and the DB and Redis come from the Marketplace; self-hosted, they're Docker containers. The shape of the architecture doesn't depend on the platform; the platform only decides what implements each box.
Step 4: deep dives
① Summary dedupe: a content-addressed cache
Normalize the URL (lowercase host, strip tracking params like utm_*, drop fragments, unify trailing slashes) and hash it as the key. Summaries live in a user-independent page_summaries table; a user's bookmark just references it. Popular articles get summarized once. If 60% of saves hit the cache, LLM cost drops straight to 40%.
Details to discuss: how long the cache lives (pages change); how to avoid generating 100 times when 100 people save the same URL simultaneously (lock or dedupe by URL hash in the job layer, a.k.a. "request coalescing"); how personalization (a user wanting English summaries) becomes part of the cache key.
② Queues and failure handling
- At-least-once delivery + idempotent consumers: queues may redeliver; workers check by link id and status whether it's done, and write with upserts.
- Retry policy: exponential backoff for network errors and LLM 429s; no retries for 404 pages or empty content — mark failed immediately.
- Dead-letter queue (DLQ): jobs that fail repeatedly go to a DLQ for investigation instead of burning money on endless retries.
- Backpressure: LLM providers rate-limit. Worker concurrency should be capped and the queue absorbs peaks.
③ Rate limiting and abuse
Free users get at most N saves per day and M chat messages per minute. Implement with a token bucket or sliding window, with counters in Redis / KV or a per-user Durable Object. For AI features, limit cost (tokens), not just request counts.
④ Search
Two needs: keyword search ("react compiler") and semantic search ("how to make components render less"). Postgres can do both: a full-text index for keywords, pgvector for vector similarity, with results fused (hybrid retrieval). Every query must filter by user_id — easy to forget in vector search, leading to cross-user data leaks. As data keeps growing, consider a dedicated search service.
⑤ Realtime status updates
| Option | How | Fits |
|---|---|---|
| Polling | Client asks for status every few seconds | Simplest; plenty for a "within 30s" summary |
| SSE | One-way server push over plain HTTP | Status notifications, LLM streaming |
| WebSocket | Two-way, long-lived connection | Chat, collaborative editing; on serverless needs Durable Objects or a dedicated service |
In an interview, saying "polling first, because the requirement allows it; switch to SSE if X happens" shows more judgment than jumping straight to WebSockets.
Step 5: trade-offs and evolution
| Decision | Choice | Cost / when it changes |
|---|---|---|
| Monolith vs microservices | Monolith + separate workers | Split when teams or deploy cadences clearly diverge |
| SQL vs NoSQL | Postgres | Clear relations, transactions, flexible queries; writes are nowhere near a bottleneck |
| Sync vs async summaries | Async | Users see a "generating" state; in exchange, stable write latency and retryability |
| Summary consistency | Eventual | Seconds to tens of seconds of delay is acceptable |
| Model choice | Tiered by task: a cost-effective config for summaries, a stronger one for chat | Needs an eval set to guard quality; can vary by user tier |
At 10× scale: LLM cost is still pressure #1 → more aggressive caching, batch APIs, model tiering; DB read load → read replicas, list caching; search → a dedicated service; multi-region → distribute data by region, considering compliance.
Issues specific to AI system design
- Cost model: per-call cost × call volume for each feature, written into the design doc. Set budget alerts.
- Latency budget: LLM calls on the synchronous path should stream; keep them off the synchronous path unless necessary.
- Degradation and failover: when the model provider is down or rate-limiting, switch to a backup model, queue, or show "summary temporarily unavailable"?
- Quality: eval sets, production sampling, user feedback (thumbs up/down) flowing back into evals.
- Security: prompt injection (fetched pages), cross-user isolation (filter at retrieval), output escaping.
- Data and privacy: which user data goes to third-party models? Is consent needed? How long is it kept?
Common interview questions and answer outlines
Grouped by area. Expand to see the key points — note they're outlines; in a real answer, bring in your own experience and examples.
JavaScript / TypeScript
What is the event loop? Tasks vs microtasks?
JS runs on one thread; after sync code finishes, drain the microtask queue (Promise callbacks, code after await), then take one task (timers, I/O, events). Microtasks are fully drained after every task, so Promise callbacks always run before setTimeout. Give and trace an output-order example (Chapter 03).
Promise.all vs allSettled vs race vs any, and when to use each?
all: succeeds only if all succeed, fails fast; allSettled: waits for all, reports each; race: first to settle decides; any: first to succeed decides. Use allSettled when batches may partially fail, and mention concurrency limits.
Do TypeScript types exist at runtime? What follows from that?
No, they're erased. So: external data must be validated at runtime (Zod); `as` checks nothing; you can't branch on types at runtime — use a discriminant field.
interface vs type? any vs unknown vs never?
interface supports declaration merging and describes objects only; type can express unions, tuples, mapped types, etc. any disables checking; unknown must be narrowed before use; never is an impossible value, used for exhaustiveness checks.
What's a closure? What problem can it cause in React?
A function captures variables from the scope where it was created. In React each render's state is a snapshot; callbacks or effects that captured an old render's values produce "stale closures" — e.g. a timer that always reads the old count. Fixes: functional updates, correct dependency arrays, useEffectEvent / refs.
React / frontend
How does React rendering work? What causes a re-render?
setState → schedule an update → re-run component functions (render) → diff old and new element trees (reconciliation) → apply DOM changes (commit). Triggers: own state change, parent re-render, consumed Context change.
When is useEffect appropriate? Common misuses?
Syncing with external systems: subscriptions, WebSockets, browser APIs. Misuses: computing derived data, responding to user events, hand-rolled data fetching, resetting state from props. Unstable references in the dependency array cause loops.
What are SSR, SSG, CSR and RSC? How do you choose?
Compare on "where rendered, when rendered, how much JS shipped"; RSC is about whether a component exists only on the server, complementary to SSR. Choose per page: static content SSG, personalized live data SSR/streaming, highly interactive dashboards CSR; discuss hydration cost and caching.
How would you speed up a page with a slow first load?
Measure first (LCP, network waterfall, JS size), then treat the cause: server rendering / streaming; code splitting and lazy loading; fewer client components; parallelize data fetching to avoid waterfalls; image optimization; CDN caching. Emphasize "measure first, then optimize".
Full-stack / backend
How do you achieve end-to-end type safety?
DB schema (Drizzle) infers row types → service return types inferred → server functions / tRPC / Hono RPC carry types to the client; inputs use Zod schemas for both validation and types. Limit: covers only code in the same compilation; external boundaries still need runtime validation.
Sessions vs JWTs? Where should tokens be stored?
Sessions: server keeps state, client holds an ID, easy to revoke; JWTs: self-contained, stateless verification, but hard to revoke early and larger. In browsers, prefer HttpOnly + Secure + SameSite cookies over localStorage; discuss XSS and CSRF defenses.
Serverless vs container deployment? What suits each?
Differences in lifecycle, state, cold start, execution time, billing (Chapter 09's table). Serverless suits request handling with variable traffic; containers suit long jobs, WebSockets, heavy compute; discuss the DB connection problem.
AI / agents
How do you get stable structured data from an LLM?
Structured output / constrained decoding + Zod validation + retry or degrade on failure; examples in the prompt; simplify the task; an eval set for regression testing.
Walk through RAG. When don't you need it?
Chunk → embed → store → embed the query → retrieve top-k → generate with citations; optionally hybrid retrieval and reranking. With little data, put it all in context + caching; agentic retrieval (a search tool) is another option.
What is an agent? How do you design a reliable one?
A model using tools in a loop until a goal is reached. Reliability: step and budget caps, least-privilege tools, human confirmation for sensitive actions, sandboxing, context management, observability, eval sets. First check whether a fixed workflow would do.
What is prompt injection? How do you defend against it?
Instructions smuggled in data hijack the model. Defense in depth: no dangerous tools when processing untrusted content; identity and permissions decided by the server; confirmation for risky actions; output escaping; isolation and auditing. Admit no perfect defense exists; the focus is limiting blast radius.
How do you reduce the cost of an LLM app?
Measure token usage per feature first; clean input; prompt caching; result caching and dedupe; match model and reasoning effort to the task; batching; cap output length and agent steps; confirm with evals that quality holds.
Self-check: a small design exercise
Design a "weekly digest email" for LinkNote: every Monday, email each active user a summary of what they saved last week. Give estimates, architecture points and the biggest risks.
Estimate: 100k active users, ~20 bookmarks each last week, ~200 tokens per summary → ~4,000 input tokens per email, ~400M input tokens per week. Use a batch API to cut cost since it doesn't need to be realtime.
Architecture: a cron job early Monday enqueues users in batches → a worker reads that user's existing summaries (reused, no re-fetching pages) → calls the LLM to write the digest → sends via an email service → records send status (idempotent: one email per user per week).
Risks: duplicate sends after batch failures (needs an idempotency key); deliverability and unsubscribe compliance; cost spikes (budget caps); injection content in a user's bookmarks (give digest generation no tools, escape output).