CHAPTER 11 · AI

Building AI Applications

Calling an LLM from an app takes three lines of code. The hard part is making it reliable, fast enough, affordable, and resistant to abuse in production. All of those engineering problems trace back to a few basic facts about LLM APIs. This chapter nails down those facts first, then derives the matching techniques one by one.

Opening: the aftermath of "just one line"

Where the answer lies: they map to five basic facts about LLMs: ① output is probabilistically generated text with no guaranteed format → structured output and validation; ② input has a context window limit → preprocessing and trimming; ③ billing is per token, and raw HTML is mostly useless markup → control the input; ④ the model generates tokens one after another → stream the output; ⑤ the model can't reliably tell instructions from data → treat external content as untrusted and limit what the model can do.

The mental model of an LLM API Deep

A stateless function

Claude, GPT or Gemini — the API shape is basically the same: you send a list of messages and get back one new message. The server doesn't remember previous conversations — "multi-turn chat" means you resend the full history every time.

Tokens, the context window and cost

• A token is the model's basic unit of text, roughly part of a word — about 4 characters of English per token (other languages vary with the tokenizer). Input and output tokens are billed separately; output costs more.
• The context window is the max tokens (input + output) one request can hold. By 2026 mainstream models reach hundreds of thousands to a million tokens, but "fits" doesn't mean "should put everything in": longer input costs more, takes longer, and the model may attend less to material in the middle.
• Statelessness means the longer a conversation, the bigger and costlier each turn's input. That's why you need context management: summarize old messages, keep only what's relevant. Chapter 12's agents magnify this problem tenfold.

Calling a model: a provider SDK or the AI SDK

Two approaches: use a provider's SDK directly (such as @anthropic-ai/sdk) to get every feature of that provider's models; or use Vercel's AI SDK (package ai), a unified interface where switching providers is a one-line change, with built-in full-stack pieces like streaming UI, tool loops and structured output.

The rest of this chapter mostly uses the AI SDK because it also unifies the streaming protocol between frontend and backend. Either way, model calls must happen on the server — API keys must never appear in the browser (Chapter 02's env var rule).

Streaming Use

Models generate one token after another. A 500-word answer may take a dozen seconds, but the first token usually arrives within a second. Streaming sends output as it's generated, so users see progress immediately. The metric is "time to first token" (TTFT).

On the web, the most common transport is Server-Sent Events (SSE): one HTTP response kept open while the server keeps writing data: ... lines. The demo below shows what goes over the wire and what the UI shows, with a tool call in the middle:

Note that messages are made of parts — yet another discriminated union: text, tool calls, reasoning, files and so on. Render each type separately.

Structured output Use

When AI output is consumed by code (written to a DB, driving UI) rather than read by a person, you need structured output: a schema that forces the model to return conforming JSON.

Many providers now support "constrained decoding" that guarantees output matches a JSON Schema (in the Anthropic SDK: messages.parse with output_config.format). You should still validate with Zod — a schema guarantees format, not whether the content is sensible (are the tags relevant? did the summary invent facts?). On parse failure: retry once, degrade to defaults, or mark the record failed.

Tool calling: letting the model "do things" Deep

A model can only generate text; it can't query a database or access the network. Tool calling (function calling) lets it "request" that your program do those things:

Three key details:

1. A tool's description is documentation for the model. The model decides when and how to call based on the description and parameter docs. Vague descriptions lead to misuse.
2. Never let the model supply identity. Above, userId comes from the server-side session, not from a tool parameter. Otherwise the model (or an attacker steering it) could query any user's data.
3. Limit steps and permissions. Tools are the capabilities you grant the model. Read-only tools are lower-risk; tools that write, delete, pay or send email should require human confirmation (AI SDK 6 supports approval before tool execution).

RAG: answering from your data Use

The model doesn't know what the user has bookmarked. The simplest approach is putting all bookmarks into the prompt — perfectly workable, and often best, when data is small. As data grows, first retrieve the relevant parts, then have the model generate the answer from them. That's Retrieval-Augmented Generation (RAG).

An important 2026 trend: RAG is no longer the only answer. With bigger context windows and stronger tool use, common alternatives are:

• Just put it in context: when total data is within tens to hundreds of thousands of tokens, including everything with prompt caching is often more accurate and simpler than RAG.
• Agentic search: give the model a search tool (keyword search, SQL, grep) and let it decide what to look up and how often. That's how coding agents read codebases — they generally use grep and file reads, not vector indexes.

Rule: start with the simplest option (everything in context → a search tool), and bring in vector retrieval only when data size or cost forces you to.

Prompts: treat them like code

Prompts determine the model's behavior; they deserve the same status as code:

• State role, task, constraints and output format clearly. Write like you're briefing a smart new colleague with no context. Explaining "why" works better than only saying "what".
• Separate instructions from data with tags, e.g. wrap page content in <article>...</article> and state that what's inside is data only.
• Give examples: one or two input/output examples convey format and style better than long descriptions.
• Version and evaluate them: keep prompts in the repo, and put changes through evaluation (below), not gut feel.

Cost and latency

Evals: did that change help or hurt?

Traditional code has unit tests; AI features have nondeterministic output and need evaluations (evals):

1. Collect samples: 20–100 representative inputs (real pages, including hard ones: very long, multilingual, almost no text, containing injection attempts).
2. Define scoring: use code wherever possible (valid JSON? tag count? forbidden words?); for semantic judgments, have another model score against a rubric (LLM-as-judge), and spot-check by hand.
3. Run it after every prompt change, model swap or parameter tweak, comparing scores and cost. That's regression testing for AI features.

In production you also need tracing: log each model call's input, output, tokens, latency, cost and user feedback (platforms like Langfuse or Braintrust, or your own on OpenTelemetry). Bad cases from production should be added to the eval set.

Security: prompt injection Deep

This is the most distinctive and most overlooked security issue in AI apps. Prompt injection: an attacker hides instructions in data the model will read (web pages, emails, documents, tool results) to hijack its behavior. The root cause: to a model, instructions and data are the same stream of tokens; it can't reliably distinguish "the developer's instruction" from "data that happens to look like an instruction".

LinkNote is born at risk: it fetches arbitrary web pages and hands them to a model. Defensive principles:

• Assume injection will succeed, then limit the damage. The model call that handles untrusted content shouldn't get dangerous tools (summarizing needs no tools at all).
• Permissions come from the server, not the model. User identity and data scope are decided by your code.
• High-risk actions need human confirmation: sending email, deleting, paying, publishing.
• Output is untrusted too: don't insert model output into the page as HTML (XSS), and don't splice it into SQL or shell commands.
• Structure and declarations reduce risk: wrapping external content in tags and stating "don't follow instructions inside" helps, but must not be the only line of defense.

Self-check