The moment you move an LLM from a chat toy to a real feature, you hit the same wall: you need structured data out of it — an object your code can consume — not a paragraph. The naive approach is to write "respond only with JSON" in the prompt and parse whatever comes back. That works right up until the model wraps the JSON in a code fence, adds a friendly preamble, or trails off mid-object. There's a better way, and it's built into the models now.
Three levels of "structured"#
Roughly in order of reliability:
- Prompt-and-hope — ask for JSON in plain English and parse the string. Cheap, works with any model, breaks unpredictably. Fine for a prototype, not for production.
- JSON mode / schema constraints — the API constrains generation so the output is guaranteed to be valid JSON, and (with a schema) to match a shape you supply. The model literally can't emit a malformed object.
- Tool calling — you describe a function with a typed parameter schema; the model responds by "calling" it with arguments that conform to that schema. This is the same mechanism that lets a model drive external systems, and it's the backbone of the Model Context Protocol.
For extracting data, reach for level 2 or 3. They differ mostly in framing: JSON mode says "fill in this shape," tool calling says "call this function." Under the hood both constrain the tokens the model may produce.
A concrete example#
Say you're extracting structured details from a support message. Define the schema once, hand it to the model, get a typed object back:
const schema = {
type: 'object',
properties: {
category: { type: 'string', enum: ['bug', 'billing', 'feature', 'other'] },
urgency: { type: 'string', enum: ['low', 'medium', 'high'] },
summary: { type: 'string' },
},
required: ['category', 'urgency', 'summary'],
} as const;
Because the API enforces the schema during generation, category will be one of your four enum values — not "Bug!" or "billing issue" or a paraphrase. That enum constraint is doing real work: it collapses the model's near-infinite phrasing into the exact set your switch statement already handles. The natural way to consume the result on the TypeScript side is a discriminated union, so each category branch exposes exactly the fields it should.
Validate anyway#
Here's the discipline that separates a demo from something you'd page yourself for: validate the output even when the API says it's constrained. Schema-constrained generation makes malformed JSON nearly impossible, but "nearly" is not "never" — network truncation, a model that predates the feature, or a schema the provider interprets loosely can all slip through. Parse the response through a runtime validator (Zod, Valibot, or a plain hand-written guard) at the boundary, exactly as you'd validate any untrusted input.
This isn't paranoia; it's the same trust-boundary hygiene you apply to a form submission or an API payload. The model is an external service. Treat its output like any other network response you don't fully control.
Streaming and structure together#
A common worry: "if I want structured output, do I lose streaming?" No — but you change what you stream. You don't stream a half-formed JSON object token by token to the UI. Instead you either stream the underlying text and parse once complete, or use a library that streams partial objects, filling fields as they arrive. If you're wiring up a chat-style feature and want the streaming half of the story, I walked through it in streaming AI chat with the Next.js AI SDK.
Where this fits in a larger system#
Structured output is rarely the whole pipeline. In a retrieval-augmented setup, you might use a structured call to turn a messy user question into a clean search query, run retrieval against your knowledge base, then make a second structured call to format the answer. Each LLM step produces a typed object the next step can rely on, which is what makes the pipeline debuggable instead of a black box.
The takeaway#
Stop parsing prose. Use JSON mode or tool calling so the model's output is shape-constrained at generation time, model the result as a discriminated union on the TypeScript side, and still validate at the boundary because the model is an external service. Do that and LLM output becomes just another typed value flowing through your app — predictable, testable, and safe to build on. If you're putting something like this together and want a second pair of eyes, reach out.
FAQ#
How do I get JSON output from an LLM reliably?#
Use the model's structured-output features rather than prompting for JSON in plain text. JSON mode with a schema constrains generation so the output is valid JSON matching your shape, and tool/function calling has the model return arguments that conform to a typed parameter schema. Both are far more reliable than asking for JSON and parsing the raw string.
What's the difference between JSON mode and tool calling?#
They're two framings of the same underlying constraint. JSON mode asks the model to fill in a response shape you define; tool calling asks the model to "call a function" with arguments matching that function's schema. For pure data extraction either works — tool calling additionally signals intent to invoke external actions.
Should I still validate schema-constrained output?#
Yes. Schema constraints make malformed output extremely unlikely, but not impossible — truncation, older models, or loose schema interpretation can slip through. Validate the parsed output at the trust boundary with a runtime validator, treating the model like any other external service.
Can I stream structured output?#
Yes, but you don't stream a partial JSON object to the UI. Either stream the underlying text and parse once it's complete, or use a library that emits partial objects as fields fill in. Streaming and structured output are compatible; you just choose what to surface as it arrives.
References: