Designing for Production: Where AI Systems Actually Break

Part six of the AI foundations series. Last time was the build/buy/rent decision. This post is the other two decisions: architecting for real load, and handling the specific ways models fail.

The course started from the version everyone writes first. One request goes to the model, one comes back. It ships quickly and it works, until a launch or a seasonal spike hits rate limits. Then requests start returning 429s, timeouts climb, users wait.

Most of this section was new to me, so what follows is the course’s path from that naive call to something that holds up, plus the parts I expect to reach for first.

Shorten the user-facing path

The first change: keep user-visible work short. Move slow model calls off the web request using a queue and a worker. The page responds immediately; the worker finishes the model call in the background.

This isn’t optional at AI latency. Your web endpoint expects sub-second responses. A model might need 20 seconds. Leave that on the request path and the user hits a web tier timeout while the model finishes in the background with nowhere to deliver the result.

Keep timeouts short on the web tier and longer on the worker tier, so each layer behaves as intended.

And enable streaming when the API supports it. Time to first token matters enormously, users watching output appear tolerate far more total latency than users watching a spinner.

Then make the dependency safe

With the path short, add the pieces that keep it stable:

Circuit breaker: a switch that stops calling a service that keeps failing, and fails fast while it’s unhealthy. Without it, threads pile up waiting on timeouts and the failure spreads into your own system.
Caching: sits in front of the model so common questions don’t trigger identical compute. The course put this forward as the single highest-leverage cost lever, with an e-commerce example that cached product description enhancements and cut costs nearly in half.
Retries with backoff and jitter: random delays so clients don’t all retry simultaneously and manufacture a second spike.

If I only get to do two of these on a given project, I think it’s these: move the slow call off the request path, and put a circuit breaker in front of it. Caching next.

Processing patterns evolve in a predictable order

The patterns were laid out in the order teams tend to hit them, which I found more useful than a menu of options. It starts sequential, one request at a time. Simple, easy to reason about, and correct until requests start overlapping and queue depth grows.

Then parallel. When you go concurrent, make repeated writes safe: if the same action happens twice, data shouldn’t duplicate or corrupt. Put shared state behind clear guards, transactions, single writer per key, or a small queue so only one update happens at a time.

Watch the right signal here. If CPU rises but throughput stays flat, you’re contending, not scaling.

Then distributed, but don’t just add servers randomly. Split work into logical groups, by customer, document type, or region, so each group operates independently. The technical term is partitioning, and the key is keeping related work on the same partition. Otherwise you’re reaching across the network to complete a single request, which defeats the point. Think dedicated checkout lanes rather than one crowd.

Running alongside all of this is a question I want to remember to ask: who’s waiting? If a person is watching a screen, keep it real time and short. If nobody’s watching, meaning reports, analytics, backfills, batch it on a schedule. Batch is roughly 10x cheaper but adds latency, and the course’s line was that teams start real-time, get the bill, and then redesign for batch.

Moving work off the user path isn’t giving up. It’s spending your latency budget where it’s noticed.

The failure cliff

Systems don’t degrade in a straight line. This caught my attention because it reframes capacity planning entirely.

At 80% capacity, everything looks healthy.
At 90%, you feel some delay.
At 100%, the queue backs up, timeouts fire, retries add traffic, and the system tips over.

It’s not a slope, it’s a cliff. When requests arrive faster than you complete them, backlog grows exponentially, and every safety net you added (timeouts, retries, autoscaling) can push you further over instead of pulling you back.

AI makes the cliff steeper. Models claim large amounts of memory just to load, even a 7B parameter model wants 15–20GB before serving a single request. Add buffers, caches, and a backlog and memory pressure builds fast. The runtime starts spending more time managing memory than working. Pauses lengthen, more requests wait, memory climbs again. It does not fix itself.

The honest response is to shed load deliberately. Cap concurrency to what memory actually supports. It’s better to return “try again shortly” than to accept work you can’t complete. Both are frustrating, but one tells the user to wait and the other invites them to refresh forever.

Storage and network have their own cliffs, verbose logging, model checkpoints nobody prunes, cached embeddings growing with every document. A response that’s a few kilobytes in testing becomes megabytes in production once you include alternates and traces; multiply by concurrent users and the internal network saturates. Everything slows down and no code changed.

Practical guards: cap queue length, set hard limits on logs and caches and enforce them automatically, and watch free memory and disk alongside latency.

Failures that don’t look like failures

This is the genuinely AI-specific category, and it’s where traditional integration instincts mislead you.

Output format drifts. Ask for plain text and Markdown shows up, because the model learned those patterns. Ask for JSON with answer and sources and you might get numbers as strings, an extra field, or the whole thing wrapped in backticks. Your code didn’t change; the behavior did. The contract can shift while you’re using it.

When a database returns malformed data, that’s a bug. When a model returns a slightly different shape, that’s normal variation. You’re not fixing errors, you’re handling expected variance.

A workable validation ladder:

Try to parse. If it succeeds, done.
If it fails, check for code-block wrapping and strip it.
Check common issues, single quotes, trailing commas, unescaped characters.
If it still won’t parse, stop trying. Take the fallback path: a generic response, a simpler model, or a human review queue.

The point isn’t to handle every case. It’s that every case has a planned destination.

Some things that reduce the variance in the first place: use the provider’s JSON mode when offered, built-in constraints beat asking politely in a system prompt. Pin model versions where you can. Where you can’t, send a test prompt at startup and log when the format drifts. Normalize encoding, timezones, and units at the edge before they become bugs.

Keep a small set of test prompts and run them daily. If responses start changing shape, you see it in your morning check instead of in customer complaints.

Shared rate limits. If multiple services share one API key, a batch job can burn the quota and cause 429s for interactive users. Split keys and quotas by use case, or put a single limiter in front of the provider that enforces your priorities. Decide in advance which traffic wins when you’re near the limit.

Data path multiplication. One document becomes many chunks, chunks become tokens, tokens become embeddings. Counts and payload sizes multiply, and queues sized for small web events start to struggle. Set hard caps early, max document size, max chunks per document, max tokens per request, and add backpressure so workers slow their intake instead of crashing.

Duplicate work. A browser refresh, a retry, or a background job can schedule the same task twice. Tag work with a stable request ID or content hash and treat repeats as the same job. The same write happening twice shouldn’t change the result, and you shouldn’t pay for it twice.

Don’t let the slowest component hold the page. Load the base page immediately and fill in AI sections as they’re ready, using server-sent events or websockets. If a result is late, show a basic or cached answer rather than blocking the whole surface.

Four metrics, not fifty graphs

The claim here was that teams monitor too much and still learn too little: 50 graphs, 20 alerts, and no idea why something broke. The proposed answer is a much smaller set chosen to tell the truth.

Metric	What it catches
P95 latency at the user boundary	Provider trouble. Averages look fine while 5% of users wait too long
User-visible errors (not internal retries)	What people actually experience
Cost per request	Prompt bloat, context creeps until you’re paying 10x per query
One quality signal (confidence distribution, lightweight accuracy check)	Slow degradation. Answers look fine at a glance but get less useful week over week

Add queue depth / backlog age if you have a data pipeline, it tells you the pipeline is falling behind before users notice.

Why P95 and not the average: if P95 is 200ms but P99 is 2,000ms, one request in a hundred takes ten times longer than typical. At thousands of requests per minute, that’s dozens of frustrated users every hour, invisible in your mean.

Degrade gracefully, on purpose

Design levels, like a phone’s low battery mode:

Full service when everything is healthy.
Reduced service under pressure, slower, simpler model, still functional.
Minimum viable service when things are bad, basic features only, but the product works.

Map these while you’re calm. If the advanced model is unavailable, fall back to a simpler one. If real-time processing is overloaded, offer batch with a wait. If the AI feature is completely down, the core product should still function. Users prefer limited features to an error page.

Feature flags help enormously here, because the fastest fix is always the one that doesn’t require a deploy.

One exercise worth doing on paper: a 10x tabletop. At ten times current traffic, what breaks first, database connections, provider rate limits, queue depth, memory? At 10x data volume, do storage and indexes keep up? At 10x cost, is this still a viable product? For each breaking point write two things: the immediate response, and the longer fix that needs an architectural change.

The other suggestion was a one-pager called When AI Goes Wrong: symptoms, checks, actions, rehearsed quarterly. I have not written one yet. I am noting it here mostly so that I do.

Choose boring technology

The last piece, and the one I suspect is most underrated. Use queues you know, databases you trust, frameworks your team understands. The exciting vector database might be powerful, but the boring Postgres you already run may well be good enough. The cutting-edge model might be more capable, but the established one has known behavior.

The argument that landed for me is that a clever solution saving 10% on compute is a bad trade against being able to understand your own system while it is on fire.

Boring scales. Boring is debuggable. Boring lets you sleep.

Next: what changes once the system is live, monitoring, drift detection, lifecycle management, and keeping humans meaningfully in the loop.