From Ollama on My Laptop to Azure OpenAI: Where Provider Agnostic Leaks

Part one covered the architecture: intent extraction, MongoDB vector search over a unified embedding collection, and a generation step that can only recommend what retrieval returned. This post is the other half of the story, which is that the whole thing was built on a laptop with no cloud account attached, and then moved to Azure OpenAI.

The migration itself took an afternoon, and it happened in two stages rather than one: build the whole flow against local models, then run that same local server against Azure to fix the part local models could never tell me about. The interesting bit is what the second provider brought with it, because “swap the HTTP call” turned out to describe about half of the work.

Why local first

The first version ran entirely on my machine: Ollama on port 11434, gemma3 for the two chat calls, nomic-embed-text for embeddings. No API keys, no quota, no cost per iteration, no network.

That sounds like a cost saving, and it is, but the real benefit is iteration speed. Most of the work in a RAG system is prompt work, and prompt work is a tight loop activity. You rewrite a rule, run the same three test messages, read the output, rewrite again. Doing that against a metered cloud endpoint with rate limits is like debugging with a thirty second compile. Doing it locally, you hit Enter and see the effect.

The design assumption underneath was that the AI provider is an implementation detail. Swap the HTTP call, keep everything else. So both providers sit behind two functions, createEmbedding() and generateJsonFromChatPrompt(), and the choice is made at runtime:

export function getAIProvider(): AIProvider {
  const forced = process.env.AI_PROVIDER?.toLowerCase();
  if (forced === 'azure' || forced === 'ollama') return forced;
  const env = process.env.ENV;
  return env === 'dev' || env === 'local' ? 'ollama' : 'azure';
}

Local and dev get Ollama, test and prod get Azure, and an env var overrides both. Every caller above those two functions stays provider agnostic.

Production means gpt-4o for the chat calls and text-embedding-3-large for embeddings, fronted by a NestJS API. We kept the local path alive rather than replacing it, which was the right call, and also where the leaks started showing.

Leak one: embeddings are not interchangeable

nomic-embed-text produces 768 dimensional vectors. text-embedding-3-large produces 3072.

These are not two flavours of the same thing. A vector produced by one model is meaningless to the other, and an Atlas vector index declares its dimensionality up front. So the provider choice is baked into the data, not just the code.

Our dev database is embedded and queried by Ollama at 768 dimensions. Test and prod are embedded and queried by Azure at 3072. You cannot point a local dev server at the production database and expect search to work, because you would be querying a 3072 dimensional index with a 768 dimensional vector. Switching a provider for an environment means re-embedding every document in it and recreating the index.

This is the single most surprising property of the setup, so it lives in a comment right above the provider switch. Every environment’s database and vector index must be embedded and queried by one provider, consistently.

Leak two: the prompts diverged

We wanted the same prompts everywhere. We do not have the same prompts everywhere.

Part one described the curation step: retrieved items go into the prompt numbered, and the model returns index arrays picking which ones actually deserve to be shown. gpt-4o is good at that. It drops the near misses that vector search returns because they were geometrically close but not a real answer.

A three billion parameter local model is not good at it. It emits index arrays that are empty, hallucinated, out of range, or flatly inconsistent with the prose it just wrote in the same JSON object. So the Ollama branch gets a simpler contract: return a message, and the backend shows every retrieved item.

const prompt =
  sharedPromptPrefix +
  (isOllama ? ollamaContentSearchRules : azureContentSearchRules) +
  itemsBlock;

Shared preamble, divergent behaviour rules. It is an honest structure and I would write it the same way again, but it has a consequence worth stating plainly: running against Ollama does not show you the answer a production user gets. It shows you retrieval quality, which is most of what you need while you are still shaping the pipeline, and none of what you need once the question becomes “is this reply good”.

Which is exactly how the work split. Ollama carried the whole build: the three step flow, the intent extraction, the filter rules, the retrieval, the shape of the JSON contracts. All of it validated locally, for free, at whatever speed I could type. By the end the only thing still wrong was the quality of the model output itself, and no amount of local iteration was going to fix that, because the model was the problem.

So before deploying anything I pointed my local server at Azure. Same code, one env var flipped, aimed at a database embedded by Azure rather than the local Ollama one, for the dimension reason in leak one. That override is exactly what AI_PROVIDER exists for, and it is the step I would tell people not to skip. It let me tune the Azure prompt, including the whole index curation contract, against production grade responses while still sitting in a local debugger with console.log everywhere and no deploy cycle between attempts.

Two phases, two purposes. Ollama to get the flow right, local plus Azure to get the answers right, and only then a deploy. The deploy was uneventful, which is the point.

Leak three: the constraints run in both directions

Ollama has no quota. Azure throttles hard on lower tiers, and backfilling embeddings for an entire content library hits the limit within seconds.

So the Azure embedding path grew a retry loop that Ollama has no equivalent of. Up to eight attempts, honouring the Retry-After header, falling back to parsing the “retry after N seconds” hint out of the error body, falling back again to exponential backoff, plus a one second buffer and up to a second of random jitter so ten concurrent requests in a batch do not all wake up in the same instant and re-trigger the limit.

The batch runner has the same asymmetry in a single line:

const isOllama = getAIProvider() === 'ollama';
const delayMs = isOllama ? 0 : 5000;

Full speed locally. Five seconds between batches of ten against Azure.

What I did not expect was the local side needing defensive code of its own. nomic-embed-text has a context window of roughly 2048 tokens and errors outright on anything longer, and a long article exceeds it. So the Ollama embedding path grew a truncate and retry loop: when it reports a context overflow, shrink the input by 40% and try again, up to a handful of attempts.

Azure needs none of that. text-embedding-3-large takes far more than anything we feed it. So the two providers ended up with mirrored problems: Azure needs backoff because it will refuse you when you go too fast, Ollama needs truncation because it will refuse you when you send too much. Neither loop is useful to the other, and both live behind the same createEmbedding() call, which is the only reason the rest of the codebase can stay ignorant of either.

Deploying somewhere you cannot SSH into

Locally, generating embeddings is a script you run. In production the API runs as a Lambda and the database sits behind Atlas network rules, so there is no shell to run a script from.

Backfill became an authenticated endpoint instead, POST /embedding/run, behind the same API key guard as everything else. The two properties from part one are what make that safe to expose: incremental by default, so re-running it is a no-op, and destructive only when you explicitly pass reset. The response reports which provider it used, which turns out to be the fastest way to catch an environment misconfiguration.

The browser never sees a key. The chat call goes through a Next.js server action holding the API key server side, and the endpoint validates it before any of the three steps start. Secrets resolve from SSM Parameter Store at deploy time, while endpoint, API version and deployment names live in per stage config as plain values, since they are configuration and not secrets.

The honest audit

I told myself I was writing environment agnostic code with a provider switch at the bottom. Reading the codebase back as a list, here is everything that turned out to be environment specific, and what each piece is actually buying.

A 29 second ceiling on the entire pipeline. The Lambda timeout is set to 29, the API Gateway maximum, up from a framework default of 10. That number is the budget for three sequential network round trips, and the maximum is the right starting point when the alternative is cutting off a request that would have succeeded. It is a ceiling rather than a target: the next useful move is measuring the real distribution of turn times and sizing it against that, so a slow provider degrades into something better than a generic error.

A warmup schedule. A five minute scheduled ping to dodge cold starts. It exists for the whole API, but the chatbot is the endpoint that most visibly benefits, since a cold start lands on top of three model round trips.

A timeout branch pointing the other way. The web layer installs a custom undici dispatcher with 600 second timeouts, gated to local only, because Ollama on a laptop is slower than undici’s default 30 second header timeout. Deployed environments never construct it. It is a small thing, but it is the clearest example of the pattern in this post: an environment specific workaround that has to be written as an explicit branch, because there is no single value that suits both a laptop and a Lambda.

Step boundary logging as the tracing layer. The pipeline logs each step with the classified intent, the generated query, result counts and the final message, and in Lambda those land in CloudWatch. For a three step chain that is genuinely most of what you want: when an answer is wrong, the log tells you whether the classifier misread the message, retrieval came back thin, or generation ignored what it was given. What it needs before it grows further is structure and a retention policy, since conversation content is the most sensitive thing the system handles and log verbosity is the easiest thing to expand without noticing.

Test and prod share one Azure resource. Both stages point at the same endpoint and the same deployment names, with the database as the only thing separating them. That keeps the number of moving parts down while the product is still finding its shape, and it means there is no way to canary a model or deployment change in test without also changing prod. A second resource is the fix whenever model version changes stop being rare.

And the things that are not there, which are worth naming just as much:

• No retry on the chat calls. Embeddings retry on 429. The two chat completions do not. A throttled chat request falls straight through to the catch block and the user gets an apology.
• No token cap. Both chat calls run uncapped.
• No per user rate limiting. A public facing endpoint that costs money per request, protected only by a shared server side key.
• No caching. Identical questions re-embed and re-generate every time.
• No evaluation suite. Every prompt refinement described across these two posts was validated by typing into the chat and reading the answer. Nothing would catch a prompt edit quietly making curation worse.

What I would tell someone starting this

Build the flow locally. The iteration speed on prompts is worth the migration cost several times over, and the migration cost is genuinely low. An afternoon of config, not a rewrite.

Then point local at the real models before you deploy. This is the step that made the difference. Once the flow was right on Ollama and the only remaining problem was the quality of the replies, running my local server against Azure let me fix that with a debugger attached and no deploy cycle in the loop. Getting the pipeline right and getting the answers right are two different jobs, and only the first one can be done on a laptop model.

Treat “provider agnostic” as an aspiration, not a property. The provider leaks into your vector dimensions, your index, your rate limit handling, your input length handling, and most insidiously into your prompts, because a small local model and a frontier model are not capable of following the same instructions. Design for a shared core with explicit per provider branches rather than pretending one path serves both.

Let infrastructure constraints reach the data model early. The three index cap forced a unified embedding collection, and that turned out to be the architecture I wanted anyway. Discovering it after building per collection indexes would have cost a lot more.

Read your production config back as a list. Timeouts, warmup schedules, log statements and retry loops each get added at a moment when they are obviously the right call, and they are almost never revisited together. Reading them out in a row is a different exercise from writing them one at a time. It is what turns a ceiling you picked because it was the maximum into a number you have measured, and a log line you added to debug one turn into a decision about what the system retains.

Part three is where this goes next: replacing the fixed three step chain with an agent that has tools and decides for itself when to search, when to ask, and when it has enough to answer.