From Pipeline to Agent: Giving the Chatbot Tools and Letting It Decide

Part one built a three step RAG pipeline: classify the message, vector search the content, generate a grounded reply. Part two took it from local models to Azure and audited what that cost.

This one is different. It is not a write up of something shipped. It is where the design is going, and why, because the ceiling on the current shape is visible from where I am standing and I would rather write down the reasoning now than reconstruct it later.

The short version: the pipeline decides everything up front, in one call, before it has seen a single piece of data. That is the wrong place for those decisions.

What the fixed chain gets wrong

The current flow is a straight line. One model call classifies and extracts, retrieval runs, a second model call writes the reply. Every message takes the same path, every time. Four problems follow from that, and they are all the same problem wearing different clothes.

Routing happens blind. The classifier picks which content types to search before anything has been retrieved. Ask “what can I do about food waste” and it has to guess, in advance, whether you want an event to attend or an article to read. It usually picks both, which is a way of not deciding. If it picks wrong, nothing downstream can recover, because retrieval already ran and the generation step can only work with what it was handed.

Clarification cannot happen before retrieval. Part one described the rule that stopped the bot hedging: if results exist, asking a question is forbidden. That rule is correct for a pipeline where search has already run by the time anyone can ask anything, and it is a constraint I want to remove rather than a rule I regret. A genuinely vague message (“I want to help the environment”) gets embedded, runs two vector searches, and comes back with eight things that are technically nearest, which is the best the shape allows. The better answer is one question asked before touching the database, and a straight line has nowhere to put it.

There is exactly one attempt at retrieval. If the extracted query was slightly off, that is the answer. A person searching would look at what came back, notice it was not right, rephrase, and search again. The pipeline cannot, because search happens once, in the middle, with no way back.

Every new content type touches everything. Adding products back into the flow means the classifier prompt, the search service, the response prompt, the result DTOs and the UI. The unified embedding collection from part one made the storage side cost nothing. The pipeline around it did not get the same treatment.

The reframing

Instead of a chain that always runs the same steps, give a model a set of tools and let it decide which to call, in what order, and when it has enough to answer.

The tools are small and boring on purpose:

• search_events(query, filters) and search_blogs(query, filters), each a thin wrapper over the same $vectorSearch from part one with the type filter already applied. Products come back as a third tool, and that is the entire cost of adding them.
• get_event_details(id) for when the conversation narrows to one thing and the summary in the search result is not enough.
• ask_user(question), which is a tool that touches no data at all. It ends the turn with a question.
• recommend(message, item_ids), the terminating action, which is what produces an actual answer.

Now trace the cases that the pipeline handles badly.

“I want to help the environment” gets ask_user on the first step. No embedding call, no vector search, no eight mediocre results dressed up as recommendations. The model can tell the message is too broad without needing to see the database, which is precisely the judgment the pipeline forces it to make blind.

“What can I do about food waste” gets search_events and search_blogs, both, then a look at what came back, then recommend with a mix. If the events are weak and the articles are strong, the reply leans on the articles. The routing decision moves to after the data arrives, where it can be made on evidence.

“Composting workshops near the northern beaches” that returns nothing useful gets a second search_events with a broadened query, because the model can see that the first attempt failed. That retry does not exist today at any price.

The shape that matters is the loop. Call a tool, look at the result, decide again. The pipeline’s steps become the agent’s options.

The grounding rule has to survive

Part one’s whole point was that the model can only recommend what retrieval returned, enforced in the backend by mapping the model’s chosen indices back to real records before anything reaches the UI. That property is not negotiable, and an agent loop makes it easier to lose, because now the model is doing more of the driving.

So recommend takes item ids, not item descriptions, and the backend resolves those ids against the database and drops anything that does not exist. If the model invents an id, the card does not render. The tool boundary is where grounding gets enforced, which is a better place for it than a prompt rule, because a prompt rule is a request and a schema is not.

The same logic applies to the search tools. They accept a query and filters and return results. They do not accept a collection name, a raw query object, or anything else that would let a confused model reach past what it is supposed to reach.

Why these tools want to be an MCP server

Right now the retrieval logic is a private service inside one NestJS API, reachable only by the chat endpoint. There is nothing wrong with that, but the tools I just described are useful well beyond the chatbot.

Expose them over the Model Context Protocol and the same tool surface becomes available to anything that speaks it. The public chatbot is one client. An internal assistant for the content team is another, asking things like which upcoming events have no article covering them, or which topics come up in searches and return nothing, which is a content gap report nobody has to build a dashboard for. My own editor is a third, during development, which means I can test a tool by talking to it rather than by writing a script that calls it.

The part I like most is that it forces the tools to be properly specified. An internal function can get away with a vague contract because there is one caller and you wrote both ends. A tool definition has to state what it takes, what it returns, and when to use it, clearly enough that a model that has never seen your codebase can pick it correctly. That is a documentation discipline the internal version never demanded.

It also lands the tools somewhere governance can reach them, which the last post in my AI foundations series covered: allow lists for which servers a client may use, budget and rate limits at the gateway, prompt and output scanning wrapping the calls. Those controls assume tools are addressable things rather than private functions. Building the tool layer as a server is what makes them applicable at all.

The arithmetic that says slow down

I wrote eight posts working through an AI foundations path earlier this year, and the one that keeps coming back on this design is the compounding error problem. Three steps at 90% accuracy each is not 90%:

0.9 × 0.9 × 0.9 = 0.729

Five steps deep is 59%. Uncertainty compounds, and the intuition that these systems get smarter by working together is exactly backwards.

The current pipeline has three steps. That is the whole chain, fixed, countable, and I can multiply it out. An agent loop has however many steps the model decides to take, which means the error math has a variable in it that I do not control. Every additional tool call is another multiplication.

That is not a reason to avoid the agent. It is a reason to build it with the brakes already fitted:

• Cap the iterations. A hard ceiling on tool calls per turn, with a deterministic fallback when it is hit rather than another model call to decide what to do.
• Validate between steps. Search results get checked for emptiness and score thresholds in code, not by asking the model whether it is happy.
• Keep the tools narrow. A tool that can only do one thing is a tool that can only fail one way.
• Make the terminating action deterministic. recommend resolves ids in the backend. The last step in the chain is code, not a model.

The foundations series also made a point I keep returning to: the capability boundary moves when you give a model tools. A model alone cannot search a database. A model with a search tool can. But moving that boundary is what adds the steps, and the steps are what compound. Both things are true at once, and the design has to hold both.

What this breaks

Being honest about the cost, because part two was about exactly this kind of accretion.

The 29 second ceiling stops working. Part two covered the Lambda timeout set to the API Gateway maximum, which comfortably covers three sequential round trips. A loop with a variable number of tool calls does not fit under any fixed ceiling, however generous, because the whole point is that the number of steps depends on the question. This is the change that forces real architectural work: streaming partial progress, or a job model where the turn is submitted and results arrive asynchronously, or a runtime that is not a Lambda. Probably streaming first, since it also fixes the loading animation problem from part one.

Cost per turn stops being predictable. Two model calls with roughly known token counts becomes n calls with a growing context, because each tool result gets appended to what the next call has to read. Budget tracking moves from nice to have to necessary, which is the other half of what the governance post was about.

Evaluation stops being optional. Part two listed the absent eval suite among the things deliberately not built yet: every prompt refinement so far was validated by typing into the chat and reading the answer. That is a defensible trade for a fixed chain, where there are exactly three places to look when something is off and the reply tells you most of what happened. It stops being defensible for an agent, because the failure mode becomes “it took six tool calls to do a one call job” or “it stopped searching too early”, and neither of those is visible in the reply text. Trajectories have to be tested, not just outputs. This is the piece I expect to be least fun and most necessary, and it moves from deferred to prerequisite.

Why write this before building it

Those eight foundations posts were notes on someone else’s material. Useful, but notes. The interesting part is what happens when the ideas meet a system that already exists and has users, because that is where they either hold up or turn out to have been slogans.

So far they are holding up. The compounding error arithmetic is the reason this post has a section about brakes instead of a section about how much the agent will be able to do. The capability boundary idea is why the tool list is the design rather than an implementation detail. The governance material is why the tools are going to be an MCP server rather than a private module, since that decision is nearly free now and expensive later.

The pipeline in part one was the right thing to build first. It is simple, its failure modes are countable, and it worked well enough to learn from. The agent is what it turns into once the limits are the interesting problem rather than the plumbing. I would rather have written down what I expected before finding out how much of it was right.