Building a RAG Chatbot: Why We Skipped Keyword Search Entirely

We wanted a search box that understood what people meant. The catalogue had events, articles and products, and the existing search only found things when the visitor happened to use the same words as whoever wrote the description. Someone looking for “something to do this weekend that helps the planet” got nothing, even though the catalogue had plenty of matching content.

So we skipped the incremental fix and went straight to a RAG chatbot: embed the content, search by meaning, and let a model write the reply from what came back. This post covers how it fits together and why each piece has the shape it has. The next one is about what changed when it left my laptop.

The problem with the obvious solution

MongoDB has had full text search for years. Put a text index on a field, run a $text query, done.

const events = await eventsCollection.find({
  $text: { $search: 'sustainable water bottle workshop' }
}).limit(6).toArray();

It is fast and cheap, and you always know why a document matched. It also breaks as soon as someone talks like a person. “I want to reduce plastic at home” looks for documents containing the literal words reduce, plastic and home. An article titled “Cutting single use waste in the kitchen” matches none of them, even though it is exactly the right answer.

Vector search compares meaning instead of words. The user’s message and every document get converted into embeddings, long lists of numbers arranged so that related concepts sit near each other, and the search returns whatever is nearest. “Single use waste” lands near “reduce plastic” because the embedding model has seen enough language to know the two phrases are related.

Keyword search is exact but brittle. Vector search is approximate and a bit slower, and it copes with users describing things in their own words. Nobody talks to a conversational assistant in catalogue keywords, so we went with vectors.

Three steps, three jobs

Early on I was tempted to do everything in one big model call: here is the user message, here is the database, figure it out. That falls apart quickly, because when the answer is bad you cannot tell which part went wrong.

The pipeline ended up as three explicit steps.

Step one: understand before searching

The cheapest design embeds the raw user message and searches with that. It saves a model call and a prompt to maintain, so it deserves a fair look.

An extraction step beats it, and a real message shows why. “Hey, so I was wondering, is there anything happening around here about composting? my neighbour mentioned it” carries maybe four useful words and a lot of conversational padding. The embedding is built from all of it, so the padding moves the vector, and two people asking the same question in different registers land in different places.

So the first call never touches the database. It asks the model to classify the message and extract a clean query, spending a round trip to get a more precise search. The rest of the pipeline makes that same trade again.

{
  "intent": "CONTENT_SEARCH",
  "query": "community composting workshop",
  "filters": ["this weekend"],
  "types": ["event", "blog"]
}

intent separates a real search from someone asking about the weather, so off topic messages get a polite redirect instead of a random database query. query is a neutral phrase tuned for similarity search. filters holds constraints the user stated, including negative ones like NOT clothing when they rejected something earlier. types decides which content to search: someone asking “what’s on near me” only wants events, while someone saying “I care about sustainability” should get both events and articles.

The filter rules needed the most care, because an over eager model can do real damage in that field. An invented constraint silently narrows the search, and the user never learns why they got three results instead of eight. The prompt is written defensively: a filter exists only if the user actually said it, and the format examples are labelled as shapes, not words to reuse. Small models will copy example text straight into their output unless you tell them not to, so the prompt tells them.

Step two: retrieve

The clean query gets embedded and run through MongoDB $vectorSearch once per requested content type, in parallel:

const rows = await this.contentEmbeddingModel.aggregate([
  {
    $vectorSearch: {
      index: VECTOR_INDEX,
      path: 'embedding',
      queryVector,
      numCandidates: 50,
      limit: RESULTS_PER_TYPE,
      filter: { type },
    },
  },
  { $project: { _id: 0, refId: 1, score: { $meta: 'vectorSearchScore' } } },
]);

This returns pointers and scores. The full records get loaded from their own collections afterwards and reordered by score, because a $in query throws away the ranking the vector search just produced.

Step three: generate, but only from what came back

The second model call receives the retrieved items and writes the reply. The model describes rows we pulled from our own database a few milliseconds earlier, so it has no reason to fall back on whatever it remembers from training data.

It has one more job. Vector search returns things that are near the query, and near is not always appropriate. The retrieved items go in numbered, and the model returns index arrays alongside its message:

{
  "message": "Here are a couple of composting sessions coming up...",
  "recommended_event_indices": [0, 2],
  "recommended_blog_indices": []
}

The backend filters the retrieved set down to the items the model chose and maps those indices back to real objects before anything reaches the UI. Every card the user sees is a database record, and the model has no way to invent one.

The rule that stopped it feeling like a robot

One turn control rule improved the experience more than all my prompt tuning did: in a single response the bot either asks one clarifying question or recommends items, never both.

Without it, the assistant would ask “are you looking for events or articles?” and then immediately list six of each, as though it had not been listening. With it, each turn has an obvious purpose.

A second rule covers the opposite case: when results exist, the bot may not ask a clarifying question. Left to its own judgment, a model will hedge. It retrieves eight perfectly good events and then asks what the user was interested in, because hedging always looks safe from inside a single turn. The prompt takes that choice away by stating outright whether items were found and what that allows.

Conversation history matters here too. Previous messages go back into the intent step along with the items already shown, so “not that one” or “something cheaper” has enough context to mean something. The subtlest rule in the prompt is about rejections: a rejection is a hard constraint unless the current message asks for that exact thing again. People change their minds, and a bot that holds a grudge about clothing from four turns ago is worse than one that forgets.

The index budget shaped the data model

The natural way to store embeddings is a vector field on each collection: events get one, articles get one, products get one, and each has its own Atlas index.

We could not do that. The shared Atlas cluster allows three vector and full text indexes across the whole cluster, and we have three databases: dev, test and prod. That leaves one index per database, with nothing spare for a per collection layout or for future growth.

So everything goes into a single collection:

@Schema({ collection: 'content_embeddings', timestamps: true })
export class ContentEmbedding {
  type: EmbeddableType;           // 'product' | 'event' | 'blog'
  refId: mongoose.Types.ObjectId; // _id of the source document
  title: string;
  text: string;                   // exactly what was embedded
  embedding: number[];
}

Each database has one vector index with type declared as a filterable field, and each search scopes itself with filter: { type }. A compound unique index on { type, refId } makes re-running the embedding job upsert instead of creating duplicates.

This is the decision I would have to defend hardest in review, because a number in a pricing tier dictated the architecture, and that is usually a smell. It still produced the better model. Adding a new entity type now needs no new indexes and about twenty lines of code: one text builder and one branch. If we had built three per collection indexes first and hit the cap later, we would have had to rewrite it.

The text field stores exactly what was fed to the embedding model. That looks redundant until a result makes no sense and you need to know whether the problem is the query or a document that got embedded as its title plus an empty description.

Embedding runs as a backfill job

Content arrives from scrapers, a CMS and manual entry, so embedding on write would have meant changing every one of those paths. There is one job instead. It walks each type, finds source documents that have no row in content_embeddings, and processes them in batches of ten.

It runs behind an authenticated endpoint, which is safe because of how it behaves. By default it only processes what is missing, so running it twice does nothing the second time. A reset flag clears the collection and re-embeds everything, which you need when the embedding model changes or stale rows have to go.

The text that gets embedded is built per type. Events combine title, description, host, location and tags. Articles combine title, introduction, category, tags and the body with its HTML stripped, since raw markup in an embedding costs tokens and adds nothing but noise.

What it cost

Latency is the biggest cost. A single turn is a model call to classify, an embedding call, a vector search, and a second model call to respond. That is three sequential round trips where a $text query would have been one database hit. This design cannot avoid it. The best it can do is hide it.

Data quality is the other cost. A recommendation is only as good as the text behind it, and an event with a one line description gets a weak vector no matter how good the model is. Vector search can’t fix thin content either. It just fails more politely than keyword search does.

Where it goes next

Hybrid search is what I am most interested in: combining vector similarity with structured filters like date, location and category, so that the query itself enforces “this weekend” and the model doesn’t have to.

Filters are deliberately soft in this version. The model extracts them and passes them to the generation step as text. That keeps the retrieval layer simple and let us ship without a schema for every constraint a user might state. The downside is that a filter can influence the answer but cannot guarantee it. Turning date and location into real query predicates is a contained change now, since the extraction step already outputs them in a structured shape, and it will be worth doing once there is enough content that a soft filter stops being good enough.

Streaming the final generation step would also hide most of the latency. Three sequential model calls is a long time to watch a loading animation, and the last one produces the only text the user actually reads.

The next post covers what happened when this moved off my laptop: the local Ollama setup, the switch to Azure OpenAI, and the parts of “provider agnostic” that turned out to be aspirational.