Hybrid search in a RAG system: a knowledge base for AI agents

In brief

• Corpus size: midway through the measurements the knowledge base held about 5,700 documents and 49,000 chunks, and it kept growing.
• Result: on a random sample of live AI agent queries from the server's log, the current search puts the right document in the top five for 96% of the queries answered in the knowledge base and first for 81%. The main cause of the former search's misses turned out to be automatic directory selection: the server guessed a knowledge base directory from the words of the question and searched only in it. On 76 questions written after it was dropped, the former and the current search were compared in the evaluation harness: the right document is in the top five for 63 instead of 45, p < 0.001.
• Quality gate: after the switch to BM25, before every rollout of the search server a copy with the change is compared with the production version on the development and held-out sets against tolerances written down in advance, and the change is not rolled out if the copy is worse.
• References: the search design was checked component by component against the NVIDIA RAG Blueprint and Anthropic's Contextual Retrieval method; chunk context made with that method is tested in the evaluation harness.
• Where the search runs: the Qdrant database, the embedding model and the reranker are deployed on local servers, and indexing and search do not depend on cloud APIs.
• Answer model: the answer is written by the agent's language model, and the agent connects to the search over the MCP protocol; the model can be replaced. Qwen on a local GPU server was tried as an agent: it searched through the production MCP server and opened documents itself, and by the gold answers it answered 169 of 177 questions correctly, 95%. The same model in a single call on the search's top five in the evaluation harness answered 133 correctly, 75%.

Risks and requirements

The case is addressed to heads of support and knowledge management, CIOs and CTOs at organizations whose staff and AI assistants look for answers across thousands of documents. According to a Gartner press release of 26 February 2025, "through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data." Gartner advises adding AI-specific practices to existing data management and names among them vector data stores, chunking, embedding and RAG integration.

Since 1 March 2026, FSTEC Order No. 117 of 11 April 2025 has been in force; FSTEC of Russia is the Federal Service for Technical and Export Control. The order approves the Requirements for the protection of information contained in state information systems and in other information systems of state bodies, state unitary enterprises and state institutions. Clause 60 of the Requirements does not allow restricted-access information held in these systems to be passed to the developer of an artificial intelligence model.

So in a RAG system it matters what data each component receives. Here indexing and search do not depend on cloud APIs: the Qdrant database, the embedding model and the reranker run on local servers; the chunk context for the experiments in the evaluation harness was also written by a local model. The answer is written by a separate component: it receives the question and the retrieved chunks, connects to the search over the open MCP protocol and can be replaced. In an isolated network the answer has to be written by a locally hosted model.

The task

Midway through the measurements the knowledge base held about 5,700 documents: project descriptions, architecture decisions, procedures, research. They are written in Russian but contain English terms, service names, addresses and port numbers. AI agents work with the knowledge base: developer assistants and chat assistants. An agent asks in its own words and needs to get the document with the answer straight away.

Search here serves as the retrieval component of a RAG system. RAG means generating an answer from search results: a large language model writes the answer from the retrieved chunks passed into its context. If the right document is not in the results, the model answers without it: the search determines what data the answer is built on.

The search had to take the meaning of a question into account without losing exact word matches: service names and terms that semantic search can miss. Indexing and search had to be independent of cloud APIs. Search quality had to be measured quantitatively.

How the hybrid search works

Once a minute the indexer reconciles the index with the knowledge base repository: it reindexes changed documents and removes deleted ones. A document is split into chunks by headings: a chunk corresponds to a section, and the path of headings is written at its start. A section longer than 3,500 characters is split by paragraphs into parts of about 3,000 characters; the last paragraph of a part, if it is no longer than 200 characters, is repeated at the start of the next one.

Each chunk is stored in two representations: a dense vector from the BGE-M3 model for semantic search, and a sparse BM25 vector with a Russian stemmer for keyword search. Combining the two kinds of search is called hybrid search. Before the search, the query is expanded with a synonym dictionary, and both query vectors are built from the expanded text. Both queries go to the Qdrant vector database in a single call, and it merges their result lists with RRF into 30 candidates; the bge-reranker-v2-m3 cross-encoder rescores the first 15 of them for relevance, and the remaining candidates never reach the results. These figures apply to the default output: the agent receives the five highest-scoring chunks.

from qdrant_client import models
 
# text: the question after the synonym dictionary;
# query_vec: dense BGE-M3 vector of the same text
candidates = qdrant.query_points(
    collection_name=COLLECTION,
    prefetch=[
        models.Prefetch(query=query_vec, limit=30),
        models.Prefetch(
            query=models.Document(
                text=text,
                model="qdrant/bm25",
                # at indexing time these options also get
                # avg_len=170: the mean chunk length in words
                options={
                    "stemmer": {"type": "snowball", "language": "russian"},
                    "stopwords": "russian",
                    "tokenizer": "word",
                },
            ),
            using="bm25",
            limit=30,
        ),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=30,
)
# bge-reranker-v2-m3 rescores the first 15 candidates;
# the agent gets the best five

Agents connect through an MCP server: the open MCP protocol defines how AI applications connect to external tools and data. Through the server the agent searches for chunks and reads a document from the results by its path; the server's instructions suggest when to search by meaning and when an exact name is faster to find with a file search. Every set of results states how the search actually ran: if the reranker is unavailable, the agent sees this instead of silently receiving degraded results. Agents reach the server through an authenticating proxy. Every knowledge base query through the MCP server is written to a Grafana Loki log and kept for 90 days; secrets in common formats are masked in it by patterns.

Chunks of external documents, such as copies of web pages, are marked by the server when it returns them and begin with a warning that they are data, not instructions. Marking untrusted content is part of the prompt injection mitigations OWASP lists in LLM01:2025.

A probe watches the search: every 10 minutes it calls the search the same way the agents do, and two failures in a row raise an alert. If the search answers in degraded mode, without the reranker, the embedding model or BM25, for more than two hours, this goes into a daily summary. The code inside the server container is checked against the repository once a day. Variants of the search logic are compared in an evaluation harness: a program that repeats the server's search steps, switches between variants and calls Qdrant, the embedding model and the reranker directly.

Checking against reference architectures

The search design was checked component by component against the open reference architecture NVIDIA RAG Blueprint version 2.6.2 and against the Contextual Retrieval method that Anthropic described in 2024: a language model adds to each chunk a short context, which document it comes from and what it is about. NVIDIA's default values are taken from the 2.6.0 documentation, as there is none specific to 2.6.2. The search design was also compared with 100 public sources on RAG system practice. Among them are the Qdrant documentation on hybrid queries, Microsoft's 2023 measurements of hybrid search in Azure AI Search, and the Russian-language benchmarks ruMTEB of 2024 and RusBEIR of 2025; 20 numbers and 15 quotations were spot-checked against the source pages themselves and confirmed.

The table compares only the retrieval components; chunk context made with the Contextual Retrieval method is covered in its own section below.

Evaluation method

A search that finds the wrong document does not crash and logs no errors. That is why search quality is checked by measurement on labelled questions, separately from answer generation: this way a search error is visible separately from an error of the answering model. Anyscale separates evaluation the same way in its 2023 guide to building a RAG application: retrieval gets its own score there, based on a match with the labelled source.

• Development set: 27 questions written from the documents; each has a pre-labelled gold document with the answer, and 8 questions have two equally valid ones. Of these, 21 concern the knowledge base and 6 concern two other document corpora served by the same search server; in those corpora keyword search still uses the former filter. The causes of misses were investigated and the changes chosen on this set, so a separate held-out set is needed to guard against overfitting to it.
• Held-out set: 40 questions on knowledge base chunks chosen by a random number generator with a fixed seed, so the sample can be reproduced on the same snapshot of the knowledge base. A top-level directory is chosen with a weight equal to the square root of its number of files, and at most one chunk is taken from a file. The questions for the chosen chunks were written with no access to the search; 11 questions have two or three equally valid gold documents or sections. The set was frozen after the first changes and before any variant was run on it; the first changes, that is, dropping directory guessing and two smaller ones, are described in "Technical decisions".
• Chunk context sets: 36 questions collected after the switch to BM25. The chunks were chosen by a random number generator with a fixed seed: 15 chunks whose subject is named only in the file path and not in the chunk text or its headings, and 21 ordinary chunks whose subject is also named in the document heading. The questions and the chunk context were prepared independently of each other. Together with the held-out set these are 76 questions written after the first changes.
• Live agent queries: 60 queries picked at random from the server's log of AI agent queries. Where in the knowledge base each query is answered was labelled from the repository itself, with no access to the search results; documents from the results that the labels had not named were checked separately, and the accepted ones counted. A hit here is counted by document.
• Labels: a hit is counted by document, and by section where the labels name one; the compiled sets are not edited after the fact.
• Metrics: Hit@1 and Hit@5, the share of questions where one of the gold documents is ranked first or appears in the top five; MRR@5, the mean over questions of the reciprocal rank of the first gold document in the top five, a question without it in the top five counting as zero; median search latency, without answer generation.
• Comparing variants in the evaluation harness: on the same questions; for Hit@1 and Hit@5 a two-sided exact McNemar test, for MRR@5 a 95% percentile bootstrap interval and a two-sided sign-flip permutation test on paired differences, 10,000 resamples each. A difference is considered significant at the 0.05 level; for Hit@k this means at least 6 questions won with no losses, or 8 with one loss.

Both stages of changes were compared with the former search reconstructed in the evaluation harness, as described in "Results".

Since the switch to BM25, every rollout of the search server goes through a quality gate: a script brings up a staging copy of the server with the change next to the production server, compares the two and stops the rollout if the copy is worse. The rule was written in advance. Both servers are queried alternately on every question of the development and held-out sets; a question on which they differ is run twice more and decided by the majority of the three runs. The rollout stops if:

• any question drops out of the top five or becomes a zero-result query;
• Hit@1 of a set falls by more than one question, or MRR@5 by more than 0.02;
• the copy answers in degraded mode more often than the production server.

The tolerances are relative to the production version. The gate can be bypassed only with an explicit flag stating the reason. Before the quality gate, changes to the search logic are compared in the evaluation harness.

Technical decisions

The changes came in two stages. First, automatic directory selection was removed, documents indexed long ago were no longer down-weighted and repeated text began to be filtered out: these are the first changes. Then keyword search was switched to BM25.

A knowledge base directory only on explicit request. The server used to guess a directory from the words of the question and apply it as a hard filter. Self-querying in LangChain follows a similar idea: there a language model builds a metadata filter from the question text, as Elastic described in 2025, while here it was picked by a substring rule. The hard filter cut off the right document before the search even started. Now the server searches the whole knowledge base unless the agent itself asks to narrow the search.

Older documents on a par with recent ones. The down-weighting of documents indexed long ago was removed: the indexing date tells when a file was edited, not how current it is.

Repeated text filtered out. Identical text from different copies of a document appears in the results once. On the 27 development-set questions this did not change Hit@5 or MRR@5 in the evaluation harness, but it frees places in the top five.

BM25 with a Russian stemmer instead of a keyword filter. Keyword search used to only select chunks that contained every word of the query, so a long question could find nothing. Now every word contributes to the score, and Qdrant reduces words to their stems.

No semantic cache. A semantic cache returns a stored result if the vector of a new query is close to the vector of an earlier one; GPTCache for language model responses is one example. The results cache of this server had a similarity threshold of 0.96. Two long queries that differ only in the port, 8080 and 8081, have a cosine similarity of 0.965, so for the second query the cache would have returned the results of the first. In a knowledge base where exact numbers matter this is unacceptable, so the results cache is turned off.

Rerank depth unchanged. On 27 questions after the first changes, still with the keyword filter, one alternative was compared: 100 candidates with 60 reranked. It gave no gain: its Hit@5 was 22 versus 23, while the median latency in the evaluation harness grew fourfold, 2,063 versus 487 ms.

Results

Measured through the production server on the same 27 development-set questions, before the changes and after both stages; the knowledge base changed between the measurements:

Before the changes, 7 of the 9 top-five misses came from automatic directory selection: the server guessed a knowledge base directory from the words of the question and applied it as a hard filter, so the right document did not even reach the candidates. The two other corpora have no such directories, so when the auto-selection fired for a question on them, the results stayed empty. Automatic selection was removed in the first changes: after that, Hit@5 through the production server rose from 18 to 23, and in the evaluation harness the right document reached the candidates for 25 of 27 questions instead of 18; the two other changes of the first stage did not change Hit@5 in the evaluation harness. After the switch to BM25, Hit@5 became 24 instead of 23, and Hit@1 became 19 instead of 17. After the switch, the median search latency was 483 ms on the development set and 471 ms on the held-out set: measured through the production server, with queries sent one at a time and time measured on the caller's side.

Before the switch, BM25 and the former filter were compared in the evaluation harness. On the held-out set, frozen after the first changes, their Hit@5 and MRR@5 were the same: 37 of 40 and 0.82, while Hit@1 was 29 with BM25 versus 30. With dense vectors alone, without keyword search, Hit@5 in the evaluation harness was 22 of 27 on the development set and 37 of 40 on the held-out set, versus 24 and 37 with BM25. Among the 30 candidates before reranking, in the evaluation harness, with BM25 the right document was present for all 67 questions of the development and held-out sets, with the former filter for 64; the reranker, however, sees only the first 15 candidates, and the 6 questions on the two other corpora use the former filter in both variants.

Hit@5 on the development and held-out sets is already about 90%, 61 or 62 of 67 in the runs through the production server and 60 or 61 in the evaluation harness, and small improvements cannot be told apart on them, while the changes were chosen on the development set. That is why both stages were tested on 76 questions written after the first changes.

Check on 76 questions. These are the held-out set and the chunk context sets from "Evaluation method". The former search logic was reconstructed in the evaluation harness: the directory auto-selection rule was copied verbatim from the code history, and with it came back the down-weighting of long-indexed documents, the keyword filter and results without removal of repeated text. On the 27 development-set questions the reconstructed search gave the same 4 zero-result queries as the measurement through the production server before the changes, and the right document was among the candidates for the same 18 of 27. The index kept updating as usual, and all three search variants were run on each question back to back, seconds apart.

The current search is significantly better than the former: on Hit@5 it wins on 20 questions and loses on 2, on Hit@1 it wins on 16 and loses on 1, p < 0.001 in both cases. The contribution of each stage is significant too. On Hit@5 the first changes have 11 wins against 1 loss, p = 0.006; BM25 has 10 wins against 2 losses, p = 0.039. On the 36 questions of the chunk context sets, collected after the switch to BM25, BM25 has 6 wins with no losses, p = 0.031.

Contribution of the reranker. The comparison ran in the evaluation harness on all 103 questions: 27 development-set, 40 held-out and 36 from the chunk context sets. Without the reranker the candidates are the same, only their order changes. With the reranker the right document is first for 67 questions, without it for 51: 21 wins for the reranker against 5 losses, p = 0.002. In the top five it is 87 and 81: 9 wins against 3 losses, not significant, p = 0.15. The median search time in the evaluation harness is 21 ms without the reranker and 355 ms with it: almost all of the search time goes to reranking 15 candidates.

Embedding model. In the evaluation harness BGE-M3 was compared with the runners-up: multilingual-e5-large-instruct in ruMTEB and USER-bge-m3 in RusBEIR. There were 179 questions: 97 knowledge-base questions from the four sets in "Evaluation method" and 82 new questions of the second chunk context check, described below. Of the 97, 21 come from the development set, 40 from the held-out set and 36 from the chunk context sets. A snapshot of the knowledge base was copied into three temporary collections that differ only in the dense chunk vectors, while BM25, fusion and the reranker are shared. The decision rule was written down before the run. With all search components working, BM25 and the reranker included, the right document is in the top five for 154 questions with BGE-M3, 151 with e5-instruct and 153 with USER-bge-m3. On Hit@5, against BGE-M3, e5-instruct has 3 wins against 6 losses, p = 0.51; USER-bge-m3 has no wins against one loss, p = 1.0. The right document is first for 110, 108 and 109 questions: e5-instruct has 1 win against 3 losses, p = 0.63, USER-bge-m3 no wins against one loss. Neither model differs significantly from BGE-M3.

Searching by vectors alone, without BM25 and the reranker, e5-instruct puts the right document first more often, 97 versus 87, but this is not significant either, p = 0.19; USER-bge-m3 is slightly behind BGE-M3: the right document is first for 82 questions versus 87 and in the top five for 125 versus 128, also not significant. Moreover, e5-instruct reads only the first 512 tokens of a chunk, while BGE-M3 and USER-bge-m3 read the whole chunk.

Live agent queries. On the 57 live queries whose answer is in the knowledge base, the current search puts the right document in the top five for 55, 96%, and first for 46, 81%; MRR@5 is 0.87.

Search time. On a copy of the production server queried from the same node, over 179 questions one at a time, the median search time is 288 ms and the 95th percentile 408 ms, with no errors and no zero-result answers. In the production server's log the median on agent queries is 760 ms, the 95th percentile 1.5 s. Agents more often ask for more than five results and narrow the search to a directory; when an agent asks for at most five results and sets no directory, the median is 548 ms. Zero-result answers in the log occurred only when the agent itself had limited the search to a directory with not a single chunk in the index.

Chain with a local model. In the evaluation harness, on 97 knowledge-base questions from the four sets, all five retrieved chunks were given to Qwen 3.8 27B on a local GPU server in a single call, without MCP and without an agent with tools. The model wrote an answer and a list of sources, and its citations and refusals were checked programmatically. A hit is counted as in the compiled sets: by document, or by section where the labels name one. There was a hit for 82 questions, and the model cited the right document in 79 answers and in the other three replied that the retrieved chunks contain no answer. No answer cited a document outside the top five. For 15 questions there was no hit: the model replied that there is no answer in eight cases, cited the right document through another of its sections in three and answered from other documents in four. The median time of the model's answer, without the search, is 7.5 s, the 95th percentile 22 s.

Correctness of the chain's answers. To these 97 questions, the 82 questions of the second chunk context check were added, and the model answered them the same way. For each question a gold answer was written: 1–4 key facts from the labelled section. The 177 questions whose answer was in the labelled section at the time of the search were included. The answers were scored blind against the gold answers, without knowing which run each came from. 133 of the 177 answers are correct, 75%, 95% interval 68–81%; 26 are partly correct, 6 wrong and 12 refusals. When the right document was in the top five, 127 of 155 answers are correct, 82%.

Agent over MCP. Then the same Qwen model was tried the way agents work: it received the question and two tools of the production MCP server, search over the knowledge base and document reading, decided itself what to search for and what to open, and only then answered. Long tool results were truncated, as in the tools agents run in, and the number of turns and the total amount of retrieved text were limited. The questions, gold answers and scoring are the same. 169 of the 177 answers are correct, 95%, 95% interval 91–98%; 5 are partly correct, 3 wrong, and there are no refusals. The agent saw the right document for 164 questions, while it was in the harness's top five for 155, p = 0.035; when the agent saw it, 161 of 164 answers are correct, 98%. Only the agent answered 38 questions correctly, only the chain 2, p < 0.001. The median number of searches per question is 1, and the agent opened documents 58 times in 51 questions. When the model was serving no other requests, the agent's median answer time was 19 s, and the median time spent on search and document reading per question was 0.75 s.

Chunk context with the Contextual Retrieval method

Chunk context made with the Contextual Retrieval method for all chunks the knowledge base had midway through the measurements, about 49,000, was written by a Qwen model. It ran on a separate local GPU server, not the one that runs the search models; no knowledge base text was sent to external providers for this. On top of the context, the model is asked to add to each chunk up to 8 words and phrases that someone might use to search for it but that do not occur in it. Every number in the chunk context is checked against the text the model saw: a word or sentence with an unconfirmed number is discarded.

Chunk context enters production search only if it passes the check in the evaluation harness. It was checked twice, the second time on 82 questions about chunks from a new random sample. The questions were prepared with no access to the search or to the chunk context, and questions with a word found only among the search words the model had added were dropped. The variant tested gives the chunk context to the reranker only and leaves the vectors unchanged. In the second check the condition was a significant gain on the new questions; there was none, so production search was not made more complex: chunk context stays in the evaluation harness.

Conclusions

1. Most misses came from the rule that guessed a directory from the words of the question and applied it as a hard filter: on the development set it accounted for 7 of the 9 top-five misses and all the zero-result queries. The first changes, which removed this rule, raised the number of questions with the right document in the top five from 45 to 55 on 76 questions written after them, p = 0.006.
2. Keyword search paid off here once it ranked chunks instead of selecting them: on 76 questions the right document is in the top five for 63 with BM25 and for 55 with a filter that requires every word of the query, p = 0.039. On the 36 questions collected after the switch, BM25 has 6 wins with no losses, p = 0.031.
3. The reranker mainly affects which document comes first: with it the right document is first for 67 of 103 questions, without it for 51, p = 0.002. Its effect on the top five is not significant, 87 versus 81, while the median search time in the evaluation harness is 355 ms with it versus 21 ms without it.
4. With BM25 and the reranker working, BGE-M3 and the two runners-up of ruMTEB and RusBEIR show no significant difference: the right document reaches the top five for 154, 151 and 153 of 179 questions.
5. Chunk context made with the Contextual Retrieval method was checked twice in the evaluation harness and did not enter production search; in the second check there was no significant gain on new questions.
6. Significance of the changes was tested on 76 questions written after directory guessing ended. Every rollout goes through a quality gate, and it stopped a deliberately broken version.
7. Tried as an agent through the production MCP server, the local Qwen model answered 169 of 177 questions correctly by the gold answers, 95%; the simplest chain in the evaluation harness, a single call on the search's top five, answered 133, 75%, p < 0.001. The agent found the right document by itself more often, for 164 questions versus 155, p = 0.035, and when it did, 98% of its answers were correct.
8. On a random sample of live AI agent queries from the server's log, the current search puts the right document in the top five for 96% of the queries answered in the knowledge base and first for 81%.