A coding assistant does not need to understand an entire repository before it can answer a developer’s question. It needs to find the right code, show where it found it, and avoid presenting stale guesses as facts. For a first implementation, I would choose live repository search over a vector index. Embeddings become worthwhile when real questions repeatedly defeat symbol and text search—not simply because the feature uses generative AI.
Live repository search is the better default for code questions
Consider a developer asking, “Where do we reject an expired session token?” The likely answer has a function name, an error string, a test, or a call site that ripgrep can find in the current checkout. A coding assistant can search those candidates, read bounded excerpts, and ask the model to explain the path from request handling to rejection. This approach wins when questions mention identifiers, configuration keys, endpoints, stack traces, or behavior with recognizable vocabulary, because those clues already exist as searchable text.
Live search also gives a backend developer a familiar consistency model: the assistant reads the same revision the developer is inspecting. git supplies the commit SHA; rg supplies file paths and line numbers; Tree-sitter can expand a hit to its containing function when a few matching lines are insufficient. No background indexing job has to catch up after a merge. Treat Generative AI for Software Development: Practical Use Cases as a menu of candidate tasks rather than evidence that every coding task needs an embeddings index.
The first version can be deliberately small. This Python 3 script searches a checked-out repository for a literal phrase, excludes two common dependency directories, and prints a bounded set of matches. Run it from the repository root after installing ripgrep:
#!/usr/bin/env python3
import subprocess
import sys
if len(sys.argv) != 2:
raise SystemExit("usage: python3 search.py PHRASE")
cmd = ["rg", "-n", "-F", "-m", "2", "--glob", "!vendor/**",
"--glob", "!node_modules/**", "--", sys.argv[1], "."]
result = subprocess.run(cmd, text=True, capture_output=True)
if result.returncode not in (0, 1):
raise SystemExit(result.stderr)
print("\n".join(result.stdout.splitlines()[:40]))
Here, -F would normally be written as an inline flag, but the important behavior is that ripgrep treats the query as a literal phrase rather than a regular expression. The 40-line output cap is a starting limit to tune against answer quality and prompt size, not a measured optimum. In a service, avoid collecting every match before applying that cap: search within authorized paths, stream results, and stop once enough distinct files have been inspected. Add a second pass for call sites or tests only when the first pass leaves an unresolved question.
I would not embed the entire repository on day one, because an index introduces update, deletion, and access-control work before there is evidence that lexical search misses important answers. Live search has costs too: repeated disk reads, poor matches for questions that use different words from the code, and the need to rank noisy hits. Those costs are visible in a request trace, which makes them easier to justify or fix than an index built on assumption.
Vector retrieval wins when developers ask about concepts the code never names
Now change the question to, “Why can a cancelled job still produce an invoice?” The relevant implementation may be spread across a queue consumer, a state transition, and an event handler without using the words “cancelled” and “invoice” together. A preindexed vector search can surface semantically related passages that literal matching misses. It is more likely to win across several repositories, long design documents, or renamed subsystems, because users cannot reliably guess the identifiers that tie those sources together.
The competing approaches have distinct bills. Live search with ripgrep wins for fresh, identifier-rich questions and costs request-time I/O plus ranking logic. Preindexed retrieval with pgvector wins when conceptual discovery repeatedly matters and costs embedding generation, storage, reindexing, permission filtering, and investigation of stale results. A vector hit is only a candidate, because similarity does not establish that a passage describes the current behavior.
If you add an index, make its unit of retrieval useful to a programmer. Chunking at arbitrary character boundaries can separate a condition from the branch it guards, so prefer a function or similarly coherent unit identified with Tree-sitter. Store repository, path, revision, symbol, and access scope beside each chunk. PostgreSQL with pgvector can hold embeddings and metadata together; an HNSW index can accelerate approximate nearest-neighbor search, although its recall and query cost still need testing. The hnsw.ef_search setting is a search-effort knob to tune rather than a value to copy from another workload.
Do not force a choice between semantic and exact matches inside the indexed approach. A developer who pastes SessionExpiredError expects the exact symbol to rank highly, so combine vector candidates with BM25 results from a search engine such as Elasticsearch, then rerank the merged set. An initial top-k of 8 is an experiment setting: increase it if relevant files are missed, or decrease it if excerpts crowd out the question and answer. Keep the final context tied to a revision so the model can cite code the developer can still open.
I agree with Your GenAI Feature Scales Until Retrieval and Latency Collide that retrieval latency becomes a production constraint, but index freshness deserves equal attention because a fast answer from yesterday’s code can be worse than a slower answer from the current commit. The vector approach earns its extra machinery only when it improves grounded answers under both constraints.
Answer quality, not search relevance alone, should decide the winner
A search result can look relevant and still lead the model to the wrong conclusion. Build an evaluation set from questions developers actually ask in code review, incident follow-up, and onboarding. For each question, record the revision, acceptable source locations, and what a correct answer must say. A 30-question pilot is a planning size for the first comparison, not a statistically persuasive benchmark; its purpose is to expose failure modes before anyone optimizes a leaderboard number.
Run both retrieval approaches against the same repository revisions and the same answer model. Count whether a necessary file appears in the retrieved set using Recall@k, and use MRR to notice when it appears only after several distractions. Neither metric says whether the final explanation is sound, so also check whether the answer cites lines that support its claims, distinguishes tests from production behavior, and says “I cannot establish that” when the evidence is incomplete. Reviewers should mark a correct conclusion with an unsupported citation as a failure, because the assistant has given the developer no reliable way to verify it.
Include adversarially ordinary questions: a symbol renamed last week, documentation that contradicts implementation, and two functions with similar names in different services. Re-run the set after a merge to see whether indexed results still point at the intended revision. If vector retrieval improves conceptual questions but regresses exact-symbol questions, add hybrid ranking before declaring it the winner; that fix targets the observed failure rather than replacing a useful search method wholesale.
Measure the request path with OpenTelemetry spans for search, excerpt loading, model input preparation, and generation. Use p95 latency rather than only the mean, because slow repository reads or index queries affect the developer waiting on an answer even when most requests are quick. An 800 ms retrieval budget is an example service objective to adjust after measuring your own environment, not a claim about either approach’s speed. Record input tokens and retrieval calls as well, since an apparently cheap search can become expensive when it sends large, repetitive excerpts to the model.
The decision rule should be hard to game: adopt the index for a question class only if it raises supported-answer quality without breaching that class’s latency and freshness requirements. Keep live search available for exact identifiers even after adding vectors, because a semantic ranking model has no reason to treat an exact error code as more important than a nearby conceptual match unless the system explicitly tells it to.
Permissions and freshness can overturn a retrieval win
A repository assistant is also an authorization system. Live search should run only over paths the requester may read; an indexed service must apply the same restriction before returning candidates. Filtering forbidden excerpts after a model has seen them is too late, because the answer can disclose their content. Store access metadata with indexed chunks, invalidate it when permissions change, and test that an unauthorized user cannot retrieve a restricted file through either exact or semantic queries.
Freshness needs an explicit contract. Suppose an index is allowed to lag the main branch by 15 minutes; treat that as a proposed policy ceiling, not a property of the indexing software. Attach the indexed commit SHA to every result and compare it with the requested revision. If they differ beyond the permitted lag, fall back to live search or state plainly that the answer uses an older revision. Silent fallback to stale context makes a code explanation difficult to debug, because the developer and assistant may be reasoning about different programs.
Retrieved repository text is untrusted input, even when it comes from a file the developer can read. A comment can tell an assistant to ignore instructions or reveal credentials; it remains source material, not an instruction from the user. Keep tool permissions separate from the language model’s retrieved context, and avoid passing secrets or whole environment files merely because they matched a search. These controls apply to both approaches, while a persistent index adds the obligation to remove deleted or newly restricted content from stored copies.
Start by collecting real developer questions and implementing bounded, permission-aware live search at a pinned commit. Ask reviewers to mark the files and lines each answer actually needed. Once that baseline exists, test an indexed retriever on the questions it misses. If vectors cannot produce more supported answers at an acceptable freshness and latency cost, keep the simpler system and improve its ranking instead.
