Generative AI

Your GenAI Vendor Demo Hides Half the Project Scope

A GenAI vendor can produce a convincing answer in a demo and still leave your team with an unscoped retrieval project. I would buy the option that makes failures cheap to detect and correct, even if its answers look less polished in a sales call. For a product manager, the evaluation is not “Which model sounds best?” It is “What work remains ours after we sign?”

The best demo is the one you can reproduce without the vendor

Give every candidate the same small, permissioned corpus and the same questions before discussing architecture. A demo built on a vendor’s prepared documents tells you little about your own workload, because document structure, access rules, and update frequency determine much of the retrieval work. Ask your team to supply documents with awkward tables, conflicting revisions, missing answers, and two test tenants whose files must never cross.

Make the vendor ingest that corpus while your team watches. Record who cleans files, splits text, assigns metadata, configures access control, and resolves failed imports. Those tasks belong in the project estimate even when the platform advertises “managed RAG,” because managed infrastructure does not make source documents consistent or permissions self-enforcing. Require an export of the processed chunks and their source identifiers; without it, moving providers later may mean repeating the least visible part of implementation.

Use a fixed evaluation set: for example, reserve 80 questions as a proposed starting size, including answerable, unanswerable, and access-denied cases. Have a product owner approve the expected source for each answer. Score retrieval with recall@10 and NDCG@10, then score the final answer separately for factual support and appropriate refusal. A high retrieval score cannot excuse a fabricated conclusion, while a good answer on a few handpicked prompts cannot establish that the right evidence is consistently retrieved.

I would not choose a framework from its chatbot screenshot, because the screenshot hides the ingestion decisions and error handling that determine the delivery schedule. Ask instead for the exact failing question, retrieved chunk IDs, prompt version, model identifier, and permission decision behind any disputed result. If a vendor cannot produce those artifacts from an ordinary request, your team will have to build its own diagnostic layer or spend support cycles guessing.

The architectural argument in RAG vs Fine-Tuning in Your First GenAI Backend is a useful starting point, but it should not become a purchasing rule: a vendor must prove that its chosen approach handles your update and permission requirements. Price a re-index after a changed document, and ask how quickly that change becomes visible. A model that answers yesterday’s policy beautifully can still fail the product requirement if today’s correction is unavailable.

Ownership of the retrieval layer determines the real migration cost

For a first release, I would compare a managed retrieval service with a stack the team can inspect. Pinecone Serverless wins when the team needs a hosted vector service and does not want to operate index capacity; it costs usage-based service spend, integration work, and a migration plan for vectors and metadata. PostgreSQL 16 with pgvector 0.8.0 wins when the corpus and query load fit a database the team already operates and when SQL-level inspection matters; it costs database tuning, backup planning, and engineering time as the index grows. Neither choice removes the need to test filtering and ranking against your documents.

Pin the evaluation environment rather than accepting a comparison between unlike defaults. For a pgvector trial, record the embedding model, distance metric, index type, and hnsw.ef_search; changing the last setting can change recall and query time, so it belongs beside the results. For a hosted service, record namespace design, metadata filters, and the index configuration supplied by the vendor. Demand the same filter semantics in both trials: a permission failure is a release blocker, not a small deduction from an accuracy score.

Libraries create a different kind of ownership. LlamaIndex may shorten the path from documents to a retrieval prototype, while LangChain may suit a team already using its application components. Neither is the retrieval service itself. Ask the engineer proposing either one to show where parsers, embeddings, prompts, retries, and source IDs are configured, and to lock package versions in the trial. Otherwise, a product estimate may treat connector setup as the whole job while leaving document repair, evaluations, and operational support unassigned.

Include security and data handling in the proof, not in a later questionnaire. Confirm whether the service accepts your OIDC-based identity flow, where tenant identity is enforced, how deletion propagates to indexes and caches, and whether prompts or documents are retained for training. A SOC 2 Type II report can support a controls review, but it cannot demonstrate that this particular integration prevents one tenant from retrieving another tenant’s text. Run that test yourself.

For each option, put the residual work in a named column: ingestion, evaluation, access control, observability, incident response, and exit. If the vendor owns a task, specify the contract or product setting that makes it theirs. “The platform handles it” is not a scope assumption you can schedule against.

A latency claim is credible only when it includes retrieval and failure

Ask for end-to-end measurements at the application boundary. A model provider’s token-generation figure omits query embedding, permission filtering, retrieval, reranking, network hops, and citation assembly; those steps can dominate the user’s wait. The warning in Your GenAI Feature Scales Until Retrieval and Latency Collide deserves a procurement consequence: require evidence for the whole request path before accepting a latency commitment.

Set an initial p95 target of 3 seconds as a value to tune with your product team, not as a universal benchmark. Measure time to first token as well as completed response time if the interface streams; a quick first token can make a slow answer feel responsive, but it does not help a workflow that needs the complete cited answer before continuing. Track HTTP 429 responses and timeouts alongside latency, because excluding failed requests makes a strained service appear faster.

Require a stable trial endpoint that returns an answer and citation identifiers. The following Python probe runs with standard-library modules after you set RAG_URL and RAG_TOKEN. Its 40 calls and 5 workers are a deliberately small smoke-test setting, not a production load model:

import concurrent.futures, json, math, os, time, urllib.request
url, token = os.environ["RAG_URL"], os.environ["RAG_TOKEN"]
def probe(_):
    body = json.dumps({"question": "What is the refund window?", "tenant_id": "test-a"}).encode()
    req = urllib.request.Request(url, body, {"Authorization": f"Bearer {token}", "Content-Type": "application/json"})
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=15) as response:
        data = json.load(response)
    return time.perf_counter() - start, bool(data.get("citations"))
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(probe, range(40)))
times = sorted(seconds for seconds, _ in results)
print("p95_seconds", times[math.ceil(0.95 * len(times)) - 1])
print("citation_presence", sum(cited for _, cited in results) / len(results))

Agree the endpoint schema with each candidate first. Citation presence is only a wiring check: an evaluator must still open the cited source and decide whether it supports the answer. The script also does not record failures as results, so a serious trial should add exception counts rather than quietly discarding them. Use k6 for a repeatable load profile, and instrument the application with OpenTelemetry spans so retrieval, reranking, and generation have separate timings. A Prometheus histogram can expose p95 over time; keep the underlying request count visible so a quiet interval is not mistaken for improved performance.

Test a warm index, a fresh document, and a degraded dependency. Those cases matter because caches, indexing delays, and retries change what a user sees even when the model and prompt are unchanged. Request the vendor’s rate-limit policy and retry guidance in writing, then estimate what your application must do when the service is unavailable: show an error, fall back to search results, or queue the request. Each behavior has a different product and engineering cost.

A short paid trial should end in a scope decision, not a winner’s slide

Run a paid trial with an explicit stop date; two weeks is a planning allowance to adjust to your team’s document readiness, not a promise that integration finishes in two weeks. Agree on the corpus, question set, tenant tests, expected traffic shape, and acceptance thresholds before the vendors see the results. Otherwise, a candidate can optimize for whichever examples were easiest to present.

Score the trial in three ledgers. The first is user outcome: supported answers, correct refusals, and failures that a reviewer can reproduce. The second is operating behavior: end-to-end latency, errors, indexing delay, and the time needed to identify a bad source. The third is ownership: tasks your engineers must build, tasks the vendor contractually covers, and tasks nobody has accepted. Keep those ledgers separate because a strong answer score does not pay for an undocumented permission system.

Ask each candidate for the cost per 1,000 successful, cited answers, including embeddings, storage, retrieval, generation, retries, and any required observability tier. This is a budgeting unit to calculate from the trial’s usage, not a vendor-published price. “Per request” can conceal the cost of retries and unusable answers; “successful, cited” ties the estimate more closely to the feature you intend to ship. Request an export and deletion exercise before approving the contract, because an exit plan that has never been tested is only a clause.

Finish with a decision record that names the rejected option, its advantage, and the condition under which you would reconsider it. That record prevents a later team from treating the choice as an eternal verdict and gives the product manager a defensible change trigger. The vendor that wins should leave you with fewer unknown work items, not merely the highest-scoring demo.

The first purchase decision is a test fixture

Before booking another demo, commission a permissioned sample corpus and an approved question set with source-level expected answers. Give the same fixture to two candidates and require request traces, failed cases, and a written ownership map. That first deliverable lets you scope ingestion, evaluation, and support work even if neither vendor passes—and keeps a persuasive sales presentation from becoming your project plan.