A GenAI backend can pass a demo and still be unfit to operate: the answer looks plausible while its source document belongs to another tenant, its index is stale, or its retrieval call consumes the entire request budget. I would build the first production setup around an enforceable retrieval boundary, not model optimization, because access and failure behavior are harder to retrofit than prompts.
The first deployable unit is an authorized document pipeline
Begin with one source of truth: a document store whose records have a stable document ID, tenant ID, source version, and access policy. Give the ingestion worker read access to that source and write access to the retrieval database; do not give the request-serving API permission to rewrite source documents. That separation makes a bad reindex recoverable without allowing a compromised API process to alter the evidence it retrieves.
For a first deployment, I would use PostgreSQL 16 with pgvector, an object store for original documents, and a queue such as Amazon SQS for change events. PostgreSQL wins here because the team can inspect document metadata, transaction state, and retrieval candidates with ordinary SQL. A dedicated vector service such as Qdrant wins when vector volume or search load has outgrown that arrangement; it costs another service to secure, back up, and reconcile with document permissions. Neither option removes the need to authorize each result.
Make ingestion idempotent before generating an embedding. An event should identify a document and its source version, not contain an untrusted copy of its text. The worker fetches the current document, checks its deletion state, splits it into chunks, embeds those chunks, and replaces the previous version in one database transaction. A uniqueness constraint on document ID, source version, and chunk position prevents an SQS redelivery from creating duplicates. On deletion, remove the chunks and record the processed version; otherwise, a delayed older event can restore text that should no longer be searchable.
For an initial tokenizer setting, try chunks of 400 tokens with 60 tokens of overlap; both are values to tune against your documents and evaluation queries, not universal limits. Record the splitter version alongside the embedding model ID, since changing either can make old and new vectors incomparable. OpenAI publishes a default dimension of 1,536 for text-embedding-3-small; choose the table dimension from the embedding configuration actually deployed, and reject vectors with the wrong dimension at ingestion.
The model-choice framing (RAG vs Fine-Tuning in Your First GenAI Backend) is premature as a launch gate because neither choice specifies who may retrieve a chunk. I would not put all tenants into a shared vector search and ask the model to ignore unauthorized results: once the text has reached the prompt, a prompt instruction is not an access control.
A small database you can recreate beats an elaborate search cluster you cannot audit
Bring up a local retrieval database first. The following Bash snippet starts a PostgreSQL 16 image with pgvector and creates the minimum chunk table. It binds the port to loopback for local development; it is not a production networking or password configuration.
docker run -d --name rag-db -e POSTGRES_PASSWORD=localonly \ -p 127.0.0.1:5432:5432 pgvector/pgvector:pg16 until docker exec rag-db pg_isready -U postgres >/dev/null 2>&1; do sleep 1; done docker exec -i rag-db psql -U postgres -v ON_ERROR_STOP=1 <<'SQL' CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE chunks ( tenant_id text NOT NULL, document_id text NOT NULL, chunk_id text PRIMARY KEY, source_version text NOT NULL, body text NOT NULL, embedding vector(1536) NOT NULL, indexed_at timestamptz NOT NULL DEFAULT now() ); CREATE INDEX chunks_tenant ON chunks (tenant_id, document_id); SQL
Pin the container image by digest in deployment so a rebuild cannot silently change database binaries. Put PostgreSQL on a private network, use a secret manager instead of the local password, enable TLS for database connections, and run migrations with a separate role. Back up both the database and original documents: vectors can be regenerated, but rebuilding them takes time and requires the exact source versions and embedding configuration.
Next, implement a query that joins chunks to authorized documents before selecting the nearest matches. Use the authenticated principal from the API gateway or OIDC token to resolve allowed document IDs on the server; never accept a tenant ID from the request body as proof of access. A tenant predicate is a useful first filter, but document-level permissions still matter when people within one tenant have different access. Consider PostgreSQL row-level security as a second guard, with tests run under the restricted application role; table owners can bypass policies unless configured appropriately.
Start with exact pgvector cosine-distance search using the <=> operator over the authorized candidate set. It may be fast enough when that set is small, and its behavior is easier to verify than an approximate index. Add an HNSW index only after query plans and load tests show the exact scan is the bottleneck: approximate search can return fewer useful matches after a selective permission filter, so test result quality as well as latency. Keep the retrieved chunk IDs and source versions with each response for audit and reproducible debugging.
A working setup is more than a successful SQL query. The request path should authenticate the caller, embed the question, fetch authorized chunks, construct a prompt with source identifiers, call the generation model, and return an answer with citations that map to those identifiers. If retrieval returns no authorized evidence, return an explicit “no supporting source found” response rather than letting the model answer from general knowledge; that behavior keeps a document-grounded feature from silently changing its contract.
Timeouts and traces are part of the feature, not later hardening
Set one end-to-end deadline and divide it among embedding, retrieval, and generation. As an initial service-level objective, 900 ms at p95 might be a target to test for the retrieval portion, not a claim about pgvector performance. Leave room for the model call and network transit in the overall deadline. Use cancellation across HTTP and database calls so a disconnected request does not keep consuming connections and model quota.
Instrument the path with OpenTelemetry spans for authentication, query embedding, authorized-candidate lookup, vector search, prompt assembly, and model response. Export duration histograms and error counters to Prometheus; calculate p95 with histogram_quantile() over an appropriate rate window. Track candidate count, returned chunk count, PostgreSQL pool wait time, and the age of the newest indexed source version. These measurements distinguish a slow search from a queue backlog or a model-provider delay, which require different fixes.
The scaling warning (Your GenAI Feature Scales Until Retrieval and Latency Collide) deserves a narrower operational test: retrieval and latency collide only when traces show which resource is saturated. Run a staging workload with, for example, 30 concurrent virtual users as an initial test setting, then vary concurrency while holding the query set and authorization distribution fixed. Include common queries, low-result queries, and tenants with very different document counts. A single aggregate p95 can hide the tenant whose authorized search space is much larger.
Configure separate timeouts and concurrency limits for ingestion and serving. An embedding provider outage should stop new indexing without exhausting API workers, while an SQS dead-letter queue should retain failed events for inspection. During a generation-provider outage, do not automatically retry every live request: retries can multiply traffic precisely when capacity is scarce. Return a bounded error, preserve the trace ID, and let the client retry according to its own backoff policy.
A release is safe only when access, freshness, and recovery have tests
Build a small evaluation set before opening traffic. Include questions with a known supporting chunk, questions with no supporting chunk, and questions where a tempting matching chunk belongs to another principal. Test the retrieved IDs separately from the generated prose; a fluent answer cannot prove that retrieval honored permissions. For each deployment, run these cases against the same source versions and record whether the expected chunk appears in the authorized top results.
Add three failure drills to the deployment checklist. First, revoke document access after it has been indexed and verify that the next request cannot retrieve it; this tests authorization at query time rather than assuming index updates are instantaneous. Second, delete a document and replay an older ingestion event to confirm that version checks prevent resurrection. Third, stop the embedding provider and verify that serving continues from existing indexed data while ingestion failures go to a visible queue.
Define freshness as an observable delay between a committed source version and its searchable version. A five-minute maximum lag is one possible acceptance threshold to adjust to the update rate and user expectation; alert on sustained breaches rather than every short queue spike. Also alert when the dead-letter queue grows, database connections approach their configured limit, or authorized-result counts drop abruptly after a deployment. Each alert should point to a runbook action: pause ingestion, roll back a splitter change, drain a bad event, or restore a database snapshot.
Roll out to a small group with citations and trace IDs visible to support staff. Keep the previous embedding model and index available until the new version passes access tests and query evaluation, because changing embedding versions in place makes regressions hard to isolate. If costs or latency rise, inspect span data and query plans before buying a faster vector service; moving the index will not repair an authorization join, an unbounded connection pool, or a slow provider call.
The first production action is to prove a forbidden document stays forbidden
Create two test principals with overlapping source material but different document permissions. Index one document each, then issue the same question under both identities and inspect the retrieved chunk IDs, not merely the answers. Run that test after an access revocation as well. If it fails, stop there: tuning prompts, indexes, and latency against a retrieval path that leaks evidence only makes the wrong setup faster.
