“Build or buy the AI” is the wrong question for a CTO in 2026. The useful boundary is narrower: buy model capability until your own evidence shows that owning inference or training improves the economics. Build the evaluation, data controls, and fallback paths yourself. Those assets remain valuable when an API price changes, an open-weight model improves, or a supplier’s terms stop fitting your product.
The default has shifted from owning a model to owning the decision boundary
Two years ago, advice to prototype with an API and then train a custom model often sounded like a natural progression. I would no longer treat training as the next stage. Better multimodal APIs have made a purchased baseline more useful, while serving software such as vLLM has made self-hosting a credible alternative without making training necessary. These changes separate three decisions that older build-versus-buy advice bundled together: whose weights to use, where inference runs, and who controls the product’s acceptance criteria.
My default is to buy access to a capable model, own the evaluation set, and keep the inference boundary replaceable. I would not train a proprietary vision model or fine-tune a language model merely because a pilot works, because pilot traffic rarely reveals enough about failure distribution or operating cost to justify an irreversible investment. Training becomes defensible when a labeled dataset represents production conditions, a purchased baseline misses a quantified requirement, and the expected gain exceeds annotation, retraining, and serving costs.
This is not an argument for letting a supplier define success. A CTO should own the ground-truth examples, review rules, data-retention requirements, and release gate. For generated answers, that gate may require an answer to cite an authorized source and refuse when evidence is absent. For images, it may require a specified precision at a tolerable false-rejection rate. Those are product decisions; a vendor’s aggregate benchmark cannot substitute for them because its test distribution and error costs differ from yours.
Custom Vision Model or Pretrained API Which Fits Your First App frames the first application as a model choice; I would make the first choice a reversible contract and test harness instead, because a promising first model says little about how cheaply the second one can be substituted. Put API calls behind an internal interface that records model identifier, prompt or preprocessing version, request cost, latency, and adjudicated outcome. That interface is worth building even if the supplier is never replaced.
Recent platform changes have made old procurement shortcuts unreliable
OpenAI’s Structured Outputs, introduced in 2024 with JSON Schema enforcement through strict: true on supported models, weakened the old claim that teams must fine-tune simply to get parseable fields. Schema conformity does not establish factual correctness, however, because a well-formed JSON object can still contain an invented value. Similarly, multimodal APIs can establish an image baseline quickly, but their availability does not prove that their terms permit every image to be sent off-site. The current procurement question is therefore less “Can an API do this?” and more “Which obligations survive a provider change?”
Self-hosting has also changed. vLLM’s OpenAI-compatible server and Hugging Face Transformers reduce the work needed to try open-weight models, while ONNX Runtime can simplify deployment of some vision pipelines. None removes the cost of operating them: somebody still owns GPU capacity, upgrades, security patches, load testing, and incident response. An open-weight license must also be checked model by model, because “downloadable” does not establish permission for a particular commercial use.
Older advice to compare only token or per-image prices is now especially weak. A quote omits retries, image resizing, prompt caching rules, minimum capacity, and human review. It can also hide a costly exit. Ask vendors for retention and deletion terms, region controls, model-version notification, rate limits, and an export path for logs and annotations. If personal data is involved, check the processor agreement against GDPR Article 28 rather than treating an API’s security page as a contract, because operational assurances and legal obligations are different artifacts.
Do not infer that a standards badge settles model risk, either. ISO/IEC 42001:2023 describes an AI management system, and the NIST AI RMF 1.0 offers a risk-management framework; neither certifies that a specific response is correct for your application. Use those frameworks to organize ownership and evidence. Use your own test set to decide whether to ship.
Pretrained APIs win the first test; self-hosting wins only under a proven constraint
The explicit comparison is between a managed pretrained API and self-hosted open-weight inference. The managed API wins when demand is uncertain, model quality changes quickly, and a small team needs to test a product assumption: its cost is variable request spend, vendor dependence, and whatever data-handling restrictions the contract imposes. Self-hosting wins when sustained utilization, residency requirements, or a necessary model modification outweigh operations overhead: its cost is GPU capacity, engineering time, on-call responsibility, and often idle headroom. Neither option wins merely because its demo is faster.
Set thresholds before running the comparison. For example, a p95 end-to-end latency target of 800 ms is a value to tune to the user interaction, not a universal benchmark. A 30-day shadow run is a proposed observation window, long enough to include at least some traffic variation but not a substitute for seasonal testing. A 95% precision floor might be an application’s chosen acceptance criterion when false positives trigger expensive review; another product may rationally favor recall. Make the assumptions explicit so a team cannot move the goalposts after seeing a preferred model’s results.
Measure cost per accepted result, not cost per request. A cheap call that fails and is retried twice may cost more than one expensive successful call; a response requiring manual correction has another cost again. Record p50 and p95 latency, error rate, retry rate, abstention rate, and reviewer time by model version. OpenTelemetry can connect application traces to inference spans, but pin the semantic-convention version used by your instrumentation because GenAI attribute names have evolved. Export the underlying events as well as a dashboard so a provider switch does not erase your comparison.
The following standard-library Python example runs without an SDK. Its 100 generated rows and dollar amounts are illustrative inputs, not measured performance; replace the rows with request logs to calculate a meaningful decision metric.
python3 - <<'PY'
import math
rows = [(110 + (i * 37) % 390, 0.002 + (i % 4) * 0.001,
i % 9 != 0) for i in range(100)]
latencies = sorted(ms for ms, cost, accepted in rows)
accepted = sum(ok for ms, cost, ok in rows)
spend = sum(cost for ms, cost, ok in rows)
p95 = latencies[math.ceil(0.95 * len(latencies)) - 1]
print(f"p95_ms={p95}")
print(f"accepted={accepted}/{len(rows)}")
print(f"cost_per_accepted_usd={spend / accepted:.4f}")
PY
Run the same scoring code on both options’ logs, with reviewer decisions attached to identical requests. If a self-hosted model misses the acceptance floor, lower per-request spend is not a saving, because the missing quality must be paid for through retries, review, or lost users.
Separate the evidence that changes a model decision from the evidence that changes a system decision
Not every bad AI result calls for a different model. In a retrieval-backed feature, a correct model cannot cite a document that retrieval failed to supply. In a vision pipeline, a stronger classifier cannot recover detail discarded by an aggressive image resize. Before approving training or a hosting migration, require an error review that assigns each failure to input quality, retrieval or preprocessing, model behavior, policy, or application integration. The categories make spending accountable because each has a different plausible remedy.
Your GenAI Feature Scales Until Retrieval and Latency Collide correctly warns that inference is not the whole response path; I would resist treating that collision as a reason to own model serving, because a slow search index remains slow after an API is replaced with a GPU. Instrument request ingress, retrieval, reranking, inference, and post-processing separately. For image requests, record decode and preprocessing time as well as inference. Compare p95 traces at the same concurrency, since a single-request benchmark conceals queueing.
Keep model and system scorecards separate. A model scorecard can show precision, recall, calibration, and abstention on a versioned test set. A system scorecard should show end-to-end latency, cost per accepted result, review load, availability, and the proportion of requests excluded by data policy. If both improve after a change, trace the mechanism before attributing the gain to the model; otherwise, the next procurement decision may reward the wrong supplier.
This separation also makes an exit plan credible. Store evaluation examples in a portable format, version transformations, and preserve enough request metadata to replay cases without retaining prohibited content. Test a second provider or a locally served model against the same gate before negotiating a long commitment. A fallback that has never passed that gate is a negotiating story, not an operational option.
Run a shadow test before signing an annual contract
Take one representative week of requests, remove data that cannot be used for testing, and have reviewers label a failure sample before comparing suppliers. Then run a managed API and a self-hosted candidate in shadow mode against the same acceptance rules. Ask finance for fully loaded costs and legal for the actual data terms. Sign for capacity only after the traces show which constraint ownership would remove.
