A vision API can win a demo and lose the procurement decision: the invoice may be predictable while data retention, camera drift, and migration remain untested. For a first production app, I would buy inference only from a vendor that passes a priced exit test. If the contract or interface prevents that test, I would start with a licensed model running in-house, even if its initial accuracy is lower.
A vendor trial should start with an exit test, not a leaderboard
Before asking vendors for accuracy claims, write down what you would need to replace them. That means access to the original images where your rights permit it, human-reviewed labels, model outputs, confidence values, timestamps, and enough request metadata to replay a decision. A supplier that returns only a category name makes replacement harder because you cannot reconstruct thresholds or diagnose disagreements after the fact.
Ask each candidate to process the same held-out images through the interface you would actually deploy. A useful trial set includes ordinary images, low-light captures, motion blur, empty frames, and examples from every camera type you expect to support. Set 2,000 labeled images as an initial trial size to tune, then increase it if important slices contain too few examples to estimate error rates. Keep near-duplicate frames from the same event together when splitting data; otherwise, a test can reward recognition of a scene rather than performance on a new one.
Record results by slice, not just in one aggregate score. A measured 95% overall agreement in your trial would still be unacceptable if one camera’s false-accept rate exceeded the limit you set for that camera. For each vendor, report false accepts and false rejects at a threshold selected on a separate development set. Do not move the threshold after inspecting held-out results, because doing so turns the test set into another tuning set.
The simplest comparison artifact is a JSON Lines file with one reviewed truth value and one prediction per candidate. The following script runs with Python’s standard library; give it an eval.jsonl file whose rows contain slice, truth, api, and onnx fields. Use the same categorical output vocabulary for both candidates.
python3 - <<'PY'
import json
from collections import defaultdict
rows = [json.loads(s) for s in open("eval.jsonl") if s.strip()]
by = defaultdict(lambda: [0, 0])
for r in rows:
for name in ("api", "onnx"):
key = (name, r["slice"])
by[key][0] += int(r[name] != r["truth"])
by[key][1] += 1
for (name, sl), (errors, n) in sorted(by.items()):
print(name, sl, f"errors={errors}/{n}", f"rate={errors/n:.3f}")
PY
That script counts disagreements, not detection quality; use intersection over union and class-specific precision and recall when the app depends on bounding boxes. Make a human review procedure part of the trial, since “ground truth” produced by one of the competing services would favor that service. I would not accept a vendor’s public benchmark as the procurement test, because its image distribution, annotation rules, and decision threshold are unlikely to match yours.
Amazon Rekognition wins on setup; ONNX Runtime wins on control
Compare two concrete options: Amazon Rekognition DetectLabels as the bought service and ONNX Runtime 1.20 running a properly licensed vision model as the in-house path. Rekognition wins when its labels match your decision, request volume is uncertain, and you need an operational endpoint quickly; you pay per applicable API request, plus engineering time for mapping labels, handling failures, and reviewing AWS service terms. ONNX Runtime wins when you need a fixed preprocessing path, local inference, or tighter control over output formats; you pay for model selection, compute, deployment, monitoring, and an owner for every regression.
Neither option should receive credit for work it does not perform. Rekognition’s MinConfidence parameter can filter returned labels, but a setting of 80 is a threshold to tune against your error costs, not proof that returned results are correct 80% of the time. ONNX Runtime executes a model efficiently, but it does not supply an appropriate model, usable training data, or a license to ship the weights. If you consider an Ultralytics YOLO11 export, review its AGPL-3.0 and enterprise licensing paths with counsel before treating an ONNX file as unrestricted.
Use OpenCV 4.10 to make resizing, color conversion, and image orientation explicit in the trial harness. Record the exported ONNX opset, such as 17 where compatible, and the ONNX Runtime execution provider, such as CPUExecutionProvider or CUDAExecutionProvider; these settings matter because a benchmark from one preprocessing and hardware path does not establish production latency on another. If deployment needs NVIDIA TensorRT 10 or Triton Inference Server 2.50, test the exported graph there before approving a model, since a successful notebook inference does not prove the target runtime accepts every operation.
Set a 200 ms p95 server-side inference budget as a provisional design target if that fits your app, and measure it under the concurrency you intend to support. Separate decode, preprocessing, inference, and network time using OpenTelemetry traces; a vendor may have fast model execution while upload and response transfer dominate the user-visible wait. Track a Prometheus latency histogram rather than an average, because the p95 estimated with histogram_quantile() will expose slow requests that an average conceals.
Read Custom Vision Model or Pretrained API Which Fits Your First App before soliciting model proposals. I would make reversibility the earlier decision, because an excellent model is a poor purchase if you cannot reproduce its inputs and outputs elsewhere.
The contract can erase an apparent accuracy advantage
Run the technical trial in parallel with a contract review. Ask whether uploaded images are retained, whether they can be used to improve the supplier’s services, where processing occurs, how deletion is verified, and what changes require notice. Get the answers in the governing agreement or service terms, not only in a sales email, because your team will need enforceable terms if policies change.
Price three years of use with a volume model you can explain. For example, treat 1 million images per month as a planning scenario to revise, and include retries, duplicate submissions, minimum commitments, storage, outbound transfer, and human review. Price the in-house alternative with compute, a spare-capacity allowance, model maintenance, and on-call time. Compare both at low and high volume: per-request pricing can be attractive while demand is uncertain, whereas owned capacity becomes easier to justify when utilization is steady.
Then ask the vendor to perform an exit drill. Give its team a written request for your available images, annotations, outputs, and configuration, and time how long it takes to receive a usable export. Require stable identifiers so exported decisions can be joined to your records. A promise that “data is portable” is weak evidence because portability depends on file formats, identifiers, rights, and the effort needed to replay decisions.
Read CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated after pricing that exit. A revised build-versus-buy argument is useful, but I would reject any shortcut from “buying is faster” to “this supplier is replaceable”; the second claim requires an export and replay test.
Open-source choices need a contract review too. Check model-weight terms separately from framework terms, and retain a software bill of materials using SPDX 3.0 or CycloneDX 1.6 if your organization supports those formats. ONNX Runtime’s license does not grant rights to every model loaded into it, so record the provenance of weights, training datasets where known, and any commercial-use restrictions before counting self-hosting as the safer option.
A purchase decision should have a failure gate and an owner
Turn the trial into a short approval memo with gates that can fail. One gate should cover error by image slice; another should cover latency under load; a third should cover deletion, export, and contract terms. Assign an owner to each gate, since an unowned exception is likely to become an implicit approval. Set a 30-day shadow period as an initial operating window to adjust, during which the candidate produces results without controlling the final decision.
During shadow operation, log the model or API version, request identifier, preprocessing version, threshold, and human override. Sample disagreements for review rather than sending every image back to an annotator, because review cost can overwhelm a low per-request inference price. Give reviewers a documented rule for ambiguous images; inconsistent labels can make either option appear unstable even when its predictions have not changed.
Write the rollback trigger before launch. For example, choose a 1% false-accept ceiling for a high-cost decision only if your product owner agrees that it reflects the real cost of errors; otherwise, set a different value and document why. Trigger rollback on a sustained breach in the affected slice, not merely on an overall average. Keep the previous inference path callable until you have tested rollback, because a feature flag that points to a retired endpoint is not a recovery plan.
Finally, decide who can change the system. A managed API still needs an owner for changed label taxonomies, quotas, authentication, and vendor notices. A self-hosted model needs an owner for patches, capacity, model updates, and drift investigations. Approve the option whose ongoing duties have named people and budget, since an apparently cheaper path can become expensive when those duties surface during an incident.
The first approval artifact should be a paired test set
Tomorrow, select one real decision your app makes and collect its first reviewed, consent-appropriate image sample. Define the output schema and error cost before inviting either vendor demos or model proposals. Then send both candidates the same test and ask for a priced export. The winner is the option your team can operate and replace, not the one with the most persuasive demo.
