A vision feature can be cheap to launch and expensive to keep honest. For a solo freelancer, the recurring work is not usually retraining; it is finding out that yesterday’s camera, lighting, API response, or customer expectations changed. I would choose a vision system whose failures I can inspect and reverse myself, even if a more sophisticated model wins the initial demo.
The first model choice matters less than the exit path
I treat the first-app comparison, Custom Vision Model or Pretrained API Which Fits Your First App, as a launch checklist rather than a maintenance plan. The choice between a pretrained API and a custom model matters, but neither choice tells you who will diagnose a bad result on a Tuesday night. If the client cannot show you the input, the version that processed it, and the decision that followed, you have inherited an incident without evidence.
A pretrained vision API wins when labels are scarce and its existing categories closely match the job, because you can test a useful workflow before collecting a training set. It costs per request, exposes you to provider changes, and may impose retention or data-transfer terms you need to check. A custom model run with ONNX Runtime wins when the task is narrow, you have permission to retain representative examples, and an API’s outputs repeatedly miss the same requirement. It costs annotation time, evaluation time, and responsibility for serving a pinned model file. Those are different maintenance bills, not a free-versus-paid choice.
My default would be a pretrained API behind one small adapter, with a manual-review path that remains usable if the API fails. I would not train a custom model merely to avoid an API bill, because a small monthly invoice is easier for one person to manage than an unmeasured labeling and retraining obligation. I would change that position only after saved examples show a persistent error the API cannot be configured to avoid.
Make the adapter return your own stable fields, such as decision, confidence, provider, and model_version, rather than passing the provider’s full JSON response into the app. That boundary limits a provider migration to one place because the rest of the code depends on your contract. Pin the provider endpoint or model identifier where the service permits it, and keep a small set of approved test images; a green HTTP response proves transport worked, not that the answer stayed the same.
There is also an exit cost in the images themselves. If your examples contain client data, confirm who may keep them before building a training archive, because a model you cannot legally or contractually retest is difficult to maintain. Record only what you need, set a deletion date, and keep a SHA-256 hash when you need to tell whether two files are identical without keeping duplicate copies.
Every accepted prediction creates a small support obligation
The tempting launch metric is “the model was correct on our test set.” The maintenance question is narrower: which mistakes reach the customer without review? A false positive that merely opens a review screen has a different cost from a false positive that sends an irreversible notification. Set the review threshold around the action’s cost, because the same model score can be acceptable for one action and unacceptable for another.
Start with an explicit record for each decision: a timestamp, a pseudonymous input ID, the model or API version, the score, the threshold in force, the action taken, and whether a human later corrected it. SQLite with PRAGMA journal_mode=WAL is sufficient for a single-process pilot because it makes local writes and later inspection straightforward. If multiple workers will write to the same database across machines, use a managed database instead, because a shared SQLite file is the wrong coordination mechanism.
Do not confuse a confidence score with a probability of being right. If a service returns 0.90, that number may be a ranking signal rather than a calibrated 90% chance, because providers and model architectures define scores differently. The threshold must be chosen against labeled examples from the actual workflow. A provisional threshold of 0.85 is a value to tune, not a recommendation to ship: compare errors above and below it before allowing automatic acceptance.
A workable solo routine is to review a fixed sample of accepted decisions and all customer-reported mistakes each week. A sample of 30 reviewed accepted decisions is an operational starting count, not a statistically sufficient guarantee; small or changing traffic can make its error rate swing sharply. Log the denominator alongside the rate so “zero errors” cannot disguise “zero labels.” Include some low-confidence decisions in the review queue as well, because examining only accepted outputs will hide cases the system repeatedly sends to manual work.
Keep the review record deliberately plain. A CSV export can carry IDs, scores, outcomes, and model versions; a separate, access-controlled folder can hold the few images you have permission to retain. Strip unnecessary EXIF metadata before sharing examples, because camera locations and timestamps can reveal information unrelated to the task. This is less elegant than an annotation platform, but it is easier to audit when you are the only operator.
Monitoring must tell you when to stop trusting automation
I use the production-metrics guide, How to prove your vision model works in production metrics, as a starting point for a stronger rule: a metric earns its place only if it changes what I do next. A dashboard of request counts cannot tell you whether accepted predictions are wrong, because correctness usually arrives later through review or complaint. Write the pause rule before launch so you are not inventing one while a client waits for an answer.
The following Python 3.11 smoke test calculates the error rate among reviewed, automatically accepted examples. It runs as written and intentionally refuses to call a four-example sample healthy. Replace the inline rows with a reviewed CSV export before using it for an operational decision.
python3 - <<'PY'
import csv, io
rows = csv.DictReader(io.StringIO("""score,correct
0.93,1
0.88,0
0.94,0
0.89,0"""))
accepted = [r for r in rows if float(r["score"]) >= 0.85]
n = len(accepted)
rate = sum(r["correct"] == "0" for r in accepted) / n if n else 0
print(f"sample={n} error={rate:.1%}")
print("REVIEW" if n >= 30 and rate > 0.10 else "INSUFFICIENT LABELS" if n < 30 else "OK")
PY
The 10% pause boundary is an example policy to set with the client, not evidence that a 9% error rate is safe. The script also needs an important production companion: report how many requests bypassed review, because an apparently good reviewed sample may not represent the requests the model accepted. If corrections arrive days late, calculate outcomes by decision date and mark recent cohorts incomplete; otherwise, late labels can make a new version appear better than it is.
Track availability separately from correctness. Log HTTP status codes and latency at the adapter, and honor HTTP 429 and the Retry-After header rather than retrying immediately, because a burst of retries can prolong a provider rate limit. In an illustrative pilot, a measured p95 latency of 420 ms across 200 requests would be a baseline for that pilot only; it would not predict a future client’s traffic. If requests start timing out, fall back to manual review rather than silently treating missing predictions as negative results.
I would not deploy Prometheus and OpenTelemetry just to watch one low-traffic freelancer project, because maintaining collectors and dashboards can consume more time than reading a daily export. A scheduled check, an emailed failure notice, and a weekly review appointment are enough until missed incidents show that the simpler arrangement is failing.
A rollback plan saves more time than a retraining plan
Many vision regressions start outside the model. A phone update can change image dimensions; an upload path can rotate pictures differently; a provider can change response fields. Keep a compact fixture set with expected decisions and image properties, then run it with pytest against the adapter before every release. Store the version of the adapter alongside the model identifier, because “same model” does not imply “same preprocessing.”
For a self-hosted model, pin the ONNX file and its checksum, the ONNX Runtime version, and any preprocessing constants such as resize dimensions and color order. A Docker image pinned by digest helps reproduce the software environment because a mutable tag can point to different contents later. It does not preserve the customer’s input distribution, so keep separately approved examples that exercise glare, rotation, blur, and the kinds of mistakes reviewers have actually seen.
For an API, save redacted request and response examples and test your adapter against them. Disable automatic acceptance with a configuration switch that does not require retraining or redeployment, because a freelancer needs a safe response even when the provider is unavailable. Tell the client what the manual path costs in time. An honest delay is preferable to an automatic answer you cannot validate.
Keep the rollback small enough to rehearse: restore the previous adapter release, restore its threshold, process one known fixture, and confirm that new requests enter the intended review path. Run that rehearsal after a change rather than assuming a saved model file is a rollback; configuration, preprocessing, and routing can all change the result while the weights stay identical.
The first maintenance task is to write the pause rule
Before signing off on a demo, take ten recent inputs you are allowed to retain and record the output, version, threshold, and human verdict for each. Then write down who switches off automatic acceptance, what evidence triggers that switch, and how work continues manually. If that procedure cannot fit on one page, simplify the feature before promising the client it will run unattended.
