A model isn't shipped because it works — it's shipped because we know how well it works. Cross-sport benchmarking across three production model families: cricket ball tracking, shuttle tracking, badminton player classification. Held-out clips. Held-out venues. Composite scoring across accuracy, latency, and pixel error. The evaluation rigor is the deliverable — the model selection follows from it.
Sports CV teams ship models. They train them, they push them into production, they watch the metrics. The decision which model to ship — between five variants that all look about the same on a single accuracy number — is where most ML teams quietly hand-wave and pick whichever was trained most recently.
That doesn't work at production scale. A model that's 0.5% more accurate but 30% slower isn't better — it's worse for on-device deployment. A model that's recall-optimized at the cost of 4,700 false positives isn't a regression — it might be exactly what the downstream pipeline needs. Single-number comparisons are how teams ship the wrong model.
The benchmarking work documented here treats model selection as its own engineering problem. Composite scores that weight accuracy, latency, and pixel error together. Held-out evaluation sets across both broadcast and KourtEye in-house captures. Honest framing when a number looks too good to trust. The work below is what makes the model selection defensible — not the model training.
Each benchmark below was conducted with held-out evaluation, multi-metric reporting, and (where relevant) variant comparison. The goal isn't to declare a model "good" — it's to know precisely how the model behaves under conditions the model hasn't seen, against alternatives that might have been chosen instead.
The job: pick the best player-classification model for the KourtEye stack from six candidate variants spanning three YOLO families (v8n, v11n, v26n) and multiple training revisions. Constraint: deploys on the same broadcast and KourtEye footage, evaluated on the same held-out clips, scored on the same composite metric.
Methodology: each variant evaluated on overall accuracy, frame-level detection rate, mean pixel error, and ms-per-frame inference speed. A composite score combines all four — so a model with 0.1% higher accuracy but 30% slower inference doesn't win automatically. The composite is the decision; the per-metric numbers are the diagnostic.
| Model | Overall Acc | Detection | Pixel Error | Speed | Composite |
|---|---|---|---|---|---|
| YOLOv8n V2 | 98.29% | 98.52% | 8.95 px | 31.70 ms | 0.923 |
| YOLOv8n V1 | 98.20% | 98.72% | 8.93 px | 32.10 ms | 0.911 |
| YOLOv11n V3 | 97.58% | 97.66% | 9.77 px | 33.80 ms | 0.702 |
| YOLOv11n V2 | 96.70% | 96.79% | 9.38 px | 31.29 ms | 0.804 |
| YOLOv26n V1 | 95.88% | 95.96% | 9.96 px | 33.10 ms | 0.621 |
| YOLOv26n V2 | 85.17% | 85.22% | 10.51 px | 29.83 ms | 0.150 |
The shuttle tracker benchmark is the canonical proof-point for the data-flywheel architecture built into the KhiladiPro Annotator. Same model architecture (TrackNetV5) across multiple retraining cycles, 80% → 93% accuracy — the lift is attributable to training data quality, loss-function tuning, and targeted augmentation, not architecture changes.
Held-out methodology: evaluation set spans both broadcast badminton footage AND KourtEye in-house captures, with venue-level separation between training and evaluation. Random clip splits on the same venue inflate accuracy — the held-out venue layer is what makes the 93% defensible.
Two variants compared at the end of the benchmark cycle: heavy TrackNetV5 (server-side, 93% accuracy) and TrackNetV5Lite (INT8 dynamic, 0.29 MB on-device, 69.5 FPS on Apple CPU). The Lite variant carries a measurable accuracy delta from the heavy version — acceptable for on-device deployment, not for analytics paths.
Cricket ball tracking benchmark across 17 clips, 1,757 frames. The model: TFLite FP16 ball detector, 3.41 MB on-disk, 16.5 FPS CPU inference. The benchmark output produced a number that looked great in isolation — 96.2% per-frame recall — and a number that looked bad in isolation — ~44% precision, 738 false positives across the same frames.
The honest read: this is a recall-optimized localization layer, not a fully-deployable ball detector. The model finds the ball on essentially every frame where the ball is visible (recall) — but it also fires on frames where the ball isn't (precision). For the production cricket pipeline this is acceptable because landing, release, and contact frames are manually marked, so the downstream analytics only consume ball positions on human-confirmed event frames, not raw detections.
Separately, the cricket stumps annotation accuracy was benchmarked at 99% sub-pixel — the prediction is essentially landing on the corrected box on both bowler-end and batter-end stumps. This is the strongest single number in the cricket pipeline and is what makes downstream line-and-length analytics geometrically reliable.
cric_clip_9 had a mean error of 17.9 px — about 2× the typical clip. Reporting median alongside mean is what makes the benchmark honest — a single number would have hidden the outlier behavior. This is exactly the kind of failure-mode signal the next flywheel cycle needs to target.
What "good benchmarking" actually means at production scale. These aren't theoretical — every benchmark above followed them.
Single held-out splits aren't enough. Production benchmarks use held-out clips AND held-out venues — venues the model has never seen any training data from. If you train on Wankhede footage and test on Wankhede footage, you're testing the venue, not the model. Cross-venue separation is the only way to surface the systematic biases (lighting, court color, camera setup) that random splits inherit.
Production models don't optimize for one metric — they balance accuracy, latency, pixel precision, and memory. The badminton player classification benchmark used a composite score across all four, and the selected model (YOLOv8n V2) wasn't the leader on any single metric — it was the best tradeoff. Single-metric leaderboards ship the wrong model.
Reporting just mean pixel error hides outlier clips. Reporting just median hides systematic skew. Always both — the gap between them is the signal that tells you whether the model has consistent error or occasional catastrophic failures. The cricket ball benchmark's mean-vs-median gap is what flagged cric_clip_9 as a failure-mode candidate worth investigating.
Detection rate alone is incomplete. The cricket ball tracker's 96% recall and ~44% precision tell different stories — and both matter, depending on how the downstream pipeline consumes the output. Bundling them into "accuracy" hides the architectural decision (recall-optimized for manual event filtering vs precision-optimized for autonomous detection).
Benchmarking quality is bounded by data quality. The work below is the foundation that every model above sits on — datasets curated, annotations validated, ground truth generated, research projects scoped. The unglamorous work that makes the headline numbers trustworthy.