Work · KhiladiPro · Evaluation Methodology

Benchmarking.

A model isn't shipped because it works — it's shipped because we know how well it works. Cross-sport benchmarking across three production model families: cricket ball tracking, shuttle tracking, badminton player classification. Held-out clips. Held-out venues. Composite scoring across accuracy, latency, and pixel error. The evaluation rigor is the deliverable — the model selection follows from it.

Model Families
3 Sports
Player Class Variants
6 Compared
Best Selected Model
98.29%
Methodology
Composite scoring
01 / The Problem

Picking the best model is harder than training a good one.

Sports CV teams ship models. They train them, they push them into production, they watch the metrics. The decision which model to ship — between five variants that all look about the same on a single accuracy number — is where most ML teams quietly hand-wave and pick whichever was trained most recently.

That doesn't work at production scale. A model that's 0.5% more accurate but 30% slower isn't better — it's worse for on-device deployment. A model that's recall-optimized at the cost of 4,700 false positives isn't a regression — it might be exactly what the downstream pipeline needs. Single-number comparisons are how teams ship the wrong model.

The benchmarking work documented here treats model selection as its own engineering problem. Composite scores that weight accuracy, latency, and pixel error together. Held-out evaluation sets across both broadcast and KourtEye in-house captures. Honest framing when a number looks too good to trust. The work below is what makes the model selection defensible — not the model training.

02 / Benchmarks

Three model families. One methodology.

Each benchmark below was conducted with held-out evaluation, multi-metric reporting, and (where relevant) variant comparison. The goal isn't to declare a model "good" — it's to know precisely how the model behaves under conditions the model hasn't seen, against alternatives that might have been chosen instead.

— Benchmark 01 / KourtEye

Badminton Player Classification · 6-variant comparison

Selected: YOLOv8n V2 · 98.29%

The job: pick the best player-classification model for the KourtEye stack from six candidate variants spanning three YOLO families (v8n, v11n, v26n) and multiple training revisions. Constraint: deploys on the same broadcast and KourtEye footage, evaluated on the same held-out clips, scored on the same composite metric.

Methodology: each variant evaluated on overall accuracy, frame-level detection rate, mean pixel error, and ms-per-frame inference speed. A composite score combines all four — so a model with 0.1% higher accuracy but 30% slower inference doesn't win automatically. The composite is the decision; the per-metric numbers are the diagnostic.

Model Overall Acc Detection Pixel Error Speed Composite
YOLOv8n V2 98.29% 98.52% 8.95 px 31.70 ms 0.923
YOLOv8n V1 98.20% 98.72% 8.93 px 32.10 ms 0.911
YOLOv11n V3 97.58% 97.66% 9.77 px 33.80 ms 0.702
YOLOv11n V2 96.70% 96.79% 9.38 px 31.29 ms 0.804
YOLOv26n V1 95.88% 95.96% 9.96 px 33.10 ms 0.621
YOLOv26n V2 85.17% 85.22% 10.51 px 29.83 ms 0.150
Selected Accuracy
98.29%
Detection Rate
98.52%
Mean Pixel Error
8.95 px
Inference Speed
31.70 ms/frame
— What the composite score reveals
Note that YOLOv8n V1 has higher detection rate (98.72%) than the selected V2 (98.52%) — and slightly better pixel error (8.93 vs 8.95). On a single-metric comparison, V1 might have been picked. But the composite score (0.911 vs 0.923) accounts for V2's marginal speed advantage (31.70 vs 32.10 ms) and the overall accuracy bump, and selects V2 as the operating point. This is the decision the composite was designed to make.
— Benchmark 02 / KourtEye

Shuttle Tracking · flywheel-driven iteration

Production: 93% accuracy · lifted from 80%

The shuttle tracker benchmark is the canonical proof-point for the data-flywheel architecture built into the KhiladiPro Annotator. Same model architecture (TrackNetV5) across multiple retraining cycles, 80% → 93% accuracy — the lift is attributable to training data quality, loss-function tuning, and targeted augmentation, not architecture changes.

Held-out methodology: evaluation set spans both broadcast badminton footage AND KourtEye in-house captures, with venue-level separation between training and evaluation. Random clip splits on the same venue inflate accuracy — the held-out venue layer is what makes the 93% defensible.

Two variants compared at the end of the benchmark cycle: heavy TrackNetV5 (server-side, 93% accuracy) and TrackNetV5Lite (INT8 dynamic, 0.29 MB on-device, 69.5 FPS on Apple CPU). The Lite variant carries a measurable accuracy delta from the heavy version — acceptable for on-device deployment, not for analytics paths.

Heavy Accuracy
93%
Lite Detection
90.08%
Median Px Error
3.67 px
Lite Size
0.29 MB
Lite Speed
69.5 FPS
Lift
+13 pp
— Benchmark 03 / Cricket

Cricket Ball Tracking · recall-vs-precision analysis

96.2% recall · 44% precision · honest framing

Cricket ball tracking benchmark across 17 clips, 1,757 frames. The model: TFLite FP16 ball detector, 3.41 MB on-disk, 16.5 FPS CPU inference. The benchmark output produced a number that looked great in isolation — 96.2% per-frame recall — and a number that looked bad in isolation — ~44% precision, 738 false positives across the same frames.

The honest read: this is a recall-optimized localization layer, not a fully-deployable ball detector. The model finds the ball on essentially every frame where the ball is visible (recall) — but it also fires on frames where the ball isn't (precision). For the production cricket pipeline this is acceptable because landing, release, and contact frames are manually marked, so the downstream analytics only consume ball positions on human-confirmed event frames, not raw detections.

Separately, the cricket stumps annotation accuracy was benchmarked at 99% sub-pixel — the prediction is essentially landing on the corrected box on both bowler-end and batter-end stumps. This is the strongest single number in the cricket pipeline and is what makes downstream line-and-length analytics geometrically reliable.

Per-Frame Recall
96.23%
Per-Frame Precision
~44%
Median Px Error
7.57 px
Mean Px Error
10.82 px
Stumps Accuracy
99% sub-pixel
Eval Set
17 clips · 1,757 frames
— The number behind the number
The mean pixel error (10.82 px) is meaningfully higher than the median (7.57 px), which signals outlier clips dragging the mean up. Specifically: cric_clip_9 had a mean error of 17.9 px — about 2× the typical clip. Reporting median alongside mean is what makes the benchmark honest — a single number would have hidden the outlier behavior. This is exactly the kind of failure-mode signal the next flywheel cycle needs to target.
03 / Methodology

Four principles. One standard.

What "good benchmarking" actually means at production scale. These aren't theoretical — every benchmark above followed them.

— 01

Held-out across two layers.

Single held-out splits aren't enough. Production benchmarks use held-out clips AND held-out venues — venues the model has never seen any training data from. If you train on Wankhede footage and test on Wankhede footage, you're testing the venue, not the model. Cross-venue separation is the only way to surface the systematic biases (lighting, court color, camera setup) that random splits inherit.

— 02

Multi-metric, composite-scored.

Production models don't optimize for one metric — they balance accuracy, latency, pixel precision, and memory. The badminton player classification benchmark used a composite score across all four, and the selected model (YOLOv8n V2) wasn't the leader on any single metric — it was the best tradeoff. Single-metric leaderboards ship the wrong model.

— 03

Mean and median, always.

Reporting just mean pixel error hides outlier clips. Reporting just median hides systematic skew. Always both — the gap between them is the signal that tells you whether the model has consistent error or occasional catastrophic failures. The cricket ball benchmark's mean-vs-median gap is what flagged cric_clip_9 as a failure-mode candidate worth investigating.

— 04

Recall and precision, separately.

Detection rate alone is incomplete. The cricket ball tracker's 96% recall and ~44% precision tell different stories — and both matter, depending on how the downstream pipeline consumes the output. Bundling them into "accuracy" hides the architectural decision (recall-optimized for manual event filtering vs precision-optimized for autonomous detection).

04 / Foundation

The data that makes the benchmarks possible.

Benchmarking quality is bounded by data quality. The work below is the foundation that every model above sits on — datasets curated, annotations validated, ground truth generated, research projects scoped. The unglamorous work that makes the headline numbers trustworthy.

— 01
Cricket Shot Type Dataset · 10 shot classes
Cover drive, on drive, off drive, straight drive, pull, square cut, sweep, front-foot defence, back-foot defence, lofted shot, left-alone. Minimum 50 videos per shot type, 2-second clips (bat-lift through follow-through), curated across multiple cricket formats (T20I, ODI, Test, IPL, BBL). Mixed ball types (red/white leather), mixed jersey colors, umpire-view camera. Foundation for the cricket action recognition model.
10 classes
500+ clips
— 02
Bat Detection Dataset · per-frame annotations
Frame-level bat detection annotations on Roboflow — 1,953 images labelled with top-hand keypoints for each frame. Used to train the bat detection model that the bat 4-point quadrilateral tracker (in the Annotator platform) builds on top of. The bat is the hardest-to-annotate object in the cricket pipeline because of its fast rotation and self-occlusion during the follow-through.
1,953
images
— 03
Badminton Doubles Clips · shuttle annotations
25+ YouTube clips, 15-second duration each, trimmed and annotated for shuttle tracking on doubles matches. Doubles is structurally harder than singles for shuttle tracking — 4 players occlude the trajectory more frequently, mixed-doubles court coverage variance, more frequent racket-on-shuttle visual confusion. Foundation for the next-cycle doubles model.
25+ clips
doubles
— 04
TrackNet Ground Truth Labels · model comparison set
Ground truth labels for shuttle position used in TrackNet vs custom models comparison — latency, speed, model size, params, precision, recall. Distance metrics between manually labelled (x,y) pixel coordinates and model inference, with PCK@0.2 tolerance thresholds. This is the dataset that makes the shuttle benchmark defensible — without these labels, every accuracy number is a model marking its own homework.
Multi-clip
+ landing-spots
— 05
KourtEye Ground Truth CSV Generation · evaluation pipeline
Production-grade ground truth CSV generation pipeline for the KourtEye stack. Per-clip CSV outputs containing shuttle position, court keypoints, player classifications — formatted so downstream evaluation scripts can compute accuracy/detection/pixel-error metrics automatically. The benchmark numbers above are produced by this pipeline.
Production
pipeline
— 06
Audio Annotations · hit/miss training data
LabelStudio-based audio annotation pipeline for the bat-ball hit/miss event detection model. Reviewed dataset composition, annotation guidelines, audio modality choice, evaluation metrics. This is the dataset that powers the only event-detection method in the production stack that remains reliable in noisy training-environment footage where visual-only methods fail.
LabelStudio
audio events
— 07
Pickleball Research Exploration
Research scoping for a new sport — pickleball trajectory tracking and rally segmentation. Reviewed reference footage, surveyed existing CV approaches, evaluated feasibility against current TrackNet variants. The kind of exploration that decides whether a sport gets a dedicated production pipeline or stays in research. Currently scoping, not production.
Sport scoping
R&D
05 / What's Next

The benchmark's own evolution.

— A note on detail
This page describes benchmarking methodology, measured results, and the data foundation behind them. Specific deployment infrastructure, internal repository paths, and proprietary algorithms are not shown — this is production work for KhiladiPro, and the engineering substance is what's discoverable here without exposing IP.