Auto-acceptp(match) = 1.000success-6865 · label correct
Claimed labelPut the pieces of string into the white basket
Trio-Spark · robot data QA · experiment
Does the clip match its label? Zero-shot checks on 256 DROID episodes
Teleop datasets are sold by the usable hour, and a wrong task label quietly poisons training. We ran Trio-Spark, unchanged and with no robot-data tuning, over 256 episodes of the public DROID set (Cosmos3-DROID release; 160 failure, 96 success). Each check sees 4 frames from one camera and returns a probability, so a pipeline can auto-accept, auto-reject, or route to a human.
What works and what doesn't yet
| Check | Question asked | Third-person cam | Wrist cam | Status |
|---|---|---|---|---|
| Label match | Which of 4 instructions is this clip doing? Distractors from other scenes. | top-1 84.4% AUC 0.95 | top-1 85.4% AUC 0.976 | works |
| Label match, hard | Same, but distractors from the same lab and scene (n = 88). | top-1 55.7% AUC 0.79 | top-1 51.1% AUC 0.78 | triage only |
| Task completion | Given the instruction, was it completed by the end? | AUC ≈ 0.50 | AUC ≈ 0.50 | not yet |
| Outcome, no instruction | Did the episode end in a clean success? | AUC 0.50 | AUC 0.44 | not yet |
994 calls, 0 errors. Chance for 4-way top-1 is 25%; chance AUC is 0.50. Also tried for completion: before/after frame pairs and end-weighted frames (AUC 0.31–0.58), still chance.
Triage on label checks
384 claimed labels (96 correct, 288 swapped in from other episodes). Accept when p ≥ 0.80, reject when p ≤ 0.05, send everything else to a person.
Wrist cam · cross-scene
accept 78, 6 wrongreject 244, 3 wrongreview 62 (16%)
Third-person cam · cross-scene
accept 87, 11 wrongreject 263, 9 wrongreview 34 (9%)
Both cams · same-scene (hard)
accept 44, 12 wrongreject 164, 14 wrongreview 144 of 352 (41%)
Read: on cross-scene swaps the wrist camera clears 84% of claims automatically at about 3% error. Same-scene swaps are harder; a human still sees 41%, and some of those distractors are near-paraphrases of the true label.
Examples
Third-person camera, 4 frames from start to end, as the model saw them.
Auto-acceptp(match) = 1.000success-6871 · label correct
Claimed labelRemove the white cords from the chair and put them in the white basket
Auto-acceptp(match) = 0.999success-47993 · label correct
Claimed labelPress a key on the keyboard.
Auto-rejectp(match) = 0.000success-36235 · label wrong
Claimed labelPut the peach plush toy in the box then put all the fruit toys in the box on the black tray
Actually recordedPut the blue packet inside the plastic bowl
Auto-rejectp(match) = 0.000success-23294 · label wrong
Claimed labelLift the green marker from the white board eraser, use it to draw on the white paper and then put it on the paper
Actually recordedMove the pineapple plushie from the right box to the left box then put the right box on the black plate
Auto-rejectp(match) = 0.000success-46885 · label wrong
Claimed labelPut the ropes inside the basket then remove them
Actually recordedTear one paper towel from the roll.
Send to humanp(match) = 0.720success-3421 · label correct
Claimed labelPut the black objects in the drawer and close it
Competing optionPut the pieces of string into the white basket
Send to humanp(match) = 0.288success-4448 · label correct
Claimed labelPut the ropes in the box one at a time
Competing optionRemove the white cords from the chair and put them in the white basket
Limits and next step
- Success vs. failure is not detectable zero-shot from 4 frames. Telling a near-miss grasp from a clean place needs robot-data training, which is the natural place for a co-training pilot with a data supplier or buyer.
- Swapped labels here are synthetic (taken from other episodes). Real mislabels may be subtler.
- The hard set is small (88 episodes) and has some label noise.
- Each call sees 4 frames; long episodes need windowed checks.