Trio-Lumen 1.0: subtask annotation for robot and egocentric video, by API
If you record egocentric or teleoperation video for robot learning teams, annotation sits between the recordings in your storage and the dataset you deliver. Trio-Lumen is a robot video annotation model behind a batch API. You send video files. Each video that is processed successfully returns one JSON document with timestamped subtask annotations that say what each hand does.
factory063_worker001_00004.mp4). We trimmed and scaled it and added the overlay. Download the video: 16:9, 1:1.Text version of the video
Silent video. A worker in red sleeves, seen from a head-mounted camera, wipes gray plastic cups with a cloth at a green workbench, wraps each cup in a clear plastic bag and sets it aside. The segments shown, unchanged from the result:
- 45.25–47.65 s: hold cup with right hand, place cloth on table with left hand
- 45.25–54.65 s: hold plastic bag with right hand, place tissue on table with left hand, wrap plastic bag around cup with both hands
- 47.65–54.65 s: hold plastic container with left hand, wrap plastic bag around plastic container with right hand
- 54.65–58.25 s: pass plastic bag from left hand to right hand, straighten plastic bag with left hand
- 58.25–59.85 s: place the plastic bag on the top right of the table with the right hand
- 59.80–61.80 s: pick up cup with right hand, remove plastic bag from cup with left hand
- 59.85–64.35 s: hold cup with right hand, remove plastic bag from cup with left hand, pick up tissue with left hand
- 61.80–64.00 s: hold cup with right hand, pick up tissue with left hand
- 64.00–69.00 s: hold cup with both hands
Trio-Lumen 1.0 is the first generally available release. In the API, 1.0 is the task version egocentric-actions@2026-11-14-v6. New batches use it by default. Send it as version when you create a batch to pin that batch to it, for as long as the version stays enabled for new submissions. Every batch records the version it ran as processing_version, and so does every result.
Subtask annotation output
This is the file your customer’s training pipeline receives. Each video that succeeds produces one JSON document. Its segments list is ordered by start time. A segment has four fields: index, start_s, end_s and caption. The caption is English free text that says what the left and right hands do. The vocabulary is open, so there is no fixed action list.
{
"segments": [
{
"index": 3,
"start_s": 7.6,
"end_s": 8.6,
"caption": "place the blue cloth on the table with the left hand while holding the hanger with the right hand"
},
{
"index": 4,
"start_s": 8.6,
"end_s": 10.2,
"caption": "hang hanger on rack with right hand"
},
{
"index": 5,
"start_s": 8.65,
"end_s": 12.4,
"caption": "place hanger on rack with right hand, pick up blue shirt with both hands"
}
]
}This excerpt is from the public sample on the Trio-Lumen page. Segments 4 and 5 overlap: the right hand hangs the hanger while the next action begins. Segments may overlap when actions are concurrent, with at most three at any instant.
- 00.00–3.60shold hanger with right hand, pick up shirt with left hand, place shirt on hanger with both hands
- 13.60–5.00shold hanger with left hand, place shirt on hanger with right hand
- 25.00–7.60shold hanger with right hand, pick up shirt with left hand
- 37.60–8.60splace the blue cloth on the table with the left hand while holding the hanger with the right hand
- 48.60–10.20shang hanger on rack with right hand
- 58.65–12.40splace hanger on rack with right hand, pick up blue shirt with both hands
- 610.20–12.00spick up blue cloth with both hands
event_0001_video_0001, spanning 0 to 12.4 seconds. Segment 5 overlaps segments 4 and 6. See the sample output.The document also carries input metadata, coverage and automated checks:
inputcarries the file’ssha256, the video stream duration and the timestamp of the first frame. Segment times use the source player’s clock.covered_secondsis the union of the captioned intervals. Overlap is counted once, for coverage and for billing.qareports four automated checks (overlap_depth_ok,no_gaps,covers_input,valid_captions) and two counts,segmentsandfallback_segments. These checks validate structure and coverage. They do not grade whether a caption is correct.
The exact JSON Schema for any version is served by the API, and the objects reject unknown fields, so a consumer can validate every file it ingests.
Egocentric video annotation in a collection pipeline
The API is asynchronous and works in batches. A batch holds up to 2,000 videos and at most 200 hours of video in total. Each video is an item and is processed independently. An account can have 20 batches open at once.
preflightEach input is fetched, inspected and priced.awaiting_confirmationThe quote is ready. Nothing is charged yet.webhook: batch.awaiting_confirmationrunningThe quoted amount is held. Each video is processed on its own.webhook: item.completed (as items finish)completedEvery item succeeded. Unused hold is released.webhook: batch.completedcompleted_with_errorsSome items succeeded, some did not. Only succeeded items are charged.webhook: batch.completedblockedPaused; blocked_reason says why. A batch blocked for insufficient funds resumes once the balance covers it.webhook: batch.blockedfailedNo item succeeded. Nothing is charged.webhook: batch.failedcancelledCancelled by you, or left unconfirmed for 144 hours.webhook: batch.cancelledstatus values and the webhook event sent for each. A quote within the max_amount_usd you set is confirmed automatically when your balance covers it, so the batch goes from preflight straight to running. Not every item outcome sends an event, so list the batch’s items when it finishes.- Inputs from your storage. Pass presigned
httpsURLs from S3, R2 or GCS. Keep them valid for at least 24 hours. The file is copied during preflight and the URL is not kept after that. Reading from a registered bucket prefix, and writing results back to it, is available and is set up with us. - Formats. MP4 or MOV with H.264 or H.265 video, at 4 frames per second or more, frames up to 4,096 pixels on each side, files up to 10 GB and 70 minutes. 4K HEVC is within these limits. Preflight checks every file, including its timestamps, before it is quoted.
- Quote before spend. Preflight inspects every file and returns a quote. Nothing is charged until you confirm it. Set
max_amount_usdand any quote within that budget confirms itself. The quote is an upper bound: you are never charged more. - Idempotent retries. Creating a batch requires an
Idempotency-Key. Retrying the same request with the same key returns the existing batch instead of creating a second one. Changing any field other thanmetadataunder that key returns a conflict. - Signed webhooks. Events are signed with HMAC-SHA256 and carry a per-batch sequence number. Deliveries can be duplicated or arrive out of order, so deduplicate on
event_idand read the batch from the API before acting on money or results. Polling every 15 to 30 seconds also works. - Results. Hosted results expire after 30 days; check
result_expires_at. Each download link is valid for 15 minutes and comes with the document’ssha256.
Failures and review
A file that fails preflight becomes a skipped item with a reason such as unsupported_codec, too_long or bad_timestamps. An item that cannot be processed, or whose result fails verification, ends failed. A result that does not pass the quality checks is held as needs_review: it is not delivered and not charged. We review held items. An approved item becomes succeeded and is charged then, never more than its share of the confirmed quote. Skipped, failed and cancelled items are not charged, and one bad file does not stop the rest of the batch. If you cancel a batch, queued items stop and items already running finish and are charged normally.
Price for robot data labeling at volume
$5 per hour of video, billed per minute, with a 2-minute minimum per charged video. There is no subscription. Usage draws on the same prepaid balance as Trio-Spark. A succeeded item is billed on the video time it covers, rounded up to whole minutes.
Re-submitting the same file (identical bytes) from the same account under the same task version within 90 days of being charged for it is free. A re-run after a failure on your side does not cost twice. See pricing.
One production run
On November 20, 2026 we processed three 4K egocentric recordings through the production API. This is one run, not a turnaround guarantee.
| Measure | Value |
|---|---|
| Videos | 3, about 20 minutes each (59:53 in total) |
| Billed | 62 minutes (21 + 21 + 20) |
| Quote | $5.17 |
| Charge | $5.17, equal to the quote |
| Outcome | 3 of 3 succeeded, all four checks passed on each |
| Segments | 192, 212 and 195 |
| Submitted | 5:56 PM PT |
| Quote ready, confirmed | 6:17 PM PT |
| Batch completed | 6:42 PM PT |
Submission to quote took 20 minutes 31 seconds. Confirmation to completion took 25 minutes 16 seconds. Preflight fetches every file before it quotes, so transfer speed from your storage is part of the first number. Larger batches take longer, and the first job after a quiet period can take longer to start.
What 1.0 does not do
- One task,
egocentric-actions. Captions are English only. - Captions are free text. You cannot supply your own taxonomy or label set, and a segment has no per-hand fields or confidence score beyond its caption.
- We have not published accuracy benchmarks for Trio-Lumen. Run a sample of your own footage and compare it with your current annotation before you commit volume.
- No turnaround SLA. Webhook delivery is retried for up to 24 hours but is not guaranteed, so reconcile against the API.
- Confirming and cancelling a batch are API calls. Webhook signing secrets and bucket access are issued by our team, not self-serve.
- Videos over 70 minutes or 10 GB need to be split first. Other containers and codecs need transcoding.
Start with one batch
Get an API key with Trio-Lumen enabled, add a prepaid balance, and submit a few files. The full request and response shapes are in the API docs.
export TRIO_API_KEY=... # a key with Trio-Lumen enabled
LUMEN_IDEMPOTENCY_KEY=$(uuidgen) # once per new batch; reuse it on retries
# 1. Create the batch. Preflight inspects each file and computes the quote.
curl -X POST https://platform.machinefi.com/api/lumen/v1/batches \
-H "Authorization: Bearer $TRIO_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $LUMEN_IDEMPOTENCY_KEY" \
-d '{
"task": "egocentric-actions",
"version": "egocentric-actions@2026-11-14-v6",
"inputs": [{"url": "https://storage.example.com/raw/episode-001.mp4?X-Amz-Signature=..."}],
"output": {"type": "hosted"}
}'
# 2. Poll the batch. BATCH_ID is the id field of the create response.
# If status is awaiting_confirmation, check quote.amount_usd, then confirm.
# A $0 quote confirms automatically.
curl https://platform.machinefi.com/api/lumen/v1/batches/$BATCH_ID \
-H "Authorization: Bearer $TRIO_API_KEY"
curl -X POST https://platform.machinefi.com/api/lumen/v1/batches/$BATCH_ID/confirm \
-H "Authorization: Bearer $TRIO_API_KEY"
# 3. When the batch finishes, list its items and get each result link.
# ITEM_ID is the id of an item in that list.
curl https://platform.machinefi.com/api/lumen/v1/batches/$BATCH_ID/items \
-H "Authorization: Bearer $TRIO_API_KEY"
curl https://platform.machinefi.com/api/lumen/v1/items/$ITEM_ID/result \
-H "Authorization: Bearer $TRIO_API_KEY"Send a few hours of your own footage.
The quote arrives before any charge, and the result schema is public.
Get an API key Read the API docs · Talk to us about volume