Trio-Lumen 1.0 · November 21, 2026

Trio-Lumen 1.0: subtask annotation for robot and egocentric video, by API

If you record egocentric or teleoperation video for robot learning teams, annotation sits between the recordings in your storage and the dataset you deliver. Trio-Lumen is a robot video annotation model behind a batch API. You send video files. Each video that is processed successfully returns one JSON document with timestamped subtask annotations that say what each hand does.

A selected 24-second excerpt of a 3-minute first-person factory recording, between a title card and an end card. The overlay is the unedited Trio-Lumen 1.0 output for the excerpt, drawn from the result JSON, with times on the source video’s clock. We chose a stretch where the captions follow the action closely. Elsewhere in the same result some object names are wrong (the cup is called a speaker in several segments), so read the full result for the source video as well as the segments in the excerpt. Footage: Egocentric-10K by Build AI, licensed under Apache-2.0 (factory063_worker001_00004.mp4). We trimmed and scaled it and added the overlay. Download the video: 16:9, 1:1.
Text version of the video

Silent video. A worker in red sleeves, seen from a head-mounted camera, wipes gray plastic cups with a cloth at a green workbench, wraps each cup in a clear plastic bag and sets it aside. The segments shown, unchanged from the result:

  1. 45.25–47.65 s: hold cup with right hand, place cloth on table with left hand
  2. 45.25–54.65 s: hold plastic bag with right hand, place tissue on table with left hand, wrap plastic bag around cup with both hands
  3. 47.65–54.65 s: hold plastic container with left hand, wrap plastic bag around plastic container with right hand
  4. 54.65–58.25 s: pass plastic bag from left hand to right hand, straighten plastic bag with left hand
  5. 58.25–59.85 s: place the plastic bag on the top right of the table with the right hand
  6. 59.80–61.80 s: pick up cup with right hand, remove plastic bag from cup with left hand
  7. 59.85–64.35 s: hold cup with right hand, remove plastic bag from cup with left hand, pick up tissue with left hand
  8. 61.80–64.00 s: hold cup with right hand, pick up tissue with left hand
  9. 64.00–69.00 s: hold cup with both hands

Trio-Lumen 1.0 is the first generally available release. In the API, 1.0 is the task version egocentric-actions@2026-11-14-v6. New batches use it by default. Send it as version when you create a batch to pin that batch to it, for as long as the version stays enabled for new submissions. Every batch records the version it ran as processing_version, and so does every result.

Subtask annotation output

This is the file your customer’s training pipeline receives. Each video that succeeds produces one JSON document. Its segments list is ordered by start time. A segment has four fields: index, start_s, end_s and caption. The caption is English free text that says what the left and right hands do. The vocabulary is open, so there is no fixed action list.

{
  "segments": [
    {
      "index": 3,
      "start_s": 7.6,
      "end_s": 8.6,
      "caption": "place the blue cloth on the table with the left hand while holding the hanger with the right hand"
    },
    {
      "index": 4,
      "start_s": 8.6,
      "end_s": 10.2,
      "caption": "hang hanger on rack with right hand"
    },
    {
      "index": 5,
      "start_s": 8.65,
      "end_s": 12.4,
      "caption": "place hanger on rack with right hand, pick up blue shirt with both hands"
    }
  ]
}

This excerpt is from the public sample on the Trio-Lumen page. Segments 4 and 5 overlap: the right hand hangs the hanger while the next action begins. Segments may overlap when actions are concurrent, with at most three at any instant.

01234560s2s4s6s8s10s12s
  1. 00.00–3.60shold hanger with right hand, pick up shirt with left hand, place shirt on hanger with both hands
  2. 13.60–5.00shold hanger with left hand, place shirt on hanger with right hand
  3. 25.00–7.60shold hanger with right hand, pick up shirt with left hand
  4. 37.60–8.60splace the blue cloth on the table with the left hand while holding the hanger with the right hand
  5. 48.60–10.20shang hanger on rack with right hand
  6. 58.65–12.40splace hanger on rack with right hand, pick up blue shirt with both hands
  7. 610.20–12.00spick up blue cloth with both hands
Seven segments from the public sample event_0001_video_0001, spanning 0 to 12.4 seconds. Segment 5 overlaps segments 4 and 6. See the sample output.

The document also carries input metadata, coverage and automated checks:

The exact JSON Schema for any version is served by the API, and the objects reject unknown fields, so a consumer can validate every file it ingests.

Egocentric video annotation in a collection pipeline

The API is asynchronous and works in batches. A batch holds up to 2,000 videos and at most 200 hours of video in total. Each video is an item and is processed independently. An account can have 20 batches open at once.

Main path, left to right
preflightEach input is fetched, inspected and priced.
awaiting_confirmationThe quote is ready. Nothing is charged yet.webhook: batch.awaiting_confirmation
runningThe quoted amount is held. Each video is processed on its own.webhook: item.completed (as items finish)
completedEvery item succeeded. Unused hold is released.webhook: batch.completed
Other outcomes
completed_with_errorsSome items succeeded, some did not. Only succeeded items are charged.webhook: batch.completed
blockedPaused; blocked_reason says why. A batch blocked for insufficient funds resumes once the balance covers it.webhook: batch.blocked
failedNo item succeeded. Nothing is charged.webhook: batch.failed
cancelledCancelled by you, or left unconfirmed for 144 hours.webhook: batch.cancelled
Batch status values and the webhook event sent for each. A quote within the max_amount_usd you set is confirmed automatically when your balance covers it, so the batch goes from preflight straight to running. Not every item outcome sends an event, so list the batch’s items when it finishes.

Failures and review

A file that fails preflight becomes a skipped item with a reason such as unsupported_codec, too_long or bad_timestamps. An item that cannot be processed, or whose result fails verification, ends failed. A result that does not pass the quality checks is held as needs_review: it is not delivered and not charged. We review held items. An approved item becomes succeeded and is charged then, never more than its share of the confirmed quote. Skipped, failed and cancelled items are not charged, and one bad file does not stop the rest of the batch. If you cancel a batch, queued items stop and items already running finish and are charged normally.

Price for robot data labeling at volume

$5 per hour of video, billed per minute, with a 2-minute minimum per charged video. There is no subscription. Usage draws on the same prepaid balance as Trio-Spark. A succeeded item is billed on the video time it covers, rounded up to whole minutes.

Re-submitting the same file (identical bytes) from the same account under the same task version within 90 days of being charged for it is free. A re-run after a failure on your side does not cost twice. See pricing.

One production run

On November 20, 2026 we processed three 4K egocentric recordings through the production API. This is one run, not a turnaround guarantee.

MeasureValue
Videos3, about 20 minutes each (59:53 in total)
Billed62 minutes (21 + 21 + 20)
Quote$5.17
Charge$5.17, equal to the quote
Outcome3 of 3 succeeded, all four checks passed on each
Segments192, 212 and 195
Submitted5:56 PM PT
Quote ready, confirmed6:17 PM PT
Batch completed6:42 PM PT

Submission to quote took 20 minutes 31 seconds. Confirmation to completion took 25 minutes 16 seconds. Preflight fetches every file before it quotes, so transfer speed from your storage is part of the first number. Larger batches take longer, and the first job after a quiet period can take longer to start.

What 1.0 does not do

Start with one batch

Get an API key with Trio-Lumen enabled, add a prepaid balance, and submit a few files. The full request and response shapes are in the API docs.

export TRIO_API_KEY=...            # a key with Trio-Lumen enabled
LUMEN_IDEMPOTENCY_KEY=$(uuidgen)   # once per new batch; reuse it on retries

# 1. Create the batch. Preflight inspects each file and computes the quote.
curl -X POST https://platform.machinefi.com/api/lumen/v1/batches \
  -H "Authorization: Bearer $TRIO_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $LUMEN_IDEMPOTENCY_KEY" \
  -d '{
    "task": "egocentric-actions",
    "version": "egocentric-actions@2026-11-14-v6",
    "inputs": [{"url": "https://storage.example.com/raw/episode-001.mp4?X-Amz-Signature=..."}],
    "output": {"type": "hosted"}
  }'

# 2. Poll the batch. BATCH_ID is the id field of the create response.
#    If status is awaiting_confirmation, check quote.amount_usd, then confirm.
#    A $0 quote confirms automatically.
curl https://platform.machinefi.com/api/lumen/v1/batches/$BATCH_ID \
  -H "Authorization: Bearer $TRIO_API_KEY"
curl -X POST https://platform.machinefi.com/api/lumen/v1/batches/$BATCH_ID/confirm \
  -H "Authorization: Bearer $TRIO_API_KEY"

# 3. When the batch finishes, list its items and get each result link.
#    ITEM_ID is the id of an item in that list.
curl https://platform.machinefi.com/api/lumen/v1/batches/$BATCH_ID/items \
  -H "Authorization: Bearer $TRIO_API_KEY"
curl https://platform.machinefi.com/api/lumen/v1/items/$ITEM_ID/result \
  -H "Authorization: Bearer $TRIO_API_KEY"

Send a few hours of your own footage.

The quote arrives before any charge, and the result schema is public.

Get an API key Read the API docs · Talk to us about volume