Documentation

Read the schema before you talk to sales.

Every field you will receive, every delivery format, and the exact shape of a skill spec. Converters ship as readable source rather than a binary, so you can retarget any of this to your own loader.

Formats: RLDS · LeRobot · WebDataset
Also: HDF5 · zarr · Rerun .rrd
Timestamps: int64 ns, PTP domain
Units: Metric — millimetres and metres

Episode schema

One episode, field by field.

This is the canonical layout. Every delivery format below is a projection of it, and nothing in it is optional or upsold.

EP_0117_KITCHEN_A — episode layout

EP_0117_KITCHEN_A/
├── meta.json
│     episode_id            str
│     skill_id              str          # matches your spec line
│     site_id               str          # anonymised capture site
│     participant_id        str          # stable, non-identifying
│     duration_ns           int64
│     clock_domain          "ptp"
│     calib_open            Calibration
│     calib_close           Calibration  # closing solve, for drift
│     calib_drift_mm        float32
│     reject_flags          list[str]    # empty on delivered episodes
│     license               "commercial-buyer-owned-v3"
│
├── streams/
│     head_rgb              uint8   [T, 2160, 3840, 3]   60 Hz
│     wrist_l               uint8   [T, 1080, 1920, 3]   60 Hz
│     wrist_r               uint8   [T, 1080, 1920, 3]   60 Hz
│     depth_mm              uint16  [T,  480,  848]      30 Hz  # metric mm
│     depth_conf            uint8   [T,  480,  848]      30 Hz  # 0-255
│     imu_head              float32 [N, 6]              200 Hz
│     imu_wrist_l           float32 [N, 6]              200 Hz
│     imu_wrist_r           float32 [N, 6]              200 Hz
│     *_timestamp_ns        int64   per stream, PTP domain
│
├── annotations/
│     hand_joints           float32 [T, 2, 21, 3]   # metres, head frame
│     hand_visible          bool    [T, 2, 21]
│     body_joints           float32 [T, 24, 3]
│     contact_events         list[{t_ns, hand, object_id, kind}]
│     instance_masks        uint16  [T, 480, 848]   # tracked object ids
│     objects                list[{object_id, noun, first_seen_ns}]
│     action_segments        list[{start_ns, end_ns, verb, noun}]
│     gaze_target            list[{t_ns, object_id}]
│
└── calibration/
      intrinsics             per camera: fx fy cx cy k1 k2 p1 p2 k3
      extrinsics             4x4 SE(3), head-camera frame is identity
      rms_reproj_px          float32   # must be < 0.28 to ship

T is the number of head-camera frames; N is the number of IMU samples. Streams are stored at native rate and resampled by the loader, never in the delivery.

Delivery formats

Five ways out, same fields.

RLDS

#rlds

Container: TFDS-compatible tfrecord shards
Step keys: observation.head_rgb, observation.depth_mm, observation.hand_joints, action, is_terminal
Episode keys: episode_metadata.skill_id, .site_id, .calib_drift_mm
Shard size: ~1.9 GB, one skill line per directory

Default for anyone building on the Open X-Embodiment tooling.

LeRobot

#lerobot

Container: parquet + mp4 per camera, HF datasets layout
Features: observation.images.head, observation.images.wrist_l/r, observation.state, action
State vector: 2×21×3 hand joints flattened, then 24×3 body joints
Version: v2.1 layout, loads with lerobot.common.datasets

Video is stored as mp4 rather than raw frames, so this is the smallest on-disk option.

WebDataset

#webdataset

Container: .tar shards, 512 samples per shard
Keys per sample: __key__, head.jpg, wrist_l.jpg, wrist_r.jpg, depth.png, ann.json
Ordering: Shuffled across episodes, sequence preserved inside a sample
Streaming: Signed URLs, works directly with wds.WebDataset(url)

Use this when you are training over more hours than you want to stage locally.

HDF5 & zarr

#hdf5

Container: One .h5 or .zarr group per episode
Chunking: Time-major, 30-frame chunks, blosc-zstd level 5
Timestamps: int64 nanoseconds per stream, PTP domain
Use case: Random access into long episodes, sync verification

This is the format our own QA tooling reads, so it is the most complete.

Rerun .rrd

#rrd

Container: Single .rrd recording per episode
Entities: world/head_cam, world/hands, world/points, plots/sync_error
Use case: Look at the data before you write a loader
Open with: rerun EP_0117_KITCHEN_A.rrd

Every stream on one timeline. This is how we recommend a new team first meets the dataset.

Python quickstart

From access to a training loop.

The CLI and the loader are thin wrappers over signed URLs. Nothing here phones home, and nothing is obfuscated.

shell

pip install firsthand-ego

# 1. list what you have access to
fh ls
# EP_0117_KITCHEN_A   78.4 s   kitchen/knife_dice        shipped
# EP_0118_KITCHEN_A   41.2 s   kitchen/pour_decant       shipped

# 2. pull one episode in the format your loader wants
fh get EP_0117_KITCHEN_A --format rlds --out ./data

# 3. or stream a whole skill line without staging it locally
fh stream --skill kitchen/knife_dice --format webdataset | your_trainer

torch_loader.py

from firsthand_ego import Episode
import torch

ep = Episode.open("data/EP_0117_KITCHEN_A")

# frames and hand pose arrive already aligned on the head-camera clock
for step in ep.steps(rate_hz=30):
    rgb    = torch.from_numpy(step.head_rgb)          # [2160, 3840, 3]
    depth  = torch.from_numpy(step.depth_mm)          # [480, 848] uint16
    hands  = torch.from_numpy(step.hand_joints)       # [2, 21, 3] metres
    valid  = torch.from_numpy(step.hand_visible)      # [2, 21] bool
    action = step.action                               # ("dice", "onion")
    assert step.sync_error_ms < 2.0

Guide

Writing a skill spec that produces trainable coverage.

The single largest predictor of whether a capture program is useful is whether the spec names an acceptance criterion a reviewer can apply without arguing. Vague specs produce hours; sharp specs produce coverage.

kitchen_knife_dice.yaml

skill_id: kitchen/knife_dice
one_line: Dice an onion into ~8 mm cubes with a chef's knife.

must_contain:
  - both hands visible at grasp and at first cut
  - the board and the onion in frame for the whole episode
  - the completed pile in frame at episode end

conditions:
  daylight:      120 h
  low_light:      40 h        # evening kitchen, single overhead
  cluttered:      60 h        # counter left as found
  failure_cases:  20 h        # slips, roll-aways, recoveries

reject_if:
  - the cut is off-frame for more than 2 s
  - a second person's hands enter the manipulation region
  - depth invalid over 35 % of the board region

accept_metric: >
  A held-out reviewer can name the verb and noun from the
  first 15 frames without reading the label.

Name the verb and the noun: Not "cooking". Dice, pour, decant, torque, fold, scan.
State the geometry: What must be in frame, at which moment, and for how long.
Budget conditions explicitly: Hours per condition cell, including failure cases.
Write the reject rule first: If you cannot write it, the skill is not specified yet.
Give one acceptance test: Something a reviewer can apply in fifteen frames.

We write the first draft with you on a call and hand it back as a file in your repository. It becomes the contract the coverage report is measured against, so both of us are looking at the same document when a batch is accepted or rejected.

What ships with every clip

Ten streams, one clock, one episode file.

Not an à la carte menu. Every validated hour we deliver carries the whole stack, in the schema below, whether you asked for depth or not.

Stream	Spec	Detail
Head camera	3840 × 2160 · 60 fps	Global-shutter, 120° HFOV, rolling-shutter-free, H.265 + lossless keyframes
Wrist cameras × 2	1920 × 1080 · 60 fps	Left + right, 100° HFOV, rigid mount, extrinsics re-solved per session
Depth	848 × 480 · 30 fps	Active stereo, 0.3–4 m range, metric millimetres, per-frame confidence map
IMU	200 Hz · 6-DoF	Accel + gyro, bias-calibrated, hardware-timestamped on the same clock domain
Hand pose	21 keypoints × 2 hands	3D metric, per-joint visibility flag, contact events on grasp and release
Body pose	24 joints	3D, root-relative and world-frame, torso and forearm chains resolved
Segmentation	Instance masks	Manipulated objects + target surfaces, tracked IDs across the episode
Action segments	Verb + noun taxonomy	97 verbs, 512 nouns, start/end to the frame, human-reviewed
Formats	RLDS · LeRobot · WebDataset	Also HDF5, zarr, and .rrd for Rerun. Converters shipped as source.
License	Commercial · buyer-owned	Perpetual, irrevocable, model-weights-clean. Exclusivity available.

Colour dots map to the modality legend used in every chart on this site. Full field-level schema in the episode schema docs.

The sample pack uses this exact schema.

Download it, point your loader at it, and find out in an afternoon whether it fits your stack.

Get sample data →Capture methodology