The bottleneck in robot learning is data. This note describes a pipeline that converts a few minutes of a person working with handheld grippers into robot-ready demonstrations: 6-DoF gripper trajectories are recovered from optical trackers, retargeted to an arbitrary arm by inverse kinematics, and rendered back into the original camera. A video world model then re-stages each demonstration across new visual domains.
One tracked human demonstration with handheld UMI grippers (top-left) becomes the same task performed by a retargeted YAM robot, re-staged by a world model across five different environments. The whole pipeline in one shot.
Modern foundation models inherit their generality from scale: language models are trained on the order of a trillion tokens of web text, and vision-language models on billions of image–caption pairs. Robot manipulation has no comparable web-scale corpus. Every grasp, wipe, and fold must be physically performed and recorded, and the relevant axes of variation (object, lighting, background, and crucially the robot embodiment itself) multiply quickly.
The largest open effort to date, Open X-Embodiment, pooled 60 existing datasets spanning 22 distinct robot embodiments into roughly one million real trajectories, and showed that policies trained on the mixture (the RT-X models) transfer across robots better than single-robot baselines.¹ That is a landmark, but it is still ~106 trajectories assembled from 34 labs over years, orders of magnitude short of the data regimes that made language and vision models general, and even a single-robot system such as RT-1 required ~130k teleoperated episodes to reach its reported performance.²
The data that would make robots general does not exist yet.
Largest aggregated training corpora by modality, log scale: each step is roughly a thousandfold less data. The biggest pooled manipulation corpus, Open X-Embodiment, holds about a million trajectories across 22 robot embodiments.1
Teleoperating physical robots to grow this corpus is slow, capital-intensive, and, most limiting, it binds each demonstration to one specific arm. The result is a hard ceiling on the rate at which embodied policies can improve. Breaking that ceiling requires two things at once: collecting demonstrations faster, and extracting more transferable supervision from each one.
Starpilot collects demonstrations both in instrumented collection cells and in the wild, with egocentric cameras and handheld, UMI-style parallel-jaw grippers³ carried into real homes and workplaces. Both streams yield hardware-agnostic demonstrations, following a line of work (UMI³ and DexCap⁴ among others) showing that portable, handheld capture works at a fraction of the cost and setup of a teleoperated rig. The rest of this note follows one demonstration captured in a collection cell: a person performs the task naturally with the grippers while the cell records everything needed to reconstruct the scene in 3D.
What "fully tracked" means
A calibrated scene camera records the task while HTC Vive trackers rigidly mounted on each gripperstream 6-DoF pose. The trackers use SteamVR Lighthouse optical tracking: base stations sweep the workspace with infrared laser planes whose timing is fused with an onboard IMU, producing a metric-scale pose stream at 100+ Hz. Benchmarked against a Vicon reference, the Lighthouse-tracked Tracker 3.0 holds low-millimeter precision under favorable conditions, degrading toward ~1 cm during fast or partially occluded motion;⁶ VIVE's newer camera-based Ultimate Tracker shows a similar ≈2.6–5 mm envelope.⁵This is the relevant accuracy budget for retargeting, and we treat it as such (see Limitations).
Beyond the pixels, we keep the metric ground-truth trajectory of every gripper. Because we know where each gripper was in space, we can re-target the motion onto a different robot and re-render the scene from scratch. Collecting with a handheld tool is roughly an order of magnitude faster and cheaper than teleoperating a real arm,³,⁴ and it is not tied to any one robot.
A second modality: egocentric video in the wild
Outside the cell there are no base stations, so our in-the-wild collection uses a worn egocentric camera time-synchronized with an IMU, and its trajectory is recovered by visual-inertial SLAM: tightly-coupled estimators such as ORB-SLAM3¹⁷ fuse image features with inertial preintegration to produce a metric, drift-corrected camera trajectory, reaching centimeter-level accuracy on standard visual-inertial benchmarks and ~9 mm under quick hand-held motion.¹⁷ A person simply wears the camera through everyday tasks, and that is where the bulk of our egocentric data comes from.
That first-person video is increasingly training data in its own right. Large egocentric corpora such as Ego4D and Ego-Exo4D (the latter explicitly logging synchronized IMU and camera pose¹⁸) now underpin visual pretraining for manipulation,¹⁹ and recent systems treat human egocentric video and robot/teleop demonstrations as one pooled embodiment dataset, co-training a single policy on both;²⁰ in that regime, cheap human hours measurably raise policy success on top of a fixed teleoperation budget.²⁰ Starpilot feeds both kinds into that mix: UMI-gripper trajectories from the cell that retarget to any arm, and field-collected egocentric video that drops directly into co-training.
The pipeline below turns one tracked human demonstration, a bimanual packing task between a plastic tray and cardboard boxes, into a clip of a robot built from two YAM arms (6-DoF, I2RT Robotics⁷) performing the same task. Each stage is a video; a fuller step-by-step breakdown of this exact shot, including the edge-control stages that feed the world model, is published as a companion page.
1 · The raw demo. A person packs items between a tray and boxes with handheld grippers; Vive trackers log each gripper's 6-DoF pose against a calibrated scene camera.
2 · Segment the scene. SAM 2 video segmentation13 tracks the task objects (tray, boxes) to keep, and marks the operator and grippers for removal.
3 · Solve IK & overlay the robot. Tracked gripper poses are mapped onto the YAM arms by inverse kinematics9,10 and rendered into the original camera with MuJoCo.12
4 · A robot demonstration. The operator is painted out with the recovered background; the YAM arms do the task alone: same motion, new embodiment.
The retargeting problem has three sub-problems: recover where the camera sits relative to each tracker (calibration), find robot joint angles that reproduce each gripper pose (inverse kinematics), and place the robot back into the scene (rendering and compositing).
Calibration: tracker frames → one camera
The two Vive trackers report pose in separate per-tracker coordinate frames, neither of which coincides with the scene camera. We work from an intrinsically calibrated camera (focal length and lens distortion recovered from a standard checkerboard calibration) and solve for the rigid transform between each tracker frame and the camera with a robot hand–eye calibration.⁸ Moving a tracked gripper through a sequence of poses produces paired camera and tracker motions; we solve the resulting AX = XB system for the fixed tracker→camera transform, run per arm so each gripper gets its own extrinsic. Because the trackers already provide metric scale and the camera intrinsics are known, this pins absolute scene geometry without a depth sensor.
Retargeting: damped least-squares IK
Retargeting runs as an automated solver, and every result is then reviewed by a human before the clip can become training data. With both grippers expressed in a single camera frame, we solve inverse kinematics per frame for the YAM arm. We use a damped least-squares (Levenberg–Marquardt) Jacobian solver⁹,¹⁰, which trades a small amount of tracking accuracy for numerical stability near kinematic singularities,¹¹ exactly the regime a human wrist drives the arm into. The 6-DoF target is the gripper's position and orientation; each solve is warm-started from the previous frame for temporal smoothness. A second pass then filters each joint trajectory and spreads elbow branch-flips over ~0.4 s with a smoothstep ramp before re-rendering, so configuration switches near joint limits never land in a single frame. Finger width is copied directly from the recorded gripper state rather than inferred. Because a human wrist can demand poses the YAM cannot reach, IK quality is never assumed: every retargeted clip passes a human review step, where the rendered robot is checked frame by frame against the source gripper motion, and any clip whose IK resolution is not acceptable for training (over-stretched reaches, large residual mismatch, or unnatural joint configurations) is flagged or dropped before it enters the dataset.
Rendering & compositing
The solved YAM configuration is rendered in MuJoCo¹² through the same calibrated camera, so the robot lands exactly where the human's gripper was. The operator and their grippers are masked out with SAM 2 video segmentation¹³ and the region is filled with a feathered median-background plate; task-relevant objects (the tray and boxes being packed) are tracked with the same model and kept in the scene. The payoff: one human demonstration becomes a demonstration for whatever robot you choose to train, with no teleoperation rig and no robot in the loop at collection time.
A robot doing the task in one room is a single visual data point. To train a policy that generalizes, you want the same motion in many rooms. We feed the robot clip into NVIDIA Cosmos, a family of video world foundation models for Physical AI,¹⁴ using its conditional-generation variant, Cosmos-Transfer.¹⁵ A Canny edge map computed deterministically from the rendered frames is used as the spatial control signal (a ControlNet-style branch), so the robot's structure and trajectory are preserved while lighting, surfaces, and background are regenerated from a text prompt. The robot's edges are combined with the segmentation edges of the task objects, so the tray and boxes keep their true geometry through the regeneration. We run the Transfer-2.5 2B checkpoint²¹ at 480p, 93 frames per clip, 35 denoising steps, guidance 3, with the task-object edges frozen to the first frame so the box holds its exact shape while the arms animate. Each clip renders in about a minute on an H100, so a whole set of environments can be generated in parallel.
Robot edges. Canny edges of the rendered robot, masked to its footprint; a contrast lift (CLAHE) keeps the dark wrist visible.
Keep the task objects. Robot edges inside the tray and box are erased (red), the objects' own edges are added back (cyan), the rest is kept (green).
The final control signal. Robot structure plus rigid task objects; everything else is left free for the world model to invent.
The same tracked demo, re-staged across five visual domains. Each is just a different text prompt over the identical control signal:
Warehouse. Fulfillment warehouse, soft even overhead lighting.
Warehouse, dramatic light. Hard key light and deep directional shadows.
Spaceship. Cargo bay, brushed-metal bulkheads, glowing panels.
Bio lab. Sterile clean-room, lab benches, fluorescent light.
One tracked human demo → a robot demonstration → an open-ended set of visual domains, picked by the text prompt. This is the same edge-conditioned sim-to-real / data-enrichment use case Cosmos-Transfer was built for.¹⁵ Lighting, surfaces, clutter, and backgrounds become a dial we can turn at render time.
This is an early pipeline, and the honest scope matters as much as the demos.
Vive Lighthouse tracking is millimeter-precise, though precision can drop during fast or occluded motion;6 a multi-lighthouse (multiple base-station) setup reduces that degradation. That is the floor on how faithfully a retargeted gripper pose can match the human's.
Hand–eye calibration recovers the fixed tracker→camera transform from paired motions (the AX = XB problem). Given a well-calibrated camera, accuracy is bounded by the pose diversity of the calibration sequence and the rotation solve's sensitivity to measurement noise; residual extrinsic error propagates directly into where the rendered robot lands.
A human wrist can reach poses outside the YAM's joint limits and workspace. Damped least squares stays stable there but returns the closest reachable pose rather than an exact match; the two-pass joint smoothing spreads, but does not eliminate, configuration flips at limits.
Cosmos preserves structure through edge control but does not guarantee physical consistency. At 480p, fine contact detail (finger–object interface) can drift, and generated frames are not validated against physics; they serve as visual augmentation rather than ground-truth dynamics.
Robot learning is gated by data, and most data is trapped: tied to one robot, one room, one day. By collecting in fully-tracked cells we keep the 3D ground truth that lets us retarget any demonstration to any embodiment with inverse kinematics, and re-stage it anywhere with a world model. Egocentric human video adds a second embodiment stream that co-trains alongside the retargeted demonstrations.²⁰ Each of these levers targets a known failure of single-robot teleoperation data: embodiment lock-in and visual narrowness. The downstream consumer is a data-hungry visuomotor policy (diffusion policies and RT-style transformers alike improve with more, more diverse demonstrations¹⁶,²), and the goal is simply that every minute a person spends in the cell becomes training data for many robots, in many worlds.
Footnotes
1
Open X-Embodiment Collaboration, "Open X-Embodiment: Robotic Learning Datasets and RT-X Models," IEEE Int. Conf. on Robotics and Automation (ICRA), 2024. arXiv:2310.08864.
2
A. Brohan et al., "RT-1: Robotics Transformer for Real-World Control at Scale," Robotics: Science and Systems (RSS), 2023. arXiv:2212.06817.
3
C. Chi, Z. Xu, et al., "Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots," Robotics: Science and Systems (RSS), 2024. arXiv:2402.10329.
4
C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, C. K. Liu, "DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation," Robotics: Science and Systems (RSS), 2024. arXiv:2403.07788.
5
J. Kulozik, N. Jarrassé, "Evaluating the precision of the HTC VIVE Ultimate Tracker with robotic and human movements under varied environmental conditions," 2024. arXiv:2409.01947.
6
S. Merker, S. Pastel, D. Bürger, A. Schwadtke, K. Witte, "Measurement Accuracy of the HTC VIVE Tracker 3.0 Compared to Vicon System for Generating Valid Positional Feedback in Virtual Reality," Sensors, 2023. PMID:37687827.
7
I2RT Robotics, "YAM Arm" (product documentation). i2rt.com/collections/yam-arm.
8
R. Y. Tsai, R. K. Lenz, "A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration," IEEE Trans. Robotics and Automation, 5(3):345–358, 1989. doi:10.1109/70.34770.
9
C. W. Wampler, "Manipulator Inverse Kinematic Solutions Based on Vector Formulations and Damped Least-Squares Methods," IEEE Trans. Systems, Man, and Cybernetics, 16(1):93–101, 1986.
10
S. R. Buss, "Introduction to Inverse Kinematics with Jacobian Transpose, Pseudoinverse and Damped Least Squares Methods," tech. report, UC San Diego, 2009.
11
Y. Nakamura, H. Hanafusa, "Inverse Kinematic Solutions with Singularity Robustness for Robot Manipulator Control," J. Dynamic Systems, Measurement, and Control, 108(3):163–171, 1986.
12
E. Todorov, T. Erez, Y. Tassa, "MuJoCo: A Physics Engine for Model-Based Control," IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), pp. 5026–5033, 2012.
13
N. Ravi et al., "SAM 2: Segment Anything in Images and Videos," Int. Conf. on Learning Representations (ICLR), 2025. arXiv:2408.00714.
14
NVIDIA, "Cosmos World Foundation Model Platform for Physical AI," 2025. arXiv:2501.03575.
15
NVIDIA, "Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control," 2025. arXiv:2503.14492.
16
C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, S. Song, "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion," Robotics: Science and Systems (RSS), 2023. arXiv:2303.04137.
17
C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, J. D. Tardós, "ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM," IEEE Trans. Robotics, 37(6):1874–1890, 2021. arXiv:2007.11898.
18
K. Grauman et al., "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives," IEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2024. arXiv:2311.18259.
19
K. Grauman et al., "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives," IEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2024. arXiv:2311.18259.
19
S. Nair, A. Rajeswaran, V. Kumar, C. Finn, A. Gupta, "R3M: A Universal Visual Representation for Robot Manipulation," Conf. on Robot Learning (CoRL), 2022. arXiv:2203.12601.
20
S. Kareer et al., "EgoMimic: Scaling Imitation Learning via Egocentric Video," 2024. arXiv:2410.24221.
21
NVIDIA, "Cosmos-Transfer2.5: Multi-ControlNet World Simulation" (code and model checkpoints), 2025. github.com/nvidia-cosmos/cosmos-transfer2.5.
