Robots and embodied agents don’t learn to act from the internet’s supply of third-person clips. They learn from the world seen through the eyes of the person doing the task — the first-person, or egocentric, view. That single shift in camera angle is why egocentric video datasets have become the most contested resource in physical AI. This guide covers the three ways to get them: open, licensed, and custom.
What is an egocentric video dataset?
An egocentric video dataset is a collection of first-person video — recorded from a camera worn on the head, glasses, or body — capturing the world from the point of view of the person performing an activity. Unlike third-person (“exocentric”) footage, it records what the actor sees, where they look, and how their hands move, usually paired with sensor streams like gaze, depth, IMU, and hand pose.
That extra signal is the point. A robot policy needs to know not just that a cup exists, but the sequence of hand movements, gaze shifts, and contact events that turn “pick up the cup” into a completed action. Egocentric data captures that chain natively.
Why egocentric video matters for physical AI
Egocentric video is the closest proxy we have to a robot’s own sensory stream, which is why it now anchors training for physical AI, embodied agents, and vision-language-action (VLA) models. It supplies the perception-plus-action signal that third-person video and static images cannot.
The clearest evidence is in recent robot-learning research. Apple’s EgoDex — 829 hours of egocentric video across 194 manipulation tasks — was built specifically because egocentric human video is “passively scalable” in a way real-robot teleoperation is not (EgoDex, 2025). Projects like EgoVLA go a step further, training vision-language-action models directly on egocentric human video and then transferring the learned policies to real robots (EgoVLA project page). The first-person view lets a model watch a human hand complete thousands of tasks and reuse that prior when a robot gripper attempts the same job.
In practice, three properties make egocentric data uniquely valuable:
- Action grounding. Every frame is tied to what the wearer is doing, so behavior — not just appearance — is learnable.
- Hand and gaze signal. Dexterous manipulation needs finger-level pose and attention cues that only a first-person rig captures well.
- Scale without robots. Humans can record demonstrations far faster and cheaper than a robot fleet, then that data pretrains the policy.
The three sourcing routes at a glance
There are only three ways to obtain egocentric video datasets, and most serious programs use more than one. Open datasets get you started; custom collection gets you to production.
| Route | What it is | Best for | Main limitation |
|---|---|---|---|
| Open / academic | Publicly released research datasets (Ego4D, EPIC-KITCHENS, etc.) | Pretraining, benchmarking, feasibility studies | Fixed scenes; often non-commercial licenses |
| Licensed / commercial | Datasets or platforms licensed for a fee with clearer usage rights | Teams needing scale plus commercial clearance | Still not tailored to your exact tasks or embodiment |
| Custom collection | Data collected to your task, environment, and sensor spec | Production models, rare tasks, specific embodiments | Requires a collection partner and lead time |
What are the best open egocentric video datasets?
The best open egocentric video datasets today are Ego4D, Ego-Exo4D, EPIC-KITCHENS-100, EgoDex, Nymeria, and HoloAssist — together they span thousands of hours of first-person activity, though each carries its own scope and license. They are the fastest way to prototype, but read the license before you build a product on them (see the next section).
| Dataset | Scale | Focus | Capture / modalities | License |
|---|---|---|---|---|
| Ego4D | 3,670 hrs, 923 participants, 9 countries | Daily-life activities at massive scale | Head-mounted cameras; video, audio, gaze, 3D scans, narrations | Signed license agreement (research/commercial tiers) |
| Ego-Exo4D | ~1,286 hrs, 800+ participants, 13 cities | Skilled activities, paired ego + exo views | Meta Aria glasses; video, IMU, gaze, point clouds, commentary | Signed license agreement |
| EPIC-KITCHENS-100 | 100 hrs, 45 kitchens, ~90k actions | Unscripted kitchen activity | Head-mounted GoPro; RGB, dense action labels | CC BY-NC 4.0 (non-commercial) |
| EgoDex (Apple) | 829 hrs, 338k demos, 194 tasks | Dexterous tabletop manipulation | Apple Vision Pro; 1080p RGB + 3D hand/upper-body pose | Research (arXiv license) |
| Nymeria (Meta) | 300 hrs mocap / 1,200 sequences, 264 participants | In-the-wild full-body human motion | Aria + miniAria wristbands + XSens suit; motion-language | CC BY-NC-SA 4.0 (non-commercial) |
| HoloAssist (Microsoft) | 169 hrs, 350 instructor-performer pairs | Interactive task assistance | HoloLens 2; RGB, depth, hand/eye/head pose, IMU | CDLA v2 (permissive) |
A few notes from working with these:
Ego4D remains the anchor of the field — 3,670 hours of daily-life footage from 923 camera wearers across nine countries, released by a Meta-led consortium (Ego4D). Its scale makes it ideal for pretraining, but it is scripted around benchmark tasks, not your product.
Ego-Exo4D adds the piece Ego4D lacks: synchronized first- and third-person views of the same skilled activity — cooking, repair, sports, healthcare — captured with Meta’s Aria glasses (Ego-Exo4D). The paired perspective is powerful for teaching a model to map what it sees to what an expert does.
EgoDex is the one to watch for manipulation: 338,000 demonstrations with precise 25-joint-per-hand pose, captured passively on Apple Vision Pro (EgoDex, 2025). It is the largest dexterous-manipulation egocentric dataset released to date.
Licensing: the trap most teams miss
The single most common mistake in sourcing egocentric video datasets is training a commercial model on a non-commercial dataset. Licenses vary sharply, and the most popular datasets are often the most restricted.
Look closely at the table above. EPIC-KITCHENS-100 and Nymeria are both released under Creative Commons NonCommercial licenses — great for research, off-limits for a product you intend to sell (EPIC-KITCHENS license). Ego4D and Ego-Exo4D require you to sign a data use agreement before access, with terms that distinguish research from commercial use. HoloAssist ships under the more permissive Community Data License Agreement.
Three practical checks before you commit:
- Commercial rights. Does the license permit training a model you’ll deploy or sell? “NC” means no.
- Consent and privacy. Egocentric footage captures faces, homes, and bystanders. Confirm the dataset was collected with informed consent and appropriate de-identification — especially under GDPR.
- Redistribution and derivatives. Some licenses (e.g., ShareAlike) require you to release derivatives under the same terms, which can conflict with a proprietary model.
This is where the “free” route quietly ends for most production teams. Open datasets are excellent for pretraining and proof-of-concept, but the moment you need clean commercial rights, task-specific coverage, and consent you can stand behind, you’re looking at licensed or custom data.
When should you commission custom egocentric data?
Commission custom egocentric data when your task, environment, embodiment, or licensing requirements aren’t met by an existing dataset — which is the norm once you move from research to production. Open datasets rarely match the exact objects, actions, camera rig, or edge cases a deployed model faces, and almost never come with the commercial and privacy clearances an enterprise needs.
Signs you’ve outgrown open data:
- Your tasks (a specific assembly line, a surgical procedure, a retail workflow) don’t appear in any public dataset.
- You need a particular capture device, resolution, or synchronized sensor stack (RGB + depth + hand pose + IMU).
- Rare, dangerous, or long-tail scenarios are underrepresented and your model fails on them.
- You require clean commercial licensing, documented consent, and de-identification.
- You need annotation depth — action segmentation, hand-object contact, gaze, intent — that public labels don’t provide.
This is where Shaip fits. Shaip specializes in custom egocentric video data collection and annotation for physical AI, embodied AI, and vision-language-action models. Rather than retrofitting a research dataset, Shaip designs the collection to your spec: the tasks, environments, participants, wearable rigs, and sensor modalities your model actually needs.
On the collection side, that means managed capture of first-person demonstrations across real-world environments — kitchens, warehouses, healthcare settings, retail, factories — with the embodiments and edge cases you define, and with consent and de-identification built into the pipeline. On the annotation side, Shaip’s teams handle the hard, action-grounded labeling that egocentric data demands: hand-pose and keypoint annotation, hand-object contact and manipulation events, action and trajectory segmentation, gaze and intent tags, and multi-sensor fusion labeling kept time-synchronized across streams.
Backed by 1,000+ vetted specialists across 60+ countries and 65+ languages, and operating under ISO 27001 and SOC 2 Type II controls, Shaip delivers egocentric datasets as training-ready and enterprise-secure — with the commercial rights and provenance that open datasets can’t offer. For teams building VLA and robotics models, Shaip also converts human egocentric video into action-labeled formats via hand-pose retargeting, bridging cheap human demonstration data to robot-ready training data. Learn more about Shaip’s AI training data services and video data collection and annotation offerings.
How to choose your sourcing route
Match the route to the stage you’re in. In our work with physical-AI teams, the pattern is consistent:
- Exploring feasibility? Start with open datasets (Ego4D for scale, EgoDex for manipulation) under their research terms.
- Pretraining a foundation policy? Combine multiple open datasets for breadth, checking each license.
- Building a product? Move to custom collection for your tasks and embodiment, with commercial rights and consent locked down.
- Fixing a model that fails in the field? Commission targeted custom data for the exact edge cases and environments where it breaks.
Most successful programs pretrain on open data, then fine-tune and evaluate on custom data. The open sets give you generality; the custom set gives you the deployment.