Multimodal AI
Why first-person video is a different kind of input
Egocentric footage preserves attention, approach and sequence — spatial context that third-person capture quietly discards.
3 min read
Egocentric Data Collection
First-person human demonstrations for robots that need to operate in the real world.
Explore first-person video data and structured human demonstrations for AI systems learning to interpret and perform real-world tasks.
Episode record
ep_0142 · seg 03
- Task
- Pick up a cup
- Action
- ReachGraspLift
- Timestamp
- 00:04.2 – 00:05.8
- Object
- Ceramic cup
- Status
- Review example
What we capture
Capture the details that matter.
Collection requirements are defined around the task, environment, recording setup, and intended use of the data.
Camera Perspectives
First-person viewpoint and where the camera sits.
Considerations
- Head- or chest-level first-person viewpoint
- Camera placement relative to the task and the hands
- Optional fixed third-person (exocentric) views alongside
Paired ego + fixed exocentric capture is described in Mvaak’s rig tiers.
Confirm during project scoping.
Recording Devices
Rig class, resolution, frame rate and audio.
Considerations
- Rig class chosen per task: action camera / smart glasses, consumer XR headset, research glasses, or paired views
- Resolution, frame rate and audio requirements
- Mvaak’s rigs or yours
The four rig classes are described in Mvaak’s materials. Exact devices and specifications are agreed per project.
Confirm during project scoping.
Task Types
Single actions, multi-step activities and sequences.
Considerations
- Individual actions (e.g. reach, grasp, place)
- Multi-step activities with a defined start and end
- Deliberate failure and recovery behaviour, where required
Confirm during project scoping.
Environments
Where the task is performed.
Considerations
- Household, commercial and agricultural settings
- Warehouse and light-manufacturing work within commercial settings
- Other settings, including industrial sites, subject to agreement
Household, commercial and agricultural domains are described in Mvaak’s materials.
Confirm during project scoping.
Recording Duration
Clip length, sessions and frequency.
Considerations
- Clip or episode length
- Session duration and number of sessions
- Recording frequency and overall volume
Confirm during project scoping.
Participant and Site Requirements
People, permissions, access and safety.
Considerations
- Participant profile and availability
- Site access and permissions
- Consent from participants and bystanders
- Safety considerations for the task and site
Confirm during project scoping.
Environments and tasks
Data grounded in real-world activity.
Choose an environment to see example tasks. Examples are illustrative, not completed projects.
Environment
Household
Everyday manipulation in real homes.
Example tasks
- Picking up an object
- Opening a drawer
- Organising items
- Cooking and prep
- Folding laundry
Described in Mvaak’s service scope. Site and task specifics are confirmed during project scoping.
Example tasks are illustrative, not completed projects.
What you receive
From recorded activity to structured data.
Define the expected files, metadata, annotation requirements, and quality documentation before collection begins.
Video Files
Recorded video, included according to the agreed collection scope — one file per episode or session.
- session_0031/
- ep_0142_rgb.mp4
- ep_0142_imu.csv
- ep_0143_rgb.mp4
- ep_0143_imu.csv
Standard deliverables
Listed as standard per episode in Mvaak’s service materials. Confirmed for each project during scoping.
- Calibrated RGB or stereo video at the agreed resolution and frame rate
- IMU clock-aligned to video frames
- Per-rig calibration and synchronisation check results
- Temporal segmentation to sub-action, with outcome markers
- Task, environment, object and operator metadata
- Consent record identifier, licence flags and provenance manifest
- Diversity report for each batch
Optional extras
Not included in the base offering. Available depending on project requirements and agreement.
- Depth, gaze or 3D hand pose (depends on the rig used)
- Hand keypoints, object tracks and contact events
- Full extraction: 3D scene, trajectory and camera pose
- Paired third-person views
- Multilingual audio or narration tracks
- A bespoke schema or additional metadata fields
- Expanded quality reporting
How the data is processed
Six steps from task to delivery.
Select a step to see what happens, what’s needed, and what follows. Details vary by project.
Step 1 of 6
Define the task
- What happens
- We work out the skill your model needs to learn and what a useful example looks like.
- What’s required
- The task or skill, the intended AI application and what the model currently gets wrong.
- What follows
- A shared description of the task family to capture.
Quality and traceability
Make every delivery easier to review.
Select a check to see what is reviewed and how issues are handled.
File integrity
- 1 · What is reviewed
- Files open, are complete, and contain no dropped or corrupted frames.
- 2 · What an issue looks like
- A recording cuts out halfway through a task.
- 3 · How it is handled
- The episode is excluded and re-captured where the agreed process allows.
Illustrative example. Handling follows the process agreed for each project.
Data governance
Considerations agreed and documented for each project.
Consent
Participants consent before capture; bystander and household consent is handled separately.
Permissions
Site access and recording permissions are confirmed before collection starts.
Secure handling
Access-controlled storage, with face and identifier redaction available on request.
Retention & transfer
Retention, transfer and deletion arrangements are agreed for each project.
Scope a collection project
Start with the task you need your AI to learn.
Not sure about the technical details? Describe the task and we can identify the requirements to discuss.
What the form asks
- Name and work email
- Organisation
- Intended AI application
- Task or activity to capture
- Required environment
- Estimated volume or duration
- Desired annotation layers
- Expected delivery format
Only your contact details and a short description are required — leave the rest blank if unknown.
FAQ
Egocentric data questions.
Video recorded from a first-person perspective — typically a head- or chest-mounted camera — showing a task from the viewpoint of the person performing it.
Mvaak’s materials describe household, commercial and agricultural settings, including warehouse and light-manufacturing work. Other environments, such as industrial sites, are confirmed during project scoping.
Yes. Tasks, environments, devices, annotation and delivery are defined in a written capture brief agreed with you before recording begins.
Typically synchronised video and sensor data, metadata, the agreed annotation layers, quality documentation and a provenance and consent manifest. The exact list is agreed during scoping.
Not all of them. Annotation depth is chosen per task — for example temporal segmentation across a corpus, with hand pose or object tracks only where needed — and agreed in scoping.
Through automated integrity checks and human review against an acceptance rubric agreed with you, with a sample of each delivery re-graded before it is accepted.
You can ask for one. Whether a sample is available, and what it contains, depends on the data and your requirements — we confirm this when you get in touch.
Describe the task you want your model to learn. We help turn it into a capture brief and an initial scope — often a small pilot — before any recording starts.
Related reading
All articlesAI Data
How to scope a first dataset
The most common mistake is committing to volume before the specification has been tested. A small end-to-end pass costs less and tells you more.
3 min read
Dataset Quality
What actually makes training data usable
Volume is the easiest property to measure and the least predictive of whether a dataset improves a model. Here is what tends to matter instead.
3 min read
Have a real-world task in mind?
Tell us what your AI needs to learn, and define a suitable collection scope with Mvaak.
Sample availability depends on the data and your requirements, and is confirmed when you get in touch.