For embodied AI teams, the useful shift is in the unit of specification. A frame is not enough to describe what a robot or person encountered over time. A dataset needs to say how space changes, how observations align with action, and what the labels mean when the camera moves or the scene is obscured.
That makes two connected specifications worth tightening: 4D occupancy annotation for embodied AI, and egocentric trajectory logging. There is no universal schema implied here. The practical job is to define a contract that annotators, data collection teams and model engineers can apply consistently.
Give 4D occupancy a testable definition
A 4D occupancy representation assigns a state to locations in space across time. The familiar three spatial dimensions describe where a cell sits; the fourth dimension tracks when that cell is observed. That description leaves many implementation choices open, and those choices affect both annotation effort and model evaluation.
Start by writing down the coordinate frame and grid resolution. State whether the origin follows the sensor, the room, or another reference frame, and define the units and axis directions. Specify how time is sampled: fixed intervals, event-driven moments, or another rule. If frames are sampled from video, record the sampling logic. Otherwise, two teams can label the same scene with different temporal coverage and still appear to follow the same brief.
Define the cell states before collecting labels. Occupied, free and unknown are useful starting categories, but the rules matter more than the names. A cell outside the sensor's observed area should not quietly become free space. Nor should an occluded region be treated as empty just because no object is visible. Set rules for partial visibility, moving objects, boundaries and transient observations.
Keep uncertainty explicit. If annotators cannot establish a cell's state from the available sensor view, give them a defined unknown or review path rather than forcing a confident label. State how disagreement is resolved and whether the delivered output preserves the original uncertainty. A quality metric is only meaningful when its denominator and excluded cases are clear.
Log trajectories, not just clips
Egocentric video puts the camera at the observer's point of view. That view can make an action legible while making motion harder to interpret: camera movement may come from walking, head turns, hand movement or a change in recording position. A useful egocentric dataset specification therefore pairs the video with a timestamped trajectory record.
At minimum, define what each timestamp refers to, how video and any other sensor streams are synchronized, and what happens when timing data is missing or uncertain. Record task start and end rules, pauses, resets and transitions between subtasks. For action labels, specify whether a segment marks an observable movement, an intended action, or a task outcome. Those are different targets; combining them under one verb label makes training and error analysis muddy.
Also settle how the annotation treats hands, objects and contact. Are object identities stable across an occlusion? Does a label mark the first touch, sustained contact or release? What counts as an action boundary when an object leaves the camera view? Teams that need a fuller treatment of turning clips into temporal labels can use a practical method for segmenting POV task video as background.
Make the two specs agree
Occupancy and trajectory labels are more useful together when they share a clock and coordinate story. Document how the camera pose relates to the occupancy grid, how pose uncertainty is represented, and which transformations are applied before delivery. If the trajectory uses one time base and occupancy another, require a stated alignment procedure and a way to flag failed alignment.
Build a small acceptance set around difficult cases: fast camera turns, hand-object interaction, occlusion, repeated actions, pauses and objects entering or leaving the view. Review these cases before volume collection. A small sample can expose ambiguous rules more cheaply than a large batch can repair them.
For each output, preserve enough metadata to trace the record to its source, annotation version and review outcome. Consent and provenance are part of the dataset specification, not administrative notes to be separated from the data. XYNTRIQ says its first-person collections are consented and that consent is documented per record; its directory description also notes sample-first work and a fixed-scope quote after review. Those details matter to teams evaluating a collection partner, but they do not replace a technical schema or acceptance test.
What builders should ask this month
Before comparing providers or estimates, ask for a sample that exercises the hardest temporal and spatial cases in the project. Ask the team to label it against written rules, then inspect disagreements, unknown states, timestamp alignment and record-level traceability. A quote without a defined sample and acceptance criteria is hard to compare across scopes.
The main takeaway is modest: do not treat 4D occupancy as a frame-labeling extension, or egocentric video as a clip with action tags attached. Specify the coordinate system, clock, state semantics, trajectory events and uncertainty together. That gives perception engineers a dataset they can audit and a collection team a brief they can actually execute.
XYNTRIQ