A labeling protocol for hand-object interaction in egocentric video

For an AR perception model, a hand near an object is not the same as a hand touching it. A hand-object interaction dataset needs to distinguish visibility, contact and action over time. If those ideas collapse into one label, the model gets weak supervision: it may learn that proximity means grasping, or that a tool’s presence means it was used.

Start with a short, representative egocentric video sample. Write the annotation rules before scaling up, and test them on clips that include occlusion, pauses and awkward viewpoints. The goal is not to label every visible detail. It is to make the physical interaction recoverable from the labels.

1. Define the annotation unit

Choose the clip boundaries and the time resolution your model needs. For an interaction dataset, annotate events as temporal spans rather than assigning one label to an entire clip. A clip might contain reaching, contact, manipulation and release, each with a different start and end.

Specify how annotators mark event boundaries. For example, define “reach” as beginning when the hand moves purposefully toward a target, and “contact” as beginning at the first visible frame of physical contact. If the action starts before the hand enters view, mark only what the video supports. Do not backdate an event based on what seems likely.

Keep the wearer’s left and right hands distinct. State whether left and right refer to the person wearing the camera, not the viewer’s left and right on screen. That distinction matters when the camera turns or the hands cross.

2. Label hand state separately from action

Use one field for visibility or observability and another for the hand’s interaction state. A practical state set can include visible, partly occluded, fully occluded and out of frame. For the interaction field, define a small set such as open, reaching, grasping, holding, manipulating and released. These are proposed categories; adapt them to the target task and keep the definitions mutually clear.

Write operational rules for transitions. “Holding” should mean the hand maintains control of an object, not merely that the fingers appear closed. “Released” should begin when the object is no longer controlled by that hand. If a hand is blocked at the critical moment, use an uncertainty or unobservable value rather than infer the state from the next frame.

Allow two hands to have separate labels at the same time. If one hand steadies a container while the other turns a lid, a single clip-level action loses the division of labor. Record the relation between each hand and the object it acts on.

3. Mark contact points with evidence

Define contact as visible physical interaction, not a small distance between hand and object. Annotators should mark the frame or time when contact starts and ends, and record the contact location in the representation required by the project. That may be an image point, a region or another agreed format; the guideline must say which, and how to handle multiple contact locations.

When fingers cover the point of contact, label the visible evidence and flag the exact point as occluded or uncertain. Do not place a point on the hidden surface just because the object’s geometry suggests where it would be. If the hand touches an object at several locations, define whether each contact gets its own mark or whether the annotation records one interaction region.

4. Track tools and what they do

Assign each object a stable ID when it first appears. Keep that ID through temporary occlusion and reappearance when the visual evidence supports continuity. If identity is genuinely unclear, mark the uncertainty under a defined rule instead of silently creating a new object. Stable IDs are especially useful when a tool moves behind a hand or out of view; the guidance in handling object IDs through long video occlusions is relevant here.

For tools, separate object identity from manipulation. Record the tool’s visible state or position only if it matters to the model, then label the action: for example, picking up, inserting, turning, squeezing, cutting or placing. Define these verbs in observable terms. “Turning” might require visible rotation relative to a fixed object; “using” on its own is too broad to adjudicate consistently.

Link the action to the hand and object involved. If the hand blocks the tool’s working end, label the action only when motion or other visible evidence makes it clear. Otherwise, use an unknown or ambiguous category. A plausible task narrative is not evidence.

5. Resolve ambiguity and check the sample

Before production, give annotators the same difficult examples: brief contact, hands crossing, tool occlusion, object handoffs and actions that begin or end outside the frame. Add a written ruling for each disagreement. Keep the rule, not just the corrected label, so the same case is handled consistently in later batches.

Review a sample for boundary consistency, hand identity, contact evidence and object-ID continuity. Compare disagreements by category. If annotators disagree mostly on whether a closed hand is grasping or holding, tighten those definitions before adding more video. A broader workflow for preserving task context in first-person footage is covered in situated tasks in embodied data.

6. Pilot before scaling

For a first pass, select a small clip set that contains ordinary interactions and the edge cases likely to break your rules. Give the annotation team the schema, examples and explicit uncertainty policy. Compare the returned labels against your acceptance criteria, revise the guideline, then decide whether the dataset is ready to expand.

XYNTRIQ says teams can start with a small sample: it labels the sample to the supplied guidelines, shares the output and sends a fixed-scope quote within 48 hours. That makes a pilot a practical place to test whether hand states, contact points and tool actions are defined tightly enough before committing to a larger egocentric video dataset.