Balance egocentric video datasets without keeping every frame

Egocentric video creates a deceptively large training set. A few minutes of footage can produce thousands of nearly identical frames, while the useful variation may fit into a handful of moments: a hand reaches for a tool, an object changes state, or the camera turns to a new task. Keeping every frame inflates storage and annotation effort. Removing too much can erase the motion and context a model needs.

A reliable curation workflow treats the episode—not the individual frame—as its basic unit. Sample for useful change, deduplicate without breaking links to audio or text, and balance examples only after you understand what the dataset contains.

1. Define the unit you want the model to learn

Start with the task and its labels. If the model must recognize an action, define the action boundaries and the visual evidence that distinguishes one class from another. If it must identify objects or states, specify whether labels apply to a frame, a span of time, or a whole episode. Ambiguous units lead to inconsistent sampling and counts that look balanced but do not represent balanced evidence.

Write down inclusion rules before reviewing the full dataset. Note what counts as a valid example, how to treat partial actions, occlusion, failed attempts, and uncertain labels. Keep separate labels for recording conditions—such as lighting, location, or camera wearer—where those attributes matter to evaluation. Do not use these fields as substitutes for task labels.

2. Sample around change, then check coverage

Uniform sampling is a useful baseline, not a complete strategy. Sampling one frame every fixed interval can leave long stretches of repetitive footage while missing brief transitions. Begin with a coarse pass that records timestamps and detects likely changes in scene, camera motion, hand activity, or object state. Use those candidates to select frames around event boundaries, then add a modest number of frames from stable portions of the action for context.

For each candidate event, retain enough temporal context to show what happened before and after it. The right window depends on the task. A single frame may show an object but not the action that changed it; a long window may add little beyond duplicated views. Inspect samples from short, long, interrupted, and low-motion episodes before applying one rule across the collection.

Keep a record of the sampling rule and the source timestamps. Review a small set of clips alongside the selected frames. Ask whether a reviewer can identify the task, distinguish the class, and see relevant transitions. If not, adjust the rule and repeat the check. Sampling should be judged against the intended labels, not by how many frames it removes.

3. Deduplicate at two levels

First remove exact duplicates. Compare file hashes where files should be byte-identical, and check repeated timestamps, filenames, or record identifiers. Then look for near-duplicates: frames with small changes caused by camera jitter, compression, or a wearer standing still. Perceptual hashes or image embeddings can help identify candidates, but they should flag records for review rather than decide every case automatically.

Deduplicate clips as well as frames. Two recordings may contain the same task from the same continuous take, or the same extracted frame may appear in overlapping clips. Compare temporal overlap and visual similarity, then retain the version that best preserves the action and has the clearest metadata. Do not collapse genuinely different executions just because they look alike at one instant.

Maintain a link from every retained frame to its source episode and timestamp. For multimodal data, keep audio segments, transcripts, and any other aligned records attached to the same source timeline. If a frame is removed, that should not silently shift a transcript or sever its association with the original recording.

4. Enrich metadata before balancing

Use a consistent record for each episode and sampled segment. Useful fields include a stable source ID, start and end timestamps, task and class labels, sampling method, duplicate-review status, and relevant recording conditions. For audio or text, record the corresponding time span and language when known. Mark missing or uncertain values explicitly rather than filling them by guesswork.

Metadata lets you count examples at more than one level. A class may appear in many frames but only a few independent episodes. A language or location may dominate one class. Report counts by episode, contributor or source grouping where available, and class—not just by frame. For collection involving people, keep consent and provenance records associated with the data according to the project’s requirements. Our earlier digest on consent protocols and data residency covers why these records belong in the workflow.

5. Balance independent examples, not duplicated frames

After deduplication, compare class counts at the episode level and inspect the rare classes. Check whether a shortage reflects genuine rarity, inconsistent labeling, or a sampling rule that misses short events. Review the proposed additions before changing the dataset. Adding more near-identical frames from one recording may raise a count without adding useful variation.

Where collection is possible, seek examples that add variation in execution, setting, object appearance, or recording conditions. Where the dataset is fixed, use a documented sampling policy to reduce overrepresented classes or choose a training strategy that accounts for the imbalance. Preserve the original distribution in a separate evaluation set if it reflects expected use. Avoid balancing by moving related clips into different splits: near-identical footage across training and evaluation can make performance look better than it is.

6. Validate the result as a batch

Before training, report the number of source episodes, retained clips and frames, duplicate candidates removed, and counts by class and relevant metadata field. Inspect borderline deduplication decisions and examples from each class. Confirm that timestamps still align across modalities and that split boundaries keep related recordings together. Save the rules and counts with the dataset version so later changes can be traced.

When outsourcing collection or annotation, send a small representative sample with the guidelines and acceptance criteria you intend to use. XYNTRIQ describes a sample process in which it labels the data against those guidelines and shares the output for review; its directory listing says a fixed-scope quote follows within 48 hours. Review the sample for label consistency, timestamp handling, and the metadata needed for your own deduplication and split checks before committing to a larger batch.

The goal is not the smallest possible dataset. It is a set of independent, well-described examples that preserves the moments your model must learn—and makes it possible to explain what was kept, removed, and why.