Can predictive self-supervision work on unlabeled sequential visual data?
This explores whether you can train models to predict masked or future frames of unlabeled video/screen-recording streams — JEPA-style predictive self-supervision — and actually learn something useful, without any human labels.
This explores whether predictive self-supervision — having a model guess masked or upcoming parts of a sequence — pays off on visual streams that nobody has labeled, like screen recordings or video. The corpus says yes, and points to the most direct case study: UI-JEPA applies JEPA-style temporal masking to unlabeled screen recordings, learning task-aware representations that a downstream language model can read to infer what a user is trying to do, all with minimal paired text Can unlabeled UI video teach models what users intend?. The key trade it makes is swapping the bottleneck of scarce labeled video for the abundance of unlabeled streams — which is exactly the bet the question is asking about.
The deeper, more surprising answer is *why* this works, and it's not just "more data." A formal sample-complexity result argues that predicting in latent space (data2vec/JEPA style) recovers compositional structure with a number of samples that stays flat as the hierarchy gets deeper, while predicting raw tokens or pixels needs exponentially more — because nearby latents are far more correlated with each other than raw inputs are Why is predicting latents more sample-efficient than tokens?. So the win isn't incidental to vision; it's structural. Sequential visual data is hierarchical and redundant, which is precisely the regime where latent prediction is exponentially more efficient than the pixel-level alternative.
The "sequential" half of the question has its own thread worth pulling. Beyond masking, a model can treat the *consequences of its own actions* — the future states it lands in — as the supervision signal. Across eight environments, agents trained this way matched expert-dependent baselines with half the data and gave better warm-starts for later RL, all without external rewards Can agents learn from their own actions without external rewards?. That's the same core move as predictive masking (the next state is the label), applied to behavior over time rather than frames in a clip.
Worth knowing about the road not taken: when you have unlabeled images but want recognition, one alternative skips visual self-supervision entirely and routes through text — describe the image with a vision-language model, then retrieve known references from a text index — and this beat direct embedding similarity for zero-shot recognition Can describing images in text improve zero-shot recognition?. The contrast is instructive: text-bridging is great for matching against a known catalog, but it can't learn the temporal dynamics of a stream the way predictive masking does. For "what is the user doing over these 30 seconds," you need the sequence model, not the descriptor.
The thing you didn't know you wanted to know: predictive self-supervision on sequences isn't confined to vision pretraining. The same logic — make the model predict a part of its own signal and use that as free supervision — recurs as models attending to their own latents to build working memory for long inputs Can models learn working memory by attending to their own latents?, and as self-supervised process rewards that replace human step annotations Can self-supervised process rewards replace human annotation?. "Predict the missing piece of your own stream" turns out to be one idea wearing many costumes across the collection.
Sources 6 notes
UI-JEPA applies JEPA-style predictive masking to screen recordings, learning task-aware temporal representations that an LLM decoder can use to infer intent with minimal paired data. This trades the bottleneck of labeled video for abundant unlabeled streams.
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
Research across eight environments shows that agents can use future states from their own actions as supervision without external rewards, matching expert-dependent baselines with half the data and providing superior warm-starts for subsequent RL training.
SignRAG demonstrates that describing an unknown image via vision-language model, then retrieving known designs from a text-indexed database, eliminates the need for recognition model training. Natural-language description bridges the visual-reference gap better than direct embedding similarity.
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.
MetaStone-S1's SPRM achieves o3-mini-level results using dynamic weighting of pseudo-labels instead of human-annotated steps. This eliminates the annotation bottleneck for process supervision, though generalization to fuzzy-outcome domains remains unproven.