Can an AI learn from words and pictures at once — and does each one cover for the other's blind spots?
How do textual and visual curriculum supervision complement each other in training?
This explores how training signals in language (descriptions, annotations, step-level feedback) and signals in images (screens, pictures) can support each other; the corpus has no note on a curriculum that literally combines the two, so this reads the question as how text and vision supervise each other.
This explores how training signals in language (descriptions, annotations, step-level feedback) and signals in images (screens, pictures) can support each other. The corpus has no paper that trains one model on a textual curriculum and a visual curriculum side by side, and I won't pretend otherwise. What it does have is several notes where the two modalities cover for each other. The pattern is that vision supplies the raw material and text supplies structure, labels, and a bridge.
The clearest case is ScreenAI. It pretrains on a screen-annotation task, where the model identifies what type of UI element sits where on a screen. Those annotations are text, and they automatically generate question-answering and navigation training data. A 5B-parameter model reaches state-of-the-art results on several benchmarks this way (Can one model understand both UIs and infographics equally well?). The visual task produces the textual curriculum, and the textual curriculum teaches richer visual reasoning.
At inference time, the same complementarity looks like a division of labor. Agent S feeds a GUI agent screenshots for understanding the environment, plus image-augmented accessibility trees (text-structured descriptions of screen elements) for pinning down what to click. Splitting planning from grounding into separate optimization paths beat the baseline by 9.37% (Can structured interfaces help language models control GUIs better?). SignRAG goes further and uses language as the bridge itself. A vision-language model describes an unknown sign in words, and the description retrieves the matching design from a text-indexed database. This needs no recognition training and beat direct embedding similarity (Can describing images in text improve zero-shot recognition?). Language does the work that a visual training set would otherwise have to do.
The curriculum half of the question is where the corpus is thinnest, and it only appears on the text side. Reverse-curriculum RL slides the reasoning start state backward from near-completion, so outcome-only feedback exposes step-level failures (Can curriculum learning approximate expensive process supervision?). Nothing in the corpus applies that idea to images. A caution from the teacher-data literature also carries over: refined data that exceeds the student's learning frontier hurts performance even when it is better data (Does teacher-refined data always improve student model performance?). A richer supervision signal, textual or visual, only helps if the model can absorb it at its current stage.
Text also has a limit as a stand-in for visual training. Across 34 models, multimodal zero-shot performance tracks how often a concept appeared in pretraining, and linear gains need exponentially more data (Does multimodal zero-shot performance actually generalize or interpolate?). Good textual supervision alone can't fill a gap in visual coverage. SignRAG's design reads as a workaround for this: it moves the burden from training to retrieval. Whether a deliberately ordered visual curriculum could beat that frequency ceiling is an open question here.
Sources 6 notes
ScreenAI unifies UIs and infographics under one schema, using screen-annotation pretraining to identify UI element types and locations. These annotations auto-generate QA and navigation data, enabling a 5B-parameter model to achieve state-of-the-art performance on multiple benchmarks.
Agent S's dual-input design—visual input for environmental understanding plus image-augmented accessibility trees for grounding—achieved 9.37% improvement over baseline by factoring planning and grounding into separate optimization paths rather than forcing end-to-end prediction.
SignRAG demonstrates that describing an unknown image via vision-language model, then retrieving known designs from a text-indexed database, eliminates the need for recognition model training. Natural-language description bridges the visual-reference gap better than direct embedding similarity.
R3 progressively slides the reasoning start state backward from near-completion, creating a curriculum that reveals step-level failure modes using only outcome feedback. This achieves process supervision granularity without expensive human step annotations.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
Show all 6 sources
Across 34 models and 5 datasets, multimodal models require exponentially more pretraining data for linear performance gains on downstream tasks. Performance correlates with how often test concepts appeared during pretraining, not genuine generalization ability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Language Modeling: An Exploration of Multimodal Pretraining
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent
- MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind
- OmniParser for Pure Vision Based GUI Agent
- ReasonVQA: A Multi-hop Reasoning Benchmark with Structural Knowledge for Visual Question Answering
- Emerging Properties in Unified Multimodal Pretraining
- Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning