Can one model understand both UIs and infographics equally well?
Screen UIs and infographics share visual structure but have been tackled separately. Can a unified schema and annotation-based pretraining bridge them in a single small model?
Can unlabeled UI video teach models what users intend? Why do planning and grounding pull against each other in agents? Can bounding boxes replace image encoders for document understanding? Can reference resolution work as a language modeling problem? Do text-based GUI agents actually work in the real world?
Temporal masking on unlabeled UI video learns user intent efficiently Foundation GUI agents need an intermediate interface separating planning from grounding Bounding boxes with disentangled attention enable layout-aware document understanding without image encoders Reference resolution including on-screen entities works as language modeling. Visual perception requires UI-specialized models, not general multimodal ones
What visual patterns transfer between infographic and UI tasks when trained jointly? How does annotation-based pretraining compare to self-supervised video masking for screen understanding? What document layouts benefit most from bounding box representations? How does serializing screen layout to text preserve spatial relationships? Can text-based and vision-based screen understanding achieve similar performance?