Can systems infer what users want from watching their screen?

Lam and colleagues, in a paper to appear at CHI 2026, argue that LLMs default to generic, "milquetoast and unopinionated" output because training objectives have to be fixed long before deployment and are optimized jointly across many competing goals at once, which — "like a design by committee" — converges model behavior toward safe, generic results. Their proposed fix is to infer a user's immediate, momentary goal from a short window of observed behavior rather than anticipate it in advance, and to call this inferred goal a "just-in-time objective" (JIT objective): for example "clarify the calculus example in the introduction" or "respond to Reviewer 2's critique about sampling." Tested across a series of experiments on participants' own tasks, outputs steered by induced JIT objectives "achieve 66–86% win rates over typical LLMs," and in-person use sessions found that the specialized tools these objectives produced were "rated as significantly higher quality than a standard LLM chat tool."

Mechanically, a JIT objective is induced from passive observation and then applied through a gen_objective operator that prepends it as a small snippet of context to a generator's existing instructions — the paper frames generators as "actors in a traditional actor-critic architecture," with a paired evaluation step scoring candidate outputs against the same objective. The architecture is demonstrated in Poppins, a browser extension and web application that observes the user's screen, infers an objective (the excerpt's example: "enhance introduction draft by clarifying system architecture"), generates and ranks candidate design specifications against it, and assembles a working interactive tool "within minutes." The authors argue scope is what makes the inference tractable: a user's long-horizon objectives (their example, "a user's writing philosophy") are too complex to anticipate at training time, but a narrow, instant-level goal drawn from "a single snapshot in time" is specific enough to infer and act on reliably.

The paper explicitly positions JIT objectives as an answer to the "Gulf of Envisioning" that How do users actually form intent when prompting AI systems? describes — rather than waiting for a user to articulate a stable intent, the system infers a provisional one from behavior and lets it steer output immediately. It also lands in the same empirical pattern as Do generated interfaces outperform text-based chat for most tasks? and Do full web pages beat markdown chat for LLM responses? — specialized, generated tools beating generic chat responses — but gets there through objective inference rather than UI generation scale or model recency. The induction step also reads as an automated version of the intent-sensing layer in How should AI interfaces handle the shift from doing to supervising?.

The excerpt does not give participant counts, task domains, or demographics for either the win-rate experiments or the in-person sessions, and it does not address how continuous screen observation was consented to or how long an inferred objective stays accurate once a session runs past the "single snapshot" window the authors say they deliberately kept minimal. The 66–86% figure is reported as a single range without a breakdown by task type. Until independent replication or a fuller report fills in those details, the result supports objective inference from passive observation can beat generic LLM defaults in the tasks and sessions tested rather than a general claim about all specialization tasks.