SYNTHESIS NOTE
Topics›Work Application Use Cases›this note

Can self-ratings replace objective performance scores for AI competence?

Do people's perceptions of their own AI competence match what they can actually do? This matters because assessment systems might rely on the wrong type of measure to evaluate workplace readiness.

Synthesis note · 2026-09-25 · sourced from Work Application Use Cases

The paper's review of AI-literacy measures ends on a narrow but usable result. An exploratory meta-analysis of three directly reported subjective-objective correlations, "all from one research program," gives a REML pooled r = .055 with a Hartung-Knapp 95% confidence interval of [-.047, .156], on a combined reported N of 2,765. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gives r = .079, interval [-.025, .181]. Both intervals include zero. The discussion states the consequence directly: the results "provide no basis for substituting self-ratings for performance scores."

The paper frames the two kinds of measure as targeting different things, which "complicate[s] assessment of competent generative-AI use in work settings." Objective tests such as AICOS-S and GLAT "assess demonstrated foundation knowledge," while self-reports "assess perceptions, confidence, and reported practice." The review organizes the measurement content it found into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. The introduction argues that the last two matter because tool-using assistants add decisions about access, state changes, execution, and evidence, and that "a person can answer conceptual questions about large language models and still make poor operational decisions." Its examples include accepting an unsupported claim because the prose sounds convincing, granting a tool more access than a task requires, and treating an agent's completion report as proof that required checks ran.

This sits beside vault notes on what people actually do with generated output. The person-side of Does polished AI output trick audiences into trusting it? is the introduction's first example: convincing prose gets accepted as support. Does AI assistance actually harm the way developers learn? and Does AI assistance help workers learn lasting skills? both turn on what people can do rather than what they feel able to do, and this paper supplies the measurement-side reason for caring about that gap. It also parallels Does a single benchmark score actually predict agent readiness?, applied to people: one score cannot stand for several separable components of competence.

The excerpt does not establish a population correlation, and the paper says as much, adding that it does not set "a workplace pass threshold." Three effects from one program cannot separate a true near-zero relationship from a small positive one, since the upper bounds (.156 and .181) leave room for weak association. The largest study also has "an unresolved discrepancy between its reported correlation and p-value," so its weight in the pooled estimate needs caution. The excerpt does not name which instruments were paired, which populations or tasks were used, or what the 24 focal publications found for epistemic oversight, reliance calibration, and operational control; it says only that other instruments address verification, reflective oversight, reliance, trust, and dependency. What follows at this strength is a limit on inference, not a finding of no relationship. A confidence rating or reported-practice questionnaire cannot be read as a stand-in for a demonstrated-performance score, and where competence at work matters, the measure needs to be a performance measure.

Inquiring lines that read this note 38

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How does AI assistance affect professional skill development and retention? How does task-replacing automation affect worker employment and remaining skill needs? How should evaluation expand to capture deployment realities? How do standardized benchmarks differ from measures of real-world AI capability? How can evaluation systems stay reliable and resistant to AI-driven gaming? Can employers detect AI-assisted applications and should they be permitted? How does AI adoption reshape workplace inequality and labor market outcomes? How can verification systems keep pace with AI generation? Does polished AI presentation mislead people about accuracy? How do confidence signals and expertise language mislead users' trust calibration? How do clinicians assess, trust, and effectively use medical AI in practice? Why does model self-revision increase confidence in incorrect answers? Do single-axis benchmarks adequately measure deployment-ready agent capability? When does AI explainability improve versus hinder human oversight and trust? How does AI exposure shape employment demand and worker earnings? Why do perceived AI productivity gains often fail to translate into measured business impact? How does increased AI involvement reshape human agency and accountability? Can AI systems achieve sustainable self-improvement without external human guidance? Which task types benefit from human-AI collaboration and how should work divide?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
22 direct connections · 218 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

self-ratings are no basis for replacing performance scores in competent generative-AI use — pooled subjective-objective r = .055, interval including zero