Can self-ratings replace objective performance scores for AI competence?

The paper's review of AI-literacy measures ends on a narrow but usable result. An exploratory meta-analysis of three directly reported subjective-objective correlations, "all from one research program," gives a REML pooled r = .055 with a Hartung-Knapp 95% confidence interval of [-.047, .156], on a combined reported N of 2,765. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gives r = .079, interval [-.025, .181]. Both intervals include zero. The discussion states the consequence directly: the results "provide no basis for substituting self-ratings for performance scores."

The paper frames the two kinds of measure as targeting different things, which "complicate[s] assessment of competent generative-AI use in work settings." Objective tests such as AICOS-S and GLAT "assess demonstrated foundation knowledge," while self-reports "assess perceptions, confidence, and reported practice." The review organizes the measurement content it found into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. The introduction argues that the last two matter because tool-using assistants add decisions about access, state changes, execution, and evidence, and that "a person can answer conceptual questions about large language models and still make poor operational decisions." Its examples include accepting an unsupported claim because the prose sounds convincing, granting a tool more access than a task requires, and treating an agent's completion report as proof that required checks ran.

This sits beside vault notes on what people actually do with generated output. The person-side of Does polished AI output trick audiences into trusting it? is the introduction's first example: convincing prose gets accepted as support. Does AI assistance actually harm the way developers learn? and Does AI assistance help workers learn lasting skills? both turn on what people can do rather than what they feel able to do, and this paper supplies the measurement-side reason for caring about that gap. It also parallels Does a single benchmark score actually predict agent readiness?, applied to people: one score cannot stand for several separable components of competence.

The excerpt does not establish a population correlation, and the paper says as much, adding that it does not set "a workplace pass threshold." Three effects from one program cannot separate a true near-zero relationship from a small positive one, since the upper bounds (.156 and .181) leave room for weak association. The largest study also has "an unresolved discrepancy between its reported correlation and p-value," so its weight in the pooled estimate needs caution. The excerpt does not name which instruments were paired, which populations or tasks were used, or what the 24 focal publications found for epistemic oversight, reliance calibration, and operational control; it says only that other instruments address verification, reflective oversight, reliance, trust, and dependency. What follows at this strength is a limit on inference, not a finding of no relationship. A confidence rating or reported-practice questionnaire cannot be read as a stand-in for a demonstrated-performance score, and where competence at work matters, the measure needs to be a performance measure.