Can expert-written questions resist web-assisted non-expert answering?

The GPQA paper reports 448 multiple-choice questions in biology, physics and chemistry, written by domain experts and checked so that they are hard for people outside the field. Experts "who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect)". "Highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web." The strongest GPT-4-based baseline reaches 39%. The figures are the paper's own measurements, taken from its contractor pool; the excerpt names no outside source for them.

The construction explains the gap. Each question went through question writing, a first expert validation, revision by the writer, a second expert validation, and then three non-expert validators who tried to answer it with internet access. The paper's stated goal is scalable oversight: as models answer harder questions, annotators "may be difficult" to supervise even when they are "themselves skilled and knowledgeable". The authors want testbeds "as close as possible to the edge of human expertise", and they claim GPQA meets seven of Irving and Askell's nine desiderata for scalable-oversight datasets.

Against the nearest notes, the GPQA result is the human baseline that oversight work would need. Why does assisted accuracy capture only half the LLM gain? measures what happens when a person works with a model, and finds assisted accuracy below the better component. GPQA measures no assisted condition, only unaided experts and web-equipped non-experts, so the two studies test different things and should not be merged. The debate note's condition, that When does debate actually improve reasoning accuracy?, is useful here: GPQA's non-experts can search, yet still reach 34%, which suggests that retrieval alone does not give a non-expert the checking that debate needs. That is an inference from the numbers, not something the paper tests. It also differs from Can crowdsourced votes reliably rank language models?, where crowd votes track expert raters on preference. GPQA asks about correctness at graduate level, and there skilled non-experts diverge from experts.

The excerpt does not establish that GPQA works as an oversight testbed. It reports no oversight experiment, so the paper's hope that the benchmark "can help devise ways for human experts to reliably get truthful information from AI systems" is a proposal, not a result. The non-expert figure is also a ceiling, not a typical rate: the authors say "our numbers will likely not directly translate to realistic settings". The discussion adds that non-expert accuracy is biased downward, because it was used to select the subsets. The set is small at 448 items, which the limitations section says gives little power for accuracy comparisons unless the effect is large, and the experts were sourced through Upwork with no enforced demographic mix. The authors "make no claim" that the set is representative.

At the strength the evidence allows, the result supports one claim: these questions are hard for both skilled non-experts with web access and a frontier model, so GPQA can serve as a hard accuracy benchmark near the edge of expert knowledge. Whether it can validate any oversight method is still open.