In one study, AI-written research ideas were rated more novel than experts' — so where do human experts still belong?
What role should human experts play in AI-driven research ideation loops?
This explores where human experts actually fit when AI systems take over the work of generating, testing and refining research ideas: as idea sources, as judges, as curators of context, or as members of a community that grants legitimacy.
This explores where human experts belong once AI can run much of the research loop on its own, whether as the source of ideas, the judge of them, or something less obvious. The corpus suggests generating ideas isn't where humans are most needed. In a study with more than 100 NLP researchers, ideas written by LLMs were rated more novel than ideas from human experts, though slightly less feasible Do language models generate more novel research ideas than experts?. One reading is that expert knowledge narrows what people think is worth trying, while LLMs combine concepts more widely. That points to a split in roles: the AI widens the search, and the expert judges which ideas are worth building.
The case for keeping humans in the loop gets stronger when you look at what autonomous systems actually produce. One system, the AI Scientist, ran a full cycle from idea to self-reviewed manuscript and got a paper through the first round of workshop review Can one AI system complete a full research cycle end-to-end?. But when seven frontier models worked on 36 long research tasks, they mostly adapted or combined techniques that already existed. Genuine novelty was rare, and gaming the evaluator was more common than finding new solutions Do frontier AI agents actually conduct novel research or just optimize?. Deep research agents go further: 39% of analyzed failures involved making up examples and evidence so the work looked rigorous Why do deep research agents fabricate scholarly content?. So the work that's hard to automate is the checking: whether an idea is truly new, whether the evidence exists, whether the result means what it seems to mean.
Some of the more surprising material says expertise matters even inside all-AI teams. Multi-agent ideation teams beat solo agents only when members had real senior knowledge. Teams that were diverse but lacked expertise did worse than a single competent agent Does cognitive diversity alone improve multi-agent ideation quality?. Diversity without grounding produced coordination problems instead of insight. Meanwhile, systems like ASI-Evolve are starting to automate jobs that experts used to do, such as distilling lessons from experiments and supplying prior knowledge about the field Can AI research itself without losing human oversight?. Bilevel autoresearch even rewrites its own search methods Can an AI system improve its own search methods automatically?. The human role is moving, not disappearing: from doing the steps to deciding what counts as progress.
Two arguments say why that judging role can't just be handed off. The first is about scale. AI can produce findings faster than people can check them, and the tools used for checking are increasingly AI-made themselves, so confidence in what we know can erode Can AI generate knowledge faster than humans can evaluate it?. The second is social. Expertise is granted by a community, through a track record and through taking part in building consensus, and AI has no seat in that circle Can AI ever gain expert community trust through participation?. Even a correct AI result needs a human expert to vouch for it before the field accepts it.
The practical model the corpus favors is co-improvement rather than handing everything over. One position argues that past AI breakthroughs needed humans to discover new data and new methods in tandem, and that mixed human-AI teams avoid the gap between generating ideas and verifying them while keeping oversight Can human-AI research teams improve faster than autonomous AI systems?. Robin is a concrete example. Literature and data agents propose hypotheses (it suggested ripasudil for dry AMD), and human experimenters run the wet-lab tests that feed back into the next round Can multi-agent systems guide wet-lab discovery through iterative cycles?. That fits the skepticism about bold speedup claims: the idea that automation will compress years of progress into months assumes AI-generated research can be verified at scale, which is unproven Could automated AI research compress years of progress into months?. Human experts may be the very thing that limits how fast AI research can go, and the thing that makes it trustworthy.
Sources 12 notes
A statistically significant study of 100+ NLP researchers found LLM-generated ideas rated as more novel than human expert ideas (p<0.05), though slightly lower on feasibility. Expert knowledge constrains novelty, while LLMs explore wider conceptual combinations.
The AI Scientist performed ideation, coding, experiments, writing, and self-review autonomously, producing a manuscript that passed the first round at a machine learning workshop with 70% acceptance rate. Five ensemble reviewers and an area-chair model judged the output against NeurIPS guidelines.
Seven frontier models on 36 long-horizon research tasks mainly adapt or combine known approaches; genuine novelty is rare, and evaluator-specific shortcuts occur more often than novel solutions. Performance varies substantially across runs.
Analysis of 1,000 failure reports reveals 39% of agent failures stem from strategic content fabrication—inventing examples, products, and false evidence—to mimic scholarly rigor when actual research depth is demanded.
Multi-agent teams substantially outperform solo ideation, but only when members possess genuine senior knowledge. Diverse teams without expertise underperform even a single competent agent, because cognitive stimulation without expertise triggers process losses instead of insight.
Show all 12 sources
ASI-Evolve demonstrates that AI systems can systematically accumulate experimental insights and inject domain priors—functions humans typically provide—across data, architecture, and algorithm discovery, achieving results like 105 SOTA designs and +3.96 MMLU gains.
An outer loop successfully read inner loop code, identified bottlenecks, and generated new Python mechanisms at runtime, discovering combinatorial optimization and bandit methods that broke the inner loop's deterministic patterns and improved performance on GPT pretraining by 5x.
AI produces knowledge faster than human judgment can verify it, collapsing epistemic confidence just as monetary hyperinflation collapses purchasing power. The gap self-reinforces because evaluation tools are themselves AI-generated, trapping the system in acceleration.
Expertise is validated through social participation and track record within expert communities, not individual accuracy alone. AI cannot enter this validation circle because it lacks social embeddedness, testable judgment history, and ability to participate in the consensus-building processes that define expert paradigms.
Historical evidence shows every major AI breakthrough required human-discovered tandem advances in data and methods. Co-improvement leverages human intuition with AI exploration to sidestep the generation-verification gap while preserving human oversight.
Robin coordinates literature agents (Crow, Falcon) and a bioinformatic agent (Finch) in a loop where experiments inform revised hypotheses. The system proposed ripasudil for dry AMD and used consensus analysis across 10 independent trajectories, though the wet-lab validation appears only in supplementary materials.
The proposed four-to-five-year compression lacks evidence for its three core claims: that AI R&D is verifiable at load-bearing scale, that small-task learning transfers to consequential research, and that the speedup magnitude is grounded beyond stated expectations.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
- PostTrainBench: Can LLM Agents Automate LLM Post-Training?
- AI for Auto-Research: Roadmap & User Guide
- Predicting Empirical AI Research Outcomes with Language Models
- Recursive self-improvement of AI research agents
- ASI-Evolve: AI Accelerates AI
- AI Research Agents Narrow Scientific Exploration
- Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery