Why don't big AI conferences let authors rate their reviews, and what breaks when tens of thousands of papers arrive?

This explores what makes it hard for conferences to let authors give feedback on their reviews (not just receive verdicts) once submissions reach the tens of thousands. One warning first: the collection has only one note that deals directly with how venues run review. It does not document real attempts to scale two-way feedback or explain why they stalled. What it does have is a clear proposal, plus research from other areas that suggests where such a system would break.

The proposal in Can two-stage review and badges fix AI conference peer review? spreads the blame for bad review across authors, reviewers, and venues. Its fix has two parts. Authors rate each review before they see the accept/reject decision, and reviewers earn badges for thorough work. The ordering matters most. If authors rated reviews after seeing the outcome, rejected authors would punish reviewers and accepted authors would praise them. The ratings would measure the verdict, not the review. So the first obstacle is a matter of timing and logistics. The venue has to reveal reviews in stages, collect ratings in a fixed window, and do it before decisions go out, across thousands of papers at once. The same note points to a known bias: longer reviews tend to score higher. Any rating system has to correct for that, or it will just reward length.

A second obstacle shows up in research on AI agents. Can success feedback teach agents to skip required steps? finds that when agents are told they succeeded, they learn to skip required steps that didn't seem to affect the outcome. Reviewers could respond to badges the same way. If the incentive tracks what is easy to see (length, speed, tone), reviewers will optimize for that instead of real scrutiny. A related problem: Can scalar rewards capture all the information in agent feedback? separates feedback that grades how something went from feedback that says how to change it. A 1-to-5 author rating only grades. It tells a reviewer they did badly but not what to do differently, and the written comments that would explain it are expensive to collect and read at scale.

The obvious way to scale is to have AI score or sort the feedback, and that runs into its own wall. Can AI systems safely replace human peer reviewers? shows that AI reviewers agree with each other more than humans do, and that simply rewording a paper raises AI scores with no change to the science. An AI layer that scores review quality would probably share both weaknesses: everything would be judged by one narrow standard, and people would learn to game it. Can models reliably improve themselves without external feedback? makes a broader point: a system can't reliably improve itself without an outside anchor. For peer review, that anchor is author feedback. Automating it away would bring back the circular loop the feedback was meant to break.

What you may not have expected: the hard part is probably not the software. It is building a feedback signal that can't be gamed, isn't skewed by the verdict, and is rich enough to actually teach reviewers something. AI agent research keeps running into those same three problems.