When an AI's medical answer gets a human edit first, do doctors rate it differently than the raw draft?

This explores whether a human editing pass changes how clinicians rate AI responses, compared with seeing the model's raw output. The short answer is that the collection has no study that tests this directly. None of the notes compares clinician ratings of edited AI drafts against unedited ones. What the collection does have is evidence about the two sides of that comparison. It suggests that the gap between edited and raw output may be smaller than you'd expect, and that ratings may not measure what we hope they do.

Start with what clinicians actually do when shown AI advice. In one blinded comparison of 104 pairs of responses, clinicians rated GPT-4's advice as more emotionally empathetic than expert advice and found no meaningful difference in scientific quality. They also guessed which response came from the AI only 45% of the time, which is no better than chance Can clinicians tell GPT-4 advice apart from expert advice?. If clinicians can't tell AI text from expert text, the question of whether editing helps becomes harder to answer through ratings alone. The baseline already reads as human-quality.

The writing-assistance research suggests why editing may change less than people assume. Writers edited AI-generated paragraphs only 23% of the time, and their edits left the text about 96% similar to the original Do writers actually edit AI-generated text before publishing?. In practice, "edited" output is often close to raw output. A related study found that AI assistance shifted readers' impressions of the writer on all 29 measured traits, toward more confident, more agreeable and more polished Does AI writing assistance change how readers perceive the writer?. If those qualities survive light editing, a clinician rating an edited response is mostly rating the model's voice.

The less obvious part is that editing out the AI's habits may lower ratings. When researchers trained reward models to remove those persona distortions, writers liked the resulting text less. The clarity and confidence readers enjoy come from the same tendencies that cause the distortions Can AI writing assistance remove distortion without losing appeal?. Evaluators also have a general weakness for style. Models trained to imitate ChatGPT fooled human raters with confident, fluent prose without becoming more factually accurate Can imitating ChatGPT fool evaluators into thinking models improved?. Fluent text also tends to read as competent text Does processing ease mislead users about their own competence?. Together these suggest a risk: careful clinical editing that adds hedges, caveats or blunt corrections could make a response more accurate while lowering its rating.

What the collection can't tell you is the actual size of the effect in clinical settings. To find out, you'd want a study that rates three versions of the same responses (raw, lightly edited and substantially edited) and scores accuracy separately from perceived empathy. Until someone runs that study, the safest reading is that clinician ratings mostly track how a response sounds. Whether a human edited it is a separate question.