Does role-play dialogue boost reasoning accuracy in ChatGPT?

The paper introduces a "zero-shot role-play prompting methodology" and reports that it "consistently surpasses the standard zero-shot approach across most datasets" on twelve reasoning benchmarks evaluated with ChatGPT. Two figures are given as evidence: "accuracy on AQuA rises from 53.5% to 63.8%, and on Last Letter from 23.8% to 84.2%." The authors also report "comparable gains" when the same method is applied to Vicuna and Llama 2-Chat, though no comparable per-dataset figures for those models appear in the excerpt.

The method is a two-stage, two-round construction, not a single prepended label like "you are a doctor." Stage one builds a task-specific "role-setting prompt" (a user-designed description of a role with "a distinct advantage for the specific task at hand") and pairs it with a "role-feedback prompt" — the model's own best sampled response elaborating that role, chosen from multiple samples as "the most representative and immersive reply." Stage two concatenates this fixed role-setting/role-feedback exchange in front of each reasoning question, so "each question of the task, in conjunction with the role-setting and role-feedback prompts, is utilized as input to the model's API." The role is established once per task and then held constant across all its questions, which the authors note also means the API is invoked "a singular time per instance" rather than requiring a separate role-elaboration call per question.

This sits a step earlier in the role-play-and-reasoning line of inquiry than Why do reasoning models lose character consistency during role-playing?. That later paper studies large reasoning models (DeepSeek-R1, o-series) that generate extended internal chain-of-thought, and finds persona immersion actively degrades during that extended reasoning — attention diverts to generic task-solving, and style drifts toward formal logic. This 2023 ChatGPT study predates that reasoning-model regime: its "role-play" is a short, two-round dialogue scaffold around an otherwise ordinary zero-shot answer, not a persona that must survive a long internal reasoning trace. The two results are not strictly contradictory, but they bound where the accuracy gain reported here should be expected to hold. Compare also Can dialogue format help models reason more diversely?, which gets a related reasoning-quality improvement from a different mechanism — scene-based internal multi-character dialogue rather than a single assigned persona.

What the excerpt does not establish: why the role-play scaffold produces the gain. The paper's own limitations note that "the design of the role-setting and role-feedback prompts" was done manually and by sampling, is "time-consuming," and "may not always guarantee optimal results" — meaning the twelve-benchmark result reflects hand-tuned roles chosen per task, not a role-selection procedure that a practitioner could apply mechanically. The excerpt also does not isolate whether the gain comes from the persona content specifically or from the extra round of dialogue context that precedes the question; see the companion note on the paper's chain-of-thought comparison for what it says about mechanism.