Your Programming Students' Cognition with ChatGPT: Higher Performance, Lower Retention, and Reduced Ownership

Generative AI can improve students’ programming performance, but successful task completion may not reflect what they retain. We examined performance, retention, cognitive load, and ownership in a controlled between-subjects experiment with 59 undergraduate computer science students, 55 were retained for analysis. Participants completed three introductory C programming tasks with access to ChatGPT-4.5 or conventional web search without generative AI. We measured task performance, self-reported mental effort and difficulty, pupillary responses, heart rate variability, and ownership, and assessed cued recall immediately and 48 hours later. ChatGPT-assisted students achieved higher coding scores (89% vs. 69%) but lower recall scores immediately (41% vs. 53%) and after 48 hours (39% vs. 52%). There was no significant difference in the loss of recall information over 48 hours between the groups. Selfreported mental effort increased less across tasks in the ChatGPT condition (Holm-adjusted p = .047), and students attributed less of the submitted code to themselves (45% vs. 81%). Confirmatory physiological tests did not detect significant differences in trajectories between conditions; substantial data loss limits their interpretation.

Introduction. Since the release of ChatGPT in November 2022, generative Artificial Intelligence (GenAI) has become part of students’ routine; they use large language models (LLMs) to draft essays, debug code, and complete assignments [46], and many tertiary institutions have replaced initial prohibition with policies permitting disclosed use [17, 54, 58]. These changes are consistent with calls from UN- ESCO and others to prepare students for an AI future [55]. Educators have traditionally relied, in part, on students demonstrating knowledge acquisition through summative assessment tasks. This practice assumes that the artefact is a sufficient proxy to measure a student’s knowledge or capability. LLMs can produce a plausible essay or functional code without the student acquiring that knowledge. Research to date has established that GenAI assistance improves immediate task performance [48, 84], but there is limited evidence on what students actually retain from AI-assisted work [24, 31].

Discussion / Conclusion. RQ1a The AI-Assisted group significantly outperformed the No AI group on the assessment with an average grade of 89% compared to 69%. Each coding challenge was an introductory level task, within the capability of the model to solve when given the full context. Several of the instances where AI-Assisted participants did not receive full marks involved participants who had not provided the full instructions or the scaffold code to the model. One AI-Assisted participant did not complete all three tasks. This participant attempted to use ChatGPT as a tutor, asking the model to respond with questions or next step hints, rather than solutions. The exchange became circular and self-contradicting and did not provide useful responses to the student, resulting in the student being unable to finish Task 2 before the timer had expired. This student was unable to attempt Task 3.