Does AI tutoring raise exam scores without leaving students less able to do the work once the AI is gone?

This explores whether AI tutoring produces real learning that lasts once the AI is gone, or only better results while it is available. The collection has no study that directly compares exam results with independent skill for a full AI tutoring system. What it does have points to one theme: the answer depends less on whether students use AI and more on how they use it, and scores taken with AI available are a poor guide to what a student can do alone.

The clearest warning concerns skill-building. In studies of learners working with and without AI, the people without AI ran into more errors and fixed them on their own, and they retained more skill. The AI-assisted learners handed debugging to the AI. Even those who did the most debugging alongside the AI scored lowest on later skill tests (Does AI assistance remove a core learning channel through error work?). Getting stuck and getting yourself unstuck is where much of the learning happens, and a helpful tutor can remove exactly that step. A four-month EEG study points the same way: participants who relied on an LLM showed the weakest brain connectivity and the poorest memory, and they had trouble recalling work they had just produced (Does AI assistance weaken our brain's ability to think independently?). Even correct AI suggestions can carry a hidden cost, because they interrupt a person's focus mid-task (Does AI assistance always help reasoning or does it carry hidden costs?).

The outcome isn't fixed, though. In a preregistered experiment with 704 people, feedback that showed learners what they lose by offloading answers to an LLM cut their answer requests in half and raised their scores on tests taken without AI by 51%. A reward for effort, which seems like the obvious fix, had no measurable effect (Can metacognitive feedback stop students from offloading to AI?). Paying people to try harder didn't change their behavior. Helping them see their own habits did. So AI tutoring can protect independent ability, but only when the design makes students notice when they're leaning on it.

A parallel from machine learning sharpens the point. When models are fine-tuned on correct answers, their benchmark accuracy goes up while the quality of their reasoning steps falls by 38.9%. They reach right answers through reasoning that doesn't actually lead there, and metrics that check only the final answer miss the decline (Does supervised fine-tuning improve reasoning or just answers?). Exams can have the same blind spot. A score that improves with AI assistance may hide weaker reasoning underneath. That's why the most informative studies above measure performance after the AI is removed.

One more thread is relevant to anyone building tutors. Researchers who simulate students to test tutoring systems found that models which accurately copy a student's behavior tend to ignore the tutor's corrections. Prompted role-play follows the tutor's guidance easily but doesn't capture what a particular student actually knows (Can student simulators match both behavior and learn from teaching?). A simulated student that simply goes along with the tutor will make any tutoring method look effective. Testing an AI tutor properly is therefore its own unsolved problem.