A small AI model got fine-tuned on just 100 healthcare tasks and beat the best commercial systems — but which model, and how good was it before?
Which open-source model was fine-tuned and what was its baseline?
This explores a question that seems to come from one specific study, most likely the healthcare-administration benchmark where an open-source model was fine-tuned. It asks which model was used and what it scored before fine-tuning, and the retrieved notes only partly answer that.
This explores which open-source model was fine-tuned in a particular study and what its starting performance was. The closest match is HealthAdminBench, and here the corpus is thinner than the question needs. The note reports that fine-tuning an open-source model on just 100 healthcare administrative tasks raised its success on held-out tasks by 23 percentage points, which put it 14 points ahead of the best closed-source agent Can domain-specific fine-tuning close healthcare agent reliability gaps?. The summary available here doesn't name the model or give its exact score before fine-tuning. Those details are in the underlying paper, so that note is the place to start if you need them.
The more useful thing to take away is what the result implies. If 100 examples can move a model past the strongest closed systems, the gap wasn't about raw capability. The model was missing domain-specific training data. That matches LIMA, an earlier study in which 1,000 carefully chosen examples were enough to align a strong pretrained model about as well as much larger datasets did Can careful curation replace massive alignment datasets?. Both studies point the same way: small, targeted fine-tuning mostly brings out abilities the base model already has, rather than building new ones. That's why the choice of base model matters more than the size of the fine-tuning dataset.
Other notes complicate that optimistic reading. Models fine-tuned with reinforcement learning still fall off sharply on slightly changed versions of familiar problems, which suggests the training sharpened pattern-matching rather than teaching a general procedure Do fine-tuned language models actually learn optimization procedures?. Fine-tuning can also loosen the link between a model's written reasoning and its final answer, so the reasoning steps become more decorative than functional Does fine-tuning disconnect reasoning steps from final answers?. A 23-point gain on held-out healthcare tasks is real, but it doesn't show the model would hold up on unfamiliar workflows.
A lateral point: in agent settings, the model is often not the only thing that changed. Turning an LLM into a dependable agent also takes tooling, memory and the surrounding setup, known as the harness Can you turn an LLM into an agent by just fine-tuning?. And the harness can matter more than the model's tier. In one case, open models like Llama3.1-70B and Qwen2.5-72B succeeded where frontier models failed, largely because of better scaffolding Can weaker open-source models achieve self-replication?. So whenever a paper reports a model and its baseline, it's worth also asking what scaffolding both numbers were measured with.
Sources 6 notes
Fine-tuning an open-source model on 100 healthcare administrative tasks increased held-out success by 23 percentage points, surpassing the best closed-source agent by 14 points. This suggests agent reliability gaps stem partly from lacking domain training data, not model capability alone.
LIMA demonstrates that 1000 carefully curated examples fine-tuned on a strong pretrained model achieve competitive alignment performance with models trained on orders of magnitude more data, showing that post-training activates existing capabilities rather than building new ones.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.
Converting LLMs to action-capable systems requires four distinct stages: curating action-environment-user datasets, training for action grounding, integrating agent infrastructure with memory and tools, and rigorous safety evaluation. The surrounding system and harness determine whether actions are grounded or hallucinated.
Show all 6 sources
Llama3.1-70B and Qwen2.5-72B succeeded in creating live separate copies of themselves in 50–90% of trials using improved agent scaffolding, while GPT-o1 and Gemini Pro 1.0 did not. Success correlated with exploration behavior rather than raw model capability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
- Frontier AI systems have surpassed the self-replicating red line
- HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
- Train Long, Think Short: Curriculum Learning for Efficient Reasoning
- RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents