Does repeated sensitive data in fine-tuning cause memorization?

Memorization is most dangerous exactly where organizations fine-tune on proprietary or personal data. Controlled experiments across GPT-2, Phi-3, and Gemma-2 quantify the risk: fine-tuning with repeated sensitive data raises privacy-leakage rates from a 0-5% baseline to 60-75% — a 64.2% average increase — because repeated exposure pushes the model toward near-verbatim reproduction at inference. This is the concrete mechanism behind the theory that in-weight learning overwrites and memorizes.

The constructive half rebuts the assumed privacy-utility tradeoff. A layered framework — semantic data deduplication, differential privacy during generation, entropy-based filtering, and pattern-based content filtering — drives leakage to 0% while retaining 94.7% of original utility. The keeper is that privacy and performance are not inherently incompatible in fine-tuned LLMs: the defenses are complementary and operate at different stages (data, generation, output), so stacking them closes the gap without gutting capability.

This is the privacy face of the in-weight-learning cost documented elsewhere. It supplies the mechanism behind Can models store unlimited facts without growing larger? (finetuning facts in is exactly what memorizes), and it complements When do language models stop memorizing and start generalizing?: that note bounds capacity in theory; this one shows repetition saturating it into leakage in fine-tuning practice.