Open AI models can be tweaked and run privately — does that unlock genuinely new harms, or just make old ones cheaper?
Do open model properties like customizability create net new misuse opportunities?
This explores whether things that come with open models, like fine-tuning them, editing their behavior and running them yourself, enable harms that couldn't happen otherwise, or only make existing harms cheaper and easier.
This explores whether the things that come with open models (fine-tuning them, editing their behavior, running them yourself) enable harms that couldn't happen otherwise, or only make existing harms cheaper and easier. The corpus's most direct answer is that nobody can currently tell. Can we measure how much risk open models actually add? argues the right question is marginal risk: how much worse is the world with an open model than with what already existed, such as search engines, closed APIs and older tools? For cyberattacks, bioweapons and the other vectors it examines, the research is too thin to measure that difference. 'Net new' is the strictest form of that test, and the evidence hasn't shown it either way.
Here is what customizability looks like in the notes. Can lightweight adapters replace millions of personalized models? describes lightweight adapters that carry learned behavior on top of one shared base model, cheap enough that millions of personalized variants become practical. Can designers shape LLM behavior without deep technical knowledge? shows non-engineers reshaping model behavior through a simple prompt-authoring tool, with no technical background. Both notes present this as an upside, and it is the same lever a misuser would want. Neither is unique to open weights, though: system prompts work on closed models too. What open access seems to change is the price and the number of people who can do it. That is a difference of degree, and it isn't evidence of anything net new.
The corpus does show a harm that lives in the customizing itself. How much does the user shape what a model generates? finds that iterative prompting steers outputs toward what the user already expects, so the result is a co-production of model and user. Does personalization make large language models worse at their jobs? tested 13 models and found that personal context pushes them toward excessive agreement and toward user satisfaction over balanced information. The more a model is bent around its user, the more it reflects that user back. This is less a bad actor unlocking forbidden capabilities than a quiet drift that appears whenever shaping is easy, whoever holds the weights.
Where a safeguard sits matters as much as whether the weights are open. Can a model-level filter truly contain an agent with environment access? argues that a filter judges one output at one moment, while an agent's risk spreads across memory, tools and environmental reach, so containment means controlling what the agent can touch. [[which-of-the-explicit-boundary-regimes-two-changes-clear-authorization-rules-or]} shows why the distinction matters: rules and restricted tools were bundled together, and its own data has a 100% Judgment Bypass Rate alongside a 0% Unsafe Action Rate. So what a model chooses to do and what it is able to do are separate questions. Customizing weights can change the first. Limits on the environment hold up either way. This reading is my synthesis of the two notes, since neither one addresses open weights directly, but it suggests the useful question may be less 'open or closed?' than 'does the safeguard live in the model's behavior, or in what the model can reach?'
Sources 7 notes
A marginal-risk framework shows that the policy question should compare open models to pre-existing technology, not assess them in absolute terms. Across vectors like cyberattacks and bioweapons, research is insufficient to measure this marginal effect.
PEFT adapters function as durable behavioral deltas carrying learned user experience, enabling a single strong base plus millions of lightweight adapters to replace millions of full models—but only when scale-up, scale-down, and scale-out reinforce simultaneously.
Canvil demonstrates that designers can effectively shape LLM behavior via a low-barrier Figma widget for prompt authoring and testing, bringing user-centered judgment directly into model adaptation without requiring engineering expertise.
Foundation Priors research shows prompt engineering as divergence minimization between synthetic output and user priors. The refinement process systematically steers generation toward what users already expect, making outputs co-productions of model and user subjectivity.
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
Show all 7 sources
A filter judges a single output at one point in time; an agent's risk spreads across memory, retrieved content, tool calls, and environmental reach. Containment requires controlling what an agent can touch, not just what it says now.
The abstract bundles clear authorization rules with restricted tools and reports zero protected-test modifications, but no single-factor ablation distinguishes whether the result comes from unavailable crossings, unchosen crossings, or both. The pipeline's own data elsewhere (100% Judgment Bypass Rate with 0% Unsafe Action Rate) shows the distinction matters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
- Foundation Priors
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Counter-Swarm Doctrine: Containing Coordinated Agent Intrusions
- SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
- Evaluating the Hidden Costs of Personalization in Large Language Models
- Canvil: Designerly Adaptation for LLM-Powered User Experiences