How do orthogonal adapter vectors avoid interference at scale?

This explores how you can stack many task- or user-specific adapters on one base model without their learned changes colliding. The literal framing — 'orthogonal adapter vectors' — implies the fix is geometric: keep each adapter's update pointing in a non-overlapping direction. The corpus pushes back on that intuition and reframes interference as a question of *which parameters an adapter is allowed to move*, not just *which direction* it moves them. The cleanest result here is that explicit structural parameter isolation beats clever merging: identifying the core parameter region each task depends on, freezing those, clustering tasks that overlap, and only geometrically merging the *non-core* parameters consistently outperforms standard multi-task fine-tuning Can isolating task-specific parameters prevent multi-task fine-tuning interference?. Notably, that same work finds that just *scheduling* tasks in time — training them in sequence to avoid stepping on each other — is not enough without the structural isolation. Interference is a property of overlapping weights, not overlapping schedules.

That reframes your question: orthogonality is one way to enforce non-overlap, but the corpus suggests the load-bearing move is deciding the boundary between shared and private parameters in the first place. Once you draw that line, adapters become durable, composable deltas rather than competing edits.

The payoff at scale is what makes this interesting. Treating PEFT adapters as *persistent local state* — small behavioral deltas that carry a user's learned history — lets one strong base model plus millions of lightweight adapters stand in for millions of full fine-tuned models Can lightweight adapters replace millions of personalized models?. But that note adds a sharp caveat: it only holds when scaling up (base capability), scaling down (adapter cost), and scaling out (population of adapters) reinforce each other simultaneously. So 'avoiding interference at scale' isn't purely an isolation problem — it's a co-design problem where the base has to be strong enough that adapters only need to encode small, separable deltas.

There's a quieter warning worth surfacing from an adjacent corner of the corpus. Any scheme that compresses many distinct behaviors into a fixed-capacity representation eventually hits a hard mathematical ceiling: for a given embedding dimension, there's a provable maximum number of distinct subsets you can represent, even with weights optimized directly on the target data Do embedding dimensions fundamentally limit retrievable document combinations?. The lesson transfers — if your adapters share a too-small subspace, no amount of orthogonalization buys you unlimited non-interfering directions. Isolation defers the collision; it doesn't repeal the capacity limit.

The through-line: interference at scale is solved less by making adapter vectors mutually perpendicular and more by *which weights you let each one own*, paired with a base model strong enough that the per-task deltas stay small and separable. If you want to go deeper, the parameter-isolation note is the mechanism and the PEFT-as-state note is the scaling argument; read them together rather than separately.