How do embedding dimensionality and ranking metrics both cause interest crowding?
This explores two separate culprits behind the same symptom — when a recommender squeezes out your smaller or niche interests in favor of your dominant ones — and how the size of the embedding vectors and the metric the system is optimized for each push in that direction.
This explores why "interest crowding" — your minority tastes getting drowned out by your dominant ones — turns out to have two independent causes that arrive at the same place: one baked into how users and items are *represented*, and one baked into what the system is *rewarded for*. The corpus is interesting precisely because these are usually discussed in separate conversations, but they reinforce each other.
Start with dimensionality. When the embedding vectors are too small, there simply isn't enough room to encode every interest faithfully, so the model takes the shortcut that helps its score most — it overfits toward popular items and lets niche ones starve, a bias that compounds over time and can't be patched after the fact Does embedding dimensionality secretly drive popularity bias in recommenders?. This isn't just an engineering tuning problem; there's a hard mathematical ceiling. Communication-complexity results show that for any embedding dimension *d*, there's a maximum number of distinct top-k result combinations the system can ever represent — even embeddings fit directly to the test data hit the wall Do embedding dimensions fundamentally limit retrievable document combinations?. A single fixed-length user vector makes it worse, because cramming many tastes into one point is lossy compression by construction How can user vectors capture diverse interests without exploding in size?.
Now the ranking metrics, which crowd interests from the other direction. Accuracy-optimized models systematically miscalibrate by over-weighting whatever interest dominates your history, so a user who is 70% into pop and 30% into jazz gets served 95% pop Why do accuracy-optimized recommenders crowd out minority interests?. This isn't an accident of the model — it's what the objective asks for. Multinomial likelihoods win on top-N benchmarks *because* they force items into direct probability competition, where the strong signals take the available mass Why does multinomial likelihood work better for ranking recommendations?. And the evaluation metric itself rewards concentration: DCG/nDCG steeply discount anything past the top few slots, binding the system to the fact that users mostly look at the head of the list How can evaluation metrics reflect graded relevance and user attention?. Optimize for that, and pushing your secondary interests off the front page is the rational move.
The thing you might not have expected: these two causes are multiplicative, not redundant. A tight embedding budget decides what *can* be represented; a winner-take-most ranking objective decides what *gets shown* from the room that's left. Even a generously dimensioned model will crowd interests if the metric rewards it, and even a calibration-aware objective will crowd them if the vectors can't hold the distinctions in the first place.
What's quietly hopeful is that the corpus also splits the *fixes* along the same seam. You can attack the representation side by giving users multiple persona vectors weighted per candidate, so a niche taste lights up exactly when relevant instead of being averaged away Can attention mechanisms reveal which user taste explains each recommendation?, or by conditioning attention on each candidate to express diverse interests without paying for more dimensions How can user vectors capture diverse interests without exploding in size?. Or you can attack the metric side after the fact, re-ranking outputs to enforce proportional representation without retraining anything Why do accuracy-optimized recommenders crowd out minority interests?. Which lever you reach for depends on which crowding mechanism is actually biting — and now you know there are two.
Sources 7 notes
Research shows that when user/item embedding dimensions are too small, recommender systems overfit toward popular items to maximize ranking quality. This compounds over time as niche items receive insufficient exposure, and cannot be fixed post-hoc without treating dimensionality as a fairness hyperparameter.
Communication complexity theory proves that for any embedding dimension d, there exists a maximum number of top-k document combinations that can be returned as results. Even embeddings optimized directly on test data hit this polynomial limit, demonstrated on trivially simple retrieval tasks.
Deep Interest Network weights historical behaviors against each candidate ad, activating only relevant interests dynamically. This preserves dimension efficiency while expressing diverse tastes without lossy compression.
Accuracy-optimized models systematically miscalibrate by over-weighting dominant user interests. A post-processing reranking algorithm that enforces calibration constraints can restore proportional representation without retraining the underlying model.
Liang et al. show that switching VAE likelihoods from Gaussian/logistic to multinomial achieves state-of-the-art results because enforced probability competition between items directly aligns training with top-N ranking objectives. Rebalancing KL regularization further improves performance.
Jarvelin and Kekalainen's DCG and nDCG metrics handle graded relevance by accumulating relevance scores with a position discount factor that devalues late-retrieved documents. This binds evaluation to observed user behavior: users examine top results more carefully than lower-ranked ones, making ranking position matter.
AMP-CF represents each user as multiple latent personas weighted dynamically by candidate item. This makes recommendations both diverse and interpretable—each suggestion traces to the specific persona preference it satisfies—without requiring post-hoc reranking.