InfoNCE loss generalizes similarity search with O(1/k) error bound
A new theoretical analysis shows that the InfoNCE loss, widely used in contrastive learning, yields a population risk that is O(1/k) close to an expected cross-entropy. This cross-entropy quantifies the deviation between softmax similarity search on unseen data using the learned embedding and an idealized search using the true similarity. The result provides a rigorous generalization guarantee for similarity search, a primary application of embedding models.
Entities
Related
Neural Collapse Is Forbidden: Information Floors in Language Models
A new paper argues that within-class variance in language model representations is not incomplete neural collapse but allocated information storage obeying a law. Across 14 models, macro-category structure carries only 4-12% of representational variance, while within-token context carries 79-91%, stable across a 100x parameter range.
Researcher tests minimal dynamical system for word embeddings without MLP or attention
A researcher proposes a word representation model using only token vectors, a start state, and two scalars, with no MLP, transformer, attention, or output matrix. The model achieves a SimLex-999 ρ of 0.3616 by updating a state vector via a cosine-based pull toward token attractors, encoding context through trajectory dynamics.
Researchers identify asymmetric generalization problem in LLM unlearning benchmarks
A new arXiv paper argues that existing machine unlearning benchmarks for LLMs suffer from under-forgetting and over-forgetting due to an asymmetric generalization problem. The authors propose that evaluation must cover diverse query formulations of target facts to reliably measure knowledge removal while preserving unrelated capabilities.
Paper derives exact dynamics of linear representation learning in neural networks
A new arXiv paper presents exact solutions for how linear concept representations emerge during neural network training, providing a mathematical framework for the dynamics of abstraction. This work formalizes the linear representation hypothesis, which underpins interpretability methods like linear probes and activation steering, and could guide future training and control techniques.
Study questions robustness of emergent misalignment in language models
A new paper systematically studies repeated alignment and misalignment cycles in language models, reproducing Emergent Misalignment (EM) but finding that both misalignment and realignment are highly sensitive to superficial dataset characteristics. The results suggest that EM may not be a robust phenomenon.