Research paper proposes reusing spectral patterns from pretrained GPT-2 checkpoints as initialization for language model pretraining
A new arXiv paper analyzes eleven pretrained GPT-2-style checkpoints and finds that they share structured weight spectra, with consistent depth trends in Frobenius norm and effective-rank entropy. The authors propose reusing these recurring spectral patterns as initialization signals for GPT-2-style language model pretraining, potentially reducing training time or improving convergence.
Entities
Related
Paper challenges text-only pretraining, proposes visual pretraining for language models
A new arXiv paper argues that current language model pretraining discards rich visual information from documents and web pages. The authors propose scalable visual pretraining to incorporate figures, equations, and layouts, aiming to improve language intelligence beyond text-only approaches.
Neural Collapse Is Forbidden: Information Floors in Language Models
A new paper argues that within-class variance in language model representations is not incomplete neural collapse but allocated information storage obeying a law. Across 14 models, macro-category structure carries only 4-12% of representational variance, while within-token context carries 79-91%, stable across a 100x parameter range.
Paper derives exact dynamics of linear representation learning in neural networks
A new arXiv paper presents exact solutions for how linear concept representations emerge during neural network training, providing a mathematical framework for the dynamics of abstraction. This work formalizes the linear representation hypothesis, which underpins interpretability methods like linear probes and activation steering, and could guide future training and control techniques.
Super-Tuning: Activation-Aware Pruning Reused for Sparse Fine-Tuning
A new arxiv paper proposes Super, a sparse PEFT method that reuses Wanda-style activation-weighted magnitude scores from pruning to select a small trainable support, and Supra, a hybrid adapter combining sparse updates with LoRA. This approach reduces memory and compute for fine-tuning LLMs while maintaining performance.
Blog post explores why LLMs produce predictable metaphors and how architecture might reduce attractor pull
A blog post titled 'Escaping the Attractor' examines why large language models tend to produce similar metaphors (e.g., 'Time is a River') when prompted, attributing this to attractors in the embedding space. The author suggests that architectural changes could make models less predictable, building on earlier ideas about shared embedding geometries across models.