Researchers identify and fix crispness penalty failure mode in legible transformers
A new paper on arXiv reveals that a crispness penalty intended to make transformer operators more legible can collapse them into dead constants. The authors derive an identity showing the penalty is a variance minimizer, and propose a per-channel variance floor as a fix.
Entities
Related
Paper derives exact dynamics of linear representation learning in neural networks
A new arXiv paper presents exact solutions for how linear concept representations emerge during neural network training, providing a mathematical framework for the dynamics of abstraction. This work formalizes the linear representation hypothesis, which underpins interpretability methods like linear probes and activation steering, and could guide future training and control techniques.
Researchers introduce fully trainable connection optimization for logic gate and lookup table networks
A new method enables partial or full optimization of connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs) by learning a probability distribution over connections per input pin. The approach outperforms standard fixed-connection LGNs on Yin-Yang, MNIST, and Fashion-MNIST benchmarks while requiring fewer resources.
SATS: Sensitivity-Aware Thresholding for MLP Activation Sparsification in LLMs
A new arxiv paper proposes Sensitivity-Aware Thresholding for Sparsity (SATS), a method to calibrate layerwise gate thresholds for MLP activation sparsification using a local sensitivity proxy instead of activation percentiles. The approach aims to reduce computation during LLM inference while preserving model quality.
Foveated Dynamic Transformer (FDT) paper proposes foveation-guided token selection for robust vision transformers
A new arXiv paper introduces the Foveated Dynamic Transformer (FDT), a vision transformer architecture that uses foveation-guided dynamic token selection inspired by the human visual system. The model demonstrates strong resilience to noise and adversarial attacks without explicit adversarial training, offering a path to more efficient and robust vision models.
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Researchers introduced SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization. It avoids SVDs of large weight matrices, additional trainable parameters, and stateful cached quantities, addressing key limitations of existing methods. This could enable more aggressive compression of modern neural networks without significant accuracy loss.