Researchers introduce fully trainable connection optimization for logic gate and lookup table networks
A new method enables partial or full optimization of connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs) by learning a probability distribution over connections per input pin. The approach outperforms standard fixed-connection LGNs on Yin-Yang, MNIST, and Fashion-MNIST benchmarks while requiring fewer resources.
Entities
Related
SATS: Sensitivity-Aware Thresholding for MLP Activation Sparsification in LLMs
A new arxiv paper proposes Sensitivity-Aware Thresholding for Sparsity (SATS), a method to calibrate layerwise gate thresholds for MLP activation sparsification using a local sensitivity proxy instead of activation percentiles. The approach aims to reduce computation during LLM inference while preserving model quality.
SAMPAT: A new interpretable neural architecture for scientific data analysis
Researchers introduced SAMPAT, a three-layer neural architecture that provably learns smooth, differentiable functions with interpretable polynomial approximations. It aims to address the lack of interpretability in deep neural networks for scientific data analysis.
Researchers identify and fix crispness penalty failure mode in legible transformers
A new paper on arXiv reveals that a crispness penalty intended to make transformer operators more legible can collapse them into dead constants. The authors derive an identity showing the penalty is a variance minimizer, and propose a per-channel variance floor as a fix.
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Researchers introduced SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization. It avoids SVDs of large weight matrices, additional trainable parameters, and stateful cached quantities, addressing key limitations of existing methods. This could enable more aggressive compression of modern neural networks without significant accuracy loss.
Paper derives exact dynamics of linear representation learning in neural networks
A new arXiv paper presents exact solutions for how linear concept representations emerge during neural network training, providing a mathematical framework for the dynamics of abstraction. This work formalizes the linear representation hypothesis, which underpins interpretability methods like linear probes and activation steering, and could guide future training and control techniques.