SATS: Sensitivity-Aware Thresholding for MLP Activation Sparsification in LLMs
A new arxiv paper proposes Sensitivity-Aware Thresholding for Sparsity (SATS), a method to calibrate layerwise gate thresholds for MLP activation sparsification using a local sensitivity proxy instead of activation percentiles. The approach aims to reduce computation during LLM inference while preserving model quality.
Entities
Related
Super-Tuning: Activation-Aware Pruning Reused for Sparse Fine-Tuning
A new arxiv paper proposes Super, a sparse PEFT method that reuses Wanda-style activation-weighted magnitude scores from pruning to select a small trainable support, and Supra, a hybrid adapter combining sparse updates with LoRA. This approach reduces memory and compute for fine-tuning LLMs while maintaining performance.
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Researchers introduced SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization. It avoids SVDs of large weight matrices, additional trainable parameters, and stateful cached quantities, addressing key limitations of existing methods. This could enable more aggressive compression of modern neural networks without significant accuracy loss.
STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
A new paper introduces STEEL, a sparsity-aware fused attention mechanism designed for AMD's XDNA NPU, enabling energy-efficient long-sequence inference on laptop-class SoCs. The approach addresses the challenge of mapping attention mechanisms onto NPUs while maintaining low power consumption, which is critical for agentic workloads that require on-device processing for reliability and privacy.
Two papers propose token-adaptive KV cache compression for long-context LLMs
Two arXiv papers from July 7, 2026 introduce token-adaptive KV cache compression methods for long-context LLM inference. DepthWeave-KV factorizes key/value states across neighboring layers using shared low-rank bases with token-specific residuals. FreqDepthKV uses shared low-frequency depth components and sparse high-frequency residuals, with an online probe assigning attention heads to different cache modes. Both aim to reduce memory bandwidth while preserving retrieval and reasoning quality.
Researchers introduce fully trainable connection optimization for logic gate and lookup table networks
A new method enables partial or full optimization of connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs) by learning a probability distribution over connections per input pin. The approach outperforms standard fixed-connection LGNs on Yin-Yang, MNIST, and Fashion-MNIST benchmarks while requiring fewer resources.