Super-Tuning: Activation-Aware Pruning Reused for Sparse Fine-Tuning
A new arxiv paper proposes Super, a sparse PEFT method that reuses Wanda-style activation-weighted magnitude scores from pruning to select a small trainable support, and Supra, a hybrid adapter combining sparse updates with LoRA. This approach reduces memory and compute for fine-tuning LLMs while maintaining performance.
Entities
Related
SATS: Sensitivity-Aware Thresholding for MLP Activation Sparsification in LLMs
A new arxiv paper proposes Sensitivity-Aware Thresholding for Sparsity (SATS), a method to calibrate layerwise gate thresholds for MLP activation sparsification using a local sensitivity proxy instead of activation percentiles. The approach aims to reduce computation during LLM inference while preserving model quality.
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Researchers introduced SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization. It avoids SVDs of large weight matrices, additional trainable parameters, and stateful cached quantities, addressing key limitations of existing methods. This could enable more aggressive compression of modern neural networks without significant accuracy loss.
Two papers propose token-adaptive KV cache compression for long-context LLMs
Two arXiv papers from July 7, 2026 introduce token-adaptive KV cache compression methods for long-context LLM inference. DepthWeave-KV factorizes key/value states across neighboring layers using shared low-rank bases with token-specific residuals. FreqDepthKV uses shared low-frequency depth components and sparse high-frequency residuals, with an online probe assigning attention heads to different cache modes. Both aim to reduce memory bandwidth while preserving retrieval and reasoning quality.
Self-Guided Test-Time Training Improves Long-Context LLM Accuracy
A new arxiv paper proposes Self-Guided Test-Time Training (SG-TTT) to improve long-context utilization in LLMs without requiring labeled data. The method uses the model's own predictions to generate pseudo-labels for fine-tuning on the test context, addressing accuracy degradation in long inputs. This approach is more efficient than full TTT and shows promise for practical deployment.
Reddit user seeks fine-tuning wisdom from experienced practitioners
A Reddit user posted a request for practical fine-tuning advice from those who have fine-tuned more than half a model, seeking tips on dataset curation, LoRA rank selection, and cost debugging. The post emphasizes real-world experience over generic documentation.