SLORR: Simple and Efficient In-Training Low-Rank Regularization
Researchers introduced SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization. It avoids SVDs of large weight matrices, additional trainable parameters, and stateful cached quantities, addressing key limitations of existing methods. This could enable more aggressive compression of modern neural networks without significant accuracy loss.
Entities
Related
Reddit user seeks fine-tuning wisdom from experienced practitioners
A Reddit user posted a request for practical fine-tuning advice from those who have fine-tuned more than half a model, seeking tips on dataset curation, LoRA rank selection, and cost debugging. The post emphasizes real-world experience over generic documentation.
Two papers propose token-adaptive KV cache compression for long-context LLMs
Two arXiv papers from July 7, 2026 introduce token-adaptive KV cache compression methods for long-context LLM inference. DepthWeave-KV factorizes key/value states across neighboring layers using shared low-rank bases with token-specific residuals. FreqDepthKV uses shared low-frequency depth components and sparse high-frequency residuals, with an online probe assigning attention heads to different cache modes. Both aim to reduce memory bandwidth while preserving retrieval and reasoning quality.
Researcher tests minimal dynamical system for word embeddings without MLP or attention
A researcher proposes a word representation model using only token vectors, a start state, and two scalars, with no MLP, transformer, attention, or output matrix. The model achieves a SimLex-999 ρ of 0.3616 by updating a state vector via a cosine-based pull toward token attractors, encoding context through trajectory dynamics.
Developer compresses GLM-5.2 MoE to run on single RTX 3090 via 79 experiments
A developer conducted 79 experiments to compress GLM-5.2, a 337 GB MoE model with 75 sparse layers and 256 routed experts, to fit on a 24 GB RTX 3090. The approach uses per-expert codecs, a batch pipeline over all MoE layers, and a patched llama.cpp runtime that loads codec-native expert binaries at inference time. The MIT-licensed repository documents the method and findings on expert similarity.
Researchers identify modality interference as root cause of full-duplex SLM degradation
A new paper on arXiv (July 7, 2026) presents a fine-grained analysis of optimization dynamics in full-duplex Spoken Language Models (SLMs), identifying severe modality interference as the root cause of knowledge degradation and compromised semantic integrity. The work aims to enable more natural and intelligent full-duplex SLMs.