Self-Guided Test-Time Training Improves Long-Context LLM Accuracy
A new arxiv paper proposes Self-Guided Test-Time Training (SG-TTT) to improve long-context utilization in LLMs without requiring labeled data. The method uses the model's own predictions to generate pseudo-labels for fine-tuning on the test context, addressing accuracy degradation in long inputs. This approach is more efficient than full TTT and shows promise for practical deployment.
Entities
Related
Researchers identify asymmetric generalization problem in LLM unlearning benchmarks
A new arXiv paper argues that existing machine unlearning benchmarks for LLMs suffer from under-forgetting and over-forgetting due to an asymmetric generalization problem. The authors propose that evaluation must cover diverse query formulations of target facts to reliably measure knowledge removal while preserving unrelated capabilities.
Two papers propose token-adaptive KV cache compression for long-context LLMs
Two arXiv papers from July 7, 2026 introduce token-adaptive KV cache compression methods for long-context LLM inference. DepthWeave-KV factorizes key/value states across neighboring layers using shared low-rank bases with token-specific residuals. FreqDepthKV uses shared low-frequency depth components and sparse high-frequency residuals, with an online probe assigning attention heads to different cache modes. Both aim to reduce memory bandwidth while preserving retrieval and reasoning quality.
NL-PAC paper introduces framework for handling specification ambiguity in LLM supervision
A new arXiv paper proposes NL-PAC, a framework that addresses specification ambiguity when LLMs provide labels or evaluations from natural language instructions. The framework uses a fixed model's thresholded decoding law to define admissible labels and candidate targets, showing that additional labels reduce sampling error but cannot resolve identification problems from ambiguous specifications.
Super-Tuning: Activation-Aware Pruning Reused for Sparse Fine-Tuning
A new arxiv paper proposes Super, a sparse PEFT method that reuses Wanda-style activation-weighted magnitude scores from pruning to select a small trainable support, and Supra, a hybrid adapter combining sparse updates with LoRA. This approach reduces memory and compute for fine-tuning LLMs while maintaining performance.
Developer seeks feedback on fine-tuning LoRA for conversation state extraction in long LLM chats
A developer is working on a side project to improve AI conversation continuity by training a small model to extract structured conversation state from chat chunks, rather than relying on summarization. They are seeking feedback on their approach involving fine-tuning a LoRA, dataset design, and long-context systems.
