Paper

Flash-MSA: Sparse Attention Kernels Enable Million-Token Training

A new paper introduces Flash-MSA, a sparse attention kernel that reduces memory and computation for long-context transformers, enabling training on sequences up to one million tokens. The method achieves up to 8x speedup over FlashAttention-2 on 128K-length sequences while maintaining model quality.

10 engagement·1 source·Sun, Jul 12, 2026, 08:46 PM

The Flash-MSA paper proposes a sparse attention mechanism that combines a fixed sparse pattern with a learned gating mechanism to reduce the quadratic cost of attention. It builds on FlashAttention's tiling approach but introduces sparsity to skip irrelevant key-value pairs. Benchmarks show Flash-MSA achieves 2-8x speedup over FlashAttention-2 on sequences of 32K to 128K tokens, with memory savings proportional to sparsity ratio. The authors demonstrate training a 1.3B parameter model on sequences up to 1M tokens, achieving perplexity comparable to dense attention. The work is relevant for long-document modeling, code generation, and genomic sequence analysis.

Entities

Flash-MSA(tool)FlashAttention-2(tool)Nanduru Ganesh(person)

PaperTue, Jul 7, 2026, 05:26 PM

Two papers propose token-adaptive KV cache compression for long-context LLMs

Two arXiv papers from July 7, 2026 introduce token-adaptive KV cache compression methods for long-context LLM inference. DepthWeave-KV factorizes key/value states across neighboring layers using shared low-rank bases with token-specific residuals. FreqDepthKV uses shared low-frequency depth components and sparse high-frequency residuals, with an online probe assigning attention heads to different cache modes. Both aim to reduce memory bandwidth while preserving retrieval and reasoning quality.

0 engagement·2 sources·arxiv

arXiv

Model ReleaseSat, Jul 11, 2026, 03:52 PM

New model with 500K-token context and $2/$6 pricing shifts cost calculus

A model offering a 500,000-token context window at $2 per million input tokens and $6 per million output tokens has been released, drawing attention for its cost-effectiveness. The pricing and context length are seen as significant for applications requiring long-context processing, potentially changing the competitive landscape before benchmark comparisons are even made.

0 engagement·1 source·rss

RSS

Model ReleaseSun, Jul 12, 2026, 07:11 AM

Xiaomi quietly uploads MiMo-V2.5-DFlash weights to Hugging Face

Xiaomi has uploaded the official DFlash weights for MiMo-V2.5-DFlash to Hugging Face. The model has 300B+ parameters and runs at 8-10 tokens/second on 2x24GB cards with offloading; DFlash could double that speed. The MTP head was shared but does not work yet.

7 engagement·1 source·reddit

Tool ReleaseSun, Jul 12, 2026, 07:26 PM

Sparsify: run MoE models bigger than your RAM — Mixtral 26GB in 3.3GB RSS on a MacBook Air, byte-identical output

Sparsify, a new tool, enables running Mixtral 8x7B (26GB) on a MacBook Air with only 3.3GB RSS, producing byte-identical output. This is achieved by exploiting the sparsity of Mixture-of-Experts models, loading only the active experts into memory. The tool allows running models larger than available RAM without performance degradation.

0 engagement·1 source·reddit

PaperFri, Jul 10, 2026, 10:20 PM

Researcher tests minimal dynamical system for word embeddings without MLP or attention

A researcher proposes a word representation model using only token vectors, a start state, and two scalars, with no MLP, transformer, attention, or output matrix. The model achieves a SimLex-999 ρ of 0.3616 by updating a state vector via a cosine-based pull toward token attractors, encoding context through trajectory dynamics.