Mach-Mind-4-Flash Technical Report Released: 35B MoE Model Matches 100B-Class Performance via Post-Training
A technical report on arxiv introduces Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts model with 3B activated parameters. Through post-training optimization alone, it achieves performance on par with or surpassing 100B-parameter-class models, enabled by scalable agentic interaction environments for large-scale reinforcement learning.
Entities
Related
Flash-MSA: Sparse Attention Kernels Enable Million-Token Training
A new paper introduces Flash-MSA, a sparse attention kernel that reduces memory and computation for long-context transformers, enabling training on sequences up to one million tokens. The method achieves up to 8x speedup over FlashAttention-2 on 128K-length sequences while maintaining model quality.
Super-Tuning: Activation-Aware Pruning Reused for Sparse Fine-Tuning
A new arxiv paper proposes Super, a sparse PEFT method that reuses Wanda-style activation-weighted magnitude scores from pruning to select a small trainable support, and Supra, a hybrid adapter combining sparse updates with LoRA. This approach reduces memory and compute for fine-tuning LLMs while maintaining performance.
Tencent Hy Team releases Hy3, a 295B-parameter MoE model
Tencent's Hy Team released Hy3, a 295-billion-parameter Mixture-of-Experts (MoE) language model with 21B active parameters and a 3.8B MTP layer. The model outperforms similar-sized models and rivals open-source models with 2-5x parameters. It is available on Hugging Face in full (598GB) and FP8 quantized (300GB) versions.
Soofi S 30B-A3B: Open-source MoE hybrid Mamba Transformer for German and English
Researchers released Soofi S 30B-A3B, a sovereign open-source Mixture-of-Experts foundation model for German and English. Its hybrid Mamba-Transformer design activates only 3B of 30B parameters per token, achieving throughput advantages for long-context deployment. Pretrained on 27 trillion tokens with up-weighted German data, it matches dense 14-27B models on English and German benchmarks while excelling in code tasks.
Developer compresses GLM-5.2 MoE to run on single RTX 3090 via 79 experiments
A developer conducted 79 experiments to compress GLM-5.2, a 337 GB MoE model with 75 sparse layers and 256 routed experts, to fit on a 24 GB RTX 3090. The approach uses per-expert codecs, a batch pipeline over all MoE layers, and a patched llama.cpp runtime that loads codec-native expert binaries at inference time. The MIT-licensed repository documents the method and findings on expert similarity.