Soofi S 30B-A3B: Open-source MoE hybrid Mamba Transformer for German and English
Researchers released Soofi S 30B-A3B, a sovereign open-source Mixture-of-Experts foundation model for German and English. Its hybrid Mamba-Transformer design activates only 3B of 30B parameters per token, achieving throughput advantages for long-context deployment. Pretrained on 27 trillion tokens with up-weighted German data, it matches dense 14-27B models on English and German benchmarks while excelling in code tasks.
Entities
Related
Tencent Hy Team releases Hy3, a 295B-parameter MoE model
Tencent's Hy Team released Hy3, a 295-billion-parameter Mixture-of-Experts (MoE) language model with 21B active parameters and a 3.8B MTP layer. The model outperforms similar-sized models and rivals open-source models with 2-5x parameters. It is available on Hugging Face in full (598GB) and FP8 quantized (300GB) versions.
Moondream releases 3.1-9B-A2B vision-language model with mixture-of-experts architecture
Moondream 3.1 is a vision language model with a mixture-of-experts architecture (9B total parameters, 2B active). It delivers state-of-the-art visual reasoning and detection while staying fast and cheap to deploy. Skills include query, detect, point, and caption, all native and all returning structured output.
Mach-Mind-4-Flash Technical Report Released: 35B MoE Model Matches 100B-Class Performance via Post-Training
A technical report on arxiv introduces Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts model with 3B activated parameters. Through post-training optimization alone, it achieves performance on par with or surpassing 100B-parameter-class models, enabled by scalable agentic interaction environments for large-scale reinforcement learning.
Community member extends Gemma 4 to 40.5B parameters via layer insertion
A Reddit user, TOTORONG, successfully extended Google's Gemma 4 31B model to 40.5B parameters by inserting additional layers, releasing the result as extGemma4-40_5B on Hugging Face. The project builds on previous attempts documented in earlier posts, overcoming challenges where new inserted layers initially degraded performance. This demonstrates community-driven model customization and fine-tuning techniques.
Flash-MSA: Sparse Attention Kernels Enable Million-Token Training
A new paper introduces Flash-MSA, a sparse attention kernel that reduces memory and computation for long-context transformers, enabling training on sequences up to one million tokens. The method achieves up to 8x speedup over FlashAttention-2 on 128K-length sequences while maintaining model quality.