qMLX: Custom inference engine for Qwen 3.5 122B on Apple Silicon released
A developer released qMLX, a custom inference engine for Qwen 3.5 122B on Apple Silicon, extending MLX with hybrid attention support, SSD-backed KV cache, and RYS layer duplication. It enables efficient serving of the hybrid MoE model on consumer Mac hardware.
Entities
Related
Developer fixes 3 bugs in qMLX serving stack to make Qwen3.5-122B long-context inference usable on Mac Studio
A developer running Qwen3.5-122B on an M3 Ultra Mac Studio with 96GB RAM identified and fixed three bugs in the qMLX fork of rapid-mlx that caused 3-5 minute cold fills during long-context agentic coding. The fixes addressed prompt instability from a unique message ID breaking KV cache matching, an interrupt path that failed to persist streaming replies, and a third unspecified bug, making long-context inference practical.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
Reame: CPU-first LLM inference server built on llama.cpp released
Reame is a new LLM inference server designed to run efficiently on cheap CPU hardware, including shared vCPUs, free tiers, and 2-core ARM boxes. Built on llama.cpp, it features disk KV cache, self-regulating speculation, generation archive, and interleaved multi-user support. The project emphasizes treating CPU hardware as a first-class citizen rather than a fallback.
Local image-to-3D model for Apple Silicon and iPhone
A port of Hunyuan3D-Paint and Hunyuan3D-Shape to Swift MLX and Python MLX, enabling image-to-3D generation on Apple Silicon Macs and iPhones with low memory usage. It solves the problem of running 3D generation locally without cloud dependencies or high-end GPUs.
PrismML claims to run compressed 27B-parameter Qwen 3.6 model on iPhone 17 Pro
PrismML, a Khosla-backed startup, announced it has compressed Alibaba's open-source Qwen 3.6-27B model to run on an iPhone 17 Pro. This is claimed to be the largest-ever AI model deployed on a mobile device, as most on-device models have only a few billion active parameters. The achievement highlights advances in model compression for edge deployment.
