Developer adds Nemotron Puzzle 75B support to mlx-lm, benchmarks 4-bit vs 5-bit quantization on M2 Max
A developer contributed native `nemotron_h_puzzle` support to mlx-lm (PR #1535) and ran benchmarks on a 64GB M2 Max. Comparing 4-bit vs 5-bit expert quantization (both with 6-bit dense layers, BF16 output head, group size 64), the 4-bit variant used 42.03 GiB checkpoint (49.68 GB peak memory) and achieved 14.27 tok/s, scoring 24/30 on local task checks, while 5-bit used 49.88 GiB (58.12 GB peak) at 10.53 tok/s scoring 21/30.
Entities
Related
User runs 100B+ MoE LLMs on low-end laptop using NVMe swap and Q3 quantization
A Reddit user with a low-spec laptop (i7-8750H, 20GB RAM, GTX 1050 4GB) reports successfully running 100B+ parameter MoE models by offloading parameters to a Samsung NVMe SSD via mmap, using Q3 quantization and quantized KV cache (Q4_0). They note that dense models are unusable on their hardware, but MoE models work with experts offloaded to CPU.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User tests GLM 5.2 on consumer hardware, finds performance comparable to Claude and GPT
A user tested GLM 5.2 on a standard computer and was impressed by its capabilities and security, finding them similar to Claude or GPT. They began converting the model to int4 and exploring MTP usage to avoid out-of-memory errors.
Developer compresses GLM-5.2 MoE to run on single RTX 3090 via 79 experiments
A developer conducted 79 experiments to compress GLM-5.2, a 337 GB MoE model with 75 sparse layers and 256 routed experts, to fit on a 24 GB RTX 3090. The approach uses per-expert codecs, a batch pipeline over all MoE layers, and a patched llama.cpp runtime that loads codec-native expert binaries at inference time. The MIT-licensed repository documents the method and findings on expert similarity.
SGLang v0.5.15 optimizes GLM-5.2 NVFP4 for Blackwell, achieves 500+ tok/s/user on 8x B300
SGLang released v0.5.15 with tuned GLM-5.2 NVFP4 inference on Blackwell GPUs, reaching 500+ tok/s/user on 8x B300 and 450 on 4x GB300 at batch size 1. The update also enables Spec V2 by default, a zero-overhead speculative decoding method that improves end-to-end throughput by 11% via CUDA-graphable draft-extend and fused metadata ops.