SGLang v0.5.15 optimizes GLM-5.2 NVFP4 for Blackwell, achieves 500+ tok/s/user on 8x B300
SGLang released v0.5.15 with tuned GLM-5.2 NVFP4 inference on Blackwell GPUs, reaching 500+ tok/s/user on 8x B300 and 450 on 4x GB300 at batch size 1. The update also enables Spec V2 by default, a zero-overhead speculative decoding method that improves end-to-end throughput by 11% via CUDA-graphable draft-extend and fused metadata ops.
Entities
Related
User benchmarks 4x RTX 5060 Ti with SGLang for Qwen3.6 27B, finds better concurrency than vLLM
A user shared benchmark results showing that SGLang handles higher concurrency better than vLLM when running Qwen3.6 27B (INT8 with bf16 KV cache) on a 4x RTX 5060 Ti (64GB VRAM) setup. The test achieved 200 successful requests at 8 concurrency over 348.87 seconds, processing 61,870 input tokens and generating 44,525 tokens. This provides a practical reference for others considering multi-GPU configurations with these consumer cards.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User tests GLM 5.2 on consumer hardware, finds performance comparable to Claude and GPT
A user tested GLM 5.2 on a standard computer and was impressed by its capabilities and security, finding them similar to Claude or GPT. They began converting the model to int4 and exploring MTP usage to avoid out-of-memory errors.
Developer adds Nemotron Puzzle 75B support to mlx-lm, benchmarks 4-bit vs 5-bit quantization on M2 Max
A developer contributed native `nemotron_h_puzzle` support to mlx-lm (PR #1535) and ran benchmarks on a 64GB M2 Max. Comparing 4-bit vs 5-bit expert quantization (both with 6-bit dense layers, BF16 output head, group size 64), the 4-bit variant used 42.03 GiB checkpoint (49.68 GB peak memory) and achieved 14.27 tok/s, scoring 24/30 on local task checks, while 5-bit used 49.88 GiB (58.12 GB peak) at 10.53 tok/s scoring 21/30.
User seeks advice on adding second cheap GPU to run larger local models
A Reddit user running Gemma 4 26B-A4B at 12-15 t/s on an RTX 3060 (12 GB) finds the model insufficiently intelligent and wants to upgrade to a 31B model, which runs at only 1.5 t/s. They ask the community about the benefits of adding a second cheap GPU to improve performance for local LLM inference.