User benchmarks 4x RTX 5060 Ti with SGLang for Qwen3.6 27B, finds better concurrency than vLLM
A user shared benchmark results showing that SGLang handles higher concurrency better than vLLM when running Qwen3.6 27B (INT8 with bf16 KV cache) on a 4x RTX 5060 Ti (64GB VRAM) setup. The test achieved 200 successful requests at 8 concurrency over 348.87 seconds, processing 61,870 input tokens and generating 44,525 tokens. This provides a practical reference for others considering multi-GPU configurations with these consumer cards.
Entities
Related
User benchmarks show 2x throughput gain from parallel agents on RTX 5090 with Qwen 3.6 35B
A Reddit user benchmarked LM Studio with Qwen 3.6 35B on an RTX 5090, finding that running at least 4 parallel agents doubles tokens/second compared to a single agent. The test with 8 configured parallel tasks showed significant throughput improvement, suggesting developers using agent frameworks like Open Code should parallelize to maximize performance.
SGLang v0.5.15 optimizes GLM-5.2 NVFP4 for Blackwell, achieves 500+ tok/s/user on 8x B300
SGLang released v0.5.15 with tuned GLM-5.2 NVFP4 inference on Blackwell GPUs, reaching 500+ tok/s/user on 8x B300 and 450 on 4x GB300 at batch size 1. The update also enables Spec V2 by default, a zero-overhead speculative decoding method that improves end-to-end throughput by 11% via CUDA-graphable draft-extend and fused metadata ops.
User seeks advice on adding second cheap GPU to run larger local models
A Reddit user running Gemma 4 26B-A4B at 12-15 t/s on an RTX 3060 (12 GB) finds the model insufficiently intelligent and wants to upgrade to a 31B model, which runs at only 1.5 t/s. They ask the community about the benefits of adding a second cheap GPU to improve performance for local LLM inference.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User tests LLMs on Intel 285HX CPU-only mini-PC, finds Llama-Swap incompatible with SYCL
A user set up a homelab server on an MS-02 mini-PC with an Intel Core Ultra 285HX and 64GB RAM, testing Qwen3, Qwen3.6, and Gemma4 via Llama.cpp. They found Llama-Swap's quick swapping helpful but incompatible with SYCL, complicating testing. The post shares early impressions of running LLMs on a CPU-only system.