Turboquant v0.3.0 fixes silent FP16 precision bug on Tesla P100 in llama.cpp
A three-line fix in turboquant v0.3.0 corrects a long-standing bug where llama.cpp's CUDA code forced FP16 math on Tesla P100 GPUs, despite the P100 having fast FP16 hardware. The fix restores correct precision and performance for P100 users running llama.cpp.
Entities
Related
User reports Qwen3.6 35B-A3B model improves with Q8_0 CPU quantization
A user on Reddit reported that switching from Q4_K_M on GPU to Q8_0 on CPU significantly improved the performance of the Qwen3.6 35B-A3B model for a complex coding task. The user noted the model 'punches far above its weight' and found the quality gain worth the slowdown.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User seeks advice on adding second cheap GPU to run larger local models
A Reddit user running Gemma 4 26B-A4B at 12-15 t/s on an RTX 3060 (12 GB) finds the model insufficiently intelligent and wants to upgrade to a 31B model, which runs at only 1.5 t/s. They ask the community about the benefits of adding a second cheap GPU to improve performance for local LLM inference.
Voodoo Quant claims 95% KLD improvement over Unsloth Dynamic 2.0 on Qwen3.5 models
A developer released two new GGUF quantizations of Qwen3.5 0.8B and 2B using a technique called Voodoo Quant, which optimizes mixed precision by assigning higher precision to more important parts of the model. The author claims Voodoo Quant beats Unsloth Dynamic 2.0 by 95% in Kullback-Leibler divergence (KLD). The quantized models are available on Hugging Face.
User tests LLMs on Intel 285HX CPU-only mini-PC, finds Llama-Swap incompatible with SYCL
A user set up a homelab server on an MS-02 mini-PC with an Intel Core Ultra 285HX and 64GB RAM, testing Qwen3, Qwen3.6, and Gemma4 via Llama.cpp. They found Llama-Swap's quick swapping helpful but incompatible with SYCL, complicating testing. The post shares early impressions of running LLMs on a CPU-only system.