User runs 100B+ MoE LLMs on low-end laptop using NVMe swap and Q3 quantization
A Reddit user with a low-spec laptop (i7-8750H, 20GB RAM, GTX 1050 4GB) reports successfully running 100B+ parameter MoE models by offloading parameters to a Samsung NVMe SSD via mmap, using Q3 quantization and quantized KV cache (Q4_0). They note that dense models are unusable on their hardware, but MoE models work with experts offloaded to CPU.
Entities
Related
User tests LLMs on Intel 285HX CPU-only mini-PC, finds Llama-Swap incompatible with SYCL
A user set up a homelab server on an MS-02 mini-PC with an Intel Core Ultra 285HX and 64GB RAM, testing Qwen3, Qwen3.6, and Gemma4 via Llama.cpp. They found Llama-Swap's quick swapping helpful but incompatible with SYCL, complicating testing. The post shares early impressions of running LLMs on a CPU-only system.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User seeks advice on adding second cheap GPU to run larger local models
A Reddit user running Gemma 4 26B-A4B at 12-15 t/s on an RTX 3060 (12 GB) finds the model insufficiently intelligent and wants to upgrade to a 31B model, which runs at only 1.5 t/s. They ask the community about the benefits of adding a second cheap GPU to improve performance for local LLM inference.
User reports Qwen3.6 35B-A3B model improves with Q8_0 CPU quantization
A user on Reddit reported that switching from Q4_K_M on GPU to Q8_0 on CPU significantly improved the performance of the Qwen3.6 35B-A3B model for a complex coding task. The user noted the model 'punches far above its weight' and found the quality gain worth the slowdown.
User benchmarks AMD EPYC 9374F for LLM inference, finds 48-thread sweet spot
A user replaced their EPYC 9135 with a cheap 9374F (8 CCDs) for LLM inference. Initial benchmarks showed no decoding advantage until they used 48 threads; 64 or 32 threads performed worse than the 9135 in some scenarios. The 9374F is worse for gaming.