User tests LLMs on Intel 285HX CPU-only mini-PC, finds Llama-Swap incompatible with SYCL
A user set up a homelab server on an MS-02 mini-PC with an Intel Core Ultra 285HX and 64GB RAM, testing Qwen3, Qwen3.6, and Gemma4 via Llama.cpp. They found Llama-Swap's quick swapping helpful but incompatible with SYCL, complicating testing. The post shares early impressions of running LLMs on a CPU-only system.
Entities
Related
Reame: CPU-first LLM inference server built on llama.cpp released
Reame is a new LLM inference server designed to run efficiently on cheap CPU hardware, including shared vCPUs, free tiers, and 2-core ARM boxes. Built on llama.cpp, it features disk KV cache, self-regulating speculation, generation archive, and interleaved multi-user support. The project emphasizes treating CPU hardware as a first-class citizen rather than a fallback.
User runs 100B+ MoE LLMs on low-end laptop using NVMe swap and Q3 quantization
A Reddit user with a low-spec laptop (i7-8750H, 20GB RAM, GTX 1050 4GB) reports successfully running 100B+ parameter MoE models by offloading parameters to a Samsung NVMe SSD via mmap, using Q3 quantization and quantized KV cache (Q4_0). They note that dense models are unusable on their hardware, but MoE models work with experts offloaded to CPU.
User benchmarks AMD EPYC 9374F for LLM inference, finds 48-thread sweet spot
A user replaced their EPYC 9135 with a cheap 9374F (8 CCDs) for LLM inference. Initial benchmarks showed no decoding advantage until they used 48 threads; 64 or 32 threads performed worse than the 9135 in some scenarios. The 9374F is worse for gaming.
User seeks advice on adding second cheap GPU to run larger local models
A Reddit user running Gemma 4 26B-A4B at 12-15 t/s on an RTX 3060 (12 GB) finds the model insufficiently intelligent and wants to upgrade to a 31B model, which runs at only 1.5 t/s. They ask the community about the benefits of adding a second cheap GPU to improve performance for local LLM inference.
User benchmarks 4x RTX 5060 Ti with SGLang for Qwen3.6 27B, finds better concurrency than vLLM
A user shared benchmark results showing that SGLang handles higher concurrency better than vLLM when running Qwen3.6 27B (INT8 with bf16 KV cache) on a 4x RTX 5060 Ti (64GB VRAM) setup. The test achieved 200 successful requests at 8 concurrency over 348.87 seconds, processing 61,870 input tokens and generating 44,525 tokens. This provides a practical reference for others considering multi-GPU configurations with these consumer cards.