User tests Intel Arrow Lake iGPU with Llama.cpp: SYCL works at 12 tok/s, Vulkan fails
A user tested Intel's Arrow Lake iGPU with Llama.cpp for local LLM inference. Vulkan support was broken (1 tok/s), while SYCL achieved ~12 tok/s on Qwen3.6 35B models. CPU-only inference was more consistent at 14 tok/s, suggesting the iGPU offers no practical benefit.
Entities
Related
User tests LLMs on Intel 285HX CPU-only mini-PC, finds Llama-Swap incompatible with SYCL
A user set up a homelab server on an MS-02 mini-PC with an Intel Core Ultra 285HX and 64GB RAM, testing Qwen3, Qwen3.6, and Gemma4 via Llama.cpp. They found Llama-Swap's quick swapping helpful but incompatible with SYCL, complicating testing. The post shares early impressions of running LLMs on a CPU-only system.
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User benchmarks 4x RTX 5060 Ti with SGLang for Qwen3.6 27B, finds better concurrency than vLLM
A user shared benchmark results showing that SGLang handles higher concurrency better than vLLM when running Qwen3.6 27B (INT8 with bf16 KV cache) on a 4x RTX 5060 Ti (64GB VRAM) setup. The test achieved 200 successful requests at 8 concurrency over 348.87 seconds, processing 61,870 input tokens and generating 44,525 tokens. This provides a practical reference for others considering multi-GPU configurations with these consumer cards.
User benchmarks AMD EPYC 9374F for LLM inference, finds 48-thread sweet spot
A user replaced their EPYC 9135 with a cheap 9374F (8 CCDs) for LLM inference. Initial benchmarks showed no decoding advantage until they used 48 threads; 64 or 32 threads performed worse than the 9135 in some scenarios. The 9374F is worse for gaming.
Reame: CPU-first LLM inference server built on llama.cpp released
Reame is a new LLM inference server designed to run efficiently on cheap CPU hardware, including shared vCPUs, free tiers, and 2-core ARM boxes. Built on llama.cpp, it features disk KV cache, self-regulating speculation, generation archive, and interleaved multi-user support. The project emphasizes treating CPU hardware as a first-class citizen rather than a fallback.