Developer compresses GLM-5.2 MoE to run on single RTX 3090 via 79 experiments
A developer conducted 79 experiments to compress GLM-5.2, a 337 GB MoE model with 75 sparse layers and 256 routed experts, to fit on a 24 GB RTX 3090. The approach uses per-expert codecs, a batch pipeline over all MoE layers, and a patched llama.cpp runtime that loads codec-native expert binaries at inference time. The MIT-licensed repository documents the method and findings on expert similarity.
Entities
Related
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
User runs 100B+ MoE LLMs on low-end laptop using NVMe swap and Q3 quantization
A Reddit user with a low-spec laptop (i7-8750H, 20GB RAM, GTX 1050 4GB) reports successfully running 100B+ parameter MoE models by offloading parameters to a Samsung NVMe SSD via mmap, using Q3 quantization and quantized KV cache (Q4_0). They note that dense models are unusable on their hardware, but MoE models work with experts offloaded to CPU.
User tests GLM 5.2 on consumer hardware, finds performance comparable to Claude and GPT
A user tested GLM 5.2 on a standard computer and was impressed by its capabilities and security, finding them similar to Claude or GPT. They began converting the model to int4 and exploring MTP usage to avoid out-of-memory errors.
User reports issues loading large MoE models after adding second RTX 3060
A user with dual RTX 3060 GPUs found that large MoE models like Qwen3.5-122B-A10B fail to load, while dense models work. After troubleshooting, they discovered that limiting VRAM usage to 12 layers allows the 122B model to load, and a PC restart plus GPU priority adjustment fixed the 35B model.
User seeks advice on adding second cheap GPU to run larger local models
A Reddit user running Gemma 4 26B-A4B at 12-15 t/s on an RTX 3060 (12 GB) finds the model insufficiently intelligent and wants to upgrade to a 31B model, which runs at only 1.5 t/s. They ask the community about the benefits of adding a second cheap GPU to improve performance for local LLM inference.