Developer ports Colibri streaming to Hy3, enabling 10GB VRAM inference
A developer named ErikTromp created a port of the Colibri streaming technique to work with the Hy3 model, reducing the VRAM requirement from 25GB (for GLM 5.2) to under 10GB. The tool is available on GitHub and allows running Hy3 on smaller hardware by using RAM instead of VRAM.
Entities
Related
Show HN: UATC-Closed-loop VRAM control and dynamic data pruning for LLM training
A GitHub project introduces UATC, a closed-loop VRAM control and dynamic data pruning system for LLM training. The tool aims to optimize memory usage and training efficiency by dynamically adjusting VRAM allocation and pruning data during training.
Developer compresses GLM-5.2 MoE to run on single RTX 3090 via 79 experiments
A developer conducted 79 experiments to compress GLM-5.2, a 337 GB MoE model with 75 sparse layers and 256 routed experts, to fit on a 24 GB RTX 3090. The approach uses per-expert codecs, a batch pipeline over all MoE layers, and a patched llama.cpp runtime that loads codec-native expert binaries at inference time. The MIT-licensed repository documents the method and findings on expert similarity.
User runs 100B+ MoE LLMs on low-end laptop using NVMe swap and Q3 quantization
A Reddit user with a low-spec laptop (i7-8750H, 20GB RAM, GTX 1050 4GB) reports successfully running 100B+ parameter MoE models by offloading parameters to a Samsung NVMe SSD via mmap, using Q3 quantization and quantized KV cache (Q4_0). They note that dense models are unusable on their hardware, but MoE models work with experts offloaded to CPU.
Community shares llama-server configs for 24GB GPUs
A Reddit thread collects proven llama-server startup configurations for 24GB VRAM GPUs (RTX 3090, 7900XTX, RTX 4090). Users are asked to share commands that maximize VRAM usage and provide at least 200,000 tokens KV cache, along with system RAM, OS, and CPU details.
User tests GLM 5.2 on consumer hardware, finds performance comparable to Claude and GPT
A user tested GLM 5.2 on a standard computer and was impressed by its capabilities and security, finding them similar to Claude or GPT. They began converting the model to int4 and exploring MTP usage to avoid out-of-memory errors.