Study evaluates energy, performance, and accuracy trade-offs across vLLM configurations
A new arxiv paper presents a large-scale controlled study of three vLLM configuration options—attention kernel type, prefix caching, and chunked prefill—examining their impact on energy consumption, performance, and output quality. The work addresses a gap in understanding how inference engine configuration affects these trade-offs in production LLM deployments.
Entities
Related
Study reveals text generation, not vision, is the energy bottleneck in edge VLM inference
A systematic energy profiling study of on-device Vision-Language Models (VLMs) across five models, four resolutions, and two hardware platforms (NVIDIA RTX 3070 and Jetson Orin NX) overturns the common assumption that visual processing dominates energy cost. The authors find that text generation is the true bottleneck, accounting for the majority of energy consumption during inference on edge devices.
Two papers propose token-adaptive KV cache compression for long-context LLMs
Two arXiv papers from July 7, 2026 introduce token-adaptive KV cache compression methods for long-context LLM inference. DepthWeave-KV factorizes key/value states across neighboring layers using shared low-rank bases with token-specific residuals. FreqDepthKV uses shared low-frequency depth components and sparse high-frequency residuals, with an online probe assigning attention heads to different cache modes. Both aim to reduce memory bandwidth while preserving retrieval and reasoning quality.
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
BlockServe introduces block-grained continuous batching to address convergence heterogeneity in diffusion LLM serving. By evicting completed requests at block boundaries and using mixed-state execution with dual cache and parallel decoding, it reduces compute bubbles and tail latency.
Software engineer publishes final part of LLM-from-scratch series covering inference and decoding
A software engineer published the fourth and final part of a blog series explaining LLMs from the ground up, focusing on token-by-token generation, KV cache, and decoding strategies (temperature, top-k, top-p). The series aims to help other software engineers understand the internals of LLMs.
Survey on Green Development of Large Models: Resource-Efficient Architectures and Hardware-Software Co-Design
A comprehensive survey published on arXiv reviews strategies for reducing computational costs and energy consumption of large AI models, covering efficient architectures (attention optimization, linear-complexity models, sparsification) and full-stack hardware-software co-design. The paper provides a systematic overview of recent advances in green AI development.