Study reveals text generation, not vision, is the energy bottleneck in edge VLM inference
A systematic energy profiling study of on-device Vision-Language Models (VLMs) across five models, four resolutions, and two hardware platforms (NVIDIA RTX 3070 and Jetson Orin NX) overturns the common assumption that visual processing dominates energy cost. The authors find that text generation is the true bottleneck, accounting for the majority of energy consumption during inference on edge devices.
Entities
Related
Study evaluates energy, performance, and accuracy trade-offs across vLLM configurations
A new arxiv paper presents a large-scale controlled study of three vLLM configuration options—attention kernel type, prefix caching, and chunked prefill—examining their impact on energy consumption, performance, and output quality. The work addresses a gap in understanding how inference engine configuration affects these trade-offs in production LLM deployments.
Survey on Green Development of Large Models: Resource-Efficient Architectures and Hardware-Software Co-Design
A comprehensive survey published on arXiv reviews strategies for reducing computational costs and energy consumption of large AI models, covering efficient architectures (attention optimization, linear-complexity models, sparsification) and full-stack hardware-software co-design. The paper provides a systematic overview of recent advances in green AI development.
STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
A new paper introduces STEEL, a sparsity-aware fused attention mechanism designed for AMD's XDNA NPU, enabling energy-efficient long-sequence inference on laptop-class SoCs. The approach addresses the challenge of mapping attention mechanisms onto NPUs while maintaining low power consumption, which is critical for agentic workloads that require on-device processing for reliability and privacy.
User tests Intel Arrow Lake iGPU with Llama.cpp: SYCL works at 12 tok/s, Vulkan fails
A user tested Intel's Arrow Lake iGPU with Llama.cpp for local LLM inference. Vulkan support was broken (1 tok/s), while SYCL achieved ~12 tok/s on Qwen3.6 35B models. CPU-only inference was more consistent at 14 tok/s, suggesting the iGPU offers no practical benefit.
Paper challenges text-only pretraining, proposes visual pretraining for language models
A new arXiv paper argues that current language model pretraining discards rich visual information from documents and web pages. The authors propose scalable visual pretraining to incorporate figures, equations, and layouts, aiming to improve language intelligence beyond text-only approaches.