OmniMapBench benchmark tests LVLMs on visual-centric map reasoning
Researchers introduced OmniMapBench, a benchmark of 2,096 QA pairs across 1,603 map documents from nine categories, designed to evaluate visual-centric reasoning in large vision-language models. It addresses the limitation that many document benchmarks allow high performance via text-only cues, requiring genuine visual grounding for tasks from perception to multi-step reasoning.
Entities
Related
MultiView-Bench: New Benchmark Tests VLMs' 3D Scene Integration from Multiple Views
Researchers introduced MultiView-Bench, a diagnostic benchmark to evaluate vision-language models' ability to integrate observations from multiple viewpoints into a coherent, world-centric 3D mental model. Unlike existing benchmarks focused on single-view perception or pixel-level mapping, MultiView-Bench requires models to decouple object positions from transient perspectives and ground them in a fixed global coordinate system. This addresses a core cognitive ability previously untested in VLMs.
DataGovBench: New benchmark evaluates LLMs on real-world data analysis with large multi-tabular datasets
Researchers introduced DataGovBench, a benchmark derived from governmental open data to evaluate LLMs on practical data analysis tasks. It includes Table QA and Table Insight tasks, addressing limitations of existing benchmarks that focus on small tables and fact retrieval.
CSB Dataset Reveals Decade-Long VLM Progress and Persistent Errors in Complex Social Scenes
Researchers introduced the Complex Social Behavior (CSB) dataset of 100 images depicting intricate social interactions, and analyzed vision-language models from 2017–2025. The study found significant accuracy improvements but also persistent visual-cognitive errors, highlighting that benchmarks like MS-COCO are insufficient for evaluating real-world social understanding.
WILDTRACE benchmark introduces natural evidence trails for long-context reasoning
Researchers propose WILDTRACE, a benchmark that tests long-context reasoning by requiring integration of evidence naturally dispersed across distant passages in real documents (e.g., incident reports, novels). Unlike existing benchmarks that rely on planted facts or needle probes, WILDTRACE uses source-internal evidence trails, making it more representative of real-world document analysis.
Paper challenges text-only pretraining, proposes visual pretraining for language models
A new arXiv paper argues that current language model pretraining discards rich visual information from documents and web pages. The authors propose scalable visual pretraining to incorporate figures, equations, and layouts, aiming to improve language intelligence beyond text-only approaches.