CSB Dataset Reveals Decade-Long VLM Progress and Persistent Errors in Complex Social Scenes
Researchers introduced the Complex Social Behavior (CSB) dataset of 100 images depicting intricate social interactions, and analyzed vision-language models from 2017–2025. The study found significant accuracy improvements but also persistent visual-cognitive errors, highlighting that benchmarks like MS-COCO are insufficient for evaluating real-world social understanding.
Entities
Related
VLMs encode correct object counts internally but output wrong answers, study finds
A new arXiv paper investigates why vision-language models (VLMs) fail at basic object counting. By training probes on internal activations across four VLMs and five counting datasets, researchers found that nonlinear probes can reliably detect counting errors, indicating that models often encode the correct count even when they output the wrong answer. SVCCA analysis further confirms a misalignment between internal representations and verbalized outputs.
MultiView-Bench: New Benchmark Tests VLMs' 3D Scene Integration from Multiple Views
Researchers introduced MultiView-Bench, a diagnostic benchmark to evaluate vision-language models' ability to integrate observations from multiple viewpoints into a coherent, world-centric 3D mental model. Unlike existing benchmarks focused on single-view perception or pixel-level mapping, MultiView-Bench requires models to decouple object positions from transient perspectives and ground them in a fixed global coordinate system. This addresses a core cognitive ability previously untested in VLMs.
OmniMapBench benchmark tests LVLMs on visual-centric map reasoning
Researchers introduced OmniMapBench, a benchmark of 2,096 QA pairs across 1,603 map documents from nine categories, designed to evaluate visual-centric reasoning in large vision-language models. It addresses the limitation that many document benchmarks allow high performance via text-only cues, requiring genuine visual grounding for tasks from perception to multi-step reasoning.
VGGT Implicitly Encodes Co-Visibility as Emergent Behavior in Its Internal Representations
A new paper probes VGGT, a geometric foundation model, and finds it implicitly encodes co-visibility—determining which image pairs share overlapping surfaces—without any supervision. The model's internal representations exhibit a hierarchical structure similar to LLMs, with early layers building 3D-aware scene representations and later layers acting as dedicated co-visibility reasoners, specifically layer L17 identified as a negative indicator.
OpenAI analysis reveals flaws in SWE-Bench Pro coding benchmark
OpenAI published an analysis uncovering reliability issues in SWE-Bench Pro, a popular benchmark for evaluating AI coding models. The findings raise concerns about the accuracy of benchmark scores, potentially affecting how developers and researchers trust model evaluations.