WILDTRACE benchmark introduces natural evidence trails for long-context reasoning
Researchers propose WILDTRACE, a benchmark that tests long-context reasoning by requiring integration of evidence naturally dispersed across distant passages in real documents (e.g., incident reports, novels). Unlike existing benchmarks that rely on planted facts or needle probes, WILDTRACE uses source-internal evidence trails, making it more representative of real-world document analysis.
Entities
Related
arXiv preprint introduces IdeaGene-Bench for scientific lineage reasoning
A new benchmark, IdeaGene-Bench (IG-Bench), evaluates AI systems on scientific lineage reasoning and idea generation grounded in prior work. It frames scientific ideas as inheriting mechanisms and recombining earlier pieces, akin to biological genomes.
LongMedBench: New Benchmark Tests Medical Agents on Long-Horizon Clinical Decisions
Researchers introduced LongMedBench, a benchmark using real EHR data from MIMIC-IV to evaluate LLM-based medical agents on long-horizon clinical decision-making. Unlike prior short-context QA benchmarks, LongMedBench requires agents to aggregate evidence across repeated visits, tests, and treatments over time. This addresses a key gap in realistic assessment of medical AI.
OmniMapBench benchmark tests LVLMs on visual-centric map reasoning
Researchers introduced OmniMapBench, a benchmark of 2,096 QA pairs across 1,603 map documents from nine categories, designed to evaluate visual-centric reasoning in large vision-language models. It addresses the limitation that many document benchmarks allow high performance via text-only cues, requiring genuine visual grounding for tasks from perception to multi-step reasoning.
Researchers identify asymmetric generalization problem in LLM unlearning benchmarks
A new arXiv paper argues that existing machine unlearning benchmarks for LLMs suffer from under-forgetting and over-forgetting due to an asymmetric generalization problem. The authors propose that evaluation must cover diverse query formulations of target facts to reliably measure knowledge removal while preserving unrelated capabilities.
UniClawBench benchmark proposed for evaluating proactive AI agents in real-world tasks
Researchers introduced UniClawBench, a universal benchmark for evaluating proactive agents that operate everyday tools in real-world environments. Unlike existing benchmarks that rely on sandboxed settings and single-turn evaluations, UniClawBench aims to isolate specific model capabilities to identify root causes of agent failures.