LongMedBench: New Benchmark Tests Medical Agents on Long-Horizon Clinical Decisions
Researchers introduced LongMedBench, a benchmark using real EHR data from MIMIC-IV to evaluate LLM-based medical agents on long-horizon clinical decision-making. Unlike prior short-context QA benchmarks, LongMedBench requires agents to aggregate evidence across repeated visits, tests, and treatments over time. This addresses a key gap in realistic assessment of medical AI.
Entities
Related
MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Researchers introduce MedRealMM, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions from a nationwide Chinese internet hospital. It addresses limitations of existing benchmarks that rely on synthetic data or omit patient-uploaded images. The benchmark aims to better align LLM evaluation with real clinical practice.
Long-Horizon-Terminal-Bench: New Benchmark Tests AI Agents on Long-Horizon Tasks with Dense Rewards
Researchers introduced Long-Horizon-Terminal-Bench, a benchmark of 46 long-horizon tasks across nine categories, including experiment reproduction and software engineering. Unlike existing benchmarks that evaluate only final outcomes, it uses dense reward signals to measure intermediate progress, providing a more complete picture of agent capability.
DataGovBench: New benchmark evaluates LLMs on real-world data analysis with large multi-tabular datasets
Researchers introduced DataGovBench, a benchmark derived from governmental open data to evaluate LLMs on practical data analysis tasks. It includes Table QA and Table Insight tasks, addressing limitations of existing benchmarks that focus on small tables and fact retrieval.
Question-type-specific LLM pipeline boosts BioASQ 14b biomedical QA
A new framework for BioASQ 14b Task B selects different inference procedures for yes/no, factoid, and list questions, improving answer robustness and evidence grounding. The approach uses question-type-specific prompting strategies rather than a single method for all queries.
UniClawBench benchmark proposed for evaluating proactive AI agents in real-world tasks
Researchers introduced UniClawBench, a universal benchmark for evaluating proactive agents that operate everyday tools in real-world environments. Unlike existing benchmarks that rely on sandboxed settings and single-turn evaluations, UniClawBench aims to isolate specific model capabilities to identify root causes of agent failures.