MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation
Researchers introduce MedRealMM, a large-scale benchmark for multimodal online medical consultation built from de-identified patient-doctor interactions from a nationwide Chinese internet hospital. It addresses limitations of existing benchmarks that rely on synthetic data or omit patient-uploaded images. The benchmark aims to better align LLM evaluation with real clinical practice.
Entities
Related
LongMedBench: New Benchmark Tests Medical Agents on Long-Horizon Clinical Decisions
Researchers introduced LongMedBench, a benchmark using real EHR data from MIMIC-IV to evaluate LLM-based medical agents on long-horizon clinical decision-making. Unlike prior short-context QA benchmarks, LongMedBench requires agents to aggregate evidence across repeated visits, tests, and treatments over time. This addresses a key gap in realistic assessment of medical AI.
DataGovBench: New benchmark evaluates LLMs on real-world data analysis with large multi-tabular datasets
Researchers introduced DataGovBench, a benchmark derived from governmental open data to evaluate LLMs on practical data analysis tasks. It includes Table QA and Table Insight tasks, addressing limitations of existing benchmarks that focus on small tables and fact retrieval.
OmniMapBench benchmark tests LVLMs on visual-centric map reasoning
Researchers introduced OmniMapBench, a benchmark of 2,096 QA pairs across 1,603 map documents from nine categories, designed to evaluate visual-centric reasoning in large vision-language models. It addresses the limitation that many document benchmarks allow high performance via text-only cues, requiring genuine visual grounding for tasks from perception to multi-step reasoning.
Mixture of Probes: New method for multimodal LLMs to learn from privileged modalities
Researchers propose Mixture of Probes (MoP), a method enabling multimodal LLMs to leverage auxiliary modalities available only during training. This addresses real-world scenarios where inference-time modalities are limited, improving model performance without requiring all modalities at test time.
Community impressions of MELTing point paper on mobile LLM benchmarking
A Reddit user shared their reading of the MELTing point paper, which benchmarks LLMs on edge devices like phones with a true-to-user setup. The paper is praised for its realistic evaluation methodology. This highlights growing interest in on-device LLM performance.