Long-Horizon-Terminal-Bench: New Benchmark Tests AI Agents on Long-Horizon Tasks with Dense Rewards
Researchers introduced Long-Horizon-Terminal-Bench, a benchmark of 46 long-horizon tasks across nine categories, including experiment reproduction and software engineering. Unlike existing benchmarks that evaluate only final outcomes, it uses dense reward signals to measure intermediate progress, providing a more complete picture of agent capability.
Entities
Related
UniClawBench benchmark proposed for evaluating proactive AI agents in real-world tasks
Researchers introduced UniClawBench, a universal benchmark for evaluating proactive agents that operate everyday tools in real-world environments. Unlike existing benchmarks that rely on sandboxed settings and single-turn evaluations, UniClawBench aims to isolate specific model capabilities to identify root causes of agent failures.
Meta releases SWE-Together benchmark measuring coding agent steering difficulty
Meta introduced SWE-Together, a new benchmark that evaluates coding agents on interactive, multi-turn tasks rather than single-shot problem solving. The benchmark measures how much human steering an agent requires, which correlates strongly with its capability. This addresses a key limitation of SWE-bench, which tests agents on frozen tickets alone.
LongMedBench: New Benchmark Tests Medical Agents on Long-Horizon Clinical Decisions
Researchers introduced LongMedBench, a benchmark using real EHR data from MIMIC-IV to evaluate LLM-based medical agents on long-horizon clinical decision-making. Unlike prior short-context QA benchmarks, LongMedBench requires agents to aggregate evidence across repeated visits, tests, and treatments over time. This addresses a key gap in realistic assessment of medical AI.
STS2-Bench tests LLM long-horizon decision-making in Slay the Spire 2
A developer created STS2-Bench, a benchmark using Slay the Spire 2 to evaluate LLMs on sequential decision-making under uncertainty. The benchmark tests models on reading changing game states, weighing short-term vs long-term goals, and adapting plans. Early results show 5.6Sol performing surprisingly well.
Researchers propose memory agent to combat behavioral state decay in long-horizon tasks
A new arXiv paper identifies 'behavioral state decay' as a failure mode where decision-relevant information gets buried in long trajectories. The authors propose a separate memory agent that actively updates a structured memory bank alongside an unmodified action agent, rather than relying on passive retrieval.