STS2-Bench tests LLM long-horizon decision-making in Slay the Spire 2
A developer created STS2-Bench, a benchmark using Slay the Spire 2 to evaluate LLMs on sequential decision-making under uncertainty. The benchmark tests models on reading changing game states, weighing short-term vs long-term goals, and adapting plans. Early results show 5.6Sol performing surprisingly well.
Entities
Related
Community compares local LLMs for agentic workflows using tool-eval-bench
A GitHub user published an interactive comparison report evaluating local LLMs for agentic workflows, using the tool-eval-bench benchmark (84 scenarios, 16 categories, 8 trials). The report targets single DGX Spark or other 96-128GB rigs and covers multi-turn tool orchestration, function calling, and autonomous planning as exercised by Hermes Agent.
User benchmarks Fable 5, Sol, and xhigh models on strategic tasks
A user ran a role-based benchmark comparing Fable 5, Sol, and xhigh models on strategic decision memos, execution briefs, and bug repairs. Fable 5 scored 95 on a multi-layer productization decision, slightly ahead of Sol max (94) and xhigh (90). The benchmark is local and not a general intelligence test.
LLM comparison dashboard for quality, latency, and cost
A dashboard that lets users test LLMs on their own data, comparing quality, latency, and cost side by side. It runs on Nebius Serverless and helps developers choose the best model for their specific use case rather than relying on leaderboards.
UniClawBench benchmark proposed for evaluating proactive AI agents in real-world tasks
Researchers introduced UniClawBench, a universal benchmark for evaluating proactive agents that operate everyday tools in real-world environments. Unlike existing benchmarks that rely on sandboxed settings and single-turn evaluations, UniClawBench aims to isolate specific model capabilities to identify root causes of agent failures.
Machine Beater: 5-question head-to-head guessing game against LLMs
A game where a human and an LLM each ask 5 yes/no questions to guess a hidden answer, inspired by 20 Questions. The goal is to benchmark reasoning skills not captured by standard benchmarks, with planned model-model and human-model matchups.
