Meta releases SWE-Together benchmark measuring coding agent steering difficulty
Meta introduced SWE-Together, a new benchmark that evaluates coding agents on interactive, multi-turn tasks rather than single-shot problem solving. The benchmark measures how much human steering an agent requires, which correlates strongly with its capability. This addresses a key limitation of SWE-bench, which tests agents on frozen tickets alone.
Entities
Related
UniClawBench benchmark proposed for evaluating proactive AI agents in real-world tasks
Researchers introduced UniClawBench, a universal benchmark for evaluating proactive agents that operate everyday tools in real-world environments. Unlike existing benchmarks that rely on sandboxed settings and single-turn evaluations, UniClawBench aims to isolate specific model capabilities to identify root causes of agent failures.
OpenAI analysis reveals flaws in SWE-Bench Pro coding benchmark
OpenAI published an analysis uncovering reliability issues in SWE-Bench Pro, a popular benchmark for evaluating AI coding models. The findings raise concerns about the accuracy of benchmark scores, potentially affecting how developers and researchers trust model evaluations.
Developer asks community for agent evaluation practices, cites silent breakage
A developer building AI agents reports that prompt or MCP changes often break silently despite passing manual tests. They ask the community about evaluation methods, including fixed test cases, skill-level vs. end-to-end checks, and tools like DeepEval, LangSmith, and Ragas.
Cognitive Core Skills taxonomy released with 159 skill cards, benchmarks, and CI
A universal, industry-neutral taxonomy of cognitive core skills for LLMs, SLMs, AI agents, and world models has been released on GitHub. It includes schemas, 159 skill cards, benchmarks, and continuous integration, covering perception, memory, reasoning, planning, action, verification, learning, and governance.
Developer proposes STT torture test benchmark on GitHub
A developer drafted a speech-to-text benchmark that goes beyond clean audio and single WER scores, proposing seven challenging scenarios including phone calls from moving cars, speaker interruptions, code-switching, and timestamp drift. The test set is intended as a public GitHub repo to stress-test STT systems with ugly real-world clips.