Developer proposes STT torture test benchmark on GitHub
A developer drafted a speech-to-text benchmark that goes beyond clean audio and single WER scores, proposing seven challenging scenarios including phone calls from moving cars, speaker interruptions, code-switching, and timestamp drift. The test set is intended as a public GitHub repo to stress-test STT systems with ugly real-world clips.
Entities
Related
Paper identifies word-level control bottleneck in LLM-based TTS
A new paper highlights the inability of current LLM-based TTS systems to explicitly manipulate word-level acoustic attributes, limiting precise stylistic control and temporal alignment for applications like audiobook narration and video dubbing. The authors attribute this to scarcity of fine-grained annotated datasets and architectural challenges.
Meta releases SWE-Together benchmark measuring coding agent steering difficulty
Meta introduced SWE-Together, a new benchmark that evaluates coding agents on interactive, multi-turn tasks rather than single-shot problem solving. The benchmark measures how much human steering an agent requires, which correlates strongly with its capability. This addresses a key limitation of SWE-bench, which tests agents on frozen tickets alone.
OpenAI analysis reveals flaws in SWE-Bench Pro coding benchmark
OpenAI published an analysis uncovering reliability issues in SWE-Bench Pro, a popular benchmark for evaluating AI coding models. The findings raise concerns about the accuracy of benchmark scores, potentially affecting how developers and researchers trust model evaluations.
UniClawBench benchmark proposed for evaluating proactive AI agents in real-world tasks
Researchers introduced UniClawBench, a universal benchmark for evaluating proactive agents that operate everyday tools in real-world environments. Unlike existing benchmarks that rely on sandboxed settings and single-turn evaluations, UniClawBench aims to isolate specific model capabilities to identify root causes of agent failures.
Developer seeks feedback on fine-tuning LoRA for conversation state extraction in long LLM chats
A developer is working on a side project to improve AI conversation continuity by training a small model to extract structured conversation state from chat chunks, rather than relying on summarization. They are seeking feedback on their approach involving fine-tuning a LoRA, dataset design, and long-context systems.
