Open-source 27B model on Groq achieves 85% on ExtractBench, frontier models score 0%
A 27B open-source model running on Groq scored 85% on a 369-field extraction task from SEC filings, while six frontier models scored 0%. The benchmark, ExtractBench (arXiv:2602.12247), uses real documents with human-checked answers.
Entities
Related
nfield: schema-splitting for large-scale structured LLM extraction
nfield is a method that splits a large JSON schema into smaller sub-schemas and makes separate LLM calls for each, then merges the results. It solves the problem of frontier models scoring 0% on 369-field SEC filing extraction by achieving 85% accuracy with a 27B open model.
Community crowdsources examples of open-weight model failures vs frontier models
A Hacker News user is collecting concrete task examples where open-weight models (GLM, DeepSeek, Kimi, Qwen) failed while frontier models (Opus, Fable, GPT) succeeded, or vice versa, to test the claim that open models 6 months behind the frontier are good enough for most work. The thread provides a structured template for reporting failures and successes.
OpenBenchmarks: open-source reproducible benchmarks for SaaS APIs
OpenBenchmarks provides open-source, reproducible benchmarks for SaaS APIs, starting with GTM APIs. It helps AI agents discover and evaluate the right APIs by providing standardized performance metrics, solving the problem of vendor selection in agentic workflows.
DataGovBench: New benchmark evaluates LLMs on real-world data analysis with large multi-tabular datasets
Researchers introduced DataGovBench, a benchmark derived from governmental open data to evaluate LLMs on practical data analysis tasks. It includes Table QA and Table Insight tasks, addressing limitations of existing benchmarks that focus on small tables and fact retrieval.
User benchmarks Fable 5, Sol, and xhigh models on strategic tasks
A user ran a role-based benchmark comparing Fable 5, Sol, and xhigh models on strategic decision memos, execution briefs, and bug repairs. Fable 5 scored 95 on a multi-layer productization decision, slightly ahead of Sol max (94) and xhigh (90). The benchmark is local and not a general intelligence test.