Paper 'Agentic-SQL Revisited' Proposes Autonomy-Based Taxonomy for LLM Text-to-SQL
A new paper and official code repository introduce a taxonomy for LLM-based Text-to-SQL systems along an inference-autonomy axis, from constrained to reasoning-internalized. The work reframes evaluation as a leaderboard aggregation problem and provides an empirical benchmark analysis.
Entities
Related
DataGovBench: New benchmark evaluates LLMs on real-world data analysis with large multi-tabular datasets
Researchers introduced DataGovBench, a benchmark derived from governmental open data to evaluate LLMs on practical data analysis tasks. It includes Table QA and Table Insight tasks, addressing limitations of existing benchmarks that focus on small tables and fact retrieval.
Community compares local LLMs for agentic workflows using tool-eval-bench
A GitHub user published an interactive comparison report evaluating local LLMs for agentic workflows, using the tool-eval-bench benchmark (84 scenarios, 16 categories, 8 trials). The report targets single DGX Spark or other 96-128GB rigs and covers multi-turn tool orchestration, function calling, and autonomous planning as exercised by Hermes Agent.
Researchers identify asymmetric generalization problem in LLM unlearning benchmarks
A new arXiv paper argues that existing machine unlearning benchmarks for LLMs suffer from under-forgetting and over-forgetting due to an asymmetric generalization problem. The authors propose that evaluation must cover diverse query formulations of target facts to reliably measure knowledge removal while preserving unrelated capabilities.
NL-PAC paper introduces framework for handling specification ambiguity in LLM supervision
A new arXiv paper proposes NL-PAC, a framework that addresses specification ambiguity when LLMs provide labels or evaluations from natural language instructions. The framework uses a fixed model's thresholded decoding law to define admissible labels and candidate targets, showing that additional labels reduce sampling error but cannot resolve identification problems from ambiguous specifications.
Software engineer publishes final part of LLM-from-scratch series covering inference and decoding
A software engineer published the fourth and final part of a blog series explaining LLMs from the ground up, focusing on token-by-token generation, KV cache, and decoding strategies (temperature, top-k, top-p). The series aims to help other software engineers understand the internals of LLMs.