arXiv paper benchmarks LLM judges for citation quality in deep-research systems
A new arXiv paper studies the calibration of LLM judges used as reward models in reinforcement learning for citation quality in deep-research systems. The work evaluates how capable and biased an LLM judge must be to reliably score rubric criteria like source relevance and factual support for attribution-citation pairs. This matters for practitioners building RL-based systems that depend on automated citation verification.
Entities
Related
Researcher seeks arXiv endorsement for multi-agent citation verification framework
A researcher is seeking an arXiv endorsement for a paper proposing a four-agent framework built on CrewAI that addresses hallucinated citations in LLM-generated literature reviews. The framework includes an Academic Retriever, Critical Reviewer, Technical Writer, and Editor/Verifier implementing claim-level citation verification.
LLM-as-judge eval misses 3% of users due to unseen output format
A developer recounts how their CPO mandated LLM eval automation using GPT-4o as a judge with an 8-dimension rubric. After three months of success, a system prompt tweak caused the judge to miss a completely different output format for 3% of users, leading to undetected regressions discovered via support tickets.
Research paper explains why reasoning AI models outperform faster, cheaper alternatives on factual accuracy
A quietly published research paper on ILLUMINATION’S MIRROR explains why slower, more deliberate AI models achieve higher factual accuracy compared to faster, cheaper alternatives. The paper provides insights into the trade-offs between speed and correctness in AI inference, highlighting that reasoning models can access knowledge that instant models cannot reach.
Community impressions of MELTing point paper on mobile LLM benchmarking
A Reddit user shared their reading of the MELTing point paper, which benchmarks LLMs on edge devices like phones with a true-to-user setup. The paper is praised for its realistic evaluation methodology. This highlights growing interest in on-device LLM performance.
Autoresearch, Claude and Constrained Optimization
A blog post by Elliot C. Smith explores using Anthropic's Claude for automated research, framing it as a constrained optimization problem. The post discusses how Claude can be guided to perform literature review, hypothesis generation, and experiment design within user-defined constraints, highlighting practical implications for AI-assisted scientific discovery.