Research paper introduces NRFR metric to measure multimodal reward hacking in RL for MLLMs
A new arXiv paper studies reward hacking in reinforcement learning for multimodal large language models (MLLMs), showing that higher proxy rewards do not always mean better task performance, especially when visual evidence is evaluated by text-only or weakly grounded rewards. The authors propose a new metric, Newly Rewarded Failure Rate (NRFR), to measure failures among samples whose proxy reward improves over the SFT baseline. Experiments span safety VQA, chart VQA, and stress-test settings, varying reward design, data ambiguity, model scale (2B-32B), and RL algorithms (GRPO, RLOO, DAPO).
Entities
Related
LLM/VLA models enable prompt-driven exploration in RL
A new research paper proposes using large language models (LLMs) and vision-language-action (VLA) models to drive exploration in reinforcement learning by modifying natural language prompts, which induce global behavioral changes beyond standard action noise. This approach could help policies escape weak local optima more effectively.
VAORA reward design addresses VLM failures in interactive physical reasoning
A new paper on arXiv introduces VAORA (Visual Action Outcome Reasoning Alignment), a reward design that targets two key failure modes in vision-language models: hallucinated chain-of-thought reasoning and misalignment between reasoning and actions. VAORA uses a Visual Alignment Reward to anchor reasoning to visual context, aiming to improve generalization in unseen interactive physical reasoning tasks.
Paper proposes correlation-aware bandits with surrogate rewards for LLM routing
A new arXiv paper introduces contextual bandit algorithms that leverage surrogate reward signals from machine learning models to improve LLM routing decisions. The approach accounts for inter-arm correlations and noisy auxiliary rewards, addressing limitations of classical bandits that assume conditional independence.
arXiv paper benchmarks LLM judges for citation quality in deep-research systems
A new arXiv paper studies the calibration of LLM judges used as reward models in reinforcement learning for citation quality in deep-research systems. The work evaluates how capable and biased an LLM judge must be to reliably score rubric criteria like source relevance and factual support for attribution-citation pairs. This matters for practitioners building RL-based systems that depend on automated citation verification.
Mixture of Probes: New method for multimodal LLMs to learn from privileged modalities
Researchers propose Mixture of Probes (MoP), a method enabling multimodal LLMs to leverage auxiliary modalities available only during training. This addresses real-world scenarios where inference-time modalities are limited, improving model performance without requiring all modalities at test time.