LLM/VLA models enable prompt-driven exploration in RL
A new research paper proposes using large language models (LLMs) and vision-language-action (VLA) models to drive exploration in reinforcement learning by modifying natural language prompts, which induce global behavioral changes beyond standard action noise. This approach could help policies escape weak local optima more effectively.
Entities
Related
VAORA reward design addresses VLM failures in interactive physical reasoning
A new paper on arXiv introduces VAORA (Visual Action Outcome Reasoning Alignment), a reward design that targets two key failure modes in vision-language models: hallucinated chain-of-thought reasoning and misalignment between reasoning and actions. VAORA uses a Visual Alignment Reward to anchor reasoning to visual context, aiming to improve generalization in unseen interactive physical reasoning tasks.
Paper proposes correlation-aware bandits with surrogate rewards for LLM routing
A new arXiv paper introduces contextual bandit algorithms that leverage surrogate reward signals from machine learning models to improve LLM routing decisions. The approach accounts for inter-arm correlations and noisy auxiliary rewards, addressing limitations of classical bandits that assume conditional independence.
CLAP paper proposes direct VLM-to-VLA adaptation to isolate semantic transfer
A new arXiv paper introduces CLAP, a method that converts pretrained vision-language models (VLMs) into vision-language-action models (VLAs) with minimal architectural changes, avoiding the distribution mismatch caused by predicting actions as bare numeric tokens. The approach aims to clarify how VLM capabilities transfer across model scales for robot control.
Research paper introduces NRFR metric to measure multimodal reward hacking in RL for MLLMs
A new arXiv paper studies reward hacking in reinforcement learning for multimodal large language models (MLLMs), showing that higher proxy rewards do not always mean better task performance, especially when visual evidence is evaluated by text-only or weakly grounded rewards. The authors propose a new metric, Newly Rewarded Failure Rate (NRFR), to measure failures among samples whose proxy reward improves over the SFT baseline. Experiments span safety VQA, chart VQA, and stress-test settings, varying reward design, data ambiguity, model scale (2B-32B), and RL algorithms (GRPO, RLOO, DAPO).
Mixture of Probes: New method for multimodal LLMs to learn from privileged modalities
Researchers propose Mixture of Probes (MoP), a method enabling multimodal LLMs to leverage auxiliary modalities available only during training. This addresses real-world scenarios where inference-time modalities are limited, improving model performance without requiring all modalities at test time.