生成时间: 2026-07-31 18:41:19 (UTC+8); Arxiv 发布时间: 2026-07-31 20:00 EDT (2026-08-01 08:00 UTC+8)

今天共有 42 篇相关文章

Keyword: reinforcement learning

RLPF: Reinforcement Learning from Performance Feedback for Code Generation

RLPF：通过性能反馈进行代码生成的强化学习

• Authors: Huihao Jing, Haozhe Cui, Wenbin Hu, Shaojin Chen, Haochen Shi, Changxuan Fan, Yuxuan Liu, Hanyu Yang, Sirui Zhang, Ziyi Chen, Haoran Li, Yangqiu Song
• Subjects: Subjects: Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.27271
• Pdf link: https://arxiv.org/pdf/2607.27271
• Abstract Code models are increasingly trained with execution feedback, but most training signals still stop at correctness. This leaves an important gap for systems code: two programs can pass the same tests while differing greatly in runtime. We study how to train code agents to prefer faster correct implementations, rather than treating efficiency only as an evaluation metric. The key difficulty is that runtime is a fragile reward. It is meaningful only after a program is correct, varies across tasks, and gives little guidance when most sampled programs fail to compile or run. We propose \textbf{RLPF}, reinforcement learning from performance feedback, which turns execution outcomes into a staged reward. Failed programs are ordered by execution progress, while correct programs are ranked by their relative improvement from the baseline toward the expert reference. This gives useful feedback before correctness and performance-sensitive feedback after correctness. Fine-tuning Qwen3-32B with RLPF on PerfCodeBench raises correct-and-runnable solutions from $11.1\%$ to $54.6\%$ and improves relative efficiency from $8.1\%$ to $38.6\%$. The trained model becomes competitive with stronger open-weight systems, and its optimization behavior transfers modestly to EffiBench-X. Additional studies show that model-generated references provide useful but weaker supervision, and that the full composite reward is more reliable than correctness-only or runtime-only baselines. These results suggest that code agents can be trained not only to pass tests, but also to optimize the programs they write.
• 中文摘要 代码模型越来越多地通过执行反馈训练，但大多数训练信号仍停留在正确性。这给系统代码留下了一个重要空白：两个程序可以在运行时间上差异很大的情况下通过相同的测试。我们研究如何训练代码代理，使其偏好更快且正确的实现，而不仅仅是将效率视为评估指标。关键难点在于，游戏时长是一种脆弱的奖励。只有在程序正确、任务间差异较大且大多数采样程序无法编译或运行时，它才有意义。我们提出了 \textbf{RLPF}，即基于绩效反馈的强化学习，将执行结果转化为分阶段奖励。失败的程序按执行进度排序，正确程序则按基线到专家参考的相对改进程度排序。这在正确性之前提供了有用的反馈，在正确性之后提供性能敏感性反馈。在PerfCodeBench上用RLPF微调Qwen3-32B，可将正确且可运行的解从$11.1%$提升到$54.6%，相对效率从$8.1\%$提升到$38.6%。训练好的模型在与更强的开权重系统竞争时具有竞争力，其优化行为在 EffiBench-X 上有所体现。其他研究表明，模型生成的参考提供了有用但较弱的监督，且完整复合奖励比仅正确性或仅运行时基线更可靠。这些结果表明，代码代理不仅可以训练通过测试，还能优化他们编写的程序。

SkillMentor: LLM Agent Self-Evolution via Learning Blind-Spot Diagnosis

SkillMentor：通过学习盲点诊断实现LLM代理自我进化

• Authors: Xiaoyi Bao, Yuanzhen Xie, Yunzhi Tan, Jinghang Gu, Zhongqing Wang, Chu-Ren Huang, Bo Hu, Zang Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27360
• Pdf link: https://arxiv.org/pdf/2607.27360
• Abstract Agent self-evolution has primarily focused on learning how to act, while overlooking an equally important capability: learning to discover what an agent does not know. Existing approaches typically assume that failure discovery is given, focusing on how to repair failures once they are identified. We ask whether blind-spot diagnosis itself can be learned. We thus study diagnosis as an agent capability separate from execution, and exclude two alternative sources of progress: executor adaptation and human supervision. Under these constraints, performance cannot improve through executor updates or annotated examples, forcing all improvements to originate from the learned diagnostic capability. We propose SkillMentor, which trains a Mentor policy via reinforcement learning to generate diagnostic tasks, identify recurrent failure modes, and curate them into reusable corrective skills. Across AppWorld and BFCLv3, SkillMentor improves executor performance by an average of 44.2%. These results suggest that blind-spot diagnosis is a learnable capability, enabling self-evolution without updating executor weights or relying on human-curated data.
• 中文摘要 代理自我进化主要关注学习如何行动，同时忽视了同样重要的能力：学会发现代理所不知之事。现有方法通常假设故障发现机制，重点是一旦发现故障后如何修复。我们会问盲点诊断本身是否可以被学习。因此，我们研究诊断作为独立于执行的代理能力，排除两个替代进展来源：执行者适应和人类监督。在这些约束下，性能无法通过执行器更新或注释示例来提升，所有改进都必须源自已学习的诊断能力。我们提出SkillMentor，通过强化学习训练导师政策，生成诊断任务，识别反复失败模式，并将其策划为可重复使用的纠正技能。在AppWorld和BFCLv3中，SkillMentor平均提升执行者性能44.2%。这些结果表明盲点诊断是一种可学习的能力，能够实现自我进化，无需更新执行者权重或依赖人类整理的数据。

Simulation of Surgical Suturing Using Position-Based Dynamics and the Material Point Method for Robot Reinforcement Learning

利用基于位置动力学和材料点方法进行手术缝合模拟，用于机器人强化学习

• Authors: Tleukhan Mussin, Yafei Ou, Mahdi Tavakoli
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.27494
• Pdf link: https://arxiv.org/pdf/2607.27494
• Abstract Recent advances in robotics research have created a strong demand for high-performance simulators. Surgical robotics simulation faces unique challenges due to the need to model diverse objects, such as rigid instruments, soft tissue, and fluids. While many studies simulate sutures or soft tissue independently, only a few have considered the complete soft-tissue suturing scenario, including the contact between sutures and deformable tissue during suture insertion. Building on previous work, this paper presents a novel suturing simulation environment using sutures modelled by position-based dynamics (PBD) and soft bodies modelled by the material point method (MPM) while considering two-way contact with frictional and drag forces. We introduce a contact coupling method between the PBD suture and the MPM soft tissue, enabling visually plausible suture-tissue interactions. The simulator is optimized for GPU execution with parallel scenes using multiple CUDA streams, and we present a Reinforcement Learning (RL) environment for autonomous suturing sub-tasks, including needle insertion, driving, and extraction. Using ML-Agents, RL agents trained in the simulator show stable learning and achieve 80% and 68% success rates in needle insertion and extraction, respectively, under the strictest distance threshold.
• 中文摘要 机器人研究的最新进展带来了对高性能模拟器的强烈需求。手术机器人仿真面临独特挑战，因为需要建模各种物体，如刚性器械、软组织和液体。虽然许多研究独立模拟缝线或软组织，但只有少数研究考虑了完整的软组织缝合场景，包括缝线插入时缝线与可变形组织的接触。基于之前的工作，本文提出了一种新颖的缝合模拟环境，利用基于位置动力学（PBD）建模的缝合线和材料点法（MPM）建模的软体，同时考虑摩擦力和阻力的双向接触。我们引入了PBD缝线与MPM软组织之间的接触耦合法，实现了视觉上合理的缝线-组织相互作用。该模拟器优化为GPU并行场景执行，使用多CUDA流，我们提供了一个强化学习（RL）环境，用于自主缝合子任务，包括针头插入、驱动和拔针。使用机器学习代理，模拟器训练的强化学习代理表现出稳定的学习能力，在最严格距离阈值下针头插入和拔针成功率分别达到80%和68%。

Training Skills Like Parameters via Self-Supervised Semantic Diffusion

通过自我监督语义扩散训练参数等技能

• Authors: Mo Li, Zixin Yin, Ting Cao, Yunxin Liu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.27557
• Pdf link: https://arxiv.org/pdf/2607.27557
• Abstract While Large Language Models (LLMs) demonstrate remarkable general instruction-following capabilities, they often fall short of human experts in highly specialized, open-ended domains such as creative screenwriting. Prior approaches typically adopt post-training, yet both supervised fine-tuning and reinforcement learning require weight access that closed-source frontier models do not offer, and demand heavy compute. Moreover, what is learned is tied to a single checkpoint and cannot be inspected by humans. Recent advancements in agentic continual learning instead attempt to bridge this gap by accumulating external textual skills. However, these methods heavily rely on costly human expert annotations or unreliable LLM-as-a-judge feedback for reflection. To overcome this bottleneck, we propose a novel, unsupervised self-evolving agent framework inspired by the corruption-and-reconstruction paradigm of diffusion models. Instead of relying on explicit external scoring, we leverage existing high-quality human artifacts to construct self-supervised signals. Training then follows the familiar loop of neural network training, forward, loss, and backward, with the loss coming from contrasting the agent's reconstruction against the human original. What is updated is not model weights but an external library of textual skills. We evaluate our framework on the challenging task of short drama screenwriting. Experimental results demonstrate that our method enables the agent to autonomously extract and internalize highly generalizable skills, significantly enhancing its domain-specific generation capabilities. Furthermore, this self-contrastive reflection paradigm offers a scalable pathway for agents to teach themselves the production of complex, high-quality human artifacts, without requiring external supervision.
• 中文摘要 虽然大型语言模型（LLMs）展现了卓越的通用指令跟随能力，但在高度专业、开放的领域如创意剧本写作中，往往未能达到人类专家的水平。以往的方法通常采用训练后，但监督式微调和强化学习都需要权重访问，而闭源前沿模型不具备，且需要大量计算。此外，所学内容仅限于单一检查点，无法被人类检查。而近期的智能持续学习进展则试图通过积累外部文本技能来弥合这一差距。然而，这些方法严重依赖昂贵的人工专家注释或不可靠的LLM作为评判的反馈来进行反思。为克服这一瓶颈，我们提出了一种受扩散模型中腐败与重建范式启发的新颖无监督自演化代理框架。我们不再依赖显性外部评分，而是利用现有的高质量人工产物构建自我监督信号。训练随后遵循熟悉的神经网络训练循环，前进、丢失和倒退，损失来自于对比代理的重建与人类原始状态。更新的不是模型权重，而是外部文本技能库。我们评估了我们对短剧剧本写作这一挑战性任务的框架。实验结果表明，我们的方法使智能体能够自主提取和内化高度通用的技能，显著增强了其领域特异性的生成能力。此外，这种自我对比反思范式为代理提供了一条可扩展的路径，使其能够自学制作复杂且高质量的人造物，而无需外部监督。

DeepResearch Agent System

DeepResearch 代理系统

• Authors: Yong Huang, Yulu Huang, for the team Collaboration
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27562
• Pdf link: https://arxiv.org/pdf/2607.27562
• Abstract The DeepResearch Agent System is a large language model system engineered for deep information retrieval, multi-step reasoning, and autonomous research tasks. Built upon a sparse activation architecture with 30 billion total parameters of which only 3 billion are activated per token, the system achieves state-of-the-art performance on multiple agent search benchmarks while delivering 3.2 times faster inference compared to dense counterparts of equivalent scale. The system supports a 128K-token context window with hierarchical attention mechanisms that yield 18.7% accuracy and 23.4% recall improvements over standard long-context approaches. A dual-mode reasoning engine provides both a ReAct paradigm for basic multi-step problem solving and an IterResearch mode for high-performance iterative research with up to 20 reasoning steps, collectively delivering a 31.2% accuracy improvement over single-pass baselines. Multi-tool coordination integrates retrieval, computation, web search, and file parsing modules to achieve 92.1% tool-use accuracy. A reinforcement learning optimization framework based on the GRPO algorithm provides token-level policy gradients that improve training stability by 35% and accelerate convergence by 42%. An automated data synthesis pipeline with seed-based expansion achieves a 92.5% usability rate. Benchmark results include 87.3% on Humanity's Last Exam, 85.3% on BrowserComp Chinese, and 91.2% on WebWalkerQA. The system is fully open-sourced, including data synthesis, training, and inference code, and supports applications in academic research, business analysis, R&D support, and education.
• 中文摘要 DeepResearch 代理系统是一个大型语言模型系统，专为深度信息检索、多步推理和自主研究任务而设计。该系统基于稀疏激活架构，拥有300亿个总参数，其中每个代币仅激活30亿个，在多个代理搜索基准测试中实现了最先进的性能，同时推理速度是同等规模密集基准测试的3.2倍。系统支持128K令牌上下文窗口，采用分层注意力机制，比标准长上下文方法提升18.7%的准确率和23.4%的回忆率提升。双模式推理引擎既提供了用于基础多步问题解决的ReAct范式，也为高性能迭代研究提供了IterResearch模式，最多可达20个推理步骤，总计比单次计算基线提高了31.2%的准确性。多工具协调整合检索、计算、网页搜索和文件解析模块，实现92.1%的工具使用准确率。基于GRPO算法的强化学习优化框架提供了令牌级策略梯度，使训练稳定性提升35%，并加速收敛42%。基于种子扩展的自动化数据综合流水线可实现92.5%的可用性。基准测试结果包括 Humanity 最后考试的 87.3%，BrowserComp 中文版的 85.3%，以及 WebWalkerQA 的 91.2%。该系统完全开源，包括数据综合、培训和推理代码，支持学术研究、商业分析、研发支持和教育等领域的应用。

Policy Gradient Steering: Interventions from Behavioral Objectives

政策梯度引导：行为目标干预

• Authors: Yoann Poupart, Aurélie Beynier, Nicolas Maudet
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.27574
• Pdf link: https://arxiv.org/pdf/2607.27574
• Abstract Activation steering has emerged in large language models as a lightweight alternative for dynamically changing a model's behavior at inference time. However, we show that existing steering methods fail to steer even a simple policy in a two-route gridworld environment. To address this limitation, we propose Policy Gradient Steering (PGS), which formulates steering as a reinforcement learning problem. PGS accumulates gradients of a temporary behavioral objective over a small set of rollouts or demonstrations to construct a removable task vector. We first demonstrate the calibration and reversibility of PGS in a two-route gridworld environment. Using chess puzzles, we then evaluate independently fitted PGS vectors both in isolation and in combination, finding that compatible tactical objectives accumulate constructively. Finally, in competitive football, we show that PGS can alter specific team behaviors and that its effects transfer across opponents. Together, these results show that policy gradients provide a natural interface for constructing temporary and composable behavioral adaptations across diverse decision-making domains.
• 中文摘要 激活引导已成为大型语言模型中一种轻量级的替代方案，用于在推理时动态改变模型行为。然而，我们表明现有的引导方法在双路径网格世界中甚至无法引导一个简单的策略。为解决这一限制，我们提出了策略梯度引导（PGS），将引导表述为一种强化学习问题。PGS通过在一小部分推广或演示中积累临时行为目标的梯度，构建可移除的任务向量。我们首先展示了PGS在双路径网格世界环境中的校准和可逆性。利用国际象棋谜题，我们对独立拟合的PGS向量进行单独评估和组合，发现兼容的战术目标会建设性地累积。最后，在竞技足球中，我们展示了PGS可以改变特定球队行为，其影响会在对手间传递。这些结果共同表明，政策梯度为构建跨多样决策领域构建临时和可组合的行为适应提供了自然的接口。

Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning

卡尔曼遇见课程：自适应强化学习的高效动态提示选择

• Authors: Haodong Zhu, Yangyang Ren, Yanjing Li, Sheng Xu, Haiguang Liu, Linlin Yang, Baochang Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.27610
• Pdf link: https://arxiv.org/pdf/2607.27610
• Abstract Reinforcement learning (RL) finetuning significantly enhances the reasoning capabilities of large language models (LLMs), yet its effectiveness critically depends on selecting prompts of appropriate difficulty for the current policy. This is challenging because prompt difficulty evolves throughout training. Existing online methods therefore face a trade-off: evaluation-based approaches are accurate but expensive, while prediction-based approaches are efficient but typically assume stationary difficulty, making them ill-suited to RL's non-stationary training dynamics. To address these issues, we propose a Kalman-Guided Prompt Selection method (KGPS), which reformulates prompt selection as a dynamic state estimation problem rather than static difficulty prediction. KGPS models each prompt's latent success rate in logit space using a linear-Gaussian state-space model, with process noise coupled to the magnitude of policy updates so that uncertainty increases when the policy changes more substantially. A Kalman filter then maintains a calibrated Gaussian posterior over prompt difficulty, and prompts are selected by maximizing a posterior-expected training utility that favors intermediate-difficulty prompts while naturally revisiting uncertain ones. The resulting procedure is adaptive to policy drift and requires no additional rollouts beyond standard policy training. Extensive experiments across mathematics, planning, and geometry reasoning benchmarks, as well as multiple RL algorithms, show that KGPS consistently improves both final accuracy and rollout efficiency over strong baselines, establishing state-of-the-art performance among online prompt selection methods. For example, on DeepSeek-R1-Distill-7B, KGPS uses 83% fewer rollouts than DS while even improving the average performance by 0.12 point across six math reasoning benchmarks.
• 中文摘要 强化学习（RL）微调显著增强了大型语言模型（LLMs）的推理能力，但其有效性关键在于选择适合当前策略的难度提示。这很有挑战性，因为即时难度会随着训练而变化。现有的在线方法因此面临一个权衡：基于评估的方法准确但成本高昂，而基于预测的方法效率高但通常假设难度是平稳的，因此不适合强化学习的非平稳训练动态。为解决这些问题，我们提出了一种卡尔曼引导提示选择方法（KGPS），将提示选择重新表述为动态状态估计问题，而非静态难度预测。KGPS利用线性高斯状态空间模型对每个提示在logit空间中的潜在成功率进行建模，过程噪声与策略更新幅度耦合，使策略变更更明显时不确定性增加。卡尔曼滤波器随后保持对提示难度的校准高斯后验，提示通过最大化后验期望训练效用来选择提示，该效用偏向中等难度提示，同时自然重访不确定提示。该程序适应政策漂移，无需额外推广，超出标准政策培训。数学、规划和几何推理基准测试以及多种强化学习算法的广泛实验表明，KGPS在强有力的基线条件下持续提升最终准确率和推广效率，确立了在线提示选择方法的先进性能。例如，在DeepSeek-R1-Distill-7B上，KGPS的部署次数比DS少83%，但平均性能在六个数学推理基准测试中提升了0.12个百分点。

Real-Time Hard Peak Age-of-Information Safety with No-Regret Learning

实时硬峰值信息时代安全与无悔学习

• Authors: Wentao Zhang, Wentao Mo
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.27626
• Pdf link: https://arxiv.org/pdf/2607.27626
• Abstract Safety-critical IoT systems such as industrial closed-loop control, V2X coordination, and remote teleoperation require every sensor's peak Age of Information (peak AoI, also abbreviated PAoI) to stay below a hard per-slot deadline, not merely an average bound. Existing approaches meet this requirement only under restrictive assumptions: stochastic channels for Whittle-index AoI, simulator rollouts for deep reinforcement learning, or sublinear cumulative violation for long-term constrained online convex optimization. Under adversarial coefficients, OCO-PAoI-Hard guarantees zero per-slot violation of the modeled AoI state under one-step viability and O(sqrt(T)) regret against any static safe comparator; packet-level safety requires stronger service assumptions. Our key observation is that the fractional peak-AoI deadline collapses exactly to an affine half-space constraint on the resource-allocation vector, turning hard real-time scheduling into time-varying constrained online convex optimization over a polyhedral safe set. A strictly causal proposal-shield-update loop enforces feasibility through one Euclidean projection per slot, the gradient step preserves no-regret behavior, and the classical virtual queue is reduced to an a-posteriori certificate. We establish closed-form static and dynamic regret bounds, a matching Omega(sqrt(T)) minimax lower bound, a margin-safe variant against execution noise, and a deadline-induced competitive ratio. On a four-sensor adversarial fluid-model trap channel, OCO-PAoI-Hard attains zero modeled-state deadline violations across all ten seeds, while four representative baselines miss between 1.65 percent and 64.0 percent of slots, and the empirical normalized regret stays below the theoretical envelope across two orders of magnitude in T.
• 中文摘要 安全关键的物联网系统，如工业闭环控制、V2X协调和远程远程操作，要求每个传感器的峰值信息时代（峰值AoI，也缩写PAoI）保持在每个时隙的硬性截止时间以下，而不仅仅是平均限。现有方法仅在限制性假设下满足这一要求：基于Whittle指数AoI的随机通道、深度强化学习的模拟器展开，或长期受限在线凸优化的亚线性累积违背。在对抗系数下，OCO-PAoI-Hard 保证在一步可行性和 O（sqrt（T）） 遗憾下，对任何静态安全比较器，每个槽对建模的 AoI 状态为零违规;分组级安全需要更强的服务假设。我们的关键观察是，分数峰值AoI截止时间恰好塌缩到资源分配向量的仿射半空间约束，将硬实时调度转变为多面体安全集上的时变在线凸优化。严格因果的提案-屏蔽-更新循环通过每个槽进行一个欧几里得投影来强制执行可行性，梯度步骤保持无遗憾行为，经典虚拟队列被简化为后验证书。我们建立了封闭式静态和动态遗憾界限、匹配的Omega（sqrt（T））极大极大值下界、一个边际安全、抗执行噪声的变体，以及截止时间诱导的竞争比率。在四传感器对抗流体模型陷阱通道中，OCO-PAoI-Hard 在所有十个种子中均实现零模型状态截止日期违规，而四个代表性基线的缺失率在1.65%至64.0%之间，经验归一化后悔在理论包线以下，持续两个数量级的T。

ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

ReDiPPO：参考引导值校准与数学推理中的差异感知令牌重权

• Authors: Zhenrong Zhang, Fei Wu, Jun Du, Jianshu Zhang, Si Wei
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.27631
• Pdf link: https://arxiv.org/pdf/2607.27631
• Abstract Reinforcement learning has emerged as an effective paradigm for enhancing the mathematical reasoning capabilities of large language models. Among existing policy optimization methods, Proximal Policy Optimization (PPO) remains particularly appealing because its learned critic can, in principle, provide token-level credit assignment. However, in mathematical reasoning tasks characterized by long reasoning horizons and sparse outcome rewards, reliable token-level credit assignment remains challenging. The standard critic often fails to accurately evaluate intermediate reasoning states, resulting in noisy advantage estimates and suboptimal policy updates. In this paper, we propose ReDiPPO, a Reference-guided and Discrepancy-aware PPO framework for mathematical reasoning. ReDiPPO introduces a reference-guided critic that uses reference answers as training-time privileged signals to provide more accurate value estimation. Meanwhile, it retains a standard critic and quantifies the token-level reference-standard discrepancy between the standard value estimate and the reference-guided value estimate. This discrepancy serves as an indicator of difficult reasoning states and is used to reweight the corresponding token-level advantages during PPO optimization. Extensive experiments on diverse mathematical reasoning benchmarks demonstrate that ReDiPPO improves value-estimation accuracy and consistently outperforms strong policy optimization baselines, including PPO, DAPO, and GSPO, in final reasoning performance. Our code is available on this https URL.
• 中文摘要 强化学习已成为提升大型语言模型数学推理能力的有效范式。在现有的策略优化方法中，近端策略优化（PPO）尤其具有吸引力，因为其有学问的批评者原则上可以提供代币级的信用分配。然而，在具有较长推理视野和稀疏结果奖励的数学推理任务中，可靠的代币级学分分配仍然具有挑战性。标准批评者常常无法准确评估中间推理状态，导致优势估计噪声大，政策更新不理想。本文提出了ReDiPPO，一种以参考文献为导向且对差异感知的PPO数学推理框架。ReDiPPO引入了参考引导批评器，将参考答案作为训练时间特权信号，提供更准确的数值估计。同时，它保留了标准的批评者，并量化了标准值估计与参考指导价值估计之间的代币级参考-标准差异。这种差异作为难以推理状态的指示，并在PPO优化过程中用于对应的代币级优势进行加权。在多种数学推理基准测试上的大量实验表明，ReDiPPO提升了价值估计的准确性，并在最终推理表现上持续优于强力的政策优化基线，包括PPO、DAPO和GSPO。我们的代码可以在这个 https URL 上获取。

Harness-G: A Graph-Structured Harness for Search Agents

Harness-G：一种用于搜索代理的图结构工具

• Authors: Yanning Hou, Haoyuan Chen, Sihang Zhou, Xiaoshu Chen, Xirui Liu, Duanyang Yuan, Lingyuan Meng, Quan Liu, Jian Huang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.27652
• Pdf link: https://arxiv.org/pdf/2607.27652
• Abstract Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.
• 中文摘要 强化学习（RL）搜索代理通常将检索建模为自由形式的自然语言查询生成，并通过最终答案奖励优化多回合交互。当前研究主要通过更密集或结构化的信用信号来改善培训，但很少探讨在政策与环境界面上是否正确表述检索。在Search-R1训练过程中，我们观察到明显的检索锯齿：同一问题的推销仍持续生成不同的查询字符串，但它们积累的证据集却越来越重叠。我们称这种现象为反演等价坍缩;在这种模式下，轨迹在检索决策上趋近效用等价，导致组内返回的有效检索对比度较低。为解决这一问题，我们提出了 Harness-G，一种图结构检索框架，重新设计了该接口。它将自由形式查询生成重新表述为有限动作选择：策略选择证据句子或实体，或选择回答，而环境构建菜单，跟踪检索状态，并验证和执行每个选择。该接口减少了语言混叠，使同状态替代方案能够直接比较。基于该接口，我们引入了结构性非近视信用（SNC），它使用固定的答案评分器将所选行为与其备选行为进行比较，并将后续收益分配给先前促成这些行动的收益。在六个质量保证基准中，Harness-G在两个评估模型尺度上均获得最高的平均F1，比最强基线Graph-R1高出10.74个百分点（1.5B），比3B高3.98个百分点。

RefineSVG: Visual Feedback-Driven Reinforcement Learning for Image-to-SVG Generation

RefineSVG：用于图像到SVG生成的视觉反馈驱动强化学习

• Authors: Shaobo Liu, Feiqiao Mao, Shuaishuai Zhou, Yan Zhan, Weiqi Tan, Zhiqiong Lu, Zhengping Liang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27699
• Pdf link: https://arxiv.org/pdf/2607.27699
• Abstract We propose RefineSVG, a single-step closed-loop visual feedback framework that enables multimodal large language models (MLLMs) to perform high-fidelity image-to-SVG generation through self-correction. Existing MLLM-based approaches rely on single-pass open-loop inference, where the model receives visual input only once and must generate thousands of SVG code tokens without intermediate verification. This paradigm inevitably leads to geometric drift, error accumulation, and visual hallucination on complex images. RefineSVG overcomes this limitation by invoking an external rendering engine after an initial SVG generation pass to compare the rendered output against the target image. The comparison yields a multi-dimensional visual residual map (Diff-Map) that is fed back to the model as a ReAct-style correction signal, driving a targeted correction step. To support this render-observe-correct interaction, we further introduce an SVG-oriented semantic vocabulary that compresses token sequences by over 52%. A progressive training pipeline spanning supervised fine-tuning, rejection-sampling cold-start data construction, and end-to-end agentic reinforcement learning aligns the model with closed-loop visual correction. Extensive experiments show that RefineSVG consistently outperforms existing baselines in reconstruction fidelity, structural accuracy, and code this http URL is available at this https URL.
• 中文摘要 我们提出了RefineSVG，这是一种单步闭环视觉反馈框架，使多模态大型语言模型（MLLM）能够通过自我校正实现高精度的图像生成SVG。现有基于MLLM的方法依赖单次开环推断，模型仅接收一次视觉输入，且必须生成数千个SVG代码令牌，且无需中间验证。这种范式不可避免地导致几何漂移、误差累积和复杂图像上的视觉幻觉。RefineSVG 通过在初始 SVG 生成后调用外部渲染引擎，将渲染输出与目标图像进行比较，从而克服了这一限制。比较后会得到一个多维视觉残差图（Diff-Map），该图作为ReAct式修正信号反馈给模型，驱动目标修正步骤。为支持这种渲染-观察-正确交互，我们进一步引入了面向SVG的语义词汇，将令牌序列压缩超过52%。一个涵盖监督微调、拒绝-采样冷启动数据构建和端到端代理强化学习的渐进训练流水线，使模型与闭环视觉修正对齐。大量实验表明，RefineSVG在重建的忠实度、结构准确性和代码方面始终优于现有基线。该http URL可在此HTTP访问。

Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO

Cocktail-Talker：在嘈杂社交环境中采用转动GRPO的多扬声器对话建模

• Authors: Xilin Jiang, Riki Shimizu, Sukru Samet Dindar, Junkai Wu, Zhongweiyang Xu, Nima Mesgarani
• Subjects: Subjects: Sound (cs.SD); Computation and Language (cs.CL); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2607.27756
• Pdf link: https://arxiv.org/pdf/2607.27756
• Abstract Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, are often more ambiguous: multiple speakers may participate in the same conversation amid irrelevant speech and background noise. Each utterance may be directed to the assistant, addressed to another speaker, or completely irrelevant. In such settings, the assistant must decide not only what to say, but also whether to speak at all. In this paper, we introduce Cocktail-Talker, a speech LLM framework for multi-speaker spoken dialog modeling in noisy social environments. We model the assistant's behavior with three action tokens: <|respond|>, <|listen|>, and <|ignore|>, placed before a response or silence. Cocktail-Talker is trained via supervised finetuning and reinforcement learning to generate the appropriate action token and, only in <|respond|> mode, a speech response. To prepare the training data, we develop Cocktail-DialogGen, an LLM-based data pipeline that simulates realistic multi-speaker dialogs with speaker roles across diverse social settings. Together, these components take a step toward spoken dialog systems that interact more naturally and selectively in complex social environments.
• 中文摘要 语音对话系统通常设计为干净、双向的互动，由单一用户和助手轮流发言。然而，现实中的社交对话往往更为模糊：多个说话者可能在同一对话中参与，伴随着无关紧要的语音和背景噪音。每句话语都可能针对助理，对其他发言者说，或完全无关紧要。在这种情况下，助理不仅要决定说什么，还要决定是否要发言。本文介绍了Cocktail-Talker，一种用于在嘈杂社交环境中进行多说话口语对话建模的语音LLM框架。我们用三个动作标记来建模助手的行为：<|respond|>、<|listen|> 和 <|ignore|>，置于响应或静默之前。Cocktail-Talker 通过监督微调和强化学习训练，生成合适的动作标记，并且仅在 <|response|> 模式下生成语音响应。为了准备训练数据，我们开发了Cocktail-DialogGen，这是一个基于LLM的数据流水线，能够模拟多说话者角色在多样社交环境中真实对话。这些组成部分共同迈向在复杂社会环境中更自然、更有选择性互动的口语对话系统。

Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold

超越最佳教师：扩展与压缩推理解流形

• Authors: Songshuo Lu, Zhi Chen, Yaohua Tang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.27770
• Pdf link: https://arxiv.org/pdf/2607.27770
• Abstract A single reinforcement-learning run can produce a strong reasoner yet an incomplete teacher: it often amplifies only a subset of the valid solution modes. We argue that reinforcement learning (RL)-trained policies should therefore be viewed as local probes of a multi-basin reasoning solution manifold, rather than as globally reliable supervisors. Based on this view, we propose an expand-then-compress framework that couples teacher construction with multi-teacher policy distillation. In the expansion stage, Residual Group Relative Policy Optimization (RGRPO) trains a sequence of teachers from a common initialization and redirects each later round toward examples not yet covered by the accumulated teacher union. In the compression stage, reliability-gated Teacher-Union On-policy Distillation (TU-OPD) lets the student learn from its own response prefixes. For each example, only reliable teachers contribute, and their sampled-token OPD losses are weighted by their per-example quality. We further introduce Consensus-Residual Decomposition, which preserves a winner teacher's excess token preferences over its reliable peers, preventing specialist behavior from being suppressed during teacher aggregation. Experiments on mathematical reasoning, code generation, and instruction following show that the resulting Qwen3-1.7B student consistently outperforms the strongest individual teacher across all three domains, yielding relative improvements of 2.0%, 8.3%, and 6.9%, respectively, while retaining single-model inference. These results establish a simple but powerful principle: stronger students can be obtained not by selecting a single better teacher, but by deliberately constructing and compressing a complementary teacher union.
• 中文摘要 一次强化学习运行可能产生强有力的推理者，但教师不完整：它通常只放大有效解法模式的部分。因此，强化学习（RL）训练的策略应被视为多盆地推理解的局部探测，而非全局可靠的监督者。基于这一观点，我们提出了一种将教师建设与多教师政策提炼相结合的扩展-再压缩框架。在扩展阶段，残差组相对策略优化（RGRPO）从共同初始化中训练一系列教师，并将后续每轮重定向到尚未被积累教师工会覆盖的示例。在压缩阶段，可靠性门槛教师-工会政策提炼（TU-OPD）让学生从自己的响应前缀中学习。对于每个例子，只有可靠的教师贡献，其抽样标记的OPD损失会根据其每实例的质量加权。我们进一步引入共识残差分解，该方法保留获胜教师相对于可靠同伴的多余代币偏好，防止专业行为在教师聚合过程中被抑制。数学推理、代码生成和教学跟随的实验显示，最终的Qwen3-1.7B学生在三个领域中持续优于最强的个别教师，分别带来2.0%、8.3%和6.9%的相对提升，同时保留了单模型推断能力。这些结果确立了一个简单但有力的原则：更优秀的学生不是通过选择一个更优秀的教师，而是通过有意构建和压缩互补的教师工会来实现。

RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy

RedFlow：将失败重定向为动作级修正，以实现流量匹配的VLA策略

• Authors: Zhengyang Yan, Junhao Li, Fangqi Zhu, Zijun Wang, Quanxin Shou, Yikun Miao, Xiaoyi Pang, Zicong Hong, Song Guo
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27782
• Pdf link: https://arxiv.org/pdf/2607.27782
• Abstract Flow-matching Vision-Language-Action (VLA) policies have shown strong potential for robotic manipulation but often suffer from compounding errors caused by distribution shifts during deployment. While offline reinforcement learning (RL) provides a practical way to improve deployed policies using rollout data, existing methods either ignore failure data or exploit it only at the trajectory level, resulting in low learning efficiency and persistent errors. We propose RedFlow, a fine-grained offline RL framework that redirects failure experiences into action-level corrective supervision for flow-matching VLA policies. RedFlow consists of two key components: (1) a Context-Aware Corrective Matching mechanism that identifies failure-inducing actions and retrieves successful alternatives from similar contexts as corrective targets, and (2) an Adaptive Redirection Objective that jointly reinforces successful actions, suppresses undesirable ones, and redirects recoverable failures toward corrective targets. By converting both successful and failed experiences into dense supervision, RedFlow enables robust recovery learning from mixed-quality data. Experiments on the LIBERO benchmark and three real-world manipulation tasks show that RedFlow consistently outperforms state-of-the-art offline RL baselines, improving the real-world success rate from 56.7% to 74.7%. It also matches strong on-policy methods (PPO, GRPO, and DDPO) while requiring roughly an order of magnitude fewer training samples.
• 中文摘要 流程匹配视觉-语言-行动（VLA）策略在机器人操作方面展现出强大潜力，但常因部署过程中分配偏移而出现叠加错误。虽然离线强化学习（RL）提供了利用推广数据改进部署策略的实用方法，但现有方法要么忽视失败数据，要么仅在轨迹层面利用失败数据，导致学习效率低下且错误持续存在。我们提出了RedFlow，一个细粒度的离线强化学习框架，将失败经验重定向到动作级纠正监督，以匹配流的VLA策略。RedFlow由两个关键组成部分：（1）上下文感知纠正匹配机制，识别导致失败的动作并从类似纠正目标上下文中检索成功替代方案;（2）自适应重定向目标，共同强化成功行动，抑制不良行为，并将可恢复的失败引导至纠正目标。通过将成功与失败的经验转化为密集的督导，RedFlow实现了从混合质量数据中稳健的恢复学习。基于LIBERO基准测试和三项真实操作任务的实验显示，RedFlow持续优于最先进的离线强化学习基线，将现实世界成功率从56.7%提升至74.7%。它还匹配了强有力的政策内方法（PPO、GRPO和DDPO），同时所需的训练样本数量大约少一个数量级。

LoRA Scaffolded Policy Optimization (LSPO): A Sampling-Time Low-Rank Scaffold for Recovering Reinforcement-Learning Gradient on Zero-Reward Cliff Prompts

LoRA支架策略优化（LSPO）：一个采样时间低秩支架，用于在零奖励悬崖提示上恢复强化学习梯度

• Authors: Ken Ding
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27787
• Pdf link: https://arxiv.org/pdf/2607.27787
• Abstract Reinforcement learning from verifiable rewards (RLVR) for mathematical reasoning suffers from a structural blind spot: on "cliff" prompts-those on which every sampled rollout in a group fails-the group-normalized advantage is identically zero, so GRPO produces no gradient on precisely the prompts at the frontier of the model's capability. We introduce LoRA Scaffolded Policy Optimization (LSPO), a sampling-time mechanism that recovers this lost gradient. Each RL step, LSPO detects cliff prompts, fits a small low-rank (LoRA) adapter by a brief supervised step on their ground-truth solutions, re-rolls the cliffs with the base-plus-adapter model, splices the now-successful completions back into the RL batch with an importance-sampling correction, and takes a GRPO step on the base alone; the adapter receives only the supervised gradient and is discarded at checkpoint, yielding a base-only model. On DeepMath-103K with DeepSeek-R1-Distill-Qwen-1.5B, evaluated over n=5 paired seeds per arm at a matched 1000-step reporting horizon, LSPO's 5-seed mean matches or beats a DAPO baseline on all 16 (benchmark, pass@k) cells (15 strict wins and one exact tie), with gains of up to +10.7 points on AIME24/pass@4, +6.7 points on AIME24 and AIME26 at pass@16, and +2.4 points on MATH500/pass@1; averaged over the 16 cells the improvement is +3.8 points.
• 中文摘要 基于可验证奖励的强化学习（RLVR）存在结构盲点：在“悬崖”提示上——即每个抽样的组内展开都失败的提示——群体归一化优势完全为零，因此GRPO在模型能力前沿的提示上不产生梯度。我们介绍了LoRA支架策略优化（LSPO），这是一种采样时间机制，能够恢复这一失去的梯度。每一步强化学习，LSPO检测悬崖提示，通过简短监督步骤拟合一个小型低秩（LoRA）适配器，使用基础加适配器模型重新滚动悬崖，将成功完成的匹配通过重要性采样修正拼接回强化学习批次，并单独对基础进行GRPO步骤;适配器仅接收监督梯度，并在检查点丢弃，得到仅基数的模型。在DeepMath-103K中，采用DeepSeek-R1-Distill-Qwen-1.5B，在匹配的1000步报告期内，每臂n=5对种子进行评估，LSPO的5种子平均在所有16个（基准、pass@k）单元格上均匹配或超过DAPO基线（15场严格胜出和1场完全平局），在AIME24/pass@4提升最高+10.7分，AIME24和AIME26在pass@16提升+6.7分。 MATH500/pass@1+2.4分;16个单元的平均提升为+3.8分。

Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning

并非所有代币都应获得同等的信用：反事实敏感性信用重新分配用于长期CoT推理

• Authors: Qiangqiang He, Zhongheng Wu, ZiJian Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27888
• Pdf link: https://arxiv.org/pdf/2607.27888
• Abstract Reinforcement learning with verifiable rewards (RLVR) is central to improving long-CoT reasoning in large language models. Critic-free methods such as GRPO convert response-level rewards into advantages and uniformly broadcast them across tokens, overlooking their unequal contributions to the final outcome. On-policy self-distillation (OPSD) instead provides dense distributional supervision by minimizing the forward KL divergence between an unprivileged policy and a privileged self-teacher, implicitly assuming that the resulting likelihood shifts encode reliable answer-aligned information. We test this premise by fixing each sampled trajectory and re-scoring it under two opposing outcome conditions, one asserting correctness and the other incorrectness. Most affected tokens shift in the same direction under both conditions, with few sign reversals and substantial overlap in the induced optimization signals. Large shifts also concentrate on highly substitutable surface-form tokens, whereas tokens carrying problem-specific reasoning content are less sensitive. These findings show that privileged shifts fail to provide reliable answer-aligned directions, while their magnitudes primarily reflect counterfactual sensitivity rather than token-level learning value. Based on these observations, we propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages to preserve both the original credit budget and verifier-determined direction. On long-CoT mathematical reasoning benchmarks, CSCR consistently outperforms GRPO baseline with the same number of policy updates. Targeted ablations further corroborate our diagnosis: privilege-induced directions are unreliable, moderate downweighting is most effective, and stronger modulation destabilizes optimization.
• 中文摘要 带有可验证奖励的强化学习（RLVR）是提升大型语言模型中长CoT推理的核心。无批评的方法如GRPO将响应级奖励转换为优势，并在代币间均匀广播，忽略了它们对最终结果的不平等贡献。策略自蒸馏（OPSD）则通过最小化非特权策略与特权自学者之间的前向 KL 分歧，隐含假设结果似然转移编码可靠且符合答案的信息，从而实现密集的分布监督。我们通过固定每个抽样轨迹并在两种相反的结果条件下重新评分来检验该前提，其中一个断言正确，另一个为错误。大多数受影响的代币在两种条件下都会朝同一方向移动，符号反转很少，优化信号中重叠较大。大转移还集中于高度可替代的表表形式符号，而携带特定问题推理内容的符号则不那么敏感。这些发现表明，特权转移无法提供可靠且符合答案的方向，其幅度主要反映反事实敏感性，而非代币级学习价值。基于这些观察，我们提出了反事实敏感性信用重分配（CSCR），这是GRPO的简单扩展，通过减少高度敏感代币的信用，并重整化代币层面优势，以保持原始信用预算和验证者确定的方向。在长期CoT数学推理基准测试中，CSCR在相同政策更新次数下持续优于GRPO基线。靶向消融进一步印证了我们的诊断：特权诱导的方向不可靠，适度降权最有效，而更强的调制会破坏优化。

One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting

一个补丁就足够了：针对基于MLLM的场景文本发现优化的视觉令牌接地

• Authors: Rui Tang, Wentao Yang, Peirong Zhang, Yongxin Shi, Shun Zhang, Huiguo He, Lianwen Jin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.27902
• Pdf link: https://arxiv.org/pdf/2607.27902
• Abstract Scene text spotting requires high-precision alignment between textual recognition and spatial localization. While visual-token grounding has emerged as a promising formulation for Multimodal Large Language Models (MLLMs), the previous multi-patch paradigm often introduces redundant noise and localization ambiguity, particularly for dense or small text instances. To address this, we propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token and then recovers geometry via full-image refinement. To accurately identify this anchor without oracle labels, we introduce Single-Patch Selective Optimization (SPaSO), a reinforcement learning framework that optimizes discrete visual-token selection using patch-level rewards. To further improve representation robustness and localization precision, we introduce Directional Embedding Alignment (DEA) to suppress unstable norm bias by decoupling feature magnitude and direction, and Patch-Enhanced Decoding (PED) to fuse the routed anchor with language semantics and cross-attend over the full-image feature map for geometry-aware boundary regression beyond coordinate-space surrogates. Extensive experiments demonstrate that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs. Code will be released soon.
• 中文摘要 场景文本定位需要文本识别与空间定位之间的高精度对齐。虽然可视化令牌基础已成为多模态大型语言模型（MLLMs）的一种有前景的表述，但之前的多补丁范式常常引入冗余噪声和定位歧义，尤其是在密集或较小的文本实例中。为此，我们提出了单块文本检测（SPaTS）的框架，这是一种以视觉为中心的框架，将每个文本实例通过单一锚点视觉标记路由，然后通过全图像细化恢复几何体。为了准确识别该锚点而无需预言机标签，我们引入了单补丁选择性优化（SPaSO），这是一种强化学习框架，利用补丁级奖励优化离散的视觉标记选择。为进一步提升表示鲁棒性和定位精度，我们引入了方向嵌入对齐（DEA）通过解耦特征大小和方向来抑制不稳定范数偏差，以及补丁增强解码（PED），将路由锚点与语言语义融合，并在全图特征图上交叉关注，实现超越坐标空间代理的边界回归。大量实验表明，SPaTS持续且显著地优于前沿的闭源MLLM和OCRMLLMs。代码将很快发布。

Class-Aware Reinforcement Learning for Counterfactual Explanation Generation

反事实解释生成的类感知强化学习

• Authors: Muhammad Adil Saleem, Syed Ali Raza, Mary-Anne Williams
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27905
• Pdf link: https://arxiv.org/pdf/2607.27905
• Abstract Counterfactual explanations (CFEs) enhance the interpretability of black-box models by generating alternative instances with adjusted feature values that achieve a contrastive outcome. Reinforcement learning (RL) offers a promising approach for CFE generation, enabling efficient exploration of counterfactual instances while ensuring control over key metrics like validity, sparsity, and proximity. Previous studies have formulated RL states exclusively using features derived from the predictors in the supervised dataset. This study explores the impact of including an instance's predicted class, alongside features derived from the predictors, in the RL state representation for generating CFEs. The hypothesis is that class-awareness enhances exploration efficiency and improves policy optimality. We compare the proposed class-aware RL method with the class-blind RL method, which is similar but excludes the instance's class information from the state representation. The comparison was conducted using seven datasets from diverse domains, varying in size. The results show that during training, class-aware RL offers benefits in terms of convergence speed, reward optimization, and episode length reduction. Moreover, it generates significantly more valid CFEs compared to class-blind RL. Finally, the instance's class-based feature consistently ranks among the most influential predictors in RL's action-selection, as shown by the SHAP and LIME values, underscoring the significance of class-awareness in RL for CFE generation. The impact is heightened clarity, faster learning, improved validity, and more effective counterfactual generation across diverse datasets.
• 中文摘要 反事实解释（CFE）通过生成带有调整特征值的替代实例，从而实现对比性结果，从而增强黑箱模型的可解释性。强化学习（RL）为CFE生成提供了一种有前景的方法，能够高效探索反事实实例，同时确保对效度、稀疏性和接近性等关键指标的控制。以往的研究仅基于监督数据集中预测变量的特征来构建强化学习状态。本研究探讨将实例的预测类与预测变量衍生特征一同纳入强化学习状态表示以生成CFE的影响。假设是类意识提升了探索效率并提高了政策的最优性。我们比较了拟议的类感知强化学习方法与类盲强化学习方法，后者相似但不包含实例的类信息。比较使用来自不同领域、大小不一的七个数据集进行。结果显示，在训练过程中，类感知强化学习在收敛速度、奖励优化和缩短集数长度方面带来了好处。此外，它生成的有效CFE显著多于类盲强化学习。最后，实例的基于类别的特性始终位列强化学习动作选择中最具影响力的预测因子之一，如SHAP和LIME值所示，凸显了类意识在强化学习中对CFE生成的重要性。其影响包括更清晰度、更快的学习、提升效度，以及在不同数据集中更有效地生成反事实。

Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control

精确的动作值不足以实现：多区 VAV 控制推理模型的推理模型的推广验证增强微调

• Authors: Takumi Shioda, Kohei Terashima, Tatsuo Nagai
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.27914
• Pdf link: https://arxiv.org/pdf/2607.27914
• Abstract Multi-zone variable-air-volume control must balance thermal comfort, indoor air quality, and electricity use across several continuous actuators. Model predictive control and reinforcement learning are widely studied, but deployment typically requires building-specific modeling or training, limiting scalability. We first test whether a frontier reasoning model (an LLM trained to use additional inference-time computation) can achieve competitive VAV control from text without building-specific training. With that capability established, we then test whether TD3-guided reinforcement fine-tuning (RFT) can transfer control knowledge into a locally deployable open-weight model. Five controllers are evaluated over three summer days in a physics-based four-zone emulator. Relative to a Guideline 36-based baseline, TD3 reduced HVAC electricity by 4.5% while improving temperature and CO$_2$ compliance. Without building-specific training, GPT-5 achieved the largest reduction (6.2%) but reduced the ventilation margin. For RFT, deterministic rollouts restore a saved state, apply one candidate, and follow TD3 to score each action. Auditing a learned critic against these rollouts exposed a failure hidden by its near-perfect across-time correlation ($r=0.9998$): within-state ranking was unreliable; the critic selected the rollout-best candidate in only 5 of 10 states. Even with the rollout verifier, 200 RFT steps produced no sustained improvement in sampled-action return; the open-weight controller used more electricity than the baseline before and after training, and its five-minute predictions remained worse than persistence. GPT-5 predicted transitions far better. Exact rollout scores rank sampled actions but reveal neither next-state effects nor an improvement direction. The unchanged transition errors motivate transition-focused supervised fine-tuning before value-based RFT.
• 中文摘要 多区可变空气体积控制必须在多个连续执行器间平衡热舒适度、室内空气质量和用电。模型预测控制和强化学习被广泛研究，但部署通常需要针对建筑的建模或训练，限制了可扩展性。我们首先测试一个前沿推理模型（即训练为使用额外推理时间计算的大型语言模型）是否能在没有特定建筑训练的情况下，通过文本实现竞争性的VAV控制。在建立该能力后，我们接着测试TD3引导加固微调（RFT）是否能将控制知识转移到可部署的本地开放权重模型中。五个控制器在基于物理的四区模拟器中，在夏季的三个天内进行评估。相较于基于指南36的基线，TD3降低了4.5%的暖通空调用电，同时提升了温度和二氧化碳$_2美元的合规性。在没有针对特定建筑的培训的情况下，GPT-5实现了最大幅度的减少（6.2%），但减少了通风裕度。对于RFT，确定性推送会恢复已保存状态，应用一个候选状态，并按照TD3对每个动作进行评分。对一位有学识的批评者进行审计，发现其近乎完美的跨时间相关性（$r=0.9998$）掩盖了一个失败：州内排名不可靠;《评论家》在10个州中仅选出了推广力最强的候选人。即使配备了推广验证器，200个RFT步骤也未能持续提升采样动作的反馈;开放式重量控制器在训练前后消耗的电量都高于基线，其五分钟预测结果仍低于持续性。GPT-5对转变的预测要好得多。确切的推广分数对采样行动进行排名，但不揭示下一阶段效应或改进方向。不变的转换误差促使在基于价值的RFT之前进行以过渡为中心的监督微调。

Learning Social Robot Navigation By Sensing Human Legs

通过感知人类腿部学习社交机器人导航

• Authors: Alberto Vaglio, Andrea Garulli, Antonio Giannitrapani, Renato Quartullo, Tommaso Van Der Meer
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.27922
• Pdf link: https://arxiv.org/pdf/2607.27922
• Abstract Robots navigating among pedestrians typically sense their surroundings with a 2D LiDAR mounted close to the ground. At that height, the sensor mostly sees moving legs rather than whole people, yet most learning-based navigation methods still treat pedestrians as simple shapes like circles. This paper addresses that gap with CALF (Convolutional Attention for Leg Features), an end-to-end neural architecture that combines convolutional layers, attention, and MLP to interpret leg motion directly from LiDAR scans and produce safe navigation commands. The CALF policy is trained using deep reinforcement learning algorithms within LegNav, a custom lightweight 2D simulator that combines 2D LiDAR ray tracing with a novel pedestrian gait model. The resulting policy is compared against classical and learning-based baselines in terms of navigation performance and social compliance. The approach is validated through real-world experiments via zero-shot deployment on a TurtleBot 4, yielding smooth and socially compliant trajectories. Written in JAX, the LegNav simulator enables the training of a deployment-ready CALF policy in under an hour on a single consumer GPU.
• 中文摘要 在行人中导航的机器人通常通过安装在靠近地面的二维激光雷达（LiDAR）感知周围环境。在那个高度，传感器主要看到的是移动的腿，而非完整的人，但大多数基于学习的导航方法仍然将行人视为简单的圆形。本文通过CALF（腿部特征卷积注意力）来解决这一空白，这是一种端到端神经架构，结合卷积层、注意力和MLP，直接从激光雷达扫描中解读腿部运动并生成安全的导航指令。CALF政策通过LegNav中的深度强化学习算法训练，LegNav是一款定制的轻量级二维模拟器，结合了二维激光雷达光线追踪和一种新型步行模型。所得政策与经典和基于学习的基线进行了导航表现和社会顺从性的比较。该方法通过在TurtleBot 4上零发射的真实实验验证，产生平滑且符合社会需求的轨迹。LegNav 模拟器用 JAX 编写，能够在单个消费级 GPU 上不到一小时内训练出部署准备的 CALF 策略。

AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization

AutoPref：自动发现神经组合优化中的任务特定偏好目标

• Authors: Shengda Gu, Kai Li, Xinyi Ke, Haobo Fu, Yifan Zhang, Jian Cheng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.27953
• Pdf link: https://arxiv.org/pdf/2607.27953
• Abstract Combinatorial optimization problems (COPs) underpin many real-world decisions, but their exponentially large search spaces make high-quality solutions costly to obtain. Neural combinatorial optimization (NCO) learns fast construction policies, typically with reinforcement learning (RL), while preference-based NCO improves sample efficiency by learning from relative solution quality. However, existing preference objectives combine two distinct design choices in manually specified, one-size-fits-all formulations: what learning signal to extract from each solution pair and how to weight each pair relative to the sampled set. We present AutoPref, the first LLM-guided framework for automated preference-objective discovery in NCO. AutoPref factorizes the objective into a pairwise loss program, which defines the learning signal, and a set-aware weighting program, which determines each pair's relative contribution. Their composition forms a unified programmatic objective space containing existing preference objectives as special cases. To make its search tractable, we introduce a staged conditional search strategy with behavioral gates that filter inadmissible programs before short-horizon training and evaluation. Across TSP, CVRP, FFSP, and JSSP, AutoPref consistently outperforms strong hand-designed baselines across problem scales, demonstrating the benefits and scalability of automated objective discovery for NCO.
• 中文摘要 组合优化问题（COPs）支撑着许多现实世界的决策，但其指数级庞大的搜索空间使得获得高质量解变得成本高昂。神经组合优化（NCO）学习快速的构建策略，通常伴随强化学习（RL），而基于偏好的NCO通过从相对解质量中学习来提升样本效率。然而，现有的偏好目标结合了两种不同的设计选择，采用手动指定的一刀切表述：从每对解中提取哪种学习信号，以及如何相对于采样集对每对进行权重。我们介绍AutoPref，这是首个基于LLM引导的NCO自动偏好目标发现框架。AutoPref将目标分解为两对损耗程序，定义学习信号，以及一个集合感知加权程序，确定每对的相对贡献。它们的组成构成了一个统一的程序客观空间，包含现有的偏好目标作为特例。为了使其搜索可操作，我们引入了分阶段的条件搜索策略，通过行为门筛选不可接受程序，然后再进行短视野训练和评估。在TSP、CVRP、FFSP和JSSP等领域，AutoPref在各问题尺度上始终优于强力的手工设计基线，展示了自动化目标发现对士官的优势和可扩展性。

MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperation

MARS-RA：通过多模态比较实现学分分配的排名聚合，实现具象多代理合作中的应用

• Authors: Dawei Wang, Di Zhao, Xinyuan Liu, Marci Chi Ma, Xiaoyang Liu, Chengming Zhou, Gary Ushaw, Richard Davison
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27967
• Pdf link: https://arxiv.org/pdf/2607.27967
• Abstract Credit assignment is a fundamental challenge in cooperative multi-agent reinforcement learning, particularly in embodied AI settings characterized by limited and delayed feedback as well as dynamically changing numbers of active agents. We propose MARS-RA, a framework that reformulates credit assignment as a rank aggregation problem using contribution-based pairwise comparisons among agents generated by large multimodal models. This shift from absolute to relative estimation ensures robustness against noise and dynamic agent participation, converting comparison results into contribution scores for potential-based reward shaping. We provide theoretical justification for the convergence and robustness of the proposed framework, and show that Shapley values can be used as an interpretive reference. Experimental results on challenging tasks of different types indicate that MARS-RA can guide agents toward effective cooperation.
• 中文摘要 学分分配是合作式多智能体强化学习中的一个根本挑战，尤其是在具身人工智能环境中，这些环境具有有限且延迟反馈，且活跃代理数量动态变化。我们提出了MARS-RA框架，该框架通过基于贡献的两两比较，将学分分配重新表述为秩聚合问题，这些方法由大型多模态模型生成的代理间进行贡献比较。从绝对估计向相对估计的转变确保了对噪声和动态代理参与的鲁棒性，将比较结果转化为基于潜在奖励形成的贡献分数。我们为所提出框架的收敛性和鲁棒性提供了理论依据，并证明Shapley值可作为解释参考。不同类型具有挑战性的任务的实验结果表明，MARS-RA能够引导特工实现有效合作。

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

超越二元奖励：强化去学习奖励设计的比较研究

• Authors: Efstratios Zaradoukas, Davide Gabrielli, Bardh Prenkaj, Gjergji Kasneci
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.27968
• Pdf link: https://arxiv.org/pdf/2607.27968
• Abstract Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act. Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem, where models are optimized against verifiable rewards computed directly from their outputs. However, existing methods rely on sparse binary rewards that provide minimal learning signal, indicating only whether forbidden content was avoided, and limiting convergence speed. In this paper, we study how reward design affects unlearning efficiency within the Reinforcement Unlearning (RUL) framework. We introduce a principled reward decomposition framework that decouples verifiability from sparsity, and propose two new reward functions: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic importance. We conduct experiments on the Real World Knowledge Unlearning (RWKU) benchmark, demonstrating that both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to $3\times$ faster and preserving general model utility. Our results show that reward design is a key driver of unlearning efficiency offering a practical path toward scalable and efficient machine unlearning.
• 中文摘要 机器学习去学习旨在选择性地从训练好的语言模型中移除特定知识，而无需进行完全的重新训练，这在GDPR和欧盟人工智能法案等隐私法规下日益必要。近期研究将去学习重新表述为可验证奖励强化学习（RLVR）问题，模型基于直接从输出计算出的可验证奖励进行优化。然而，现有方法依赖稀疏的二元奖励，提供最小的学习信号，仅指示是否避免了禁止内容，并限制了收敛速度。本文研究奖励设计如何影响强化去学习（RUL）框架下的去学习效率。我们引入了一个原则性的奖励分解框架，将可验证性与稀疏性解耦，并提出了两种新的奖励函数：一种指数奖励，基于禁止概念发生次数的分级惩罚;另一种是受PageRank启发的奖励，按语义重要性加权惩罚。我们在现实世界知识解构（RWKU）基准测试上进行了实验，证明这两种奖励都能持续优于二元设置，同时以3美元×1美元的速度达到相似的遗忘性能，同时保持了模型的通用效用。我们的结果表明，奖励设计是实现去学习效率的关键驱动力，为实现可扩展且高效的机器学习提供了切实可行的路径。

TAPO: Transition-Aware Policy Optimization for LLM Agents

TAPO：LLM代理的过渡感知策略优化

• Authors: Cong Li, Peixi Peng, Yisen Zhao, Xinyu Hu, Shudong Liu, Zhan Su, Zhuojian Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.27973
• Pdf link: https://arxiv.org/pdf/2607.27973
• Abstract Recently, Reinforcement Learning (RL) has emerged as a crucial paradigm for the post-training of Large Language Model (LLM) agents. However, existing methods predominantly rely on sparse task rewards for policy optimization, failing to fully exploit another class of inherently dense supervisory signals naturally present during online interaction: environmental feedback following action execution. Recent theoretical studies suggest that generalization in multi-step, goal-oriented tasks hinges on predictive knowledge of environmental consequences. Inspired by this, we propose TAPO: Transition-Aware Policy Optimization for LLM Agents, a unified training framework that alternates between policy optimization and transition supervision. Beyond standard RL updates, TAPO repurposes rollout data to apply action-conditioned next-observation prediction supervision on a shared backbone model. This approach enhances the model's sensitivity to environmental transition dynamics and action consequences while concurrently optimizing the policy. It serves as a computationally lightweight, plug-and-play enhancement module for existing agent RL algorithms, requiring no additional expert data, extra sampling costs, or inference-time overhead. We conduct systematic experiments on WebShop and ALFWorld, integrating foundation models of various scales with different policy optimization algorithms. Empirical results demonstrate that TAPO consistently improves task performance over pure policy optimization baselines.
• 中文摘要 近年来，强化学习（RL）已成为大型语言模型（LLM）代理后训练中的关键范式。然而，现有方法主要依赖稀疏的任务奖励来优化策略，未能充分利用在线交互中自然存在的另一类固有密集监督信号：行动执行后的环境反馈。近期理论研究表明，多步骤目标导向任务的泛化依赖于对环境后果的预测性知识。受此启发，我们提出了TAPO：LLM代理的过渡感知策略优化，这是一个在策略优化和过渡监督之间交替进行的统一训练框架。除了标准的强化学习更新外，TAPO还将推广数据重新利用，应用于共享骨干模型上的动作条件下一次观测预测监督。这种方法增强了模型对环境转型动态和行动后果的敏感性，同时优化了政策。它作为一个计算量轻量、即插即用的增强模块，适用于现有的智能体强化学习算法，无需额外的专家数据、额外的采样成本或推理时间开销。我们在WebShop和ALFWorld上进行系统实验，整合不同规模的基础模型与不同的策略优化算法。实证结果表明，TAPO在任务表现上始终优于纯策略优化基线。

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

通过特权自蒸馏进行对比强化策略优化

• Authors: Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu, Jianing Wang, Linsen Guo, Xiaoyu Li, Xuezhi Cao, Xunliang Cai
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28026
• Pdf link: https://arxiv.org/pdf/2607.28026
• Abstract Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level supervision, it inherently suffers from exposure bias due to the privileged information of the self-teacher. In multi-turn agentic settings, this leads to reasoning route convergence and the loss of clear optimization directions. To tackle these challenges, we introduce Contrastive Reinforced Policy Optimization (CRPO), which reformulates agentic OPSD from a contrastive learning perspective. By leveraging predictive entropy to distinguish between positive positions (reflective exploration) and negative positions (exposure bias), CRPO conducts group-wise contrast to preserve reliable, fine-grained optimization signals. Extensive evaluations across 13 challenging reasoning and deep-search benchmarks demonstrate that CRPO consistently outperforms existing reinforcement learning and self-distillation baselines, significantly enhancing training stability and generalization in long-horizon interactions.
• 中文摘要 大型语言模型（LLMs）的后续训练进展越来越依赖可验证奖励强化学习（RLVR）或策略上自我蒸馏（OPSD）。虽然OPSD提供密集的logit级别监督，但由于自学者拥有特权信息，本质上存在暴露偏差。在多回合智能体环境中，这导致路径收敛推理和明确优化方向的丧失。为应对这些挑战，我们引入了对比强化策略优化（CRPO），从对比学习视角重新表述了代理OPSD。通过利用预测熵区分正向位置（反思性探索）和负位置（暴露偏差），CRPO进行组别对比，以保持可靠且细粒度的优化信号。对13项具有挑战性的推理和深度搜索基准的广泛评估表明，CRPO始终优于现有的强化学习和自蒸馏基线，显著提升了长期相互作用中的训练稳定性和泛化性。

VIG-RL: Learning to Search and Insert for Verified Image Grounding

VIG-RL：学习搜索和插入验证图像接地

• Authors: Qinhan Yu, Jun Guang, Chong Chen, Wentao Zhang
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.28055
• Pdf link: https://arxiv.org/pdf/2607.28055
• Abstract In knowledge-intensive scenarios, providing reliable interleaved text-image responses requires Verified Image Grounding (VIG): the precise integration of retrieved authentic visual evidence. Existing retrieval-augmented frameworks predominantly rely on decoupled, static pipelines, inherently failing to dynamically reason about when external knowledge is required and where visual assets should be contextually inserted. To bridge this gap, we propose VIG-RL, an autonomous agentic framework that formulates the search-selection-insertion workflow as an active decision-making process. Operating within a dynamic ReAct-style loop, VIG-RL is optimized via reinforcement learning, guided by a composite reward system that holistically evaluates the agent's step-by-step tool execution and final multimodal alignment. Extensive evaluations demonstrate that VIG-RL establishes a new state-of-the-art, significantly outperforming existing static baselines.
• 中文摘要 在知识密集型场景中，提供可靠的交错文本-图像响应需要验证图像基础（VIG）：精确整合检索到的真实视觉证据。现有的检索增强框架主要依赖解耦的静态流水线，本质上无法动态推理何时需要外部知识以及视觉资产应在何处上下文插入。为弥合这一空白，我们提出了VIG-RL，一种自主的代理框架，将搜索-选择-插入工作流程构建为主动决策过程。VIG-RL在动态ReAct式循环中运行，通过强化学习进行优化，由复合奖励系统整体评估代理的逐步工具执行和最终多模态对齐。广泛评估表明，VIG-RL确立了一种全新的最先进技术，显著优于现有静态基线。

GVR-Coder: A Visual-Feedback Framework for Structured SVG Generation in Complex Document and Meeting Scenarios

GVR-Coder：一个用于复杂文档和会议场景中结构化SVG生成的可视化反馈框架

• Authors: Yiming Xu, Jihua Kang, Chunsai Du, Qifan Zhang, Wangqiu Zhou, Yiting Wu, Tianqi Li, Qi Song
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.28073
• Pdf link: https://arxiv.org/pdf/2607.28073
• Abstract In demanding professional environments and meeting review scenarios, lengthy text often imposes a high cognitive load. To facilitate efficient information communication, transforming verbose text into logically clear diagrams is essential. Scalable Vector Graphics (SVG) provide an effective representation for this purpose due to their editability and resolution independence. However, current research on Text-to-SVG generation remains hindered by three major challenges: (1) the scarcity of datasets for complex, logic-rich diagrams; (2) the absence of explicit layout priors, which leads to chaotic spatial arrangements; and (3) the lack of fine-grained visual feedback to validate rendered outputs and correct aesthetic defects. To address these challenges, at the data level, we introduce DocMeetSVG-100K, a large-scale SVG dataset tailored for document authoring and meeting review scenarios. At the model level, we propose GVR-Coder, a novel framework designed to generate high-quality logical diagrams from lengthy professional texts. Specifically, we adopt a curriculum-driven rejection sampling fine-tuning to progressively enhance the model's capability in modeling complex structures, while explicitly incorporating layout constraint knowledge during training. In addition, we introduce reinforcement learning from dual rendering feedback, a mechanism that provides implicit feedback through reward signals to jointly optimize structural complexity and visual aesthetics. Furthermore, we design a generate-verify-repair agent loop, which improves generation quality through explicit, fine-grained feedback and targeted refinement. Extensive experiments demonstrate that GVR-Coder outperforms competitive baselines and reliably produces logically coherent and visually appealing diagrams. Code and data are available at this https URL.
• 中文摘要 在要求严格的职业环境和会议复习场景中，冗长的文字往往会带来较高的认知负担。为了促进高效的信息交流，将冗长的文本转化为逻辑清晰的图表至关重要。可缩放矢量图形（SVG）因其可编辑性和分辨率独立性，提供了有效的表示方式。然而，当前关于文本转SVG生成的研究仍受到三大挑战的阻碍：（1）复杂且逻辑丰富的图表数据集稀缺;（2）缺乏显式布局先验，导致空间布局混乱;以及（3）缺乏细致的视觉反馈来验证渲染输出和纠正美学缺陷。为应对这些挑战，在数据层面，我们推出了DocMeetSVG-100K，这是一个面向文档创作和会议审查场景量身定制的大规模SVG数据集。在模型层面，我们提出了GVR-Coder，这是一个新颖框架，旨在从长篇专业文本中生成高质量的逻辑图。具体来说，我们采用了课程驱动的拒绝抽样微调，逐步提升模型在复杂结构建模中的能力，同时在培训中明确纳入布局约束知识。此外，我们还引入了来自双重渲染反馈的强化学习机制，该机制通过奖励信号提供隐性反馈，共同优化结构复杂性和视觉美感。此外，我们设计了生成-验证-修复代理循环，通过显式、细粒度的反馈和有针对性的优化提升生成质量。大量实验表明，GVR-Coder 优于竞争基线，能够可靠地生成逻辑连贯且视觉美观的图表。代码和数据可在此 https URL 获取。

Group-Reflective Self-Distillation for Agentic Reinforcement Learning

群体反思自我蒸馏用于能动强化学习

• Authors: Binbin Zheng, Zijun Xie, Guanqun Zhao, Enlei Gong, Xing Ma, Xiaoliang Fu, Zeyu Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28076
• Pdf link: https://arxiv.org/pdf/2607.28076
• Abstract Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
• 中文摘要 带有可验证奖励的强化学习（RLVR）对于训练大型语言模型代理非常有效。然而，终极奖励仅提供粗略的轨迹级监督，成功行为、反复错误和偶发选择都纠缠在同一结果信号中。现有的能动自蒸馏方法通过自然语言技能丰富稀疏监督，但外部检索或由更强模型从单一轨迹中提取的技能可能与当前经验不匹配，超出策略能力，或仍停留在路径特定性。我们提出集团反思自提炼（GRSD），其指导基于政策自身验证的推广，以能力为对齐且具结果判断力。对于每个提示，策略会反映在政策组中已验证的轨迹，并用停止梯度快照对比成功与失败推广的反映，构建组级特权指导。在该指导的前提下，自学者通过调节基于结果的优势，同时保持验证者确定的学习方向，从而优化轮回级学分分配。跨多个能动环境和模型尺度的实验表明，GRSD始终优于竞争基线，并更有效地推广到未见任务。

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

LEEPS：大型语言模型中高效RLVR的潜在引导探索-利用提示采样

• Authors: Shuang Liang, Haoyang Zhou, Yifan Gong, Guowei Wang, Xiting Wang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.28077
• Pdf link: https://arxiv.org/pdf/2607.28077
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6\% and 3.7\% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at this https URL.
• 中文摘要 带有可验证奖励的强化学习（RLVR）提升了大型语言模型的推理能力，但具有相同推广奖励的提示组会消耗生成预算，而没有有效的学习信号。预先选择提示可以通过在生成前筛选提示来减少浪费。然而，现有的预推广方法难以平衡利用与探索：反复利用历史上有用的提示会缩小培训覆盖范围，而更广泛的探索则可能降低信息提示的比例。为解决这些局限性，我们引入了LEEPS，一种潜在引导探索-利用提示采样器，能够自适应地平衡重复使用先前观察到的信息提示与持续探索不确定提示。LEEPS将候选人划分为“利用”和“探索”两个职位，并根据其近期非平凡的比例灵活分配部署预算。它还利用表示空间邻居和历史推展结果，优先考虑可能产生非零奖励方差的不确定提示，从而使探索更具针对性，无需额外推展。在六个数学推理基准测试中，LEEPS在两个模型尺度上均获得最高平均得分，Qwen2.5-Math-1.5B和7B的较强基线分别提升了2.6%和3.7\%，且在训练过程中通常进步更快。它在三个评估的户外逻辑通用推理基准测试中均获得最高平均分，且每个训练步骤仅增加约2秒的在线采样开销。代码可在此 https URL 访问。

FinSMART: Financial Sentiment Analysis for Algorithmic Trading through Market-Aligned Reinforcement Learning

FinSMART：通过市场对齐强化学习实现算法交易的金融情绪分析

• Authors: Giorgos Iacovides, Wuyang Zhou, Danilo Mandic
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Statistical Finance (q-fin.ST); Trading and Market Microstructure (q-fin.TR)
• Arxiv link: https://arxiv.org/abs/2607.28127
• Pdf link: https://arxiv.org/pdf/2607.28127
• Abstract Recent advances in Generative AI have substantially improved financial sentiment analysis through post-trained financial large language models (LLMs). However, existing approaches remain confined to a market-agnostic, supervised learning paradigm that relies on limited, static and human-annotated datasets, and thus are incapable of adapting to evolving market conditions. To address this limitation, we introduce FinSMART, the first market-aligned reinforcement learning framework for financial sentiment analysis, which directly optimizes sentiment signals using realized market outcomes. To deal with the noisy, non-stationary, and multifactorial nature of financial markets, FinSMART incorporates a signal extraction pipeline that combines market-aware data filtering with a discrete asymmetric trading reward, enabling stable reinforcement learning from economically meaningful market feedback. Experimental results demonstrate that FinSMART significantly outperforms existing state-of-the-art methods in profitability, risk-adjusted performance, and sentiment signal quality, improving cumulative trading returns by 220% over the strongest baseline. Uniquely, the FinSMART framework naturally supports market-aware retraining, at any point in time, by replacing costly manual annotation with newly observed financial articles and their realized market outcomes. Such a retraining strategy enables the model to continuously adapt to changing market dynamics, resulting in consistent performance gains over its static counterpart. These findings demonstrate the practical applicability of market-aligned reinforcement learning and highlight its potential as a next-generation paradigm for developing adaptive financial LLMs.
• 中文摘要 生成式人工智能的最新进展显著提升了通过后训练的金融大型语言模型（LLM）对金融情绪的分析。然而，现有方法仍局限于市场无关的监督学习范式，依赖有限的静态且人工注释的数据集，因此无法适应不断变化的市场环境。为解决这一局限，我们引入了FinSMART，首个市场对齐的金融情绪分析强化学习框架，直接利用市场结果优化情绪信号。为应对金融市场的噪声、非平稳和多因素特性，FinSMART 采用了信号提取流水线，结合了市场感知数据过滤与离散非对称交易奖励，实现从经济有意义的市场反馈中实现稳定的强化学习。实验结果显示，FinSMART在盈利能力、风险调整表现和情绪信号质量方面显著优于现有最先进方法，累计交易回报比最强基线提升了220%。FinSMART框架独特地支持市场感知再培训，随时通过用新观察到的金融文章及其实现的市场结果替代昂贵的人工注释。这种再训练策略使模型能够持续适应不断变化的市场动态，从而在性能上持续优于静态模型。这些发现展示了市场对齐强化学习的实际应用性，并凸显其作为开发自适应金融大型语言模型下一代范式的潜力。

LM-GRASP: Instance-Specific Language Models for Combinatorial Construction via Online Imitation Learning

LM-GRASP：通过在线模仿学习实现组合构建的实例特定语言模型

• Authors: Mohand Mezmaz, Grégoire Danoy
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28135
• Pdf link: https://arxiv.org/pdf/2607.28135
• Abstract Machine learning for combinatorial optimization typically relies on neural constructors trained via reinforcement learning on large offline datasets for a fixed problem class-incurring high pretraining costs and generalizing poorly outside the training distribution. We propose an alternative: a metaheuristic framework that reformulates the randomized constructive phase of GRASP as an online imitation learning task, trained from scratch on each problem instance. A local search procedure acts as an expert oracle, while a decoder-only Transformer serves as the constructive policy. Unlike classical GRASP, which relies on static, myopic heuristic rules based on localized scalar costs, our approach is fully data-driven: the construction policy emerges from high-quality solutions discovered during the search itself, with no problem-specific feature engineering required. We instantiate this as LM-GRASP, a hybrid metaheuristic following an iterative learn-infer-improve cycle, training the policy online via behavioral cloning on a dynamic archive of elite trajectories-no external data or offline pretraining needed. The pipeline interfaces with the domain solely through the objective evaluator used by local search. Evaluated on the Taillard PFSP benchmark (ta51-ta60), the most discriminating block due to half its optima being unknown, LM-GRASP outperforms GPU-GRASP by 28.4 makespan units on average-comparable to the gain from GPU acceleration over sequential execution (27.2 units), though with overlapping standard deviations. This suggests instance-specific, online-trained language models are a promising, practical alternative to hand-engineered constructors, especially for landscapes resistant to classical greedy construction.
• 中文摘要 用于组合优化的机器学习通常依赖通过强化学习在大型离线数据集上训练的神经构造器，针对固定问题类——这会产生高预训练成本，且在训练分布外推广能力较差。我们提出了另一种方案：一种元启发式框架，将GRASP的随机建构阶段重新表述为在线模仿学习任务，从零开始针对每个问题实例进行训练。局部搜索过程充当专家预言机，而仅译码器的变换器则作为构造策略。与依赖基于局部标量成本的静态、短视启发式规则的传统GRASP不同，我们的方法是完全数据驱动的：构建策略源自搜索过程中发现的高质量解，无需针对特定问题的特征工程。我们将此实例化为LM-GRASP，一种混合元启发式，遵循迭代学习-推断-改进循环，通过行为克隆在线训练策略，基于精英轨迹的动态档案——无需外部数据或离线预训练。该流水线仅通过本地搜索所用的目标评估器与域接口。LM-GRASP基于Taillard PFSP基准（ta51-ta60），该区块因其一半最优情况未知而最具辨别性，平均性能优于GPU-GRASP的28.4个工时长，与GPU加速相较于顺序执行的27.2单位相当，尽管标准差会重叠。这表明，实例特定、在线训练的语言模型是手工工程构建器的有前景且实用的替代方案，尤其适用于抗传统贪婪构建的环境。

HARGO: Heterogeneity-Aware Reward-Guided Optimization for RL Post-Training of LLMs on HPC Tasks

HARGO：针对强化学习后训练的高效计算任务中，异质性感知奖励引导优化 LLMs

• Authors: Tiangang Li, Xiangbo Tian
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28301
• Pdf link: https://arxiv.org/pdf/2607.28301
• Abstract Supervised fine-tuning (SFT) can equip large language models (LLMs) with domain knowledge for high-performance computing (HPC) tasks such as data race detection and benchmark question answering. However, knowledge alone does not guarantee task-appropriate behavior: the same SFT model that correctly classifies 88.65\% of C/C++ data race samples produces verbose, imprecise answers to factual queries, with 65.9\% of MLPerf responses exceeding 40 characters. Reinforcement learning (RL) post-training addresses this gap by optimizing for task-specific rewards rather than token-level imitation. Yet HPC tasks exhibit extreme heterogeneity, with binary classification, factual QA, and semantic generation differing by 58x in answer length, spanning three distinct reward distributions, and showing widely varying SFT accuracy. This makes uniform-weight RL methods such as GRPO suboptimal. We propose HARGO, Heterogeneity-Aware Reward-Guided Optimization, which introduces per-response importance weighting via confidence-modulated advantage: computing a discrimination signal from group-level reward contrast and a confidence signal from reference model log-probabilities, then modulating the advantage before computing per-response weights, without requiring task-type labels. Across four HPC tasks and nine methods, HARGO achieves the best performance on all three primary metrics: WinRate 54.62\%, Data Race F1 91.30\%, and PLP Similarity 0.8558. Ablation confirms complementary contributions from both signals. HARGO establishes the best overall alignment quality among compared methods for heterogeneous HPC tasks.
• 中文摘要 监督式微调（SFT）可以为大型语言模型（LLM）配备用于高性能计算（HPC）任务的领域知识，如数据竞赛检测和基准测试问答。然而，仅凭知识并不能保证任务适用行为：同一SFT模型正确分类了88.65%的C/C++数据种族样本，但对事实性查询的回答却冗长且不精确，MLPerf回答中有65.9%超过40个字符。强化学习（RL）后训练通过优化任务特定奖励而非代币级模仿来弥补这一空白。然而，HPC任务表现出极大的异质性，二元分类、事实质询和语义生成的答案长度相差58倍，涵盖三种不同的奖励分布，且SFT准确率差异很大。这使得均匀权重的强化学习方法（如GRPO）不那么优。我们提出了HARGO（异质性感知奖励引导优化），通过置信度调制优势引入了每次响应重要性加权：先计算组级奖励对比的区分信号和参考模型对数概率的置信信号，然后在计算每次响应权重前调制优势，无需任务类型标签。在四项高性能计算任务和九种方法中，HARGO 在三个主要指标上均表现最佳：胜率 54.62%，数据竞赛 F1 91.30% 和 PLP 相似度 0.8558。消融确认了两种信号的互补贡献。HARGO在异构高性能计算任务中，确立了比较方法中最佳的整体比对质量。

Hierarchical Multilevel Monte Carlo for Order-Optimal Neural Actor-Critic in Average-Reward CMDPs

平均奖励CMDP中顺序最优神经演员-批评者的层级多层次蒙特卡洛

• Authors: Ankur Naskar, Vaneet Aggarwal
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28390
• Pdf link: https://arxiv.org/pdf/2607.28390
• Abstract Constrained Markov Decision Processes (CMDPs) provide a natural framework for reinforcement learning in safety-critical applications, where agents maximize long-term reward while satisfying long-term constraints. Although primal-dual actor-critic methods with linear critics are well understood, extending order-optimal convergence guarantees to neural critics in average-reward CMDPs has remained open. The main challenge is a fundamental bias-cost trade-off in neural critic estimation: under Neural Tangent Kernel (NTK) analysis, reducing critic bias substantially increases critic optimization cost, preventing order-optimal convergence in the primal-dual framework. We resolve this bottleneck by introducing a hierarchical Multilevel Monte Carlo (MLMC) neural critic that performs debiasing simultaneously across trajectory sampling and critic optimization. The resulting estimator attains the bias of a long critic optimization run with only logarithmic expected sample cost. Building on this estimator, we develop a primal-dual Natural Actor-Critic algorithm that achieves both an optimality gap and a constraint violation of order $\tilde{O}(T^{-1/2})$. This establishes the first order-optimal convergence guarantees for infinite-horizon average-reward CMDPs with general policy parameterization and neural critics, while eliminating the need to know the underlying mixing time. Our results are novel even in the unconstrained setting.
• 中文摘要 受限马尔可夫决策过程（CMDPs）为安全关键应用中的强化学习提供了自然框架，在这些应用中，智能体在满足长期约束的同时最大化长期奖励。尽管带有线性批评者的原始-对偶actor-critic方法已被充分理解，但在平均奖励CMDP中，向神经批判者扩展序最优收敛保证仍未完全开放。主要挑战在于神经批判者估计中的根本偏置-成本权衡：在神经切核（NTK）分析下，降低批判偏差显著增加批判者优化成本，阻止了原始-对偶框架中的序最优收敛。我们通过引入一个分层多层次蒙特卡洛（MLMC）神经批评器来解决这一瓶颈，该技术同时在轨迹采样和批判者优化中进行偏倚处理。所得估计量在仅有对数期望样本成本的情况下，获得长时间批判优化运行的偏置。基于该估计量，我们开发了一个原始对偶的自然演员-批判算法，既实现了最优性缺口，也实现了阶数为$\tilde{O}（T^{-1/2}））$的约束违背。这为具有一般策略参数化和神经批判的无限视野平均奖励CMDP建立了第一阶最优收敛保证，同时消除了对底层混合时间的了解需求。即使在无约束环境下，我们的结果也具有新颖性。

SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute

SVR：通过联合判决-置信度强化学习进行自适应测试时间计算的自我验证精炼

• Authors: Hongyu Chen, Liang Lin, Guangrun Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.28457
• Pdf link: https://arxiv.org/pdf/2607.28457
• Abstract Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback. We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy. At each turn, the model produces a solution together with a discrete correctness verdict and a confidence score; it retains the current answer only when the verdict is Correct and confidence exceeds a threshold, and otherwise continues refinement using its own self-verification. Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference. SVR is trained with GRPO on fixed-horizon trajectories using rewards that promote solution correctness, calibration-aware self-verification, and stop-ready correct states; adaptive stopping is activated only at inference. On seven mathematical reasoning benchmarks with Qwen3.5-2B, SVR achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average. In the evaluated complete-system comparison, it exceeds standard GRPO, strong multi-turn baselines, and a fixed-budget oracle-guided score-feedback reference while requiring substantially fewer turns than fixed ten-turn inference. These results demonstrate that learned self-verification can serve as an effective internal control signal for answer retention and adaptive test-time compute allocation.
• 中文摘要 测试时间计算的扩展可以提升语言模型推理能力，但统一预算会浪费计算在简单输入上，而验证者引导的细化则依赖外部反馈。我们介绍了自我验证细化（SVR），这是一个无预言机的多回合强化学习框架，学习将自我验证作为计算控制策略。在每个回合，模型都会生成一个解，包括离散的正确性判决和置信度分数;只有当判决正确且置信度超过阈值时，它才保留当前答案，否则通过自身的自我验证继续精炼。地面真实性仅用于构建训练奖励，从未通过精炼提示或推理时被要求暴露于策略之下。SVR通过GRPO训练固定地平线轨迹，奖励促进解正确性、校准感知自我验证和停止准备正确状态;自适应停止仅在推理时激活。在七个数学推理基准测试中，使用Qwen3.5-2B，SVR平均宏观平均精度为0.563，平均推理回合仅2.99次。在评估的完整系统比较中，它超过了标准GRPO，拥有强的多回合基线和固定预算的预言机引导评分反馈参考，且所需的回合数远少于固定十回合推断。这些结果表明，学习到的自我验证可以作为有效的内部控制信号，用于回答保留和自适应测试时间计算分配。

Cybersecurity Detection Classification with Reasoning-enabled Language Models

基于推理驱动的语言模型进行网络安全检测分类

• Authors: Amol Khanna, Manu Nandan, Cristian Viorel Popa, Joan Pujol-Roig, Diana Bolocan, Laura Vasilie, Alexandru Apostu, Chase Helwig, Mihaela Gaman, Michael Brautbar, Edward Raff, Chase Midler, Sven Krasser
• Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.28460
• Pdf link: https://arxiv.org/pdf/2607.28460
• Abstract A major issue in Security Operations Centers (SOCs) is alert fatigue, as the number of detections reported is more than staff can triage in a given day. Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat. We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections by combining automated prompt optimization, self-training, and reinforcement learning with verifiable rewards. We find that CoT reasoning also degrades the label-token probabilities that automated triage relies on, so we separately train a calibrator that reads the full reasoning trace and estimates the probability that the verdict is correct. Our system reaches 82.6% test accuracy and, at the high-confidence operating point that governs automated triage, improves benign recall by 43.0% and malicious recall by 18.3% over a direct-label LLM classifier. We further show that the trained calibrator is necessary - an untrained confidence judge collapses high-confidence recall to zero - and that a finetuned 30B model significantly outperforms frontier general-purpose models, motivating targeted training over scale.
• 中文摘要 安全运营中心（SOC）的一个主要问题是警报疲劳，因为报告的检测数量超过了员工一天内能够分诊的范围。以往的工作会提示或微调大型语言模型（LLMs）直接发出分诊标签，但并未训练它们判断检测是否为真实威胁。我们通过结合自动提示优化、自我训练和强化学习，并结合可验证的奖励，训练一个支持思维链（CoT）推理的分类器，基于真实的、人工标记的Windows端点检测。我们发现CoT推理还会降低自动分诊依赖的标签-标记概率，因此我们单独训练校准器，读取完整的推理轨迹并估算判决正确的概率。我们的系统测试准确率达到82.6%，在自动分诊的高置信度操作点上，良性召回率提升43.0%，恶意召回率提升18.3%，相比直接标签LLM分类器。我们还进一步证明，经过训练的校准器是必要的——未经训练的信心评判会将高置信度回忆归零——而微调的30B模型显著优于前沿通用模型，激励有针对性训练而非大规模训练。

Can Vision-Language Models Reason about AI Edits in Images?

视觉语言模型能否推理AI在图像中的编辑？

• Authors: Darsha Udayanga, Pin-Yu Chen, Payel Das, Qiang Ji
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.28464
• Pdf link: https://arxiv.org/pdf/2607.28464
• Abstract Detection and localization of AI-tampered images are critical for trustworthy AI, yet modern generative models have made such manipulations increasingly difficult to identify. While traditional binary classifiers can detect image tampering, they lack interpretability and generalization. Vision-Language Models (VLMs) offer a promising alternative due to their strong visual understanding and reasoning capabilities; however, existing approaches typically rely on supervised finetuning with curated explanations rather than exploiting their inherent reasoning capabilities. In this work, we investigate whether VLMs can be trained to reason about AI-generated image edits using reinforcement learning (RL) rather than explicit reasoning supervision. Motivated by the success in Group Relative Policy Optimization (GRPO), an RL technique that incentivizes the model to reason by generating thinking traces prior to giving the final answer, we propose a GRPO-based training framework that utilizes simple accuracy and format rewards. Given an input image, the model produces a structured reasoning trace and predicts whether the image has been tampered with. A lightweight segmentation model is then guided by the reasoning output to generate pixel-level localization masks. Experiments across multiple image manipulation datasets demonstrate that our approach achieves competitive detection and localization performance compared to state-of-the-art image forgery detectors, despite requiring substantially weaker supervision. We introduce effective intersection over union (eff-IoU), a unified metric to jointly evaluate detection and localization. These results suggest that reinforcement learning provides an effective and scalable mechanism for teaching VLMs to reason about AI-generated content.
• 中文摘要 检测和定位被AI篡改图像对于可信的AI至关重要，但现代生成模型使此类操作越来越难以识别。虽然传统的二进制分类器可以检测图像篡改，但它们缺乏可解释性和泛化性。视觉语言模型（VLM）因其强大的视觉理解和推理能力，提供了有前景的替代方案;然而，现有方法通常依赖监督式的微调和经过精心策划的解释，而非利用其固有的推理能力。本研究探讨VLM是否可以通过强化学习（RL）而非显式推理监督来训练VLM对AI生成的图像编辑进行推理。受群体相对政策优化（GRPO）成功启发，这是一种通过生成思维痕迹激励模型推理的强化学习技术，我们提出了基于GRPO的训练框架，利用简单的准确性和奖励格式。给定输入图像时，模型生成结构化推理轨迹，并预测图像是否被篡改。然后，轻量级分割模型会根据推理输出生成像素级定位掩码。跨多个图像处理数据集的实验表明，尽管监督强度明显较低，我们的方法在检测和定位性能上仍优于最先进的图像伪造检测器。我们引入了有效交集（eff-IoU），这是一种统一的指标，用于联合评估检测和定位。这些结果表明，强化学习为教授VLM推理AI生成内容提供了一种有效且可扩展的机制。

X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching

X-NavDP：将导航扩散政策推广至新颖行为和具象，采用Q分数重加权匹配

• Authors: Tianyu Yang, Yiming Zeng, Wenzhe Cai, Yuqiang Yang, Jiaqi Peng, Hui Cheng, Jiangmiao Pang, Tai Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.28560
• Pdf link: https://arxiv.org/pdf/2607.28560
• Abstract Pretraining navigation diffusion policies rely on large-scale expert demonstrations. These data are typically generated by a fully-informed oracle planner suited to a single nominal robot. This limits the policy's generalization to diverse embodiments and challenging scenarios (e.g., escaping dead ends or detouring long obstacles) that demand diverse local reactive behaviors with only onboard local observations. Post-training the policy with reinforcement learning (RL) offers a principled remedy. However, previous RL for diffusion approaches lead to only marginal improvements. This is because the intractable likelihood of diffusion policies renders policy gradients unstable in addition to inefficient policy exploration. To address these challenges, we propose a data-efficient diffusion RL post-training framework - GQRM (Group Q-score Reweighted Matching). Our framework introduces two complementary designs: (i) a self-bootstrapped exploration strategy with behavior perturbation that preserves the pretrained policy prior, and (ii) a group Q-score normalization mechanism that computes per-trajectory values on each state for efficient reweighted score matching. By conducting distributed online RL training across heterogeneous embodiments, the resulting fine-tuned policy, X-NavDP, achieves state-of-the-art cross-embodiment visual navigation performance, improving the overall success rate from 61.20% to 84.28% in simulation and 10% to 65% in real-world hard cases. The code and model are publicly available at this https URL.
• 中文摘要 预训练导航扩散策略依赖于大规模专家演示。这些数据通常由一个完全知情的预言机规划师生成，适合单个名义机器人。这限制了该政策对多样化体现和挑战性场景的推广（例如，逃离死胡同或绕行长障碍），这些场景需要多样的局部反应行为，仅需船载局部观察。培训后，强化学习（RL）政策提供了原则性的补救方案。然而，以往的扩散强化学习仅带来有限的改善。这是因为扩散政策的不可解决可能性使政策梯度不稳定，同时降低政策探索效率。为应对这些挑战，我们提出了一个数据高效的扩散强化学习训练后框架——GQRM（组Q分数重加权匹配）。我们的框架引入了两种互补设计：（i）一种自引导探索策略，带有行为扰动，保持预训练策略的先验;（ii）一种组Q分数规范化机制，计算每个状态的每个轨迹值，实现高效的加权分数匹配。通过在异构实例之间进行分布式在线强化学习训练，最终优化的策略X-NavDP实现了最先进的跨身体视觉导航性能，将模拟中的整体成功率从61.20%提升至84.28%，在真实艰难案例中提升至10%至65%。代码和模型在此 https URL 公开。

$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

β$-OPSD：以策略优化为导向，以自我蒸馏训练

• Authors: Jiawei Xu, Minghui Liu, Juzheng Zhang, Tom Goldstein, Furong Huang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28582
• Pdf link: https://arxiv.org/pdf/2607.28582
• Abstract On-policy self-distillation (OPSD) is a promising approach to improve reasoning language models, but it remains brittle in practice: making it work reliably often requires substantial engineering effort. We identify a structural source of this difficulty: vanilla OPSD is precisely the $\beta=1$ member of a broader policy-optimization family, where $\beta$ weights the KL penalty anchoring the student to a reference policy. This equivalence turns $\beta$ from an implicit value fixed at one into a controllable regularization parameter, yielding a more general formulation that trades off proximity to a reference policy against privileged teacher guidance. We introduce $\beta$-OPSD and derive its optimal policy as a geometric interpolation between the reference policy and the privileged teacher. Directly optimizing this objective with reinforcement learning, however, would be costly and high-variance. Rather than optimize the RL objective directly, we turn its closed-form solution into a distillation target. Each value of $\beta$ selects a target along the reference-to-teacher path, which we implement efficiently by mixing their token-level logits. In this way, inexpensive distillation approximates the solution of expensive policy optimization. Return-to-go credit assignment further aligns token updates with the sequence-level objective while retaining the simplicity of OPSD. Experiments on mathematical reasoning benchmarks show that $\beta$-OPSD consistently outperforms vanilla OPSD, improving optimization stability and downstream reasoning performance. Our results provide a principled route from self-distillation to policy optimization and back without sacrificing the efficiency that makes OPSD practical.
• 中文摘要 政策上自提纯（OPSD）是一种有前景的方法来改进推理语言模型，但实际上仍然脆弱：使其可靠地工作通常需要大量工程投入。我们识别出这一困难的结构性来源：普通OPSD正是更广泛政策优化家族中$\beta=1$的成员，其中$\beta$加权了将学生锚定于参考政策的Kle罚款。这种等价性将 $\beta$ 从固定在一的隐式值转变为可控正则化参数，从而产生一种更通用的表述，在参考政策的接近与特权教师指导之间做出权衡。我们引入$\beta$-OPSD，并将其最优策略推导为参考策略与特权教师之间的几何插值。然而，直接通过强化学习优化这一目标将成本高且方差性高。我们不是直接优化强化学习目标，而是将其封闭式解转化为蒸馏靶。每个值 $\beta$ 会选择沿教师引用路径上的目标，我们通过混合它们的标记级 logit 高效实现。这样，廉价的提纯近似了昂贵策略优化的解。退货信用分配进一步使代币更新与序列层级目标保持一致，同时保持OPSD的简洁性。数学推理基准测试的实验显示，$\beta$-OPSD持续优于普通OPSD，提升优化稳定性和下游推理性能。我们的结果提供了一条从自我提炼到策略优化再回到自我提炼的原则性路径，同时不牺牲OPSD实用的效率。

Beacon: Knowing When and How to Perform Agentic Visual Reasoning

信标：了解何时以及如何进行代理视觉推理

• Authors: Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, Yuqi Tang, Qi Zhang, Xinlei Yu, Ruizhe Chen, Tianrun Xu, Yuanxing Zhang, Pengfei Wan, Haotian Wang, Xianghua Ying
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.28595
• Pdf link: https://arxiv.org/pdf/2607.28595
• Abstract The fundamental goal of agentic visual reasoning is to improve the success rate of multimodal large language models (MLLMs) on complex tasks, rather than merely equipping them with a sophisticated yet inefficient reasoning paradigm. In this work, we rethink agentic visual reasoning through two key dimensions of tool use: Mode Adaptiveness (MA) and Tool Effect (TE). Mode Adaptiveness characterizes whether an MLLM can recognize when tools are truly necessary and invoke them accordingly, thereby avoiding unnecessary computational overhead while improving performance on challenging problems that require tool assistance. Tool Effect characterizes the actual impact of tool use: tools should extend the model's capabilities on problems unsolvable through text-only reasoning, while avoiding additional errors on problems that the model can already solve without tools. We conduct a comprehensive analysis to quantify these two properties and empirically reveal that existing agentic visual reasoning models exhibit limited Mode Adaptiveness, while the gains produced by tool use on hard examples are largely offset by the harm introduced on easy examples that the models can already solve. Motivated by these observations, we propose Beacon, a novel agentic visual reasoning model that achieves stronger overall performance, improved Mode Adaptiveness, and genuine tool-induced performance gains. At the core of Beacon are the Necessity-Aware Adaptive Reward and the Hint-Guided Capability Expansion mechanism in the reinforcement learning stage, which respectively encourage adaptive tool invocation based on task necessity and strengthen the model's tool-use capability on the most challenging problems. Extensive experiments across diverse benchmarks demonstrate the strong overall performance of Beacon and its substantial improvements in both Mode Adaptiveness and Tool Effect.
• 中文摘要 智能视觉推理的基本目标是提高多模态大型语言模型（MLLMs）在复杂任务中的成功率，而不仅仅是赋予它们一个复杂但效率较低的推理范式。在本研究中，我们通过工具使用的两个关键维度重新思考代理视觉推理：模式适应性（MA）和工具效应（TE）。模式适应性描述MLLM是否能够识别工具的真正必要性并据此调用，从而避免不必要的计算开销，同时提升在需要工具辅助的复杂问题上的性能。工具效应描述工具使用实际影响：工具应扩展模型在无法通过纯文本推理解决的问题上的能力，同时避免模型在无需工具的情况下解决的问题上出现额外错误。我们进行了全面的分析以量化这两种特性，并实证揭示现有的智能视觉推理模型表现出有限的模式适应性，而工具在困难实例上使用的收益在很大程度上被模型已能解决的简单实例所带来的伤害所抵消。基于这些观察，我们提出了信标，一种新型代理视觉推理模型，能够实现更强的整体性能、改进的模式适应性以及真正的工具驱动性能提升。信标的核心是强化学习阶段的必要性感知自适应奖励和提示引导能力扩展机制，分别鼓励基于任务必要性的自适应工具调用，并增强模型在最具挑战性问题上的工具使用能力。跨越多种基准测试的广泛实验展示了Beacon的强劲整体性能及其在模式适应性和工具效应方面的显著提升。

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

OSReward：为跨平台计算机使用奖励模型实施标准化评估

• Authors: Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding, Nuo Chen, Jialin Cao, Xingdong Gong, Zehao Li, Kaiming Jin, Xinfeng Yuan, Zhoumianze Liu, Jingyang Gong, Zhangyue Yin, Jiahui Gao, Zhiyong Wu, Tianbao Xie, Jianbing Zhang, Ben Kao, Lingpeng Kong
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.28609
• Pdf link: https://arxiv.org/pdf/2607.28609
• Abstract Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at this https URL.
• 中文摘要 计算机使用代理（CUA）正在数字世界中快速发展。CUA轨迹记录了代理人的行为、状态和推理。验证是否完成任务指令是CUA评估、数据整理和强化学习的核心。无论是人工验证器还是人工注释者，都无法大规模提供此类验证，因此该领域越来越依赖视觉语言模型（VLM）来判断CUA轨迹。但一个根本性的问题长期未被审视：这些VLM法官是否足够可靠？为了系统地研究，我们引入了OSReward，一个现实且高质量的基准测试，用于评估VLM评委的CUA轨迹。这些发展轨迹来自多样的代理骨干，在各平台上执行人工验证的指令，然后通过多阶段人工注释严格标注基于真实的判决。在此基础上，我们推导出OSReward-Hard，一个专注于真正困难案例的挑战集，以及OSReward-Multi，用于细粒度效率和对齐评分。迄今为止对VLM裁判最全面的评估发现，即使是最先进的模型也未能达到理想的评判，存在系统性的宽容偏差，将失败的跑动错误地标注为成功。少数足够可靠的模型成本过高，无法大规模运行，而价格合理的开放模型则远远落后。为弥补这一空白，我们构建并发布了OS-Shepherd-100K，这是一个面向CUA社区的推理注释轨迹判断的开放语料库。在该平台上，我们训练OS-Shepherd（9B和35B），这些开放奖励模型提供低成本、稳定且可靠的奖励信号，并以比前沿低30-60%的成本匹配商业评判。大量分析进一步为大规模可靠CUA奖励的设计提供了参考。我们的代码、基准测试、数据集和模型检查点均可在此 https URL 访问。

Keyword: diffusion policy

DexDirect: Direct Kinesthetic Arm Guidance for Efficient Dexterous Demonstration Collection

DexDirect：直接动觉臂引导，高效灵巧示范采集

• Authors: Beom Jun Kim, Shiu-Jen Wang, Jonathan Liu, Alvin Zhu, Quanyou Wang, Hanzhang Fang, Feng Xu, Mingzhang Zhu, Yuchen Cui, Dennis W. Hong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.27784
• Pdf link: https://arxiv.org/pdf/2607.27784
• Abstract Scalable collection of dexterous manipulation demonstrations remains a major bottleneck for robot learning. High-fidelity interfaces often require costly hardware and extensive setup, while low-setup, low cost alternatives tend to provide less precise control and impose greater cognitive workload on operators. We present DexDirect, a direct kinesthetic arm guidance for efficient dexterous demonstration collection. The operator drags a 6-DoF gravity-compensated robot arm directly by a handle, while a single webcam retargets operator's other hand onto a 16 joints 13-DoF dexterous robot hand. User studies suggest DexDirect collects 17.2x and 3.2x more successful demonstrations compared to purely vision (AnyTeleop) and pose-tracking (TeleDex) baselines. An adapted NASA-TLX shows DexDirect greatly reduces mental demand, effort, and frustration, despite raising physical demand. A diffusion policy trained on DexDirect demonstrations reaches a 90% success rate on a cube pick-and-place task. These results suggest that direct kinesthetic arm guidance combined with vision-based hand retargeting provides an efficient low-setup and scalable interface for collecting dexterous manipulation demonstrations
• 中文摘要 灵活操作演示的可扩展集成仍然是机器人学习的主要瓶颈。高保真度接口通常需要昂贵的硬件和广泛的设置，而低成本、低成本的替代方案往往控制精度较低，给操作员带来更大的认知负担。我们介绍DexDirect，一种直接动觉手臂引导，用于高效地进行灵巧示范采集。操作员直接通过手柄拖动6景深的重力补偿机械臂，而单台摄像头则将操作员另一只手重新定位到16关节、13景深的灵巧机器人手上。用户研究显示，DexDirect 收集的成功演示数量是纯视觉（AnyTeleop）和姿势追踪（TeleDex）基线的 17.2 倍和 3.2 倍。一个改编后的NASA-TLX显示，尽管身体需求增加，DexDirect仍大幅降低了精神负担、精力和挫折感。在DexDirect演示中训练的扩散策略在立方体选放任务中成功率达到90%。这些结果表明，直接的动觉手臂引导结合基于视觉的手部再定向，提供了高效、低配置且可扩展的界面，用于收集灵巧的操作演示

FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulation

FA-RDP：一种用于丰富接触操作的频率自适应反应扩散策略

• Authors: Lifeng Zhuo, Wendi Chen, Han Xue, Shirun Tang, Jun Lv, Cewu Lu, Chuan Wen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.28596
• Pdf link: https://arxiv.org/pdf/2607.28596
• Abstract In contact-rich manipulation, action multimodality and reactivity dominate different stages of a single episode. Before contact, multiple trajectories might be equally valid, making it important to preserve diverse action modes. After contact, geometric constraints and force limits narrow the solution space, while successful execution demands rapid responses to force feedback. However, standard diffusion policies use a fixed inference frequency and sampling steps throughout the episode, forcing a fundamental compromise: low-frequency, multi-step sampling better preserves pre-contact multimodality but responds slowly to force feedback, whereas high-frequency sampling improves reactivity but tends to collapse distinct pre-contact modes. To resolve this tradeoff, we present FA-RDP, a frequency-adaptive reactive diffusion policy. A shared multi-frequency visual-force Transformer predicts action chunks at both low and high frequencies, while a learned multimodality indicator dynamically selects multi-step low-frequency sampling before contact and one-step high-frequency sampling as action ambiguity decreases. We further introduce Manifold Consistency Distillation (MCD), which reparameterizes the diffusion network to predict actions on the robot action manifold while retaining DDPM-based residual supervision. Experiments on three contact-rich manipulation tasks show that FA-RDP achieves the highest success rate while preserving diverse pre-contact trajectory modes. Code and videos are available at this https URL.
• 中文摘要 在接触丰富操作中，作用多模态和反应性主导单个发作的不同阶段。接触前，多种弹道可能同样有效，因此保持多样的行动模式非常重要。接触后，几何约束和力极限缩小了解空间，而成功执行则需要对力反馈的快速响应。然而，标准扩散策略在整个事件中使用固定的推断频率和采样步，迫使我们做出根本性妥协：低频多步采样更好地保留接触前多模态，但对力反馈响应缓慢;而高频采样提升反应性，但往往会崩溃不同的接触前模式。为解决这一权衡，我们提出了FA-RDP，一种频率自适应的反应性扩散策略。一个共享的多频视觉力变压器预测低频和高频的动作块，而学习到的多模态指示器则动态选择接触前的多步低频采样和在作用模糊度减少时的一步高频采样。我们进一步引入了流形一致性蒸馏（MCD），该技术重新参数化扩散网络，以预测机器人动作流形的作用，同时保留基于DDPM的残余监督。在三种接触丰富操作任务上的实验表明，FA-RDP在保持多样接触前轨迹模式的同时，成功率最高。代码和视频可在此 https URL 获取。