生成时间: 2026-08-26 16:44:09 (UTC+8); Arxiv 发布时间: 2026-08-26 20:00 EDT (2026-08-27 08:00 UTC+8)

今天共有 27 篇相关文章

Keyword: reinforcement learning

Generating Biomedical Fact-Checking Reports with RL-Enhanced Agentic Search

利用强化学习增强的代理搜索生成生物医学事实核查报告

• Authors: Jiongxiao Wang, Dingli Ma, Chaoqun Ni
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23811
• Pdf link: https://arxiv.org/pdf/2608.23811
• Abstract Automated fact-checking is essential for ensuring the reliability of public health information, yet the biomedical domain poses unique challenges. Validating biomedical claims requires rigorous interpretation of scientific literature, assessment of retrieved evidence, and comprehensive justification toward the conclusion. Although Large Language Models (LLMs) enhanced by Retrieval-Augmented Generation (RAG) and agentic search perform automated fact-checking in a retrieve-then-verify paradigm, current methods still output isolated prediction labels, lacking explanatory depth and offers limited utility for human understanding. To bridge this gap, we introduce an LLM-based agent named BioCheck Agent that generates structured biomedical fact-checking reports with agentic search. Rather than merely outputting supported or refuted labels, our agent synthesizes final conclusions with retrieved evidence and rigorous analysis. To ensure domain-specific accuracy, BioCheck Agent exclusively searches high-quality scientific literature in PubMed, utilizing advanced Boolean search operators. Recognizing that direct prompting often results in hallucinations and low-quality reports, especially for lightweight open-source models, we further propose the Evidence-Grounded Group Relative Policy Optimization (EG-GRPO) to perform reinforcement learning on BioCheck Agent with a task-specific reward that incentivizes advanced search behavior and high-quality evidence retrieval while penalizing hallucinations. Our experimental results show that compared to the base model Qwen3.5-4B, BioCheck Agent with EG-GRPO improves label prediction accuracy on SciFact by 9.95%. Furthermore, it achieves a 3.7% higher evidence quality score and a 19.63% lower evidence hallucination rate, demonstrating its ability to generate biomedical fact-checking reports with improved accuracy and quality.
• 中文摘要 自动化事实核查对于确保公共卫生信息的可靠性至关重要，但生物医学领域存在独特的挑战。验证生物医学主张需要对科学文献进行严格解读，评估所检索的证据，并对结论进行全面的依据。尽管通过检索增强生成（RAG）和代理搜索增强的大型语言模型（LLM）在检索后验证的范式中实现了自动事实核查，但当前方法仍输出孤立的预测标签，缺乏解释深度，且对人类理解的实用性有限。为弥合这一差距，我们推出了基于大型语言模型的代理BioCheck Agent，它通过代理搜索生成结构化生物医学事实核查报告。我们的代理不仅输出支持或被驳斥的标签，而是结合检索到的证据和严谨分析，综合最终结论。为确保领域特定准确性，BioCheck Agent 仅在 PubMed 中搜索高质量科学文献，采用先进的布尔搜索操作符。鉴于直接提示常导致幻觉和低质量报告，尤其是轻量级开源模型，我们进一步提出基于证据的群体相对策略优化（EG-GRPO）对BioCheck代理进行强化学习，并以任务特定奖励激励高级搜索行为和高质量证据检索，同时惩罚幻觉。我们的实验结果显示，与基础模型Qwen3.5-4B相比，带有EG-GRPO的BioCheck Agent在SciFact上的标签预测准确率提升了9.95%。此外，其证据质量评分提高了3.7%，证据幻觉率降低了19.63%，展示了其生成生物医学事实核查报告的准确性和质量提升的能力。

Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency

学习在等待中行动：在推理延迟下通用机器人策略的强化分析

• Authors: Brian Zhu (1), Momen Khalil (1), E Harrison (2), Emanuele Poggi (1), Philipp Schmitt (1), Bernd Kast (1), Philine Meister (1), Pranav Atreya (2), Qiyang Li (2), Finn Ferchau (1), Cesar Colmenero (1), Yash Shahapurkar (1), Gokul Narayanan (1), Melih Erdogan (1), Kai Wurm (1), Georg von Wichert (1), Oier Mees (3 and 4 and 2), Eugen Solowjow (1), Andrew Wagenmaker (2), Sergey Levine (2) ((1) Siemens (2) UC Berkeley (3) Microsoft (4) ETH Zurich)
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.23831
• Pdf link: https://arxiv.org/pdf/2608.23831
• Abstract While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency---which can lead to pauses or jerky movements---can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.
• 中文摘要 虽然强化学习（RL）允许通用机器人策略在部署过程中不断改进，但现代通用策略（如VLAs）的大型模型规模，成为有效强化学习改进的根本障碍。尤其是它们严重的推理延迟---可能导致停顿或动作不稳---可能改变有效环境动态，如果未能正确考虑，会破坏强化学习所依赖的马尔可夫假设，导致标准强化学习算法完全失败。在本研究中，我们引入了一个带有延迟感知的框架——带中间信息的异步强化学习（ARLI），它使基于强化学习的通用策略在推理延迟下得到改进成为可能。我们的框架基于异步推理方法，将动作生成与执行交错以隐藏延迟，并通过提供低延迟的强化学习策略设计解决其与强化学习的不兼容问题，通过两项贡献最大化推理窗口内的反应性：通过包含已提交的动作恢复近马尔可夫结构的状态增强和中期推理观察。我们评估了模拟和现实操作任务的改进方法，发现在标准强化学习完全失效的情况下，它能有效微调推理延迟，甚至在理想化无延迟条件下性能匹敌甚至超越标准强化学习。

AI Finds A Way

人工智能找到出路

• Authors: Aaron Dharna, Cong Lu, Ryan Sullivan, Joel Lehman, Victoria Krakovna, Jeff Clune
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23875
• Pdf link: https://arxiv.org/pdf/2608.23875
• Abstract Artificial Intelligence (AI) algorithms frequently learn creative and unexpected solutions, surprising even expert researchers who develop and study them. They often astonish practitioners by discovering unanticipated behavior, exploiting loopholes in reward signals, or spontaneously uncovering previously unknown scientific phenomena. However, accounts of such unconventional behavior across machine learning are seldom formally documented. This work presents 26 curated firsthand anecdotes from various machine learning subfields representing the work of over 100 researchers. These anecdotes showcase the capability of modern AI systems to circumvent human-imposed design limitations and discover unexpected solutions to the tasks we train them on. Furthermore, these accounts are particularly important for the safety of future AI systems. They illustrate the fundamental challenge of aligning models with human values without diminishing their creativity, so they can make surprising discoveries without producing surprising, potentially harmful outcomes. The paper first details AI achieving superhuman success through reinforcement learning across many challenging domains. However, reward-driven optimization can fail when the model learns to hack an underspecified reward or unarticulated constraint. We then present case studies suggesting that harnessing internet-scale foundation models (FMs) has not resolved these fundamental challenges and, in fact, can supercharge them. Nevertheless, we argue that these same learning dynamics can be harnessed to accelerate scientific discovery. Finally, we hope this work provides a consolidated resource to inform future research and demonstrates that the tendency toward unexpected behaviors is commonplace in modern AI, highlighting the need to anticipate and manage AI's capacity for innovative, yet unpredictable, solutions. (abstract abridged)
• 中文摘要 人工智能（AI）算法经常学习到富有创意且意想不到的解决方案，甚至让开发和研究它们的专家研究者感到惊讶。它们常常通过发现意想不到的行为、利用奖励信号的漏洞，或自发发现此前未知的科学现象而令从业者惊讶。然而，关于机器学习中此类非常规行为的描述很少被正式记录。本书呈现了来自100多名研究者的26个机器学习子领域的精选第一手轶事。这些轶事展示了现代人工智能系统绕过人类设计限制、发现对我们训练任务的意想不到解决方案的能力。此外，这些账户对未来AI系统的安全性尤为重要。他们展示了如何在不削弱创造力的前提下，将模型与人类价值观对齐的根本挑战，从而在不产生意外且潜在有害结果的情况下做出惊人发现。论文首先介绍了人工智能通过强化学习在许多具有挑战性的领域实现超人般的成功。然而，当模型学会破解未明确的奖励或未明确的约束时，奖励驱动优化可能会失败。随后，我们展示了案例研究，表明利用互联网规模的基础模型（FM）并未解决这些根本性挑战，反而可能加剧这些挑战。尽管如此，我们认为这些学习动态同样可以被利用来加速科学发现。最后，我们希望这项工作能为未来研究提供整合资源，并展示现代人工智能中意外行为的倾向是普遍现象，强调了预测和管理人工智能创新但不可预测解决方案能力的必要性。（摘要）

Evolutionary Recurrent Decision Model in Developing Adaptive and Maladaptive Behaviors

适应性与不良适应性行为发展中的进化循环决策模型

• Authors: Andrew Hu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23932
• Pdf link: https://arxiv.org/pdf/2608.23932
• Abstract This study introduces the evolutionarily recurrent decision model (ERDM), a computational reinforcement learning framework designed to examine how evolutionary mismatch, bounded rationality, and satisficing contribute to adaptive and maladaptive behavior. ERDM simulates agents across evolutionary recurrent environments, including threat, prey/goal-pursuits, and alliances. Agents learn through competing rewards abstracted from survival metrics. A validity study under varying adverse childhood experiences demonstrates that distinct adaptive and maladaptive strategies, such as learned helplessness, avoidance, healthy relationships, and aggression, emerge naturally without being hardwired. These results align with empirical literature, showcasing ecological validity. The results suggest that many psychopathology-relevant aspects may be interpreted as bounded cognitive systems operating under modern-ancestral environmental mismatch, positioning ERDM as a key computational cognitive tool that can be extended to other studies.
• 中文摘要 本研究介绍了进化重复性决策模型（ERDM），这是一种计算强化学习框架，旨在探讨进化不匹配、有限理性和满足性如何影响适应性和不良适应行为。ERDM 模拟了进化反复出现环境中的代理，包括威胁、猎物/目标追求和联盟。代理通过从生存指标中抽象出的竞争奖励来学习。一项针对不同逆境童年经历的有效性研究表明，不同的适应性和不良适应策略，如习得性无助、回避、健康关系和攻击性，都是自然而然出现的，无需被硬接线。这些结果与实证文献一致，展示了生态学的有效性。结果表明，许多与精神病理学相关的方面可被解释为在现代祖先环境错配下运作的有限认知系统，使ERDM成为一个可扩展至其他研究的关键计算认知工具。

CoDrift: Compositional Drifting for Offline Reinforcement Learning

CoDrift：离线强化学习中的组合漂移

• Authors: Xiewei Ni, Ruofeng Mei, Xiangyu Xu
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.23939
• Pdf link: https://arxiv.org/pdf/2608.23939
• Abstract Offline reinforcement learning is intrinsically multi-objective: a policy must remain compatible with the behavioral support of a fixed dataset while preferentially selecting high-value actions. We recast these objectives in a common form by viewing each as an action-space motion field that specifies how generated actions should move. This perspective enables heterogeneous learning objectives to be combined directly through field composition. Inspired by drifting models, we propose CoDrift, a compositional framework for one-step generative policy learning. CoDrift combines three objective-level fields into a unified policy field. The conditional field preserves state-dependent behavioral structure, while the marginal field pools actions across states to provide a more stable generative signal in the single-positive-sample regime of continuous-control offline RL. The value field moves generated actions toward higher-value regions. The composed field is absorbed into a stochastic generator that produces an action with a single forward pass at deployment. We evaluate CoDrift on 73 tasks from OGBench and D4RL in both offline and offline-to-online settings. CoDrift compares favorably with state-of-the-art methods and achieves the best average rank in both settings.
• 中文摘要 离线强化学习本质上是多目标的：策略必须与固定数据集的行为支持兼容，同时优先选择高价值动作。我们将这些目标重新定义为一种通用形式，将每个目标视为一个动作空间运动场，指定生成动作的移动方式。这种视角使得异质学习目标能够通过场域组合直接结合起来。受漂移模型启发，我们提出了CoDrift，一种用于一步生成式政策学习的组合框架。CoDrift 将三个目标层级字段合并为统一的策略字段。条件场保持状态依赖的行为结构，而边际场则跨状态池化行动，在连续对照离线强化学习的单正样本环境中提供更稳定的生成信号。价值字段会将生成的动作移动到更高价值区域。合成后的场被吸收到随机发生器中，该生成子在部署时只需一次前向传递即可产生作用。我们在离线和离线转在线环境中，对OGBench和D4RL的73项任务进行了CoDrift评估。CoDrift在两种方法中均表现优异，均获得最佳平均排名。

Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignment

算法影响揭示了隐藏的社会选择结构——一致性

• Authors: Zachary Wojtowicz, Michelle Si, Finale Doshi-Velez, Ariel Procaccia
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24046
• Pdf link: https://arxiv.org/pdf/2608.24046
• Abstract When an AI algorithm makes decisions that affect more than one person, aligning it becomes a problem of social choice: how should people's divergent preferences about system behavior be reconciled and aggregated into a single coherent model? The standard approach to aligning frontier AI models$\unicode{x2013}$reinforcement learning from human feedback$\unicode{x2013}$largely sidesteps this question and has poor social choice guarantees. However, it remains unclear what alternative should replace it. We show that, by focusing directly on an algorithm's welfare consequences, the alignment problem can be reformulated as linear optimization over a convex impact space, which makes it amenable to the standard toolkit of welfare economics and mechanism design. This reformulation clarifies how alignment protocols translate into welfare consequences and, conversely, how a social planner's desired constraints on welfare consequences can be translated back into alignment protocols. We apply this transformation to show that voting-by-issues and random-dictatorship mechanisms are strategyproof and unanimous. Demonstrating the reverse direction, we also apply the impact representation to derive a family of alignment protocols that maximize utilitarian social welfare subject to various social desiderata, such as bounds on individual or group harm. We illustrate the welfare implications of these alignment protocols empirically using real human preferences over kidney allocation, charitable food distribution, LLM responses, and trolley problems.
• 中文摘要 当人工智能算法做出影响多个人的决策时，协调它就成为社会选择的问题：人们对系统行为的不同偏好应如何调和和并汇总成一个连贯的模型？标准的前沿AI模型对齐方法$\unicode{x2013}$reinforcement从人类反馈中学习$\unicode{x2013}$largely回避了这个问题，且社会选择的保障较差。然而，目前尚不清楚应由何种替代方案取代。我们表明，通过直接关注算法的福利后果，比对问题可以重新表述为凸影响空间上的线性优化，使其适合纳入福利经济学和机制设计的标准工具包。这一重新表述阐明了对齐协议如何转化为福利后果，反过来也说明了社会规划者期望的福利后果约束如何被转化回对齐协议。我们应用这一变换来证明按议题投票和随机独裁机制是策略可行且一致通过的。反向演示，我们还应用影响表征推导出一系列符合各种社会需求（如个人或群体伤害界限）的功利社会福利的对齐协议。我们通过实证地展示了这些对齐方案的福利影响，具体参考了人类对肾脏分配、慈善食物分配、LLM反应和推车问题的偏好。

Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments

Android GUI 代理对运行时异常有强健性吗？AnTrap：在动态对抗环境中评估代理

• Authors: Guo Gan, Yilun Zhao, Cong Chen, Jinbiao Wei, Tingyu Song, Zheyuan Yang, Lin Fu, Hong Zhou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24099
• Pdf link: https://arxiv.org/pdf/2608.24099
• Abstract GUI agents often encounter dynamic anomalies when deployed on Android devices, from unexpected pop-ups to action misuse, yet existing benchmarks lack systematic evaluation of agent robustness against runtime anomalies. We introduce AnTrap, a comprehensive benchmark that injects dynamic perturbations into agent execution trajectories. We propose a taxonomy organizing real-world anomalies into four layers (State, Thinking, Action and Round) with ten fine-grained subcategories, and develop a construction pipeline that preserves task solvability while introducing realistic adversarial conditions. Evaluating 16 leading GUI models, we reveal universal vulnerability to dynamic anomalies, with even the strongest models suffering significant performance degradation. Furthermore, we conduct GRPO training in both original and adversarial environments to validate our benchmark, separating environment-learnable anomalies from reasoning-bottlenecked ones. Our findings show that while single-step traps at state and action layers are largely addressable through adversarial reinforcement learning, deep contextual traps, like state deadlock, expose intrinsic limitations that cannot be resolved by training in environments with traps alone.
• 中文摘要 GUI 代理在部署于安卓设备上时常常遇到动态异常，从意外弹窗到动作滥用，但现有基准测试缺乏对代理对运行时异常的稳健性系统的系统评估。我们介绍AnTrap，一个综合基准测试，将动态扰动注入代理执行轨迹中。我们提出了一种分类法，将现实世界的异常分为四层（状态、思维、行动和圆周），并设有十个细粒度子类别，并开发了一个既保持任务可解性又引入现实对抗条件的构建流程。通过评估16个领先的图形界面模型，我们发现普遍存在动态异常的脆弱性，即使是最强的模型也存在显著的性能下降。此外，我们在原始和对抗环境中进行GRPO培训，以验证基准，区分环境中可学习的异常与推理瓶颈异常。我们的发现表明，虽然状态层和动作层的单步陷阱大多可以通过对抗强化学习解决，但深度上下文陷阱，如状态死锁，暴露出仅靠陷阱环境中训练无法解决的内在局限。

AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL

展望：环境增强蒸馏的适应性后见明，适用于能动强化学习

• Authors: Xiaolong Jin, Dingmin Wang, Vijay Lingam, Varun Kumar
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24114
• Pdf link: https://arxiv.org/pdf/2608.24114
• Abstract Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.
• 中文摘要 用强化学习训练多回合LLM代理通常依赖轨迹级奖励，这些奖励为每一步分配统一优势，无法识别哪些决策导致了成功或失败。自蒸馏方法可以通过增强学习提供特权信息，提供更细致的监督。然而，现有方法通常以无法区分的方式对每一步应用同类型的特权信息，忽视了一个关键的不对称性：常规步骤几乎不需要额外指导，而关键错误步骤则需要环境反馈无法提供的纠正方向。我们提出了AHEAD，这是一个步进感知框架，将不同的监督来源匹配到不同的步数类型。教师在所有步骤中接收环境反馈，作为接地的密集信号，同时还会收到LLM生成的错误步骤纠正提示，以弥补环境反馈所缺乏的方向。该方法对标准GRPO算法的修改极小。在ALFWorld、WebShop和基于搜索的质量保证中，以及三个模型尺度，AHEAD 提高了任务成功率（ALFWorld 上提升了+13.3分，WebShop 上比GRPO提升了70亿美元，提升了+11.0分），在更少的训练步骤内达到了一定的成功率，并且在比仅结果的强化学习和之前的自我蒸馏基线更紧密的交互预算内完成任务。

Robust Code RL via Faulty-Code-Driven Test case Synthesis and Dense Reward Shaping

通过错误代码驱动的测试用例综合和密集奖励塑造实现强健代码强化学习

• Authors: Yiwen Zhang, Xiaodong Yan, Zhenyu Huang, Deng Zhao, Liang Jiang, Qing Cui, Zujie Wen, Zhiqiang Zhang, Jun Zhou
• Subjects: Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2608.24135
• Pdf link: https://arxiv.org/pdf/2608.24135
• Abstract Reinforcement learning from verifiable rewards (RLVR) has emerged as a pivotal technique for enhancing the code generation capabilities of Large Language Models (LLMs). However, the efficacy of RLVR in coding implementations is fundamentally limited by the comprehensiveness of test cases, because insufficient test coverage in code validation often causes false positives, further leading to reward hacking and policy degradation. To mitigate the reward bias stemming from the suboptimal quality of current automated generation methods, we propose the RobustTests framework, which introduces a faulty-code-driven test case synthesis strategy that leverages "near correct" faulty codes to guide the model in precisely capturing latent logical discrepancies and further integrates validator agents with behavioral feature clustering to facilitate the granular filtering of invalid and redundant test cases. To address false negatives caused by inherent hallucination noise in synthetic test cases, RobustTests also incorporates a stepwise dense reward function based on pass rates, bolstering training robustness through fine-grained feedback. By employing this pipeline, we construct a high-quality dataset that augmented the test cases in CodeContests, encompassing a broader spectrum of faulty code scenarios and significantly enhances diagnostic utility. Experimental results demonstrate that, by leveraging a moderately challenging subset of problems from CodeContests for training, RL fine-tuning of Qwen3-32B via RobustTests achieves an absolute 3% performance gain on the LiveCodeBench benchmark compared to baseline methods, confirming the effectiveness of the RobustTests framework in advancing the code generation proficiency of LLMs.
• 中文摘要 可验证奖励强化学习（RLVR）已成为提升大型语言模型（LLM）代码生成能力的关键技术。然而，RLVR在编码实现中的有效性受限于测试用例的全面性，因为代码验证中测试覆盖不足常导致误报，进一步导致奖励被黑和策略降级。为缓解当前自动生成方法质量不佳带来的奖励偏差，我们提出了RobustTests框架，该框架引入了一种基于错误代码的测试用例综合策略，利用“接近正确”的错误代码指导模型精确捕捉潜在逻辑差异，并进一步将验证代理与行为特征聚类整合，促进对无效和冗余测试用例的细致过滤。为了解决合成测试案例中固有幻觉噪声引起的假阴性，RobustTesting还采用了基于通过率的逐步密集奖励函数，通过细粒度反馈增强训练的鲁棒性。通过采用该流程，我们构建了一个高质量数据集，丰富了CodeContests的测试用例，涵盖了更广泛的错误代码场景，显著提升了诊断效用。实验结果表明，通过利用CodeContests中中一些中等难度的问题进行训练，通过RobustTests对Qwen3-32B进行强化学习微调，在LiveCodeBench基准测试中相比基线方法实现了绝对3%的性能提升，证实了RobustTests框架在提升LLM代码生成熟练度方面的有效性。

Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation

低方差在线策略评估的稳健数据收集策略学习

• Authors: Claire Chen, Shuze Daniel Liu, Licheng Luo, Rohan Chandra, Nan Jiang, Shangtong Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2608.24146
• Pdf link: https://arxiv.org/pdf/2608.24146
• Abstract In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not account for uncertainties in the transition functions. In practice, simulator transitions often differ from the real world due to modeling errors or approximation limitations. As a result, behavior policies trained in simulation may still yield high variance when deployed in real environments, leading to costly reliance on real-world evaluation samples. In this work, we propose a double-loop gradient-based algorithm for learning behavior policies that are both efficient and robust to transition uncertainty. Theoretically, we derive novel transition-variance gradient expressions and establish global convergence guarantees for the algorithm. Numerically, we demonstrate that our method is less sensitive to transition perturbations than existing approaches, providing supportive evidence for its practical utility.
• 中文摘要 在强化学习策略评估中，经典的策略中方法在估计策略绩效时常存在较高的方差性。为缓解这一问题，有人提出行为政策搜索以学习针对减少在线评估方差的数据收集策略。然而，这些方法并未考虑转移函数中的不确定性。实际上，模拟器的转变常因建模错误或近似限制而与现实世界不同。因此，在模拟中训练的行为策略在实际环境中部署时仍可能产生较高的方差，导致对真实世界评估样本的依赖成本高昂。本研究提出一种基于双环梯度的算法，用于学习行为策略，既高效又能抵抗转移不确定性。理论上，我们推导出新的过渡-方差梯度表达式，并为算法建立了全局收敛保证。从数值上看，我们证明了该方法对跃迁扰动的敏感度低于现有方法，这为其实际效用提供了有力证据。

Task-Adaptive Rubrics for GUI Reward Modeling

GUI奖励建模的任务自适应评分标准

• Authors: Tao Xiong, Xavier Hu, Wenkai Wang, Qinzhuo Wu, Changqiao Wu, Pengzhi Gao, Wei Liu, Jian Luan, Shengyu Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24174
• Pdf link: https://arxiv.org/pdf/2608.24174
• Abstract Recent studies on GUI agents have increasingly focused on outcome reward modeling, which assigns outcome rewards by judging whether an executed trajectory satisfies the success criteria implied by the user instruction. Existing GUI reward verifiers, however, often under-specify how these criteria should be constructed for each task instance. Whether using generic rubric structures or implicit model reasoning, their judging criteria are not sufficiently task-adaptive: they can transfer checks across tasks, overlook concrete constraints in the current instruction, or become overly strict by enforcing unstated requirements. To address this limitation, we propose AdaptRubric, a Coarse-to-Fine Rubrics Framework that constructs task-adaptive judging criteria through a category-level coarse stage and an instance-level fine stage. AdaptRubric performs category-level coarse rubric retrieval by routing the instruction to a GUI task family and retrieving reusable task-family criteria, then conducts instance-level fine rubric generation to surface compact cues for concrete values, scopes, and constraints in the current instruction. Across offline reward evaluation and online reinforcement learning optimization, AdaptRubric consistently outperforms prior reward agents, improving F1 by 3.6 points over the baseline average under a matched image budget and yielding a 4.23-point task-success gain.
• 中文摘要 近期关于图形界面代理的研究越来越多地聚焦于结果奖励建模，即通过判断执行的轨迹是否满足用户指令所暗示的成功标准来分配结果奖励。然而，现有的GUI奖励验证器往往对每个任务实例应如何构建这些标准的规定不够明确。无论是使用通用的评分标准结构还是隐式模型推理，它们的判断标准都不够适应任务：它们可能在任务间转移检查，忽视当前指令中的具体约束，或通过强制执行未明说的要求而变得过于严格。为解决这一限制，我们提出了AdaptRubric，一种粗到细评分标准框架，通过类别级粗略阶段和实例级细分阶段构建任务适应性判断标准。AdaptRubric通过将指令路由到图形用户界面任务族并检索可重用的任务族标准，执行类别级粗略评分标准检索，然后进行实例级细评分规，以呈现当前指令中具体值、范围和约束的紧凑线索。在离线奖励评估和在线强化学习优化中，AdaptRubric持续优于以往的奖励代理，在匹配图像预算下，F1比基线平均提升3.6分，任务成功率提升4.23分。

CARO: Contact-Agnostic Residual Observation for Zero-Shot Robust Quadruped Locomotion

CARO：零射强健四足行走的接触无关残留观测

• Authors: Zihan Yang, Shixuan Han, Kexin Guo, Xiang Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.24217
• Pdf link: https://arxiv.org/pdf/2608.24217
• Abstract We propose CARO, a contact-agnostic residual observation framework for policy adaptation. CARO embeds a fixed-base Euler--Lagrange model into the reinforcement learning control loop and constructs a torque-level residual observation without requiring torque sensors, explicit contact estimation, or vision-based measurements of the floating-base position and linear velocity. A disturbance observer extracts a structured signal representing dynamics mismatch, while the policy learns to exploit this feedback for online adaptation. CARO is trained under the same terrain, command, and domain-randomization conditions as the nominal policy, without specialized disturbance curricula or additional adaptation supervision. Nevertheless, it achieves substantially improved zero-shot robustness in simulation and sim-to-real transfer tasks involving out-of-distribution payloads, center-of-mass shifts, terrain geometries, abrupt dynamics changes, and elevated-platform landings.
• 中文摘要 我们提出了CARO，一种与接触无关的残余观察框架，用于政策调整。CARO将固定基欧拉-拉格朗日模型嵌入强化学习控制环路，构建一个扭矩级残差观测，无需扭矩传感器、显式接触估计或基于视觉的浮动基准位置和线速度测量。扰动观察者提取代表动态不匹配的结构化信号，而策略则学习利用该反馈进行在线调适。CARO在与名义政策相同的地形、指挥和领域随机化条件下接受培训，没有专门的干扰课程或额外的适应监督。尽管如此，它在模拟和模拟到实物转移任务中实现了显著提升的零发射鲁棒性，这些任务涉及非分布有效载荷、质心转移、地形几何形状、突变动力学变化和高平台着陆。

RePolicy: Reinforcement Learning for Safety-Policy Invocation in Agent Safeguards

RePolicy：代理保障中安全-策略调用的强化学习

• Authors: Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.24275
• Pdf link: https://arxiv.org/pdf/2608.24275
• Abstract Safeguarding language model agents requires assessing complete execution trajectories under context-dependent safety policies. Existing policy-aware safeguards mainly rely on prompting or supervised fine-tuning, limiting their ability to adapt to unseen trajectories and changing policy contexts. We propose RePolicy, an agent safeguard that learns safety-policy invocation through reinforcement learning. Given an agent trajectory and a dynamic policy library, RePolicy invokes the applicable policy and uses its content to produce a policy-grounded rationale and safety judgment. We construct PolicyTraj-20K to support supervised initialization, followed by GRPO with verifiable rewards and policy-context perturbation. Experiments across six agent safety benchmarks show that RePolicy achieves strong overall safety-detection performance and robust policy invocation under varying policy contexts.
• 中文摘要 保护语言模型代理需要在上下文相关安全策略下评估完整的执行轨迹。现有的政策意识保障主要依赖于提示或监督的微调，限制了它们适应未知轨迹和政策环境变化的能力。我们提出了RePolicy，一种通过强化学习学习安全策略调用的代理保障。在给定代理人轨迹和动态政策库的情况下，RePolicy 调用适用政策，并利用其内容生成基于政策的理由和安全判断。我们构建PolicyTraj-20K以支持监督初始化，随后构建具有可验证奖励和策略上下文扰动的GRPO。在六个代理安全基准测试中的实验表明，RePolicy 在不同策略环境下实现了强劲的整体安全检测性能和稳健的策略调用。

Contrastive Branch Policy Optimization

对比分支策略优化

• Authors: Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24300
• Pdf link: https://arxiv.org/pdf/2608.24300
• Abstract Reinforcement learning with verifiable rewards (RLVR) enables language models to learn multi-turn interaction with external tools, yet its sparse outcome rewards provide no signal for identifying which intermediate decisions are responsible for success. Branch sampling induces local comparisons among alternative continuations, but existing methods tend to conflate two distinct problems: allocating a fixed rollout budget and translating branch outcomes into token-level credit. We introduce Contrastive Branch Policy Optimization (CBPO), which disentangles these two problems and assigns a dedicated mechanism to each. Generation entropy screens candidate branch positions across the entire response, while path-level and node-level decay distribute a fixed budget across trajectories and positions to prevent exploration from collapsing onto a few paths or adjacent tokens. A parent trajectory together with the branches that share an identical token prefix forms an exact-prefix group, and the reward variation within this controlled group defines the Contrastive Branch Value (CBV), an outcome-based estimate of local decision sensitivity that rescales continuation advantages without altering their sign. When multiple nodes are selected along the same trajectory, CBPO partitions it into non-overlapping credit segments, thereby avoiding duplicated gradients on shared tokens. Requiring only outcome rewards and no process-level annotation, CBPO provides a practical solution for fine-grained credit assignment in tool-integrated agent training. Extensive experiments on ten benchmarks, including five for mathematical reasoning and five for knowledge-intensive search, show that CBPO consistently outperforms state-of-the-art policy-optimization and branch-based methods, attaining the highest macro-average accuracy in both domains and across two model scales.
• 中文摘要 带有可验证奖励的强化学习（RLVR）使语言模型能够学习与外部工具的多回合交互，但其稀疏的结果奖励并未提供识别哪些中间决策是成功的关键信号。分支抽样可以引发不同延续方案之间的局部比较，但现有方法往往将两个截然不同的问题混为一谈：分配固定的推广预算和将分支成果转化为代币级信用。我们介绍了对比分支策略优化（CBPO），它将这两个问题分开，并为每个问题分配专用机制。生成熵筛查候选分支位置在整个响应中，而路径级和节点级衰减则在轨迹和位置上分配固定预算，防止探索崩溃到少数路径或相邻标记上。父脉络与拥有相同标记前缀的分支共同构成一个精确前缀组，该受控组内的奖励变异定义了对比分支值（CBV），这是一种基于结果的局部决策敏感性估计，可在不改变其符号的情况下重新量度延续优势。当沿同一轨迹选择多个节点时，CBPO会将其划分为不重叠的信用段，从而避免共享代币上的重复梯度。CBPO仅要求结果奖励，无需流程级注释，为工具集成代理培训中的细粒度学分分配提供了实用解决方案。对十个基准测试的广泛实验，包括五个数学推理基准和五个知识密集型搜索基准，显示CBPO持续优于最先进的政策优化和分支方法，在两个领域及两个模型尺度上均获得最高的宏观平均准确率。

FARCA: Fact-Aligned Reliability-Aware Credit Assignment for Reinforcement Learning with Factual Supervision

FARCA：事实对齐、可靠性意识、具事实监督的强化学习学分作业

• Authors: Qiming Xie, Wenjie Zheng, Xiangqing Shen, Rui Xia
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24350
• Pdf link: https://arxiv.org/pdf/2608.24350
• Abstract To reduce the hallucination risk caused by outcome-driven rewards in large language models trained through reinforcement learning with verifiable rewards, existing mitigation approaches introduce process-level factual supervision. However, due to coarse-grained aggregation of factual signals and the lack of reliability assessment for these signals, they create a mismatch between fact verification and policy updates. We term this noisy factual credit assignment and decompose it into two aspects: credit localization ambiguity and credit reliability ambiguity. To address these issues, we propose FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a policy optimization framework that transforms factual supervision into localized, reliability-weighted token-level training signals. FARCA achieves fine-grained credit localization by aligning the granularity of fact verification with that of policy updates. It further introduces counterfactual evidence attribution, which uses the dependence of a factual judgment on key evidence as an empirical proxy for verification reliability to compute reliability weights. These weights modulate factual rewards and local policy advantages, reducing the influence of potentially unreliable signals on policy optimization. Experiments across different models and multiple factual reasoning benchmarks show that FARCA significantly improves model factuality while preserving general reasoning capabilities.
• 中文摘要 为了降低通过强化学习训练并带有可验证奖励的大型语言模型中结果驱动奖励带来的幻觉风险，现有的缓解方法引入了过程层面的事实监督。然而，由于事实信号的粗粒度聚合以及缺乏对这些信号的可靠性评估，它们在事实验证与政策更新之间造成了不匹配。我们将这种杂乱的事实信用分配称为“信贷分配”，并将其分解为两个方面：信贷本地化模糊性和信贷可靠性模糊性。为解决这些问题，我们提出了FARCA（事实对齐可靠性感知信用分配），这是一种策略优化框架，将事实监督转化为局部化、基于可靠性加权的令牌级训练信号。FARCA通过将事实验证的粒度与政策更新的粒度对齐，实现了细致的信贷本地化。它进一步引入了反事实证据归属法，利用事实判断对关键证据的依赖作为验证可靠性的实证代理，计算可靠性权重。这些权重调节了事实奖励和地方政策优势，减少了潜在不可靠信号对政策优化的影响。跨不同模型的实验和多个事实推理基准表明，FARCA显著提升了模型的事实性，同时保持了一般推理能力。

Reinforcement Learning-Guided Evolutionary Policy Optimization for Preference-Adjustable Heterogeneous Agile Earth Observation Satellite Scheduling

强化学习引导的进化策略优化，适用于偏好可调异构敏捷地球观测卫星调度

• Authors: He Wang, Junyu Wu, Hui Li, Yanjie Song, Witold Pedrycz, Liang Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24470
• Pdf link: https://arxiv.org/pdf/2608.24470
• Abstract Heterogeneous agile Earth observation satellite (AEOS) scheduling requires task selection, satellite assignment, and observation sequencing under satellite-dependent visibility windows, attitude maneuvering requirements, energy consumption, and onboard storage constraints. Since satellites differ in orbital access, maneuvering capability, and payload resources, the same task may have different feasible windows, transition costs, and resource-consumption patterns on different platforms, which increases the difficulty of unified modeling and efficient optimization. To address this problem, this paper proposes an evolutionary policy optimization framework for heterogeneous AEOS scheduling with preference-adjustable weighted objectives. In the modeling layer, assignment-based indirect encoding is combined with decoder-based equivalent-cost evaluation to retain satellite-dependent constraints while integrating task gain, energy saving, and load balance into an interpretable scalar utility. In the optimization layer, schedule decoding, population-based search, and online actor-critic operator control are decoupled, so that reinforcement learning selects high-level search operators rather than constructing schedules directly. Based on this framework, a reinforcement-learning-assisted operator-selection memetic evolutionary algorithm (RLOSMEA) is developed to coordinate global exploration, feasibility recovery, and local refinement under a limited function-evaluation budget. Experiments on different heterogeneous AEOS scenarios show that RLOSMEA achieves higher overall weighted utility and more stable convergence than representative metaheuristic baselines. Sensitivity and learning-behavior analyses further confirm the robustness of the proposed method and the effectiveness of reinforcement-learning-guided operator selection.
• 中文摘要 异构敏捷地球观测卫星（AEOS）调度需要在卫星能见度窗口、姿态机动要求、能耗和机载存储限制下进行任务选择、卫星分配和观测序列。由于卫星在轨道访问、机动能力和有效载荷资源上存在差异，同一任务在不同平台上可能有不同的可行窗口、过渡成本和资源消耗模式，这增加了统一建模和高效优化的难度。为解决该问题，本文提出了一种针对异质AEOS调度、偏好可调加权目标的进化策略优化框架。在建模层中，基于赋值的间接编码与基于解码器的等效成本评估相结合，保留卫星依赖约束，同时将任务增益、节能和负载均衡整合为可解释的标量效用。在优化层，调度解码、基于群体的搜索和在线演员-批评操作符控制被解耦，强化学习选择高级搜索运算符，而非直接构建调度。基于该框架，开发了一种强化学习辅助操作员选择模因进化算法（RLOSMEA），用于在有限的功能评估预算下协调全局探索、可行性恢复和局部细化。在不同异构AEOS场景上的实验表明，RLOSMEA比代表性的元启发式基线实现更高的整体加权效用和更稳定的收敛。敏感性和学习行为分析进一步证实了所提方法的稳健性以及强化学习引导算子选择的有效性。

WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation

WarpSAC：通过重新思考探索与开发，迈向可扩展的非政策强化学习巅峰

• Authors: Zihao Wu, Hongyao Tang, Yi Ma, Huizhong Song, Pengyi Li, Yifu Yuan, Fei Ni, Jinyi Liu, Wei Wei, Jianrong Wang, Yan Zheng, Jianye Hao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.24479
• Pdf link: https://arxiv.org/pdf/2608.24479
• Abstract Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for data-limited replay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent: parameter normalization helps with narrow replay coverage but restricts value fitting when data are abundant, while clipped double-Q can be relaxed in high-throughput manipulation. Age-biased replay weighting improves learning efficiency across regimes, especially with limited network capacity. Based on these findings, we propose WarpSAC, a regime-aware family of off-policy RL algorithms. WarpSAC uses Sample Weight Decay for efficient exploitation and provides two variants: WarpSAC-L (Norm ON, clipped double-Q) for data-limited CPU-scale training, and WarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training. WarpSAC improves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% faster sim-to-real deployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.
• 中文摘要 大规模并行仿真改变了非策略强化学习（RL）训练的数据环境，挑战了为数据限制重放设计的稳定器。通过八个基准家族的受控实验，我们表明这些稳定器是数据体系依赖的：参数归一化有助于窄重放覆盖，但在数据丰富时限制值拟合，而截断双Q则可在高通量操作中放松。基于年龄的回放加权能提升跨体制的学习效率，尤其是在网络容量有限的情况下。基于这些发现，我们提出了WarpSAC，一种体制感知型的非政策强化学习算法家族。WarpSAC 采用样本权重衰减以实现高效利用，并提供两种变体：WarpSAC-L（标准开启，截波双 Q）用于数据限制的 CPU 规模训练，以及 WarpSAC-A（规范关闭，单 Q）用于数据丰富的 GPU 并行训练。WarpSAC 在九种 CPU 级别环境中，比 FlashSAC 提升了 4.5% 的归一化分数步 AUC，在十四种 GPU 并行环境中提升了 23.1%。它将UnitreeG1TransportBox-v1的成功率从19.8%提升到96.4%，MuJoCo Playground的平均归一化墙面时间AUC提升了19.1%，并且在Unitree G1上的模拟到现实部署速度比FlashSAC快了36.4%。这些结果表明，可扩展的非政策强化学习应根据可用数据体系调整其稳定器。

NeuralParker: A Reinforcement Learning Planner for Irregular Parking Environments

NeuralParker：针对不规则停车环境的强化学习规划器

• Authors: Zihan Wang, Bai Huang, Yang Guan, Xiao Li, Haoyu Xu, Naizheng Wang, Shengbo Eben Li
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.24485
• Pdf link: https://arxiv.org/pdf/2608.24485
• Abstract Automated parking commonly assumes marked slots and short approach maneuvers. Delivery and service vehicles, however, may need to reach an operator-specified pose in an irregular bounded environment from a distant start. Existing learning-based parking planners often rely on local observations, which can restrict long-range route reasoning. To address this problem, we present NeuralParker, a reinforcement learning-based hybrid planner for arbitrary-pose parking. NeuralParker encodes full-environment obstacle and boundary geometry in a target-relative vertex representation, allowing the policy to retain route-defining context throughout the approach. It further couples a learned curvature--length arc policy with an in-loop terminal ensemble that selects from diverse cubic Hermite connections using a curvature-regularized cost. We also establish factorial and long-range route-choice benchmarks to evaluate planning success and trajectory quality. Experiments on these benchmarks show that NeuralParker achieves higher planning success and better overall trajectory quality than the evaluated baselines, while ablation studies support the benefits of the target-relative global representation and terminal ensemble. Finally, a real-vehicle evaluation confirms that the planner transfers effectively to real delivery-vehicle perception at a working parking site, planning successfully at low computational cost.
• 中文摘要 自动停车通常假设有标记的车位和短距离进场操作。然而，送货和服务车辆可能需要在不规则的边界环境中从远处起步达到运营商指定的姿势。现有基于学习的停车规划师通常依赖局部观察，这可能限制长期路线推理。为解决这个问题，我们介绍了NeuralParker，一种基于强化学习的任意姿势停车混合规划器。NeuralParker 在目标相对顶点表示中编码了全环境障碍物和边界几何，使策略在整个方法过程中保持路径定义上下文。它进一步将学习到的曲率长度弧策略与一个环内终端系综结合，该终端系综利用曲率正则化成本从不同的三次 Hermite 联络中选择。我们还建立了因子和长期路线选择基准，以评估规划成功率和发展轨迹质量。这些基准测试的实验显示，NeuralParker比评估基线实现了更高的规划成功率和更好的整体轨迹质量，而消融研究则支持了目标相对全局表征和终端集合的优势。最后，实车评估确认规划者能有效转移至实际配送车辆感知，在工作中的停车场地实现低计算成本的成功规划。

Joint Optimization of Tool Creation and Use for Large Language Model Agents

大型语言模型代理工具创建与应用的联合优化

• Authors: Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee
• Subjects: Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2608.24571
• Pdf link: https://arxiv.org/pdf/2608.24571
• Abstract Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. Tools written by our 4B models also lifted the performance of LFM-2.5-350M and Qwen3-30B-A3B under same reasoning tasks.
• 中文摘要 工具增强的语言模型受限于人类编写的 API;现有的工具创建系统通过在推理时提示冻结的大型语言模型来修补这一点，使写入工具的模型与使用该工具的模型解耦，且不会显示其生成的模式是否可调用模式。我们提出了SMITH（基于模式的多任务迭代工具磨练），这是一个强化学习框架，在单一策略中联合培训工具创建和工具使用。每次推出要么是一个构建任务（从几个示例中编写工具），要么是一个使用任务（调用一个堆积工具，针对一个未完成的问题）。三个独立的奖励轴分别捕捉模式、代码和结果失败，因此每种失败模式都贡献了自己的梯度。一名4B的Qwen3在SMITH训练的13个程序性推理任务中，在未完成任务中达到79.8的宏观平均准确率，是所有评估方法中最佳，领先未受过训练的30B-A3B工具编写者。在TabMWP-Hard上，它在域外GQA上达到40.4（在最佳同骨干推断时间基线上+7.6），且无任何视觉或表格训练数据。我们4B模型编写的工具也提升了LFM-2.5-350M和Qwen3-30B-A3B在相同推理任务下的性能。

IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents

IAPO：多轮服务代理中信用分配的影响力感知策略优化

• Authors: Bo Ren, Yirong Mao, Yi Yang, Wenhui Que
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.24588
• Pdf link: https://arxiv.org/pdf/2608.24588
• Abstract Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks: {\tau^2}-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.
• 中文摘要 大型语言模型（LLM）代理越来越多地通过与用户和外部工具的多回合交互来解决长期任务。在这些环境中，相关任务信息往往随着时间逐渐展开，而非在初始提示时完全指定。服务代理使这一挑战更加具体：用户可以澄清或修正目标，而工具响应则提供后续决策所需的信息。因此，仅凭最终奖励无法说明哪些行为促成了任务的解决。最新方法依赖于来自其他轨迹或重抽样延续的比较证据，或单独构建的步级学习信号来细化学分。然而，完成的部署已经记录了代理操作之间的信息和错误流动。我们引入了影响力感知策略优化（IAPO），将每次部署都表示为对可训练代理动作的类型影响依赖图，用户和工具观察作为证据。IAPO将支持-使用和失败使用结构转换为路由权重，重新分配相同的轨迹层级优势。Qwen3-4B 和 Qwen3-8B 的实验显示，在三个服务代理基准测试（{\tau^2}-Bench、UserBench 和 AgentChangeBench）上，表现优于多回合强化学习（RL）基线。BFCL-v4 多回合进一步表明，这些提升并不会影响多回合函数调用性能。这项工作推进了对多回合用户交互中信用分配的理解，并提供了一种原则性的方法，帮助服务代理从稀疏的结果反馈中获得培训。

EVEREST:Endogenous Vision-Language Reinforcement Reasoning Exploration for Urban Socio-Semantic Segmentation

EVEREST：城市社会语义分割的内生视觉语言强化推理探索

• Authors: Qixiu Li, Zhongzhi He, Xiang Zhu, Xiaoyong Li, Jiarun Lin, Weifeng Xu
• Subjects: Subjects: Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2608.24640
• Pdf link: https://arxiv.org/pdf/2608.24640
• Abstract Urban socio-semantic segmentation leverages digital and satellite imagery to provide critical spatial semantic information for downstream applications such as urban resource allocation. Although existing methods achieve high segmentation accuracy, they still suffer from inaccurate delineation of target boundaries. The underlying issue is that current models primarily rely on passively aggregated global cross-modal cues, lacking active exploration of the environment. To address this limitation, we propose the EVEREST model, which adopts an egocentric exploration strategy that enables the model to actively investigate boundary cues and perform self-correction. In addition, we formulate discrete natural-language prompts as pseudocode to regularize the execution logic. Reinforcement learning is further employed to implement this irreducible process and elicit the model's structured reasoning capability. Our EVEREST achieves optimal performance on all metrics in the real world urban socio-semantic dataset, demonstrating the superiority of our model. Codes are available at this https URL.
• 中文摘要 城市社会语义分割利用数字和卫星影像，为下游应用如城市资源分配提供关键的空间语义信息。尽管现有方法实现了较高的分割精度，但仍存在目标边界划定不准确的问题。根本问题在于当前模型主要依赖被动聚合的全球跨模态线索，缺乏对环境的主动探索。为解决这一限制，我们提出了珠峰模型，采用以自我为中心的探索策略，使模型能够主动探索边界线索并进行自我修正。此外，我们将离散自然语言提示作为伪代码来规范执行逻辑。强化学习进一步用于实现这一不可约过程，并激发模型的结构化推理能力。我们的EVEREST在现实城市社会语义数据集中的所有指标均实现最佳表现，展示了模型的优越性。代码可在此 https 网址获取。

On-Policy Self-Distillation in Diffusion Models

扩散模型中的政策自提纯

• Authors: Wei Zhou, Xiongwei Zhu, Lingdong Kong, Bo Chen, Lei Zhang, Yongyuan Liang, Xiaoxia Hou, Ye Tian, Xian Sun, Yingshuo Wang, Linfeng Li, Shengqiong Wu, Leigang Qu, Feng Li, Wei Liu, Julian McAuley, Tat-Seng Chua
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.24646
• Pdf link: https://arxiv.org/pdf/2608.24646
• Abstract Reinforcement learning can align diffusion models with human preferences and task-specific objectives, but endpoint rewards do not specify how an intermediate denoising prediction should change. We introduce DiffusionOPSD as an on-policy self-distillation framework that converts image-level reward guidance into explicit targets for clean-output predictions at sampled queries. At each outer iteration, a frozen behavior policy generates trajectories and supplies query states and anchors. Reward gradients construct bounded positive and negative targets around each anchor. The trainable policy fits these targets as detached supervision through finite fitting before an exponential moving average update refreshes the behavior policy. This setup lets us measure target construction and finite realization separately. Controlled same-query experiments show that larger target-construction gains do not necessarily translate into larger realized gains after a single fitting update. Across SD 3.5-M and the step-distilled Z-Image-Turbo, our approach achieves the best final held-out scores in 19 of 20 reward-matched settings across two backbones and ten evaluators. It outperforms the strongest competing method by up to 44.0% and reduces training GPU-hours relative to DiffusionNFT by 40% on SD 3.5-M and 63% on Z-Image-Turbo. These results support on-policy self-distillation as an efficient and analyzable approach to diffusion post-training by converting image-level reward guidance into explicit and continually refreshed intermediate supervision, thereby opening a path toward more efficient and diagnosable alignment.
• 中文摘要 强化学习可以使扩散模型与人类偏好和任务特定目标对齐，但终点奖励并未明确规定中间去噪预测应如何变化。我们将 DiffusionOPSD 引入一个基于策略的自我蒸馏框架，将图像级奖励指导转化为抽样查询中清晰输出预测的明确目标。在每个外部迭代中，冻结行为策略生成轨迹并提供查询状态和锚点。奖励梯度在每个锚点周围构建有界的正负目标。可训练策略通过有限拟合来满足这些目标，在指数移动平均更新前更新行为策略。这种设置使我们能够分别测量目标构造和有限实现。受控的同查询实验表明，较大的目标构建增益不一定转化为单次拟合更新后的更大实现增益。在SD 3.5-M和逐步蒸馏的Z-Image-Turbo中，我们的方法在20个奖励匹配设置中的19个中，通过两个骨干和十个评估者，取得了最佳的最终保留分数。它在SD 3.5-M上比DiffusionNFT减少了40%的训练GPU小时数，比最强的竞争方法高出44.0%，在Z-Image-Turbo上减少了63%。这些结果支持策略自提纯作为一种高效且可分析的训练后扩散方法，通过将图像级奖励指导转化为明确且持续更新的中间监督，从而为更高效、更可诊断的对齐开辟道路。

EviGraph: Towards Verifiable Evidence Construction for Information-Seeking Agents

EviGraph：迈向信息寻求智能体的可验证证据构建

• Authors: Jiashun Chen, Yirong Mao, Wenhui Que
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2608.24667
• Pdf link: https://arxiv.org/pdf/2608.24667
• Abstract Agentic Web search can retrieve relevant information without establishing that the retrieved content actually supports the claims used in an answer. Existing agents typically keep search and evidence recording in a linear interaction trace and optimize primarily for final-answer correctness, providing limited supervision for intermediate grounding. We present EviGraph, a deep-search framework that separates search execution from evidence recording while using a shared policy for the trainable roles. An executor plans concise queries, a frozen evidence verifier inspects source pages and returns verbatim evidence items with an explicit polarity, and the policy maps those items to add/support graph requests that are checked by a deterministic structural validator. The resulting graph serves both as persistent working memory and as a source of dense process rewards, enabling reinforcement learning to directly supervise evidence construction rather than only the final answer. On BrowseComp-Plus, a Qwen3-8B EviGraph agent achieves 35.9% accuracy under a matched interaction budget, compared with 26.9% for the same dual-role architecture without reinforcement learning and 2.7% for a monolithic agent, while generating fewer tokens per rollout. Consistent gains on BrowseComp, GAIA, and XBench indicate that explicitly structuring and rewarding evidence recording improves agentic search
• 中文摘要 代理式网页搜索可以检索相关信息，而无需证明检索内容是否支持答案中所用的主张。现有代理通常将搜索和证据记录记录为线性交互轨迹，主要优化最终答案的正确性，提供有限的中间接地监督。我们介绍EviGraph，一个深度搜索框架，将搜索执行与证据记录分离，同时对可训练角色采用共享策略。执行者规划简洁查询，冻结证据验证器检查源页并返回带有显式极性的逐字证据项，策略将这些项映射为添加/支持由确定性结构验证器检查的图请求。由此产生的图既作为持续的工作记忆，也作为密集过程奖励的来源，使强化学习能够直接监督证据构建，而不仅仅是最终答案。在BrowseComp-Plus上，Qwen3-8B EviGraph代理在匹配交互预算下准确率为35.9%，而同双角色架构中无强化学习者为26.9%，单体代理为2.7%，且每次部署产生的令牌数更少。BrowseComp、GAIA和XBench的持续增长表明，明确结构化并奖励证据记录能提升代理搜索

On-policy Distillation with Verifiable Reward

带有可验证奖励的保单内提炼

• Authors: Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, Gao Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24696
• Pdf link: https://arxiv.org/pdf/2608.24696
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）和策略中提炼（OPD）已成为大型语言模型后训练中广泛采用的两种范式。然而，RLVR存在任务层级反馈稀疏的问题，而OPD提供密集的令牌级指导，但忽略轨迹正确性，仅限于教师的表现。结合它们是一个有前景的方向：OPD提供密集的监督信号，而RLVR则提供任务级别的正确性。然而，现有积分通常依赖加权组合或启发式切换，引入额外的超参数和权衡。我们提出了带有可验证奖励的策略提纯（OPDVR），这是一种简单但有效的方法，能够无缝结合OPD和RLVR，而无需添加任何超参数。我们首先基于轨迹正确性重新表述抽样代币OPD的隐式奖励，然后应用ReLU门槛机制，确保正确轨迹获得非负奖励，错误轨迹获得非正奖励---从而使蒸馏信号与任务成功率保持一致，同时保持教师的分配指导。此外，我们的修改将采样令牌OPD转换为合适的RLVR方法，使其易于与任何策略梯度算法（如GRPO）组合。六个推理基准测试的实验显示，OPDVR始终优于标准OPD。我们的代码可在此 https URL 访问。

SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents

SkillForge：强化学习代理的可验证技能演进

• Authors: Shidong Yang, Ziyu Ma, Tongwen Huang, Xucong Wang, Renda Li, Yiming Hu, Yong Wang, Xiangxiang Chu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.24747
• Pdf link: https://arxiv.org/pdf/2608.24747
• Abstract Large language model (LLM) agents are trained with reinforcement learning (RL) for complex decision-making tasks. However, most RL-trained agents remain episodic and cannot accumulate reusable knowledge across episodes. Recent skill-based approaches, such as SkillRL, attempt to address this issue by extracting skills from raw trajectories, but treat the skill bank as an append-only repository without verifying whether stored skills remain effective. In this paper, we propose SkillForge, a framework for continuous skill evolution that enables skills to be verified and refined through environment interaction. By making skill usage explicit during agent interaction, RL can directly optimize both environment actions and skill invocation decisions. SkillForge further introduces evidence-based skill verification and multi-pathway skill induction, allowing the skill bank to continuously grow while maintaining its quality. Extensive experiments on ALFWorld, WebShop, and AppWorld show that SkillForge consistently outperforms SkillRL, demonstrating the effectiveness of continuously verified skills in training stronger LLM agents.
• 中文摘要 大型语言模型（LLM）智能体通过强化学习（RL）训练，用于复杂的决策任务。然而，大多数经过强化学习训练的代理仍为片段式，无法在多个发作间积累可重复使用的知识。近期基于技能的方法，如SkillRL，试图通过从原始轨迹中提取技能来解决这一问题，但将技能库视为仅附录存储的存储库，未验证存储技能是否仍然有效。本文提出了SkillForge框架，一个持续技能演进的框架，使技能能够通过环境交互得到验证和完善。通过在代理交互中明确技能使用，强化学习可以直接优化环境动作和技能调用决策。SkillForge进一步引入了基于证据的技能验证和多途径技能导入，使技能库能够在保持质量的同时持续增长。在ALFWorld、WebShop和AppWorld上的大量实验显示，SkillForge始终优于SkillRL，证明了持续验证技能在训练更强LLM代理方面的有效性。

Improving Cross-Problem Vehicle Routing with Locally Augmented Preferences and Representation Disentanglement

通过局部增强偏好和表示解缠改进跨问题车辆路由

• Authors: Arthur Corrêa, Paulo Nascimento, Samuel Moniz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.24859
• Pdf link: https://arxiv.org/pdf/2608.24859
• Abstract Multi-task vehicle routing problem (VRP) solvers seek to handle multiple VRP variants within a single unified model, avoiding the need to train a separate model for every variant. In spite of recent progress, current approaches remain limited on two fronts. On the training side, reinforcement learning suffers from reward-scale disparities and shrinking advantage signals as policies improve, whereas preference optimization stagnates once sampled tours become near-identical and thus fundamentally limited by the quality of the policy's own generated solutions, leaving both paradigms with weak supervision as training progresses. On the architecture side, existing fully shared encoders entangle constraint-dependent representations across heterogeneous variants, which limits generalization. We address these gaps with two model-agnostic contributions. First, we propose Preference Optimization with Locally Augmented Refinement (POLAR), a novel training algorithm that applies a local search refinement pass to the best decoded tour before forming preference pairs, yielding much more informative pairwise margins. Second, a Progressive Layered Extraction (PLE) encoder routes each encoder layer through one shared expert and a set of task-specific experts via a gating mechanism, progressively separating common routing structure from constraint-specific encodings. Through extensive experiments on various VRP variants, we show that POLAR and PLE together elevate the current state-of-the-art among neural multi-task solvers. We reduce the average gap to reference solutions by 21.3% relative to the strongest published baseline on 16 in-distribution variants, and outperform prior neural methods on 27 out of 32 unseen variants. Ablation studies confirm the efficacy of each contribution, showing that both improve cross-problem generalization across multiple backbone model architectures.
• 中文摘要 多任务车辆路由问题（VRP）求解器力求在统一模型中处理多种VRP变体，避免为每个变体训练独立模型。尽管近期取得了进展，当前的做法在两个方面仍然有限。在训练方面，强化学习会受到奖励尺度差异和优势信号缩小的影响，而当抽样的巡回几乎相同、受限于策略自身生成的解决方案质量时，偏好优化趋于停滞，导致两种范式在训练过程中监督薄弱。在架构方面，现有的完全共享编码器会在异构变体间纠缠依赖约束的表示，限制了泛化。我们通过两个模型无关性贡献来弥补这些空白。首先，我们提出了带有局部增强细化的偏好优化（POLAR），这是一种新颖的训练算法，在形成偏好对前对最佳解码的巡回进行局部搜索细化，从而获得更有信息量的成对边距。其次，渐进分层提取（PLE）编码器通过门控机制将每个编码层路由到一个共享专家和一组任务专属专家之间，逐步将通用路由结构与约束特定编码分离。通过对多种VRP变体的广泛实验，我们表明POLAR和PLE共同提升了神经多任务解算器当前的技术水平。我们相较于最强已发表基线，在16个分布内变体中，平均差距减少了21.3%，并且在32个未见变体中27个表现优于以往神经方法。消融研究证实了每种贡献的有效性，表明两者都能提升跨多骨干模型架构的跨问题泛化能力。

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

SPO++：异步代理强化学习的流对齐策略优化

• Authors: Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.24870
• Pdf link: https://arxiv.org/pdf/2608.24870
• Abstract Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.
• 中文摘要 群体相对强化学习等待同一提示的兄弟姊妹部署，这对于长且多变的工具使用轨迹来说成本较高。单流策略优化（SPO）通过持续的提示级价值估计去除了这种依赖，但其配方在优化token-mean演员损失之前，会在每个轨迹中净化一个优势。我们证明轨迹中心化通常不以行为者消耗的token加权量为中心，并通过在动作token度量下标准化终端结果优势来解决不匹配。我们还根据产生该信息的政策事件而非学生收据顺序来整理即时证据。在ALFWorld两种模型尺度和Math-TIR上的匹配运行中，SPO++相比SPO提高了在线学习效率。配对消融将作用-标记-测量归一化识别为最强的测试成分。

Keyword: diffusion policy

There is no result