生成时间: 2026-07-27 19:23:53 (UTC+8); Arxiv 发布时间: 2026-07-27 20:00 EDT (2026-07-28 08:00 UTC+8)

今天共有 21 篇相关文章

Keyword: reinforcement learning

Do Modules Stay in Their Lane? Role Drift in Compound LLM Systems

模块会保持其自身的定位吗？复合式大型语言模型系统中的角色漂移

• Authors: Xiaoyang Cao, Siddarth Srinivasan, Michiel A. Bakker
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.21627
• Pdf link: https://arxiv.org/pdf/2607.21627
• Abstract End-to-end reinforcement learning can improve the accuracy of compound LLM systems, but it does not constrain how modules divide labor internally. We identify Role Drift, a failure mode in which modules preserve or improve end-task performance while deviating from their assigned roles through role-violating shortcuts that remain invisible to system-level evaluation. To make role drift observable and controllable, we propose Role Anchor, a regularizer that modulates how much each module deviates from its assigned role during end-to-end training. The key idea is to preserve how the role prompt shifts the module's next-token predictions relative to a neutral prompt, which serves as a proxy for the role's intended effect during training. Experiments on two compound LLM pipelines reveal role drift that accuracy alone fails to detect: a decomposer meant to split a question into sub-questions for a separate solver instead plants the answer in them, and a reader meant to answer from retrieved passages instead falls back on parametric memory. In fact, on the decomposer pipeline this shortcut drives most of the apparent RL gain: 86% of it vanishes once the decomposer is held to its role, indicating that terminal accuracy alone can badly overstate how much a compound system has genuinely learned. Across both pipelines, Role Anchor mitigates role drift at a tunable accuracy cost that varies by pipeline and anchor strength. Additional gradient analysis suggests that the regularizer reduces alignment with the role-drift direction rather than simply suppressing learning.
• 中文摘要 端到端强化学习可以提高复合大型语言模型系统的准确性，但它并不限制模块内部的分工方式。我们识别了角色漂移，这是一种失败模式，模块在通过角色违背的捷径偏离其分配角色的同时，保持或提升终端任务的性能，这些捷径对系统层级评估来说是隐形的。为了使角色漂移可被观察和控制，我们提出了角色锚点（Role Anchor），这是一种规范化器，可以调节每个模块在端到端训练中偏离其分配角色的程度。关键思想是保持角色提示如何将模块的下一个标记预测相对于中性提示进行调整，中性提示作为角色在训练期间预期效果的代理。在两个复合大型语言模型流水线上的实验揭示了单靠准确性无法检测的角色漂移：一个用于将问题拆分为子问题的分解器，实际上将答案植入了子问题;而一个从检索到的文章中回答的读者则依赖参数记忆。事实上，在分解器流水线上，这一捷径驱动了大部分表面的强化学习增益：一旦分解器被要求履行其角色，86%的收益会消失，表明仅靠终端准确率可能严重夸大复合系统实际学习的程度。在两个流水线中，角色锚点以可调节的准确率成本减轻角色漂移，该成本取决于流水线和锚点强度。额外的梯度分析表明，正则化器减少了与角色漂移方向的对齐，而不仅仅是抑制学习。

Quasi-Monte Carlo Initialization for Meta-Reinforcement Learning

元强化学习的准蒙特卡洛初始化

• Authors: Julian G. Soltes
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.21637
• Pdf link: https://arxiv.org/pdf/2607.21637
• Abstract This paper explores the efficacy of quasi-Monte Carlo (QMC) weight initialization for meta-reinforcement learning within modern benchmark environments. Various sampling methods are used to bound a population-based search and aggregate an optimal prior from a baseline set of tasks. The QMC meta-priors show improvements in training convergence compared to modern orthogonal (SB3) defaults when extrapolated to similar unseen continuous control environments. In dissimilar tasks, the orthogonal orientation was globally superior for an unbiased search.
• 中文摘要 本文探讨了准蒙特卡洛（QMC）权重初始化在现代基准环境中元强化学习中的有效性。采用多种抽样方法对基于总体的搜索进行界限，并从基线任务集中汇总出最优先验。QMC元先验在将训练收敛率提升至类似的未见连续控制环境中，相较于现代正交（SB3）默认值。在不同任务中，正交方向在无偏搜索中整体优越。

Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning

调整速度作为非固定强化学习的安全约束

• Authors: Timothy Tomashevskiy
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.21646
• Pdf link: https://arxiv.org/pdf/2607.21646
• Abstract Ensuring safety in reinforcement learning under nonstationarity requires determining whether a learning system can safely adapt to forecasted environmental change within the required recovery horizon. Existing safe reinforcement learning methods typically assume stationary environments and do not explicitly consider adaptation speed as a safety concern. However, when environments evolve over time, delayed adaptation may result in transient unsafe behavior. This paper proposes adjustment speed as a safety constraint for nonstationary reinforcement learning. The central idea is to define safety in terms of adaptation feasibility: future states or regions may become unsafe when the adaptation required to remain safe exceeds the learning system's calibrated recovery capacity. The proposed framework uses learned context representations and short-horizon context forecasts to estimate adaptation demand and compare it with the agent's achievable adaptation capacity. When predicted adaptation demand exceeds the calibrated recovery capacity, the framework proactively tightens the admissible action set and activates an action-level shield to reduce unsafe behavior before violations occur. Experiments in a nonstationary driving environment show that the proposed approach primarily reduces safety violations in short-horizon windows aligned with context changes. Ablation studies further show that shielding is more conservative for peak- and tail-risk suppression, while optimization-level adjustment provides additional reductions in short-horizon switch-conditioned violations. These results support adaptation feasibility as a practical safety principle for reinforcement learning under nonstationarity and demonstrate that proactive intervention can improve safety during periods of environmental change.
• 中文摘要 在非平稳性下确保强化学习的安全性，需要确定学习系统是否能在所需的恢复视野内安全适应预测的环境变化。现有的安全强化学习方法通常假设静止环境，且未明确将适应速度视为安全问题。然而，当环境随时间演变时，延迟适应可能导致暂时的不安全行为。本文提出调整速度作为非平稳强化学习的安全约束。核心思想是以适应可行性来定义安全：当保持安全所需的适应超过学习系统校准的恢复能力时，未来状态或区域可能变得不安全。所提出的框架利用学习的上下文表示和短期上下文预测来估算适应需求，并将其与智能体可达成的适应能力进行比较。当预测适应需求超过校准恢复能力时，框架主动收紧可接受动作集，并激活动作级屏蔽，以减少违规发生前的不安全行为。非静止驾驶环境下的实验表明，所提方法主要减少与情境变化相符的短视距安全违规。消融研究进一步表明，屏蔽在峰值和尾部风险抑制上更为保守，而优化级调整则能进一步减少短视距开关条件违规。这些结果支持适应可行性作为非平稳强化学习的实用安全原则，并表明主动干预能在环境变化期间提升安全性。

Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning

Molt：一个可扩展的PyTorch原生智能强化学习训练框架

• Authors: Jian Hu, Huiying Li, Hao Zhang, Binfeng Xu, Yifan Zhang, Shaokun Zhang, Hemil Desai, Michael Demoret, Pavlo Molchanov, Jan Kautz, Yi Dong
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2607.21653
• Pdf link: https://arxiv.org/pdf/2607.21653
• Abstract Agentic reinforcement learning research is constant algorithm modification, new estimators, new pipeline stages, new rollout schemes, and in mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue: the cost lands on the researcher at every iteration. Molt is a PyTorch-native training framework built to keep that cost small: a codebase compact and clean enough for a researcher to hold in their head, and for an AI coding assistant to read and reason about in its entirety, so the algorithm flow can be traced and changed end to end. The agent is an ordinary program, and one asynchronous loop trains multimodal and mixture-of-experts policies while never training on a token it did not generate, consistent in tokens, policy versions, and model semantics. Leanness does not cost performance: under a matched, fully asynchronous protocol, Molt is statistically comparable to a state-of-the-art Megatron-based stack. Molt is open source and provides recipes and containers at this https URL.
• 中文摘要 代理强化学习研究是不断修改算法、新估计器、新流水线阶段、新推广方案，在主流框架中，每个线程都通过训练器、分布式后端和滚动胶层进行转换：成本在每次迭代中都落在研究者身上。Molt 是一个基于 PyTorch 的本地训练框架，旨在保持成本低：代码库紧凑且干净，研究者可以完整地阅读和推理，从而跟踪和调整算法流程。代理是一个普通程序，一个异步循环训练多模态和专家混合策略，从不训练它未生成的令牌，令牌、策略版本和模型语义保持一致。精简度不会影响性能：在匹配的完全异步协议下，Molt在统计上可与基于Megatron的顶级堆栈相媲美。Molt是开源的，并在此https URL提供配方和容器。

Oxygen-TryOn: Fashion-Native Foundation Model for Any-item Virtual Try-On

Oxygen-TryOn：时尚原生基础模型，适用于任何物品的虚拟试穿

• Authors: Yong Liu, Xiaolong Fu, Zihang Xu, Wen Xue, Xueheng Li, Lin Song, Yuan Zhang, Chuyang Zhao, Haoyang Huang, Nan Duan, Yipeng Sun, Yan Li, Simiu Gu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.21694
• Pdf link: https://arxiv.org/pdf/2607.21694
• Abstract We present Oxygen-TryOn, a unified foundation model for any-item virtual try-on. Rather than repurposing a general-purpose image editor, Oxygen-TryOn is fashion-native, built for try-on through a dedicated data engine and try-on-specific training. Given one or more reference items (clean product shots or in-the-wild worn-on photos) and a single target subject image, it synthesizes a photorealistic image of the subject wearing the items across virtually any fashion category. Prior systems handle a single garment category in a studio setting, and recent multi-reference methods remain garment-centric; in contrast, Oxygen-TryOn supports diverse items and scenarios, including full- and half-body views, a variable number of references, and free multi-item composition, while faithfully preserving both subject identity and item appearance. Instead of mask-based inpainting, we reformulate try-on as a multi-reference, understanding-driven generation task. We build a data engine that collects, manufactures, annotates, and filters high-quality try-on data at scale, and design a three-stage recipe of continued pre-training (CPT), supervised fine-tuning (SFT), and reinforcement learning (RL). The RL stage uses a hybrid reward combining an in-house try-on reward model with a proprietary, rubric-guided general-purpose model, jointly supervising fine-grained consistency and instruction-level quality. It also follows general editing instructions (e.g., pose changes) in the same pass. Across public benchmarks and our in-house Oxygen-TryOn Bench, it achieves state-of-the-art consistency and realism on single-item try-on and leads on multi-item try-on, matching or surpassing both leading proprietary systems (Nano Banana Pro, GPT-Image-2, Seedream5 Lite) and open-source models (FLUX.2).
• 中文摘要 我们介绍Oxygen-TryOn，一个统一的基础模型，适用于任何物品的虚拟试穿。Oxygen-TryOn并非改造通用图像编辑器，而是时尚原生，通过专用数据引擎和试穿专用培训为试穿打造。给定一个或多个参考物品（干净的产品照片或野外穿过的照片）和单一目标主体图像，它就能合成几乎任何时尚类别中被试穿者的写实照片。以往系统在工作室环境中处理单一服装类别，近期多参考方法仍以服装为中心;相比之下，Oxygen-TryOn支持多样化的物品和场景，包括全身和半身视角、可变数量的参考资料以及自由的多件物品构图，同时忠实地保留了主体身份和物品外观。我们不再采用基于面具的修饰，而是将试穿重新表述为一个多参考、以理解为驱动的生成任务。我们构建了一个数据引擎，大规模收集、制造、注释和过滤高质量试用数据，并设计了持续预训练（CPT）、监督微调（SFT）和强化学习（RL）的三阶段方案。强化学习阶段采用混合奖励，结合了内部试用奖励模型与专有的评分标准指导通用模型，共同监督细粒度的一致性和指令级质量。它还在同一次中遵循通用编辑指令（如姿态变化）。在公开基准测试和我们自家的Oxygen-TryOn测试台上，它在单项试用和多项试用上实现了最先进的一致性和真实性，甚至超越了领先的专有系统（Nano Banana Pro、GPT-Image-2、Seedream5 Lite）和开源模型（FLUX.2）。

Be Consistent! Enhancing Robust Visual Reasoning in LVLMs with Consistency Constraints

保持一致性！增强具有一致性约束的LVLM中强健的视觉推理

• Authors: Liqiang Jing, Xiong Zhou, Siddharth Varia, Neha Anna John, Xinya Du, Vassilis N. Ioannidis
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.21722
• Pdf link: https://arxiv.org/pdf/2607.21722
• Abstract While Large Vision-Language Models (LVLMs) exhibit strong perceptual capabilities, they remain vulnerable in visual reasoning tasks. Existing benchmarks largely focus on symbolic mathematical or scientific problems and simple vision-centric tasks, offering limited assessment of complex visual reasoning and logical consistency, a critical requirement for reliable reasoning systems. We introduce ConVBench, a complex vision-centric reasoning benchmark in which each image is paired with two logically equivalent questions across six categories: action and state, complex counting, spatial reasoning, causal and intent understanding, commonsense reasoning, and temporal perception. To complement this benchmark, we define two evaluation metrics, logical consistency and robust accuracy, that jointly assess both the correctness and consistency of model responses. We further present ConVLM, which improves LVLM reasoning through Group Relative Policy Optimization (GRPO)-based reinforcement learning with a novel consistency reward. This method leverages automatically generated logically equivalent question-answer pairs and a dual-reward design combining accuracy- and consistency-based signals, encouraging agreement between paired responses. The framework functions effectively with or without strict answer supervision.
• 中文摘要 虽然大型视觉语言模型（LVLM）表现出强大的感知能力，但在视觉推理任务中仍存在脆弱性。现有基准主要关注符号数学或科学问题及简单的视觉中心任务，对复杂视觉推理和逻辑一致性的评估有限，而逻辑一致性是可靠推理系统的关键需求。我们介绍ConVBench，这是一个复杂的视觉中心推理基准测试，每个图像与两个逻辑等价问题分布在六个类别中：动作与状态、复杂计数、空间推理、因果与意图理解、常识推理和时间感知。为补充该基准，我们定义了两个评估指标：逻辑一致性和稳健准确性，共同评估模型响应的正确性和一致性。我们进一步介绍ConVLM，通过基于群体相对策略优化（GRPO）的强化学习提升LVLM推理，并带来新的一致性奖励。该方法利用自动生成的逻辑等价问答对和结合准确性和一致性信号的双重奖励设计，鼓励配对回答间的一致性。该框架无论是否严格的答案监督都能有效运行。

QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization

QLPO：象限加权抽样用于长度感知策略优化

• Authors: Siwei Chen, Siqi Chen, Xupeng Miao, Bin Cui
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.21793
• Pdf link: https://arxiv.org/pdf/2607.21793
• Abstract Recent large reasoning models often develop long chain-of-thought responses during reinforcement learning (RL), resulting in high inference latency and deployment cost. Existing methods for response length control typically rely on explicit length penalties or additional control modules, which require careful tuning and may compromise reasoning quality. We propose Quadrant-weighted Sampling for Length-aware Policy Optimization (QLPO), a simple resampling-based variant of GRPO that introduces implicit length control without modifying the reward function. QLPO first over-generates candidate responses and then resamples the training group by preserving the empirical correct/incorrect ratio while favoring short correct responses and long incorrect responses. This reshapes the training distribution and implicitly encourages shorter model outputs. Across models ranging from 1.5B to 32B parameters, including both base models and strong reasoning models, QLPO consistently improves the accuracy-length trade-off. It reduces response length by 30% to 70% while preserving reasoning performance. These results suggest that structured resampling provides an effective and robust approach to efficient reasoning.
• 中文摘要 近期大型推理模型在强化学习（RL）过程中常常发展出较长的思维链响应，导致推理延迟和部署成本高。现有的响应长度控制方法通常依赖显式长度惩罚或额外控制模块，这需要仔细调优，可能影响推理质量。我们提出了象限加权抽样用于长度感知策略优化（QLPO），这是一种基于重抽样的简单GRPO变体，引入隐式长度控制而不改变奖励函数。QLPO先过度生成候选响应，然后通过保持经验上的正确/错误比例，同时偏向短正确响应和长错误响应，重新采样训练组。这重塑了训练分布，隐含地鼓励模型输出更短。在参数范围从1.5B到32B的模型中，包括基础模型和强推理模型，QLPO持续改善准确率与长度的权衡。它将响应长度缩短30%至70%，同时保持推理性能。这些结果表明结构化重抽样提供了一种有效且稳健的高效推理方法。

Variance-Reduced Q-Learning over Static and Time-Varying Networks

静态和时间变化网络上的方差减少Q学习

• Authors: Sreejeet Maity, Feng Zhu, Aritra Mitra, Robert W. Heath Jr
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.21876
• Pdf link: https://arxiv.org/pdf/2607.21876
• Abstract We investigate a decentralized reinforcement learning problem involving multiple agents that interact with the same Markov Decision Process (MDP). The agents can exchange information over a network to collectively learn the optimal state-action value function. For this setting, we introduce a novel epoch-based distributed $Q$-learning algorithm called VRDQ, where within each epoch, agents locally estimate the Bellman optimality operator and diffuse information using a consensus-based protocol. For both static and time-varying networks, we establish high-probability finite-time convergence rates for VRDQ that enjoy linear speedups from collaboration. Crucially, we prove that such speedups in sample-complexity require only $\tilde{O}(1)$ communication, substantially improving upon the communication costs in prior work.
• 中文摘要 我们研究一个去中心化强化学习问题，涉及多个智能体与同一马尔可夫决策过程（MDP）交互。智能体可以通过网络交换信息，集体学习最优状态-动作值函数。在此背景下，我们引入一种新的基于跨元分布式$Q$学习算法VRDQ，在每个跨时期内，智能体局部估计贝尔曼最优算子，并使用基于共识的协议扩散信息。对于静态和时间变化网络，我们建立了高概率有限时间收敛率的VRDQ，这些收敛率因协作而获得线性加速。关键是，我们证明了这种样本复杂度的加速仅需$\tilde{O}（1）$一次通信，显著改善了以往工作的通信成本。

Adaptive Undulatory Locomotion of Snake-like Robots in Dynamic Viscous Environments via Deep Reinforcement Learning

通过深度强化学习实现蛇状机器人在动态粘性环境中的自适应波动运动

• Authors: Tsuyoshi Kimoto, Akio Yamano, Kohei Honda, Takashi Iwasa
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.21960
• Pdf link: https://arxiv.org/pdf/2607.21960
• Abstract This paper demonstrates how deep reinforcement learning (DRL) enables adaptive locomotion of snake-like robots in dynamically changing viscous environments, overcoming the inherent performance limitations of classical predefined control methods. The lack of direct onboard sensors for fluid properties necessitates formulating this task as a partially observable Markov decision process. By employing an asymmetric actor-critic framework, a teacher policy trained using privileged information available only in the physics simulator distills its knowledge into a student policy that relies solely on proprioceptive sensor information. Simulation results across a wide range of dynamic viscosity changes ($10^{-7}$ to $10^{-2} m^2/s$) reveal that the DRL agent autonomously acquires non-sinusoidal adaptive gaits. These gaits improve propulsion velocity and transport efficiency, breaking the inherent limits of conventional sinusoidal and kinematic control. The findings establish that implicit environment inference via privileged information distillation is an effective approach to bypass the constraints of classical models under unpredictable fluid dynamics.
• 中文摘要 本文展示了深度强化学习（DRL）如何使蛇形机器人在动态变化的粘性环境中实现自适应运动，克服了经典预定义控制方法固有的性能限制。缺乏直接的车载流体属性传感器，使得将该任务表述为部分可观测的马尔可夫决策过程。通过采用非对称演员-批评框架，使用物理模拟器中仅有的特权信息训练的教师策略，将知识提炼为仅依赖本体感觉传感器信息的学生策略。在广泛动态粘度变化范围内（$10^{-7}$ 到 $10^{-2} m^2/s$）的模拟结果显示，DRL 智能体自主获得非正弦的自适应步态。这些步态提升了推进速度和传输效率，打破了传统正弦波和运动学控制的固有限制。研究结果证明，通过特权信息提炼进行隐式环境推断，是绕过经典模型在不可预测流体动力学下限制的有效方法。

Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning

教授大型语言模型自我进化：通过强化学习培养核心元技能

• Authors: Shujin Wu, Cheng Qian, Xiusi Chen, Heng Ji
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.21971
• Pdf link: https://arxiv.org/pdf/2607.21971
• Abstract Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains. We hypothesize that the success of such evolution frameworks hinges on meta-skills, such as self-reflection with environment feedback, that enable effective multi-round refinement, yet are largely neglected by traditional post-training. To bridge this gap, we present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning (RL), and inference-time evolutionary search. Concretely, we ground MetaEvolve in coding, where program execution provides natural, continuous reward signals beyond binary correctness. Building on these signals, we synthesize evolution trajectories as training data, each containing a current program, its fitness score (combining correctness and efficiency), and a history of prior attempts, and train the model via RL with verifiable rewards derived from test case execution. By training on large-scale code data, we aim to inspire generalizable domain-agnostic meta-skills that can transfer broadly to open-ended problems where such rich training signals are scarce. Across seven coding benchmarks, MetaEvolve outperforms the strongest baseline by 10.01% absolute on in-distribution tasks and 24.12% on out-of-distribution tasks. On open-ended algorithm optimization problems entirely outside the training domain, it further achieves a 46.9% relative improvement. These results demonstrate that explicitly cultivating self-evolution meta-skills offers a principled path toward more capable and autonomously self-evolving AI.
• 中文摘要 通过迭代自我进化和环境反馈进行测试时间扩展，AlphaEvolve 展示了显著的性能提升。我们假设此类进化框架的成功依赖于元技能，如环境反馈的自我反思，这些技能使得有效的多轮精炼成为可能，但传统后期训练大多忽视了这一点。为弥合这一差距，我们提出了 MetaEvolve，该框架旨在通过数据综合流水线、进化感知强化学习（RL）和推理时间进化搜索来开发这些元技能。具体来说，我们将 MetaEvolve 建立在编码基础上，程序执行提供了超越二元正确性的自然、连续奖励信号。基于这些信号，我们将演化轨迹综合为训练数据，每个数据包含当前程序、其适应度评分（结合正确性和效率）及以往尝试历史，并通过强化学习训练模型，并从测试用例执行中获得可验证的奖励。通过在大规模代码数据上训练，我们旨在激发可广泛迁移的领域无关元技能，这些技能能够广泛应用于缺乏丰富训练信号的开放式问题。在七个编码基准测试中，MetaEvolve在分布内任务中绝对表现优于最强基线10.01%，在分布外任务中高出24.12%。在完全不在训练领域进行的开放式算法优化问题上，MetaEvolve 进一步实现了46.9%的相对提升。这些结果表明，明确培养自我演化元技能为迈向更强大、自主自我演化的人工智能提供了一条有原则的路径。

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

推理展开中的学习：高效强化学习的渐进推广分配

• Authors: Heyang Jiang, Henry Liu, Baharan Mirzasoleiman
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22002
• Pdf link: https://arxiv.org/pdf/2607.22002
• Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3$\times$ fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49$\times$ fewer rollouts, and improves the coding average test pass rate by 3.4 points.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为提升大型语言模型推理的高效框架，GRPO等方法就是其最成功的实例之一。然而，GRPO依赖于反复生成长链思考的展开。训练时间与展开次数的增长成比例，其中很大一部分信息量不高。因此，GRPO计算成本高且不稳定。为缓解这一问题，现有方法要么生成更大的滚动池并过滤最具信息性的提示，要么利用历史信号在训练后期进行过滤。这些策略带来适度的性能提升，但会拖慢整体过程。为此，我们提出了VarIance引导在线推广分配（VIGOR），它不是为每个示例分配固定的推广预算，而是从批次中所有样本的少量推广开始，迭代地为组奖励方差最高的部分分配额外推广，直到达到固定的总推广预算。理论上，我们证明在RLVR下，奖励方差控制梯度大小，并推导出VIGOR相对于GRPO的闭式加速比，而GRPO在帕累托分布奖励方差下，GRPO在细化轮次中增长。数学推理和编码任务的实验显示，VIGOR在数学上最多减少2.3$\时间$的推广次数即可达到目标精度，以减少1.49$\时间$的滚动次数达到GRPO的最终编码全通过率，并将编码平均测试通过率提升3.4个百分点。

On the Runtime Analysis of Reinforcement Learning Hyper-Heuristics

关于强化学习超启发式的运行时分析

• Authors: Pietro S. Oliveto, Zhenyu Wang, Peizhou Wu, Mengqing Xu
• Subjects: Subjects: Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2607.22036
• Pdf link: https://arxiv.org/pdf/2607.22036
• Abstract Selection Hyper-heuristics (HHs) automate algorithmic design by selecting from a set of low-level heuristics which one to apply at each stage of the optimisation process. Several impressive results have been recently rigorously proven regarding the performance of selection hyper-heuristics (HHs) for standard benchmark functions. However, the learning mechanisms employed by these HHs are considerably simplified compared to the machine learning techniques typically used in real world applications. In this paper we analyse a Reinforcement Learning Hyper-heuristic (RLHH) from the literature. The only previous result available proved that for a wide range of parameter settings, RLHH does not learn to select heuristics appropriately for the standard LeadingOnes benchmark function. In this paper, we rigorously prove that with appropriate parameter values RLHH equipped with two random local search operators, RLS_1 and RLS_2 optimises the LeadingOnes benchmark function in the best possible expected runtime achievable with the two operators up to lower order terms. Experiments show that for realistic problem sizes it is faster than the Generalised Random Gradient HH which was previously proven to also have optimal expected runtime up to lower order terms.
• 中文摘要 选择超启发式（HHs）通过从一组低级启发式中选择在优化过程的每个阶段应用的算法，实现算法设计的自动化。最近有若干令人印象深刻的结果被严格验证，关于选择性超启发式（HHs）在标准基准函数中的性能。然而，这些HH所采用的学习机制相比，实际应用中通常使用的机器学习技术要大大简化。本文分析了文献中的强化学习超启发式（RLHH）。此前唯一可用的结果证明，对于广泛的参数设置，RLHH并未学会为标准LeadingOnes基准函数选择合适的启发式方法。本文严谨证明，在配备两个随机局部搜索算子的适当参数值RLHH下，RLS_1和RLS_2能够在两个算符（至低阶项）下，以最佳预期运行时间优化LeadingOnes基准函数。实验表明，对于现实问题规模，它比广义随机梯度HH更快，后者此前已被证明在低阶项下也具有最优期望运行时间。

Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

够了，就像盛宴一样：强化学习如何缓解大型语言模型任务冲突的综合分析

• Authors: Zixuan Ren, Jinliang Lu, Junhong Wu, Yang Zhao, Dai Dai, Hua Wu, Haifeng Wang, Chengqing Zong
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.22039
• Pdf link: https://arxiv.org/pdf/2607.22039
• Abstract Model merging plays a crucial role in consolidating multiple specialized models into a single, unified model, especially in the era of large language models (LLMs). Recent research has primarily focused on developing strategies to enhance merging performance with the trained models, while the impact of training paradigms, such as supervised fine-tuning (SFT) and reinforcement learning (RL), on the effectiveness of model merging remains underexplored. In this study, we systematically explore the merging behavior of RL-trained LLMs compared to those trained with traditional SFT. Through comprehensive evaluations across five representative tasks, we find that RL significantly reduces task conflicts and results in less performance degradation after merging, making RL-trained models particularly well-suited for this process. To unearth the reasons behind the superior suitability of RL for model merging, we conduct extensive empirical experiments and theoretical analyses. Our findings highlight three key factors: (1) On-policy training data in RL control the gradient updates in a smaller magnitude, reducing the risk of overwriting existing knowledge for other tasks in the model. (2) The RL optimization objective, which favors ``\textit{enough is as good as a feast}", progressively reduces the magnitude and the number of conflict parameter updates as the model converges. (3) Joint optimization of positive and negative examples in RL steers the model towards an unbiased task-specific parameter subspace, ensuring robust performance while further preventing parameter conflicts.
• 中文摘要 模型合并在将多个专业模型整合为统一模型中起着关键作用，尤其是在大型语言模型（LLMs）时代。近期研究主要集中在开发提升与训练模型合并性能的策略，而训练范式如监督式微调（SFT）和强化学习（RL）对模型合并效果的影响仍未被充分探讨。本研究系统地探讨了强化学习训练LLM与传统SFT训练模型的合并行为。通过对五个代表性任务的全面评估，我们发现强化学习显著减少了任务冲突，合并后性能下降较小，使强化学习模型特别适合这一过程。为揭示强化学习在模型合并中优越适用性的原因，我们进行了大量实证实验和理论分析。我们的发现突出了三个关键因素：（1）强化学习中的策略训练数据能以更小幅度控制梯度更新，降低模型中其他任务覆盖现有知识的风险。（2）强化学习优化目标偏向“\textit{足够了就如盛宴}”，随着模型收敛，冲突参数的更新幅度和次数逐渐减少。（3）在强化学习中对正负样本的联合优化，使模型朝向无偏的任务特定参数子空间，确保稳健性能，同时进一步防止参数冲突。

Predictive Lightweight MARL for Resilient Coverage in Sparse-Signaling Aerial Networks

预测轻量级MARL用于稀疏信号天线网络的韧性覆盖

• Authors: Chuan-Chi Lai, Ang-Hsun Tsai
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.22109
• Pdf link: https://arxiv.org/pdf/2607.22109
• Abstract This letter proposes the Predictive Lightweight Multi-Agent Reinforcement Learning (PL-MARL) framework to ensure resilient coverage in bandwidth-constrained UAV swarms. To counter coordination collapse caused by sparse signaling and information aging, we introduce a Kinematic-Aware Inference Engine that proactively reconstructs neighbor trajectories via physical priors. This approach enables an efficient computation-for-communication trade-off, decoupling structural resilience from signaling frequency. Simulations confirm that PL-MARL maintains superior coverage and mission continuity under extreme signaling scarcity and node failure. Our results validate proactive inference as a scalable, low-latency solution for robust aerial coordination, effectively minimizing control overhead to preserve spectrum for payload services while ensuring resilience against interference.
• 中文摘要 本信提出了预测轻量级多智能体强化学习（PL-MARL）框架，以确保带宽受限无人机群的韧性覆盖。为应对由稀疏信号和信息老化引起的协调崩溃，我们引入了运动学感知推理引擎，通过物理先验主动重建邻居轨迹。该方法实现了高效的计算与通信权衡，将结构弹性与信号频率脱钩。模拟确认PL-MARL在极端信号稀缺和节点故障下仍能保持优越覆盖和任务连续性。我们的结果验证了主动推断作为一种可扩展、低延迟的稳健空中协调解决方案，有效减少控制开销，保留有效载荷服务的频谱，同时确保对干扰的韧性。

Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning

拆解非策略比率：异步强化学习中的熵尺度信任区域

• Authors: Guanqun Zhao, Zijun Xie, Binbin Zheng, Enlei Gong, Jiafeng Lu, Yehan Yang, Aoqi Hu, Zeyu Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22186
• Pdf link: https://arxiv.org/pdf/2607.22186
• Abstract Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization, but the resulting stale, off-policy data can destabilize optimization and ultimately cause policy collapse. Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios, applying the same threshold uniformly across token positions. In this work, we reveal that the natural scale of the importance ratio varies systematically with token entropy. Under asynchronous dynamics, this entropy-ratio scaling dictates two distinct phenomena: at low entropy, the inherent train-inference discrepancy is drastically amplified into substantial sampling noise; at high entropy, in-flight weight updates naturally induce pronounced, legitimate exploratory deviations. Consequently, magnitude-only correction inadvertently admits the amplified noise while strictly masking out the essential exploration triggered by in-flight updates. To address this, we propose the Entropy-Scaled Trust Region (ESTR), which scales each token's off-policy deviation by its local entropy, requiring no auxiliary forward passes or explicit version-switch detection. Across long-horizon agentic tasks and mathematical reasoning benchmarks, ESTR consistently outperforms existing asynchronous methods and achieves the best train-inference consistency. Compared with synchronous GRPO, ESTR attains comparable accuracy while improving training speed by $2.6\times$.
• 中文摘要 异步强化学习（RL）通过将部署生成与策略优化重叠，加速大型语言模型（LLM）训练后完成，但由此产生的陈旧、非策略数据可能破坏优化，最终导致策略崩溃。现有方法通常仅根据重要性比的大小保留或丢弃令牌，并在令牌位置间均匀应用相同的阈值。本研究揭示，重要性比的自然尺度随令牌熵系统性变化。在异步动力学下，这种熵比尺度决定了两种不同现象：在低熵下，列车推断的固有差异被大幅放大为显著的采样噪声;在高熵时，飞行中的权重更新自然会引发明显且合法的探索偏差。因此，仅值幅度的修正无意中承认了放大噪声，同时严格掩盖了由飞行中更新触发的本质探索。为此，我们提出了熵尺度信任区域（ESTR），该方法通过其局部熵来扩展每个代币的偏离策略偏差，无需辅助前向传递或显式版本切换检测。在长视野代理任务和数学推理基准测试中，ESTR持续优于现有异步方法，实现最佳的列车-推断一致性。与同步GRPO相比，ESTR在训练速度提升2.6美元/倍数的同时，精度相当。

LayoutLite: Token-Level Implicit Layout Analysis for Efficient Document OCR

LayoutLite：用于高效文档OCR的令牌级隐式布局分析

• Authors: Xudong Liu, Bicheng Wan, Yulin Jin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.22200
• Pdf link: https://arxiv.org/pdf/2607.22200
• Abstract End-to-end OCR systems based on vision-language models have achieved strong performance in complex document OCR, but their efficiency is limited by the large number of visual tokens produced from document images. Many of these tokens correspond to blank margins or visually redundant regions, yet directly applying generic visual token compression methods may remove OCR-critical fine-grained details. In this paper, we propose LayoutLite, a lightweight plug-and-play module for efficient document OCR. Instead of relying on explicit document layout detection, LayoutLite performs implicit layout analysis at the token level between the vision encoder and the language decoder. It aggregates multi-layer visual representations from the vision encoder, and predicts an importance score for each visual token with a lightweight scoring network. Low-information tokens are then removed before entering the language decoder while preserving the original spatial positional information of retained tokens. To train LayoutLite without human annotations, we cast token selection as a reinforcement learning problem and optimize it with a group-relative policy optimization objective driven by OCR output consistency, together with an auxiliary layout supervision signal to stabilize training. Experiments on OmniDocBench demonstrate that LayoutLite can substantially reduce visual token length and inference cost with negligible degradation in recognition quality. We further evaluate LayoutLite on two OCR-specialized VLMs, FireRed-OCR and Logics-Parsing-V2. Under up to 50% token compression, LayoutLite preserves almost the same score on both models while reducing prefill latency, FLOPs, and KV cache memory by over 40%, with only a small additional inference overhead. These results show that token-level implicit layout analysis is an effective and practical approach for accelerating VLM-based OCR systems.
• 中文摘要 基于视觉语言模型的端到端OCR系统在复杂文档OCR中取得了强劲表现，但其效率受限于由文档图像生成的大量视觉标记。许多标记对应空白页边或视觉冗余区域，但直接应用通用的视觉标记压缩方法可能去除OCR关键的细粒度细节。本文提出LayoutLite，一款轻量级即插即用模块，用于高效文档OCR。LayoutLite不依赖显式文档布局检测，而是在视觉编码器与语言解码器之间进行符号层面的隐式布局分析。它汇总视觉编码器的多层视觉表示，并通过轻量级评分网络预测每个视觉符号的重要性评分。在进入语言解码器前，低信息符号会被移除，同时保留保留符号的原始空间位置信息。为了在无人工注释的情况下训练LayoutLite，我们将令牌选择定位为强化学习问题，并结合由OCR输出一致性驱动的组相对策略优化目标，并结合辅助布局监督信号以稳定训练。OmniDocBench上的实验表明，LayoutLite能够显著减少视觉令牌长度和推理成本，且识别质量几乎不下降。我们还进一步在两种专门用于OCR的VLM上评估LayoutLite，FireRed-OCR和Logics-Parsing-V2。在高达50%的令牌压缩下，LayoutLite在两模型上保持几乎相同的分数，同时将预填充延迟、FLOP和KV缓存减少超过40%，且仅有少量额外推理开销。这些结果表明，令牌级隐式布局分析是加速基于VLM的OCR系统的有效且实用的方法。

Active few-shot segmentation by reinforcing data selection

通过强化数据选择实现的主动少数样本分割

• Authors: Chenlan Zhao, Benny Wong, Timothy F. Lundberg, Ahmed M. Elsayed, Abdallah Aljarkas, Hamad A. Aljamaan, Lynn Karam, Qianye Yang, Yipeng Hu, Claire C. Villette, Shaheer U. Saeed
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.22371
• Pdf link: https://arxiv.org/pdf/2607.22371
• Abstract Few-shot learning enables medical image segmentation models to adapt to new tasks using only a small number of labelled examples. However, adaptation performance depends strongly on which examples are selected for the support set. Effective support sets should capture relevant variation within the target domain and be informative for adaptation, with constituent samples providing complementary information. Despite this, existing active data selection approaches largely prioritise samples individually and do not explicitly account for interactions between examples. In this work, we propose a reinforcement learning framework for support-set selection in few-shot medical image segmentation, enabling support sets to be optimised jointly rather than through independent sample scoring. Given a pool of unlabelled candidate images, an agent directly predicts a support set that maximises downstream segmentation performance. Experiments on a cross-institutional pelvic MRI dataset demonstrate improvements over random selection and current state-of-the-art methods. Our findings highlight the importance of support-set complementarity for effective adaptation and demonstrate the potential of reinforcement learning for optimising adaptation sets.
• 中文摘要 少样本学习使医学图像分割模型能够仅用少量标记样本适应新任务。然而，适应性能高度依赖于支持集中选择的样本。有效的支持集应捕捉目标域内相关的变异，并对适应具有信息价值，组成样本提供互补信息。尽管如此，现有的主动数据选择方法大多优先处理单个样本，未明确考虑样本间的交互。本研究提出了一个强化学习框架，用于少数样本医学图像分割中支持集选择，使支持集能够联合优化，而非通过独立样本评分。给定一组未标记候选图像池，代理直接预测支持集以最大化后续分割性能。跨机构盆腔MRI数据集的实验显示，其优于随机选择和当前最先进方法。我们的发现强调了支持集互补性对有效适应的重要性，并展示了强化学习在优化适应集方面的潜力。

Conformal Constraint Tightening for Chance-Constrained Motion Planning with Unknown Dynamics

在未知动力学条件下的概率约束运动规划中的共形约束紧缩

• Authors: Shubham Natraj, Bruno Sinopoli, Yiannis Kantaros
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.22409
• Pdf link: https://arxiv.org/pdf/2607.22409
• Abstract Motion planning algorithms compute control sequences that drive autonomous robots to goal regions while avoiding unsafe states. Existing methods, from sampling-based planning to deep reinforcement learning, typically provide task-completion guarantees only with respect to a nominal model or simulator, which may be invalidated when the true dynamics are unknown or difficult to model accurately. This letter addresses this limitation for systems with unknown dynamics and an available approximate nominal model, contributing a planner-agnostic constraint-tightening procedure that equips existing planners with a probabilistic task-completion guarantee on the true system. We leverage conformal prediction to provide a probabilistic bound on the nominal-to-true trajectory deviation over a distribution of planning problems. We tighten the planning constraints using that bound, and show that solving the tightened problem under the nominal model is a sufficient condition for solving the original problem on the true system with a prescribed probability. We validate the theoretical guarantees empirically and demonstrate substantially improved task completion relative to nominal-model planning.
• 中文摘要 运动规划算法计算控制序列，使自主机器人达到目标区域，同时避免不安全状态。现有方法，从基于采样的规划到深度强化学习，通常仅针对名义模型或模拟器提供任务完成保证，当真实动力学未知或难以准确建模时，这些保证可能失效。本信件针对未知动力学且有近似名义模型的系统解决了这一限制，提供了一种与规划者无关的约束紧缩过程，使现有规划者能够在真实系统上获得概率性任务完成保证。我们利用共形预测为规划问题分布上的名义到真轨迹偏差提供概率界限。我们利用该界限收紧规划约束，并证明在名义模型下解决紧密问题是以规定概率解决真实系统原始问题的充分条件。我们通过实证验证理论保证，并展示了相较于名义模型规划显著提升的任务完成率。

Explainable Reinforcement Learning for assisting Air Traffic Controllers

可解释强化学习，协助空中交通管制员

• Authors: Anduel Mehmeti, Gabriella Gigante, Salvatore Venticinque
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22525
• Pdf link: https://arxiv.org/pdf/2607.22525
• Abstract To effectively integrate AI into high-stakes, critical environments such as healthcare, autonomous driving, and aviation--and to advance toward higher levels of automation and seamless human-AI collaboration--building trust in AI-driven solutions is essential. Trust, in turn, is closely linked to the explainability of AI systems. The rapid advancements in AI across various domains have underscored the challenges of establishing trust, raising increasing interest in AI explainability even more when applied to deep learning. In this context, the present work aims to explore the application of explainability techniques to Reinforcement Learning (RL) algorithms, specifically within the safety-critical domain of Air Traffic Control (ATC). Using a simplified ATC environment as an initial testbed, an intelligent agent is trained with a reinforcement learning algorithm to make decisions on alternative flight routes that avoid no-fly zones. As a preliminary explainability approach, a saliency map is employed, providing insights into the input features that most significantly influence the agent's decision-making process.
• 中文摘要 为了有效将人工智能融入高风险、关键环境，如医疗、自动驾驶和航空领域，并推动更高水平的自动化和无缝的人机协作——建立对人工智能驱动解决方案的信任至关重要。信任反过来又与人工智能系统的可解释性密切相关。人工智能在各个领域的快速进步凸显了建立信任的挑战，进一步激发了对人工智能可解释性在深度学习应用中的日益关注。在此背景下，本研究旨在探索可解释性技术在强化学习（RL）算法中的应用，特别是在安全关键领域——空中交通管制（ATC）领域。以简化的空中交通管制环境为初始测试平台，智能智能体通过强化学习算法训练，以决策避免禁飞区的替代航线。作为一种初步的可解释性方法，采用显著性图，提供对主体决策过程影响最显著的输入特征的洞见。

Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills

技能自玩：通过共同进化技能推动LLM能力的前沿

• Authors: Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, Yu Cheng, Xiaoxi Jiang, Guanjun Jiang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.22529
• Pdf link: https://arxiv.org/pdf/2607.22529
• Abstract LLM training is shifting from manual design and annotation to interaction-driven self-evolution. However, existing self-evolutionary methods face a fundamental dilemma between task diversity and verification reliability: environment-bound methods obtain precise feedback but confine learning to narrow domains, while open-ended self-generation broadens the task space but lacks reliable verification, allowing misleading rewards to pollute the training loop. We identify agent skills as a powerful middle ground to reconcile this tension: each skill ensures deep, verifiable execution in a specific scenario, while dynamic routing across skills maintains open-ended task variety. Leveraging this insight, we introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a dynamic skill controller. Orchestrated via a reinforcement learning loop, these components co-evolve in a continuous self-play loop: the proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library. This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration. Empirical evaluations on tool-use and reasoning benchmarks demonstrate that Skill-SP, serving as a robust evolution engine, consistently pushes the performance ceiling of competent backbones while catalyzing striking turnarounds for initially misaligned models. Our code is available at this https URL.
• 中文摘要 LLM训练正从手工设计和标注转向交互驱动的自我进化。然而，现有的自我进化方法面临任务多样性与验证可靠性之间的根本困境：环境约束方法获得精确反馈但将学习限制在狭窄领域;而开放式自我生成拓宽任务空间但缺乏可靠验证，导致误导性的奖励污染训练循环。我们将代理技能定位为调和这种张力的强大中间地带：每个技能确保在特定场景下深度且可验证的执行，而技能间动态路由则保持开放式任务多样性。基于这一洞察，我们引入了技能自我扮演（Skill-SP），这是一个由提案者、求解者和动态技能控制器组成的共进化框架。通过强化学习循环，这些组成部分在持续的自我游戏循环中共同演化：提案者生成基于动态采样技能的挑战任务;求解器探索候选方案以突破能力边界;技能控制器收集执行反馈以更新和扩展技能库。这种互动共进有效弥合了结构化验证与开放探索之间的鸿沟。工具使用和推理基准的实证评估表明，作为一个稳健的进化引擎，Skill-SP持续推动具备能力骨干的能力上限，同时催化最初错位模型实现显著的转变。我们的代码可在此 https 网址获取。

Keyword: diffusion policy

GRACE: Gradient-Free Robot Action Generation via Combined Diffusion-MPPI Posterior Mean Estimation

GRACE：通过联合扩散-MPPI后验平均估计生成无梯度机器人动作

• Authors: Leesai Park, Jiho HOng, Sanghyun Kim
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.21661
• Pdf link: https://arxiv.org/pdf/2607.21661
• Abstract Diffusion policies generate multimodal robot action sequences from demonstrations, but steering them toward deployment-time constraints typically relies on differentiable guidance costs. This excludes many practical safety constraints, such as binary collision checks, joint limits, and black-box rollout costs that are nondifferentiable. We propose Gradient-free Robot Action generation via Combined diffusion-MPPI posterior mean Estimation (GRACE), which guides a pretrained diffusion policy with Model Predictive Path Integral (MPPI) control using only forward cost evaluations. Building on the common score-ascent structure of diffusion and MPPI, GRACE constructs a cost-conditioned guidance posterior at each reverse step and estimates its mean with a single MPPI update centered at the diffusion reverse mean. For differentiable costs, GRACE recovers conventional gradient guidance under a first-order, matched-covariance approximation. GRACE attains higher success rates than diffusion-based and sampling-based baselines in simulation. On a real 7-DoF manipulator, GRACE avoids a deployment-time obstacle that the unguided prior collides with in every trial. Code and experiment videos are available at this https URL.
• 中文摘要 扩散策略从演示中生成多模态机器人动作序列，但引导其走向部署时间约束通常依赖可微导引成本。这排除了许多实际安全约束，如二元碰撞检查、关节极限和不可微分的黑盒滚动成本。我们提出通过组合扩散-MPPI后验均值估计（GRACE）生成无梯度机器人动作，该方法仅通过前向成本评估指导预训练扩散策略，结合模型预测路径积分（MPPI）控制。基于扩散和MPPI的共同得分-上升结构，GRACE在每个反向步骤构建成本条件后验指导，并以扩散反向均值为中心的单次MPPI更新估计其均值。对于可微成本，GRACE通过一阶匹配协方差近似恢复传统梯度指导。GRACE在模拟中比基于扩散和采样的基线获得更高的成功率。在真实的7-DoF操作手上，GRACE避免了每次试验中无导先验碰撞的部署时间障碍。代码和实验视频可在此 https 网址观看。