生成时间: 2026-10-21 21:32:18 (UTC+8); Arxiv 发布时间: 2026-10-21 20:00 EDT (2026-10-22 08:00 UTC+8)

今天共有 40 篇相关文章

Keyword: reinforcement learning

Navalia: Advancing Cowork Agents through Verifiable Task Synthesis and Long-Horizon Post-Training

Navalia：通过可验证任务综合和远视野后培训推进协同工作代理

• Authors: Hengqi Liu, Ziyuan Liu, Yang Qin, Wanting Zhou, Zichuan Wang, Jiachen Liang, Xu Chu, Shaowei Chen, Zhaokai Luo, Mu Chuan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26887
• Pdf link: https://arxiv.org/pdf/2610.26887
• Abstract Large language model (LLM) agents are evolving from conversational assistants into Cowork Agents capable of autonomous planning, tool use, and complex task execution. Financial analysis requires these agents to integrate cross-document evidence and perform multistep numerical reasoning, but expert task authoring is costly, while direct LLM synthesis struggles to produce challenging tasks with reliable answers. We introduce Navalia, a framework for Cowork Agents that combines verifiable task synthesis with long-horizon post-training. Using corporate financial reports as its data source, Navalia semantically selects supporting evidence from complete fact sets and constructs executable computation chains to generate analytical tasks with traceable evidence and actual data dependencies between steps. We further use independent LLM review and programmatic checks to verify evidence consistency and answer correctness. The verified tasks are then used for trajectory collection, cold-start supervised fine-tuning (SFT), and reinforcement learning (RL) to improve long-horizon task execution. Navalia, initialized from Qwen3.6-35B-A3B, achieves consistent gains over the base model across all five benchmarks: OfficeQA Pro, FinSearchComp, Finance Agent Benchmark, APEX-Agents-AA, and Workspace-Bench-Lite. These results demonstrate that Navalia improves evidence integration, tool use, and long-horizon task execution capabilities and generalizes to broader Cowork settings, including general office tasks.
• 中文摘要 大型语言模型（LLM）代理正从对话助手发展为能够自主规划、工具使用和复杂任务执行的Cowork代理。财务分析要求这些代理整合跨文档证据并执行多步数值推理，但专业的任务编写成本高昂，而直接LLM综合则难以产生具有挑战性的任务并提供可靠答案。我们介绍了Navalia，这是一个结合可验证任务综合与长期训练后培训的Cowork代理框架。以企业财务报告为数据源，Navalia语义上从完整事实集中选择支持证据，构建可执行的计算链，生成具有可追溯证据和实际数据依赖性的分析任务。我们还进一步使用独立的LLM审查和程序检查来验证证据一致性和答案正确性。验证后的任务随后用于轨迹收集、冷启动监督微调（SFT）和强化学习（RL），以提升长期任务执行。Navalia（初始化自Qwen3.6-35B-A3B）在五个基准测试中均实现了持续优于基础模型的提升：OfficeQA Pro、FinSearchComp、Finance Agent Benchmark、APEX-Agents-AA和Workspace-Bench-Lite。这些结果表明Navalia提升了证据整合、工具使用和长期任务执行能力，并可推广至更广泛的Cowork环境，包括一般办公任务。

Large language model and reinforcement learning based autonomous construction and progress updates

基于大型语言模型和强化学习的自主建构与进度更新

• Authors: Yifei Xiao, Huasheng Lin, T.Y. Yang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26899
• Pdf link: https://arxiv.org/pdf/2610.26899
• Abstract Due to the rapid growth of the global population, traditional construction methods are gradually unable to meet the current housing demand. The advancement of machine learning algorithms has encouraged researchers to explore their applications in the construction industry, leading to more intelligent and efficient building processes. In particular, the rise of large language models in recent years has provided more possibilities for the transformation from traditional construction to smart construction. In this study, a large language model and reinforcement learning based autonomous construction method is proposed. An enhanced soft actor-critic reinforcement learning algorithm is developed to plan actions for a mobile crane according to construction tasks. Robot kinematics and control algorithms are used to ensure that the actions generated by the reinforcement learning algorithm can be executed by the crane. In addition, a customized embedding model is developed to extract information from the construction environment according to user interests. Then, a large language model, ChatGPT 5, is used to update construction progress, provide safety information to control system, and generate construction progress reports. The feasibility of proposed method is verified through several full-scall retaining wall construction experiments.
• 中文摘要 由于全球人口的快速增长，传统建筑方法逐渐无法满足当前的住房需求。机器学习算法的进步促使研究人员探索其在建筑行业的应用，推动了更智能、更高效的建筑流程。尤其是近年来大型语言模型的兴起，为从传统建筑向智能建筑的转型提供了更多可能性。本研究提出了一种基于大型语言模型和基于强化学习的自主建筑方法。开发了一种增强型软演员-批判者强化学习算法，用于根据施工任务规划移动起重机的动作。采用机器人运动学和控制算法确保强化学习算法生成的动作能够被起重机执行。此外，开发了定制嵌入模型，根据用户兴趣从建筑环境中提取信息。随后，使用大型语言模型ChatGPT 5更新施工进度，向控制系统提供安全信息，并生成施工进度报告。通过多次全套拔挡土墙施工实验验证了所提方法的可行性。

Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models

通过多目标强化学习与大型语言模型实现帕累托最优RTL代码生成

• Authors: Koki Takeshita, Kenji Uebayashi, Hikaru Tomonari, Masaya Tsunokake, Koichiro Yawata, Tatsuya Sasaki, Yusuke Ohtsubo, Kota Dohi, Yasuhiro Sogawa
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26904
• Pdf link: https://arxiv.org/pdf/2610.26904
• Abstract Recent studies have proposed reinforcement learning with verifiable rewards (RLVR) methods that enable LLMs to generate high-quality RTL circuits from natural-language specifications. Existing approaches optimize circuit PPA (Power, Performance, Area) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto-optimal solutions across different trade-offs such as low power, low latency, and small area. However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high-quality candidate set is beneficial in that designers can choose a preferred trade-off a posteriori. In this work, we are the first to propose a learning method for LLM-based circuit generation that optimizes, not the quality of a single circuit, but the quality and diversity of the entire circuit set obtained through multiple generations in the PPA objective space. The proposed method consists of (1) a generation-based training pipeline that reformulates coverage improvement of the entire Pareto solution set into a sequential circuit generation problem, and (2) a reward function based on Hypervolume Contribution (HVC) that measures how much a generated circuit contributes to the Pareto front. Evaluation on the RTLLM v2 benchmark shows that the proposed method achieves +13.3 percentage points in relative Hypervolume over a baseline without PPA optimization and +14.4 percentage points over a single-objective reinforcement learning method, while also demonstrating superiority in circuit diversity.
• 中文摘要 最新研究提出了可验证奖励（RLVR）强化学习方法，使大型语言模型能够从自然语言规格生成高质量的RTL电路。现有方法通过将多个目标聚合到单一标量奖励中来优化电路PPA（功率、性能、面积），这使得在低功耗、低延迟和小面积等不同权衡下同时生成多样的帕累托最优解变得困难。然而，在实际设计中，每个目标的相对重要性取决于产品需求和设计约束;因此，获得多样且高质量的候选集有利于设计者事后选择优选权衡。本研究提出一种基于LLM的电路生成学习方法，该方法优化的不是单一电路的质量，而是通过多代PPA目标空间获得的整个电路集的质量和多样性。所提方法包括：（1）基于生成的训练流水线，将整个帕累托解集的覆盖改进重新表述为顺序电路生成问题;（2）基于超体积贡献（HVC）的奖励函数，衡量生成电路对帕累托前沿的贡献程度。RTLLM v2基准测试的评估显示，该方法在未进行PPA优化基线时，相对超音量提升了+13.3个百分点，在单目标强化学习方法中实现了+14.4个百分点，同时在电路多样性方面表现出优越性。

Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains

专家引导的评分标准作为不可验证领域强化学习的密集奖励信号

• Authors: Ashish Bansal, Hossam Amer, Keerat Kaur Guliani, Parinita Edke, Lesley Matheson, Leizer Teran
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.26932
• Pdf link: https://arxiv.org/pdf/2610.26932
• Abstract Large language models (LLMs) are increasingly deployed as autonomous agents in non-verifiable domains such as financial services, where they must interpret user intent, rewrite queries, and retrieve information under domain-specific constraints. Although reinforcement learning (RL) can train such agents from the consequences of their actions, recent successes largely rely on verifiable tasks with automatically computed binary rewards. In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert (SME) judgment that is inherently graded rather than binary. With Group Relative Policy Optimization (GRPO), sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall. We introduce a method that converts a small, fixed investment of SME judgment into a dense reward and combines it with retrieval-based metrics to form a hybrid, GRPO-compatible training signal. Experiments show improvements of up to 6\% in Recall@3 and 6.7\% in MRR@3 compared with sparse-reward baselines.
• 中文摘要 大型语言模型（LLMs）越来越多地作为自主代理部署于不可验证的领域，如金融服务，它们必须解读用户意图、重写查询并在领域特定约束下检索信息。尽管强化学习（RL）可以根据行为的后果训练此类代理，但近期成功案例主要依赖于可验证任务和自动计算的二进制奖励。在不可验证的检索任务中，评估查询质量需要稀缺的主题专家（SME）判断，这种判断本质上是分级的，而非二元的。在群相对策略优化（GRPO）中，当一组内所有采样输出都获得相同结果时，稀疏奖励不会产生有用的学习信号，导致优化停滞。我们引入了一种方法，将SME判断的小型固定投资转化为高密度奖励，并将其与基于检索的指标结合，形成混合的、与GRPO兼容的训练信号。实验显示，与稀疏奖励基线相比，Recall@3提升高达6\%，MRR@3提升6.7\%。

Replan for Error Recovery: Training Embodied VLMs through Verifiable Interaction with Action-Grouped Turn-GSPO

重新规划错误恢复：通过可验证的与动作分组回合GSPO交互来培训具身VLM

• Authors: Zhuoran Wang, Keqing Fan, Yang Li, Jinbo Su, Ke Wang, Sijia Luo, Yuanzheng Tan, Junken Peng, Jing Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.27003
• Pdf link: https://arxiv.org/pdf/2610.27003
• Abstract Embodied VLMs can execute increasingly complex manipulation tasks, yet execution errors can leave the environment in states that require replanning for recovery. Existing recovery methods either rely on inference-time mechanisms with fixed policies or learn from failure examples and reference corrections rather than verified action outcomes. To enable recovery learning through verifiable interaction, we introduce the Embodied Replanning for Recovery Environment (ERR-Env), which provides feedback on task progress and execution errors. We further construct ERR-Bench, a benchmark with held-out tasks spanning nominal, intermediate, and error states for evaluating recovery capability and execution reliability. We then propose Action-Grouped Turn-GSPO (AGT-GSPO), an online reinforcement learning method combining turn-level rewards for task progress and error states with action-grouped advantage estimation, and use it to train ERR-VLM, a compact 2B embodied VLM. Evaluation on ERR-Bench reveals a mismatch between task progress and execution reliability across diverse VLMs. AGT-GSPO improves ERR-VLM by 10.25 percentage points in Strict Success Rate for error-free completion and 22.95 percentage points in Recovery Rate, while ablations and zero-shot transfer further validate the effectiveness of our method.
• 中文摘要 具身VLM可以执行越来越复杂的操作任务，但执行错误可能使环境处于需要重新规划恢复的状态。现有恢复方法要么依赖具有固定策略的推断时间机制，要么从失败实例和参考修正中学习，而非验证的行动结果。为了通过可验证的交互实现恢复学习，我们引入了具身恢复重新规划环境（ERR-Env），它提供任务进度和执行错误的反馈。我们进一步构建了ERR-Bench，这是一个基准测试，包含涵盖名义状态、中间状态和错误状态的未完成任务，用于评估恢复能力和执行可靠性。随后，我们提出了动作分组回合GSPO（AGT-GSPO），这是一种在线强化学习方法，结合了任务进展和错误状态的回合级奖励与动作分组优势估计，并用它来训练ERR-VLM，一种紧凑的2B具身VLM。ERR-Bench上的评估显示，不同VLM在任务进度与执行可靠性之间存在不匹配。AGT-GSPO使ERR-VLM的严格成功率提升10.25个百分点（无错误完成率），恢复率提升22.95个百分点，而消融和零样本转移进一步验证了我们方法的有效性。

Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards

通过激活空间引导进行探索，进行可验证奖励的强化学习

• Authors: Yunan Zeng, Jinhao Jing, Xi Cheng, Yuwang Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27012
• Pdf link: https://arxiv.org/pdf/2610.27012
• Abstract Reinforcement learning with verifiable rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and updating the policy from the resulting rewards. The policy therefore receives direct learning signals only from trajectories that appear in its rollouts. Yet repeated sampling from the same policy can concentrate on similar reasoning paths, leaving useful alternatives unexplored under a fixed rollout budget. We introduce Activation-Space Exploration (ASE), an online method for diversifying training rollouts. By clustering activation differences between correct and incorrect responses to the same problem, ASE builds a structured bank of latent exploration vectors. Different vectors are then applied to subsequent rollouts to explore alternative reasoning paths. On Qwen3-1.7B-Base and Qwen3-8B-Base trained on DAPO-Math-17k, ASE improves average Mean@8 across six mathematical reasoning benchmarks over GRPO by 4.87 and 1.87 percentage points, respectively, under the same rollout budget.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过抽样回答、自动验证器评分并根据所得奖励更新策略来训练语言模型。因此，策略只接收来自其推广中出现的轨迹的直接学习信号。然而，重复抽样同一策略可以集中在相似的推理路径上，从而在固定的推广预算下留下未被探索的有用替代方案。我们引入激活-空间探索（ASE），这是一种用于多样化训练推广的在线方法。通过将同一问题正确与错误响应间激活差异聚类，ASE构建了一个结构化的潜在探索向量库。随后将不同的向量应用于后续的推广，以探索替代推理路径。在基于DAPO-Math-17k训练的Qwen3-1.7B-Base和Qwen3-8B-Base上，ASE在同一推广预算下，在六个数学推理基准测试中分别提升了GRPO的平均发Mean@8 4.87个百分点和1.87个百分点。

BR-MoE: Budgeted Semantic Release for Privacy-Preserving Traffic Analytics

BR-MoE：隐私保护流量分析的预算语义发布

• Authors: Abdolazim Rezaei
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27026
• Pdf link: https://arxiv.org/pdf/2610.27026
• Abstract Traffic cameras support useful analysis, but their outputs can reveal private information. Replacing images with text reduces direct visual exposure. It does not place a formal bound on disclosure. The published reinforcement learning and mixture-of-experts framework refines scene descriptions through relevance, coverage, and brevity rewards. It does not account for repeated releases under a fixed privacy budget. This paper proposes budgeted semantic release as an extension of that framework. Four scene experts first process isolated video chunks. Their outputs are mapped to a fixed, bounded semantic schema. A reinforcement learning controller selects which expert summaries to release and how much privacy budget to spend. The controller uses public task requests and previous private releases. A separate privacy filter enforces the total budget. Calibrated noise is added before the summaries are converted to text. Under explicit chunk isolation and bounded-event assumptions, the complete release transcript satisfies an event-level differential privacy bound. The design also supports reuse of previous summaries without new disclosure. An evaluation protocol covers traffic utility, privacy attacks, release freshness, and computational cost. Illustrative tables show the intended reporting format. They do not establish empirical gains. The proposed contribution is a task-aware release interface with auditable privacy accounting for repeated traffic descriptions.
• 中文摘要 交通摄像头支持有用的分析，但其输出可能暴露隐私信息。用文本替代图像减少直接视觉暴露。它不对披露设定正式界限。已发布的强化学习与专家混合框架通过相关性、覆盖范围和简洁性奖励来细化场景描述。它不考虑固定隐私预算下的重复发布。本文提出预算语义发布作为该框架的扩展。四位场景专家首先处理隔离的视频块。他们的输出映射到固定且有界的语义模式。强化学习控制器选择发布哪些专家摘要及花费多少隐私预算。控制器使用公开任务请求和之前的私密发布。一个独立的隐私过滤器执行总预算。在摘要转换为文本之前加入校准噪声。在显式块隔离和有界事件假设下，完整的发布转录满足事件级差别隐私限制。设计还支持重复使用之前的摘要，无需新披露。评估协议涵盖流量效用、隐私攻击、发布新度和计算成本。示例表显示预期报告格式。它们不建立实证收益。拟议贡献是一个任务感知的发布接口，支持可审计的隐私处理重复流量描述。

Glass-Box Deep Reinforcement Learning for High-Speed Flow Control

玻璃盒深度强化学习用于高速流量控制

• Authors: Ameya D. Jagtap
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.27132
• Pdf link: https://arxiv.org/pdf/2610.27132
• Abstract Deep reinforcement learning (DRL) is challenging long-held assumptions in active flow control. By learning directly from interaction with a flow, DRL agents can find actuation strategies for nonlinear, unsteady flows without explicit models, often exploiting mechanisms that classical control would not anticipate. High-speed flows are a demanding test of this promise: shocks, shock/boundary-layer interactions, transition, aerothermal loading and microsecond-scale dynamics strain both algorithms and hardware. Yet most demonstrations remain idealized, low-speed simulations with limited evidence of robustness, interpretability, transferability or experimental deployment. In this perspective, we examine what constrains DRL-based high-speed flow control, including compressible simulation cost and fidelity, data efficiency, sensing in extreme environments, reward design, explainability, and generalization across Mach and Reynolds number. We argue that these limits reflect a tension between data-driven optimization and the structured physics, constraints and uncertainties of real flows. We then highlight where DRL may go beyond conventional approaches: adaptive control of shock-induced separation and turbulent boundary layers, coordination of distributed actuators, and discovery of unconventional flow-manipulation this http URL central challenge, however, is translating these capabilities into reliable technology, which will require validated compressible simulation environments, physics- and constraint-aware learning, robustness to off-design conditions and sensor degradation, real-time operation, hardware-in-the-loop and wind-tunnel validation, and credible pathways to safety and certification.
• 中文摘要 深度强化学习（DRL）正在挑战长期以来在主动流量控制中的假设。通过直接从与流体的交互中学习，DRL智能体可以找到非线性、不稳定流动的驱动策略，而无需显式模型，常利用经典控制无法预见的机制。高速流动是对这一承诺的严峻考验：激波、激波/边界层相互作用、跃迁、气热载荷以及微秒级动力学对算法和硬件造成压力。然而，大多数演示仍停留在理想化的低速模拟中，缺乏鲁棒性、可解释性、可转移性或实验部署的证据。从这个角度来看，我们考察了基于DRL高速流量控制的限制因素，包括可压缩模拟的成本和保真度、数据效率、极端环境下的传感、奖励设计、可解释性以及跨马赫和雷诺数的泛化。我们认为这些限制反映了数据驱动优化与结构化物理、真实流程的约束和不确定性之间的张力。随后，我们强调了DRL可能超越传统方法的地方：冲击诱导分离和湍流边界层的自适应控制、分布式执行器的协调以及非常规流动操作的发现。然而，http URL的核心挑战在于将这些能力转化为可靠的技术，这需要经过验证的可压缩仿真环境、物理和约束感知学习、对设计外条件和传感器劣化的鲁棒性、实时运行、硬件在环和风洞验证，以及可信的安全和认证路径。

Deep catastrophic-risk-aware reinforcement learning

深度灾难风险感知强化学习

• Authors: Carlos Pérez Octavio Mendoza, Frédéric Godin
• Subjects: Subjects: Machine Learning (cs.LG); Risk Management (q-fin.RM)
• Arxiv link: https://arxiv.org/abs/2610.27214
• Pdf link: https://arxiv.org/pdf/2610.27214
• Abstract Estimating extreme risk in reinforcement learning becomes increasingly difficult as the confidence level increases, since only a small number of observations contribute to empirical estimates of Conditional Value-at-Risk (CVaR). In this paper, we extend an Extreme Value Theory (EVT) approach for CVaR optimization to high-dimensional neural-network policies. Using a peaks-over-threshold representation and method-of-moments estimation, we obtain a differentiable tail-risk objective and show that, conditional on the selected threshold, its finite-sample gradient can be computed directly through automatic differentiation. We first validate the approach in a controlled experiment with a known optimal solution and then investigate its performance in a deep-hedging problem with rare extreme market movements. Our results show similar hedging behavior for EVT- and empirical-CVaR-trained policies under normal conditions, while EVT-trained policies exhibit lower hedging losses under the extreme-scenario evaluation across the training sample sizes considered. These results suggest that EVT can provide a practical alternative for training neural policies when the observations relevant to extreme risk are scarce.
• 中文摘要 随着置信水平的提高，估计强化学习中的极端风险变得越来越困难，因为只有少量观测值对条件值风险（CVaR）的实证估计贡献。本文将极值理论（EVT）方法推广到高维神经网络策略。利用峰值超过阈值表示和矩法估计，我们获得可微的尾部风险目标，并证明在所选阈值条件下，其有限样本梯度可通过自动微分直接计算。我们首先在已知最优解的受控实验中验证该方法，然后研究其在极端市场罕见波动的深度对冲问题中的表现。我们的结果显示，在正常条件下，EVT和实证CVaR训练策略的对冲行为相似，而EVT训练策略在极端情景评估下，在所考虑的训练样本量中，对冲损失更低。这些结果表明，当极端风险相关观测值稀少时，EVT可以为训练神经策略提供一种实用的替代方案。

Sparse RL Relaxation for LLM Reasoning

LLM 推理中的稀疏强化学习放松

• Authors: Jiabin Fan, Matthias Lindemann, Zita Marinho, André FT Martins, Lili Mou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27220
• Pdf link: https://arxiv.org/pdf/2610.27220
• Abstract Recent work has explored continuous representations in place of discrete tokens during Chain-of-Thought (CoT) reasoning, often referred to as soft CoT. This is motivated by the idea that a mixture of token embeddings can represent a superposition of multiple reasoning possibilities within a single step. However, it remains challenging to train large language models to realize this potential. Existing work has explored reinforcement learning (RL)-style training for soft CoT. However, these approaches treat soft tokens as actions in a Markov decision process rather than as differentiable intermediate computation steps. Thus, they do not propagate gradients across the soft tokens and miss out on the signal of how the choice of one soft token affects the next. In this paper, we propose a sparse relaxation method for RL (called SpaRLax), which leverages the differentiability of soft-CoT reasoning over time steps and makes training more stable and effective. We adopt an entmax sparsity method that adaptively determines the degree of softness, which truncates long-tail noise at confident steps while retaining multiple plausible candidates when uncertain. Across nine reasoning benchmarks, our method achieves the highest average performance compared with baseline methods. Further analyses validate the effective use of soft-token superposition in our approach, and also demonstrate the contribution of adaptive sparsity in our approach.
• 中文摘要 近期研究探索了在Chain-of-Thought（CoT）推理中用连续表示替代离散词元，通常称为软CoT。其动机是基于这样一个想法：混合词元嵌入可以在单一步内表示多种推理可能性的叠加。然而，训练大型语言模型实现这一潜力仍具挑战性。现有研究曾探索软CoT的强化学习（RL）式训练。然而，这些方法将软词符视为马尔可夫决策过程中的动作，而非可微的中间计算步骤。因此，它们不会在软词符之间传播梯度，也不会传递一个软词符选择对另一个词的信号。本文提出了一种名为SpaRLax的稀疏松弛法，利用软CoT推理在时间步中的微分性，使训练更加稳定和有效。我们采用entmax稀疏度方法，自适应地确定软度程度，在信心步截断长尾噪声，同时在不确定时保留多个合理候选。在九个推理基准测试中，我们的方法实现了与基线方法相比最高的平均性能。进一步分析验证了软标记叠加在我们方法中的有效应用，并证明了自适应稀疏性对我们方法的贡献。

A latent control model for realistic rodent motion

用于真实啮齿动物运动的潜在控制模型

• Authors: Aidan Sirbu, Charles Y. Zhang, Yuanjia Yang, Bence P. Ölveczky, Talmo D. Pereira, Blake A. Richards
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.27279
• Pdf link: https://arxiv.org/pdf/2610.27279
• Abstract The development of embodied models of animal behavior is a major goal of the neuro-AI community. Embodied models will enhance our ability to make direct comparisons between biological and artificial agents. To make such comparisons effective, though, we need embodied models that actually move like animals. Previous work has developed virtual animal bodies and techniques for training realistic motion through imitation. However, to date, there is no system for embodied animal modeling that produces natural movements both during trained goal-directed behavior, and during exploration driven by random generation. Here, we introduce Simulated Control via Augmented Motor Priors for Embodied Reinforcement learning (SCAMPER), a system for training an embodied virtual rat model, enabling natural movements during both goal-directed behavior and random exploration. SCAMPER uses a variational architecture with two distinct priors, a motion prior for control and a behavior prior for exploration. We show that combining both of these priors allows SCAMPER to generate realistic behavior during exploration and exploit the task structure effectively. SCAMPER can also learn from sparse rewards as a result of the diverse movements induced by the behavior prior, allowing the system to model realistic sparse-reward tasks from experimental neuroscience. Altogether, SCAMPER provides the neuro-AI community with an effective new system for training embodied models of rodents on realistic tasks.
• 中文摘要 开发具身动物行为模型是神经人工智能社区的主要目标。具身模型将增强我们直接比较生物与人工代理的能力。然而，要使这种比较有效，我们需要具身模型，能够像动物一样运动。此前的研究开发了虚拟动物身体和通过模仿训练真实运动的技术。然而，迄今为止，尚无具身动物建模系统能在训练目标导向行为和随机生成驱动的探索过程中产生自然运动。本文介绍了通过增强运动先验进行具身强化学习的模拟控制（SCAMPER），这是一种训练具身虚拟老鼠模型的系统，使目标导向行为和随机探索都能实现自然运动。SCAMPER采用变分架构，具有两个不同的先验，控制的运动先验和探索的行为先验。我们证明，结合这两个先验使SCAMPER能够在探索过程中生成真实行为，并有效利用任务结构。SCAMPER还能从稀疏奖励中学习，这源于行为先验引发的多样化动作，使系统能够模拟实验神经科学中真实的稀疏奖励任务。总体而言，SCAMPER为神经人工智能社区提供了一个有效的新系统，用于训练具身啮齿动物模型执行真实任务。

VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching

VPFlow：基于不完美世界模型实现线下目标达成的价值预测规划

• Authors: Yunzhe Hu, Dong Xu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27280
• Pdf link: https://arxiv.org/pdf/2610.27280
• Abstract Offline goal-conditioned reinforcement learning enables skill acquisition by reaching any goal from any state without dense rewards. Model-based methods allow for planning by scaling test-time compute to solve complex tasks. Combining both holds appealing promise for pre-training on suboptimal demonstrations and zero-shot adaptation via goal specification and planning. Yet existing methods assume effective planning requires accurate next-state prediction in pixel or latent space. In this work, we show that value prediction is more useful for planning, whereas accurate state prediction is often unnecessary. Our key insight is that offline goal reaching naturally admits a closed-form Monte Carlo return, enabling value regression to replace temporal-difference bootstrapping. We use this return as the sole supervision for latent world models: imagined latents are trained to predict goal-reaching value rather than match future states, implicitly preventing representation collapse. These planning-oriented representations let imperfect world models imagine on the level sets of value field beyond dataset transitions. We then present VPFlow, a value-predictive planner that pairs the world models with a one-step flow policy and plans in its noise space via evolutionary search. It enables policy improvement with imagined generalized advantage and reduces out-of-distribution queries common in action-space planning. On 30 state and pixel OGBench datasets, VPFlow surpasses model-free baselines by at least 30.3 and 10.1 percentage points in success rate, respectively, and outperforms latent planners LeWorldModel, PLDM, and TD-MPC2. It achieves this despite a latent prediction error orders of magnitude larger than with state prediction, while minimizing this error lowers success rate. This demonstrates that value-predictive representations, not exact state prediction, are what offline planning requires.
• 中文摘要 离线目标条件强化学习使技能习得能够从任意状态达到任意目标，而无需密集的奖励。基于模型的方法通过扩展测试时间计算实现规划，以解决复杂任务。结合两者在预训练次优演示和通过目标指定与规划实现零机会适应方面具有吸引力。然而现有方法假设有效规划需要在像素或潜在空间中准确预测下一状态。本研究表明，价值预测对规划更有用，而准确状态预测往往并非必需。我们的关键见解是，离线目标达成自然允许闭式蒙特卡洛回归，使价值回归取代时间差分自助法成为可能。我们将该返回作为潜在世界模型的唯一监督：想象中的潜在变量被训练预测目标达成价值，而非匹配未来状态，隐含防止表征崩溃。这些以规划为导向的表示使不完美世界模型能够在数据集转移之外的价值场层级上设想。随后，我们介绍VPFlow，一款价值预测规划器，通过进化搜索将世界模型与一步流策略和计划配对。它实现了策略改进，并实现了想象中的广义优势，并减少了行动空间规划中常见的分布外查询。在30个状态和像素OGBench数据集中，VPFlow的成功率分别比无模型基线高出至少30.3个百分点和10.1个百分点，并且优于潜在规划器LeWorldModel、PLDM和TD-MPC2。尽管潜在预测误差比状态预测高出数量级，它仍能实现这一目标，而最小化该误差则降低了成功率。这表明离线规划所需的是价值预测表示，而非精确状态预测。

Vulnerability Analysis for Safe Multi-Agent Reinforcement

安全多智能体强化的脆弱性分析

• Authors: Taozheng Zhu, Shixiong Jiang, Fanxin Kong
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27293
• Pdf link: https://arxiv.org/pdf/2610.27293
• Abstract Safe Multi-Agent Reinforcement Learning (SMARL) aims to learn cooperative control policies under prescribed safety constraints. Al- though existing SMARL methods focus on enforcing safety during policy optimization, their vulnerability remains largely unexplored. This paper investigates inference-time observation attacks against SMARL, where an adversary perturbs the observations of a subset of agents, thereby altering the actions produced by their policies. We propose MinMax Attack, an attack framework that jointly selects adversarial target actions for compromised agents while accounting for the Cooperative Resistance of benign agents. We further derive a bound on the induced safety cost and characterize its relation- ship with the Compromised-Agent Budget and the Perturbation Budget. Experiments across multiple safety-critical multi-agent benchmarks and representative state-of-the-art SMARL algorithms show that MinMax Attack induces substantially higher safety costs than existing attack baselines while preserving task reward, thereby reducing its detectability and achieving both attack effectiveness and stealthiness.
• 中文摘要 安全多智能体强化学习（SMARL）旨在学习在规定安全约束下的合作控制策略。尽管现有SMARL方法专注于在策略优化过程中执行安全，但其脆弱性仍大多未被充分探索。本文探讨针对SMARL的推理时间观察攻击，即对手扰动部分智能体的观测，从而改变其策略所产生的动作。我们提出了极大攻击（MinMax攻击），该框架在考虑良性智能体的协同抵抗性时，共同选择被受攻击的智能体的对抗目标动作。我们进一步推导出诱导安全成本的上界，并描述其与受损智能体预算和扰动预算的关系。在多个安全关键的多智能体基准测试和代表性最先进的SMARL算法上进行的实验表明，MinMax攻击在保持任务奖励的同时，诱导的安全成本远高于现有攻击基线，从而降低其可检测性，同时实现攻击的有效性和隐蔽性。

Look Before You Reason: Implicit Visual Thinking for Efficient Multimodal Reasoning

先看后推理：隐含的视觉思维促进高效的多模态推理

• Authors: Zerui Chen, Changrui Chen, Fei Ni, Jiankang Deng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.27341
• Pdf link: https://arxiv.org/pdf/2610.27341
• Abstract Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet they remain limited on tasks requiring fine-grained understanding of small or easily overlooked image regions. Recent thinking with images approaches mitigate this limitation by allowing models to inspect images through operations such as zooming, cropping, or code execution. While effective, their multi-turn interactions with VLMs introduce redundant token computation and slow inference, and separate region inspection from the reasoning step that depends on it. To address these issues, we propose Look Before You Reason (LoRe), an implicit visual thinking framework for efficient visual reasoning within a single-turn VLM interaction. Given an image and a question, LoRe predicts the most relevant image region, aligns it with the corresponding image patch features, and strengthens the associated key-value (KV) cache entries, enabling subsequent decoding to directly exploit the selected visual cues without additional visual inputs or explicit tool calls. We further employ reinforcement learning to guide the VLM toward identifying informative image regions and leveraging them during reasoning. To support fine-grained evaluation of multimodal reasoning, we introduce COCO-ObjQA, a large-scale benchmark for object-centric multimodal reasoning. Experiments on three challenging multimodal benchmarks demonstrate that LoRe consistently improves VLM reasoning performance with low additional inference overhead.
• 中文摘要 视觉语言模型（VLM）在多模态推理方面进展迅速，但在需要细致理解小或易被忽视图像区域的任务上仍受限。近期图像方法的思考通过允许模型通过缩放、裁剪或代码执行等操作检查图像，缓解了这一限制。虽然有效，但它们与VLM的多回合交互引入了冗余的令牌计算和缓慢的推理，并将区域检查与依赖于其的推理步骤分离。为解决这些问题，我们提出了“先看再推理”（Look Before You Reason，LoRe）的隐式视觉思维框架，用于在单回合VLM交互中高效进行视觉推理。给定一张图像和一个问题，LoRe预测最相关的图像区域，将其与相应的图像补丁特征对齐，并强化相关的关键值（KV）缓存条目，使后续解码能够直接利用选定的视觉线索，无需额外的视觉输入或显式工具调用。我们进一步运用强化学习，引导VLM识别信息性图像区域并在推理过程中加以利用。为支持多模态推理的细粒度评估，我们引入了COCO-ObjQA，这是一个面向对象的多模态推理的大规模基准测试。在三个具有挑战性的多模态基准测试上的实验表明，LoRe能够以较低的推理开销持续提升VLM推理性能。

HealBench: A Fine-Grained Benchmark for Evaluating Psychological Comfort in Text-to-Image Generation

HealBench：评估文本转图像生成心理舒适度的细致基准

• Authors: Junjie Xu, Yuzhe Yang, Rui Yang, Luwei Xiao, Wenxin Wu, Xingjiao Wu, Liang He
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27516
• Pdf link: https://arxiv.org/pdf/2610.27516
• Abstract Despite advances in photorealism, semantic fidelity, and aesthetics, text-to-image generation and image editing remain understudied in a human-centered dimension: whether their outputs feel psychologically comforting. Grounded in cognitive evaluation theory, we define psychological comfort as an affective response shaped by viewers' appraisal of restorative and emotionally supportive visual cues. We introduce HealBench, a fine-grained benchmark of 30,000 high-quality image-text pairs spanning 19 categories and 92 subcategories, selected from 100,000 candidates through a human- and expert-in-the-loop pipeline combining vision-language model matching, human preference models, expert filtering, and exemplar-guided augmentation. We also construct HealBench-Eval, a feature-clustered evaluation set of 1,095 instances. Using HealBench-Eval, we conduct objective and subjective evaluations of 14 representative image generation models and prompt-rewriting strategies. Results reveal a gap between conventional visual quality and perceived comfort: strong semantic alignment and aesthetics do not guarantee emotionally restorative images. Reinforcement learning experiments further show that HealScorer can serve as a reward signal for assessing and improving the emotional and restorative quality of generated images. These findings establish psychological comfort as a distinct dimension of human-centered image generation and position HealBench as a foundation for affect-aware generative models.
• 中文摘要 尽管在写实、语义忠实和美学方面取得了进步，文本生成和图像编辑在以人为中心的维度上仍然缺乏研究：即其输出是否带来心理安慰。基于认知评估理论，我们将心理舒适定义为由观众对修复性和情感支持视觉线索的评估塑造的情感反应。我们介绍了HealBench，这是一个细粒度基准，包含3万对高质量图像-文本对，涵盖19个类别和92个子类别，通过结合视觉语言模型匹配、人类偏好模型、专家过滤和示例引导增强的人类与专家参与流程，从10万名候选人中选出。我们还构建了HealBench-Eval，一个包含1095个实例的特征聚类评估集。利用HealBench-Eval，我们对14个代表性图像生成模型及提示重写策略进行了客观和主观评估。结果揭示了传统视觉质量与感知舒适感之间的差距：强烈的语义对齐和美学并不保证图像具有情感修复效果。强化学习实验进一步表明，HealScorer可以作为评估和提升生成图像情感及修复性质量的奖励信号。这些发现确立了心理舒适度作为以人为本图像生成的一个独特维度，并将HealBench定位为情感感知生成模型的基础。

From POMDP Theory to Deep RL with Particle Filters

从POMDP理论到带粒子滤波器的深度强化学习

• Authors: Kevin Tan, Ziping Xu
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.27575
• Pdf link: https://arxiv.org/pdf/2610.27575
• Abstract Model-based reinforcement learning in partially observable environments requires inferring hidden states, learning latent dynamics, and planning under the resulting belief. Practical algorithms often solve one aspect of the above pipeline while relying on approximations for the others that are often poorly understood, while existing POMDP theory typically analyzes computationally inefficient theoretical algorithms far from modern deep RL practice. To fill this gap, we propose a practical end-to-end algorithm that approximately solves the hidden state filtering problem via sequential Monte Carlo (SMC), updates a latent dynamics model online, and performs approximate online planning for $d$ steps before bootstrapping leaves with a state-value critic (a QMDP approximation). We analyze each approximation and prove an end-to-end regret bound decomposing the error into model estimation error, particle filter error, Monte Carlo rollout error, and a structural bias term induced by the QMDP approximation controllable by the planning depth that we further analyze in various special cases. The resulting guarantee makes explicit how statistical sample size, particle budget, rollout budget, and planning depth affect performance. We instantiate our algorithm in an MPC-based deep RL framework, demonstrating its practicality on a variety of environments. On a POMDP that satisfies our assumptions, each term of the bound decays at its predicted rate. Our results provide a practical and theoretically grounded route for bridging theory and practice within online decision-making in POMDPs.
• 中文摘要 在部分可观测环境中进行基于模型的强化学习需要推断隐态、学习潜在动态，并在由此产生的信念下进行规划。实用算法通常解决上述流水线中的某一方面，同时依赖其他方面常被理解不足的近似，而现有的POMDP理论通常分析计算效率较低的理论算法，远离现代深度强化学习实践。为填补这一空白，我们提出了一种实用的端到端算法，通过顺序蒙特卡洛（SMC）近似求解隐藏状态过滤问题，在线更新潜在动力学模型，并在带状态值批判者（QMDP近似）前对$d$步进行近似在线规划。我们分析每个近似，并证明了一个端到端的遗憾界限，将误差分解为模型估计误差、粒子滤波器误差、蒙特卡洛展开误差以及由QMDP近似引起的结构偏置项，该偏差项可由规划深度控制，我们在各种特殊情况下进一步分析。所得保证明确说明了统计样本量、粒子预算、推广预算和规划深度如何影响性能。我们将算法实例化为基于MPC的深度强化学习框架，展示了其在多种环境中的实用性。在满足假设的POMDP上，束缚项的每个项以预测速率衰减。我们的结果为POMDP中的在线决策提供了理论与实践的结合，提供了实用且理论基础的路径。

From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents

从结果到步骤：长远视野语言代理的可靠成功意识批评者学习

• Authors: Jiaxi Liu, Shixiong Kai, Zhentao Tang, Yang Li, Yuqi Cui, Mingxuan Yuan, Junchi Yan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.27623
• Pdf link: https://arxiv.org/pdf/2610.27623
• Abstract Reinforcement learning for long-horizon language agents requires identifying whether intermediate actions make eventual task success more or less likely when supervision is available only at trajectory level. The main difficulty is that a critic with aggregate return-prediction accuracy may still fail to resolve the local value differences needed to assign credit to actions, especially under sparse feedback and function approximation. Conventional critics estimate future return, but policy improvement also depends on knowing whether each intermediate decision makes task success more or less likely. We introduce SCALE, an actor-critic framework that augments conventional return estimation with explicit prediction of eventual task success. At each interaction prefix, the critic predicts a return value and a Success Value, defined as the probability that the task will eventually be completed from that prefix. SCALE converts changes in Success Value into multi-step action credit through Success-GAE and modulates its contribution according to the estimated reliability of the predictor, while retaining the token-level PPO objective. Across five agent benchmarks and three model backbones, SCALE improves task performance over the compared RL baselines in most settings, with gains of up to 4.1 percentage points in success rate. Separate critic-learning analyses show that SCALE also improves critic estimation and training stability.
• 中文摘要 针对长视野语言代理的强化学习需要识别在仅在轨迹层面有监督时，中间行动是否使最终任务成功的可能性增加或降低。主要难点在于，拥有汇总回报预测准确率的批评者可能仍可能无法解决给予行动功劳所需的局部价值差异，尤其是在稀疏反馈和函数近似下。传统批评者估计未来回报，但策略改进还依赖于了解每个中间决策是否使任务成功的可能性增加或降低。我们引入SCALE行为者-批评者框架，它通过明确预测任务最终成功的预测来补充传统回报估计。在每个交互前缀处，批评者预测一个回报值和成功值，定义为任务最终从该前缀完成的概率。SCALE通过Success-GAE将成功值的变化转换为多步行动积分，并根据预测变量的估计可靠性调节其贡献，同时保持代币级PPO目标。在五个代理基准和三个模型骨干中，SCALE在大多数环境中提升任务表现，成功率提升多达4.1个百分点。独立的批评者学习分析显示，SCALE还提升了批评者估计和训练稳定性。

Reusing Multimodal Agents: Efficient Evaluation of Sound Designs in Games

多模态代理的重用：游戏中声音设计的高效评估

• Authors: Thai Nguyen Van, Ibrahim Khan, Mury Fajar Dewantoro, Pratch Suntichaikul, Hai Van Pham, Ruck Thawonmas
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27670
• Pdf link: https://arxiv.org/pdf/2610.27670
• Abstract This paper introduces a novel framework for efficiently evaluating in-game sound designs using multimodal learning and reinforcement learning. Sound design significantly influences gameplay and accessibility; however, objective evaluation methods, defined here as automated assessments based on agent performance rather than human ratings, often require retraining agents from scratch for each sound design. Our approach first pre-trains a multimodal encoder that encodes visual and auditory information into a shared latent space. A reinforcement learning agent is then trained using this latent representation. We fine-tune the multimodal encoder for each sound design while keeping the trained policy fixed, allowing us to isolate the effects of sound changes on agent behavior. To evaluate the proposed method, we conducted experiments on games from multiple genres, comparing the performance of blind agents (agents using only sound input) under various sound designs. Experimental results suggest that this approach can detect differences in agent performance caused by degraded or altered sound design. Moreover, we significantly reduce the computational cost compared to retraining the agent from scratch. Experimental results suggest that this approach provides a practical and scalable solution for evaluating in-game sound designs. The source code, data and supplementary document are available: this https URL.
• 中文摘要 本文引入了一种新颖框架，用于利用多模态学习和强化学习高效评估游戏内声音设计。声音设计显著影响游戏玩法和可及性;然而，客观评估方法（此处定义为基于代理表现而非人类评分的自动评估）通常需要为每个声音设计从头重新训练代理。我们的方法首先预训练一个多模态编码器，将视觉和听觉信息编码到共享的潜在空间中。然后用该潜在表示训练强化学习代理。我们在保持训练策略固定的同时，微调每个声音设计的多模态编码器，从而隔离声音变化对代理行为的影响。为评估所提方法，我们对多种类型的游戏进行了实验，比较盲代理（仅使用声音输入的代理）在不同声音设计下的表现。实验结果表明，该方法能够检测因声音设计退化或改变引起的代理表现差异。此外，我们显著降低了计算成本，而非从零开始重新训练代理。实验结果表明，这种方法为评估游戏内音效设计提供了实用且可扩展的解决方案。源代码、数据和补充文档已提供：此 https URL。

Prescribed Performance Guided Safe Reinforcement Learning for Roundabout Navigation of Autonomous Vehicles

自主车辆绕行导航的规定性能引导安全强化学习

• Authors: Wenhan Li, Yangrui Zhang, Hang Zhou, Peiran Liu, Yiding Ji
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.27732
• Pdf link: https://arxiv.org/pdf/2610.27732
• Abstract Connected and Autonomous Vehicles (CAVs) face significant challenges in roundabout navigation due to dense traffic interactions and unpredictable agent behaviors, necessitating high levels of adaptability and safety. While the combination of Reinforcement Learning (RL) and Model Predictive Control (MPC) offers a promising approach, existing frameworks suffer from a structural mismatch where pointwise RL decisions fail to align with MPC's trajectory level optimization, and often lack rigorous safety guarantees. This work proposes RLTrajGen-PPMPC-CBF, a hierarchical framework for safe, smooth and comfort roundabout navigation, which regulates the coupling between RL-based planning and MPC-based control. We employ Prescribed Performance Constraints (PPC) to enforce a gradual, stable convergence between reference trajectories and control outputs, ensuring MPC tracking performance remains consistent with intended behavior from RL. Additionally, a Control Barrier Function (CBF) layer provides real-time safety corrections that are fed back into the RL process. Simulation results in the MetaDrive environment demonstrate that our method achieves a 6.25\% increase in average success rate and a 65.87\% reduction in Motion Sickness Dose Value (MSDV) compared to the optimal baseline, meanwhile accelerating training convergence.
• 中文摘要 连接和自动驾驶车辆（CAV）在环岛导航中面临重大挑战，原因是交通交互密集且代理行为不可预测，需要高度的适应性和安全性。虽然强化学习（RL）与模型预测控制（MPC）的结合提供了有前景的方法，但现有框架存在结构不匹配的问题，即点状强化学习决策未能与MPC的轨迹级优化保持一致，且常缺乏严格的安全保障。本研究提出了RLTrajGen-PPMPC-CBF，这是一个安全、平顺且舒适的环岛导航分层框架，规范基于强化学习的规划与基于MPC的控制之间的耦合。我们采用预定性能约束（PPC）来强制执行参考轨迹与控制输出之间的渐进、稳定收敛，确保MPC跟踪性能与强化学习的预期行为保持一致。此外，控制障碍函数（CBF）层提供实时安全修正，反馈回强化学习过程。MetaDrive环境下的模拟结果表明，我们的方法相比最优基线实现了6.25%的平均成功率提升，晕动症剂量值（MSDV）减少65.87%，同时加速了训练收敛。

RSI Control: Can AI Models Perform Covert Sabotage in the Self-Improvement Process?

RSI控制：AI模型能否在自我提升过程中进行隐蔽破坏？

• Authors: Kaiwen Zhou, Udari Madhushani Sehwag
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27735
• Pdf link: https://arxiv.org/pdf/2610.27735
• Abstract Recursive self-improvement, in which frontier AI models work on training the next generation, is quickly becoming a reality. However, the safety of this process has received far less attention than it should. A misaligned model training its successor can shape what that successor knows, how it behaves, and what it wants, and the harm carries forward into the next generation. We study this risk through the lens of AI control. We introduce RSI Control Arena, a testbed in which a frontier model, acting as an autonomous R\&D agent, is given a benign main task from a stage of a successor model's training pipeline together with a covert side task to install a misaligned property in the successor. The testbed contains 16 side tasks in three classes, forming 22 task pairs with main tasks that span the pipeline from data curation to reinforcement learning. We evaluate six frontier models as agents against LLM monitors that read the transcript of the run and the agent's reasoning. We observe that every model completes a substantial share of the side tasks without degrading the main-task performance, including side tasks that make the successor itself a better saboteur. Under minimal instructions to stay covert, models already produce successful attacks that monitors fail to detect, and providing a short list of evasion tactics further reduces the detection rate. Monitors with access to agent's reasoning detects most attacks, but not all. Our results indicate that recursive self-improvement raises significant safety concerns, and that strong safety effort should be in place before models are trusted with greater responsibility of training their successors.
• 中文摘要 递归自我改进，即前沿AI模型致力于训练下一代，正迅速成为现实。然而，这一过程的安全性受到的关注远未达到应有的水平。错位的模型训练其继任者，可能会影响其知识、行为和需求，这种伤害会延续到下一代。我们通过AI控制的视角研究这一风险。我们介绍RSI控制领域，这是一个测试平台，前沿模型作为自主研发代理，从继任模型训练流程的某阶段获得一个良性主任务，同时执行一个隐性任务，用于在继任模型中安装错位属性。该测试平台包含16个支线任务，分为三类，形成22对任务，主要任务涵盖从数据管理到强化学习的流程。我们评估了六个前沿模型作为代理，对抗大型语言模型监控器，后者读取运行记录和智能体推理。我们观察到，每个模型都能完成大量支线任务，同时不降低主任务性能，包括使继任者成为更好破坏者的支线任务。在最低限度的隐蔽指令下，模型已产生监控者未能侦测的成功攻击，且提供简短的规避策略列表进一步降低了侦测率。能够访问代理推理的监控者能检测大多数攻击，但并非全部。我们的结果表明，递归自我改进引发重大安全隐患，在模型被信任承担更多培训继任者的责任之前，应加强安全措施。

RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

RoboJudge：多模态语言模型作为具身视频生成的评判

• Authors: Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang, Weiyang Liu, Siyuan Wang, Yandong Wen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.27782
• Pdf link: https://arxiv.org/pdf/2610.27782
• Abstract Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.
• 中文摘要 视频生成模型是具身世界模型的有前景基础，但前提是其生成的推广在物理上有效且忠实于任务指令。多模态大型语言模型（MLLM）越来越多地用于判断这些属性，但它们与人类对具身视频判断的一致性尚未被系统评估。为弥补这一空白，我们引入了RoboJudge-Bench，这是评估MLLM作为具身视频生成评判者的基准。该基准测试了两个基本维度：物理依从性和指令对齐，涵盖8个源语料库和10个视频生成器的800个人工注释视频，涵盖握把、人手和灵巧手。实验显示现有法官与人类评分之间的一致性不均：通用MLLM在身体依从性上达成较低，而专门设计用于一般领域物理推理的视频评判则不稳定地转化到具身场景。基于这些发现，我们构建了RoboJudge-Train，包含12K视频，标注了2.5万个人验证分数和模型辅助的身体依从性和指令对齐理由。我们在该数据集上对9B的MLLM进行微调，先进行监督学习和强化学习，生成RoboJudge-9B。在RoboJudge-Bench评估的模型评审中，RoboJudge-9B整体一致性最高，Pearson相关系数为0.719，统计学上显著地优于Gemini-3.7-Flash，后者是我们评估中表现最好的专有基线。RoboJudge-Train训练还能在外部基准测试（包括VideoPhy2和Physion-Eval）上取得提升。我们还使用RoboJudge-9B作为下游评估器，根据物理依从性和指令对齐对视频生成模型进行基准测试。

SAPD: Learning from Agent Experience without Environment Interaction

SAPD：从代理经验中学习而不涉及环境互动

• Authors: Juncheng Diao, Haibin Wang, Wei Shi, Yifan Ding, Qingbin Li, Rongxiang Weng, Xunliang Cai
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.27847
• Pdf link: https://arxiv.org/pdf/2610.27847
• Abstract Scaling the training of Large Language Model (LLM) agents typically requires a large collection of executable environments, whose construction, maintenance, and operation constrain the expansion of training experience. Existing offline trajectories offer an opportunity to reuse interaction experience, yet effectively refining the current policy without revisiting these environments remains challenging: supervised fine-tuning (SFT) can lead to catastrophic forgetting, while online reinforcement learning requires real-time interaction with the environment. To address this challenge, we propose Step-Advantage Policy Distillation (SAPD), a method for learning from offline trajectories using relative step advantages. SAPD decomposes multi-turn trajectories into individual decision contexts for on-policy distillation, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training. The entire process requires neither new environment observations nor additional multi-turn interactions. Experiments on ALFWorld, WebShop, and Search-based QA show that SAPD improves performance across different teacher configurations, mitigates catastrophic forgetting compared with SFT, and runs at least $4.7\times$ faster per training step than online OPD. We will release our code soon.
• 中文摘要 大规模语言模型（LLM）代理的训练规模化通常需要大量可执行环境，这些环境的构建、维护和运行限制了训练体验的扩展。现有的离线轨迹提供了重用交互经验的机会，但在不重新访问这些环境的情况下有效优化当前策略仍然具有挑战性：监督微调（SFT）可能导致灾难性的遗忘，而在线强化学习则需要与环境的实时交互。为应对这一挑战，我们提出了阶梯优势策略蒸馏（SAPD），这是一种利用相对阶梯优势从离线轨迹中学习的方法。SAPD将多回合轨迹分解为个体决策上下文，用于策略上蒸馏，使学生能够基于记录历史生成响应，并引入相对阶梯优势信号以稳定训练。整个过程既不需要新的环境观测，也不需要额外的多回合交互。在ALFWorld、WebShop和基于搜索的QA中的实验显示，SAPD能提升不同教师配置的表现，减少灾难性遗忘，相比SFT减少灾难性遗忘，且每培训步骤比在线OPD快至少4.7倍。我们将很快发布代码。

Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

诱导还是扩展？解开智能体束带优化带来的神秘面纱

• Authors: Younghyun Kim, Wooseok Seo, Minseo Kim, Seunghyuk Oh, Minjae Lee, Donghoon Kim, Hyung Il Koo, Jaehyung Kim, Wonjun Kang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.27889
• Pdf link: https://arxiv.org/pdf/2610.27889
• Abstract Recent studies in harness optimization have shown substantial performance gains in language model agents without updating model weights. A well-designed harness more effectively leverages the underlying model's capabilities by shaping context, tool use, and execution workflows. However, naïve success rates alone do not fully characterize whether optimized harness yields more reliable success on tasks already within reach (elicitation) or a broader range of solvable tasks (expansion). This distinction is crucial for recursive self-improvement and evolutionary search, as expansion may provide a broader range of successful experiences to push the capability boundaries. Despite the growing significance, this distinction remains underexplored in harness optimization. We conduct a systematic analysis to study the effect of gains from harness optimization. Building on pass@k analyses in the reinforcement learning (RL) literature that compare models before and after RL training, we compare optimized harnesses with their pre-optimization counterparts. Specifically, we assess changes in reachable task coverage, compare harness optimization with model scaling and increased inference-time compute, and examine how gains vary with task difficulty. Across static prediction and dynamic tasks, we find that pass@1 advantages of harness over stronger interventions can reverse at larger attempt budgets and aggregate gains conceal shortfalls on hard tasks, underscoring the need for a fine-grained diagnosis.
• 中文摘要 近期关于工具束优化的研究显示，语言模型代理在不更新模型权重的情况下，性能可大幅提升。设计良好的工具框架通过塑造上下文、工具使用和执行工作流，更有效地利用底层模型的能力。然而，单靠简单的成功率无法完全说明优化框架在已可及的任务（引发）上是否能带来更可靠的成功，还是更广泛的可解决任务（扩展）。这一区别对于递归自我提升和进化搜索至关重要，因为扩展可能带来更广泛的成功体验，推动能力边界。尽管这一区别日益重要，但在工具束优化中仍未被充分探讨。我们进行系统分析，研究工具束优化带来的收益效果。基于强化学习（RL）文献中比较模型的强化学习前后pass@k分析，我们将优化后的增强器与优化前的对应工具进行比较。具体来说，我们评估了可达任务覆盖的变化，比较了束缚优化与模型扩展和推断时间增加计算，并考察了收益如何随任务难度变化。在静态预测和动态任务中，我们pass@1发现束缚相较于强干预的优势在更大尝试预算下可以逆转，整体收益掩盖了困难任务中的不足，凸显了细致诊断的必要性。

From Comparison to Selection: Enhancing Cross-Scale Generalization of Pointer Networks Trained with Reinforcement Learning for Capacitated Drone Delivery Routing

从比较到选择：增强用强化学习训练的指针网络的跨尺度泛化，用于电容无人机投递路由

• Authors: Sofiene Lassoued, Devara Tertia, Stefan Lier, Andreas Schwung
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.27936
• Pdf link: https://arxiv.org/pdf/2610.27936
• Abstract Unmanned aerial vehicles (UAVs) enable fast, flexible last-mile and emergency delivery, but their routing remains challenging as the number of clients varies. Classical approaches based on mixed-integer programming and heuristics can scale poorly or require problem-specific design, motivating learning-based alternatives. Reinforcement learning is particularly promising, provided a single policy can generalize across problem sizes without retraining. Although attention-based Pointer Networks are inherently size-agnostic, their solution quality degrades under both upscaling and downscaling. Using the Capacitated Vehicle Routing Problem (CVRP) as a controlled proxy, with fixed capacity reflecting drone payload constraints, we investigate this degradation and identify three contributing factors: shifts in node density, attention dispersion, and dilution of the mean-pooled graph context. We systematically compare architecture-wide, encoder-side, and decoder-side modifications to address these limitations and propose a learned, permutation-invariant graph-context aggregation mechanism, implemented as a weighted average and as Set2Set. Results reveal distinct requirements for transfer in each direction: entropy-based attention scaling reduces the CVRP20-to-CVRP100 performance gap by 1.41 percentage points, while the proposed aggregation methods achieve the largest improvement on the more challenging CVRP100-to-CVRP20 transfer, approximately 0.9 percentage points. Exploiting this complementarity, we introduce an approach selector that chooses the best model for each training-target size pair. The selector reduces cost relative to the reference model across all nine seen train-test pairs and eight of nine pairs involving unseen sizes (CVRP35, CVRP75, and CVRP200), achieving improvements of up to 3.2 percent.
• 中文摘要 无人机（UAV）实现了快速、灵活的最后一公里和紧急配送，但由于客户端数量变化，其路由仍然具有挑战性。基于混合整数规划和启发式的经典方法扩展性较差，或需要针对问题的设计，促使基于学习的替代方案。强化学习尤其有前景，前提是单一策略能够在不重新训练的情况下跨问题规模进行推广。尽管基于注意力的指针网络本质上不受大小限制，但其解决方案质量在放大和降尺度下都会下降。利用电容车辆路由问题（CVRP）作为受控代理，固定容量反映无人机有效载荷约束，我们研究了这种退化，并识别出三个促成因素：节点密度的变化、注意力分散以及均值池图上下文的稀释。我们系统地比较了架构范围内、编码器侧和解码器侧的修改，以解决这些限制，并提出了一种学习式、置换不变的图-上下文聚合机制，作为加权平均和Set2Set实现。结果显示，每个方向的转移都有不同的需求：基于熵的注意力尺度将CVRP20到CVRP100的性能差距减少了1.41个百分点，而所提出的聚合方法在更具挑战性的CVRP100到CVRP20转移方面取得了最大提升，约0.9个百分点。利用这一互补性，我们引入了一个方法选择器，为每个训练目标大小对选择最佳模型。该选择器在所有九对可见的列车测试对以及涉及未见尺寸的九对中八对（CVRP35、CVRP75和CVRP200）相较于参考模型降低成本，实现了高达3.2%的改进。

RANGER: Retrieval Agent for Navigating Graphs with Semantic Evaluation Rewards

RANGER：带语义评估奖励的导航图的检索代理

• Authors: Artem Mikhaylov, Pavel Konoplev, Nikolay Efanov, Artyom Sorokin, Evgeny Burnaev
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28010
• Pdf link: https://arxiv.org/pdf/2610.28010
• Abstract The integration of Large Language Models and Reinforcement Learning offers a promising direction for complex sequential decision-making tasks, such as multi-hop question answering over knowledge graphs. While LLMs provide strong semantic understanding and reasoning capabilities, they often struggle with precise, multi-step navigation in structured environments. Conversely, RL agents excel at learning navigation policies but are hindered by the difficulty of designing semantically meaningful reward functions. This paper introduces a novel hybrid approach in which an LLM serves as a mentor to an RL agent during training by providing dense, semantically informed reward signals. The RL agent, built on a Transformer-based policy and trained with Group Relative Policy Optimization, learns to navigate a knowledge graph to retrieve relevant information. The LLM evaluates the semantic relevance of the agent's traversed paths, generating reward signals that align the agent's behavior with the information needs expressed in a query. We integrate this Retrieval Agent for Navigating Graphs with Semantic Evaluation Rewards (RANGER) into a complete question-answering pipeline and evaluate it on the open question answering datasets. Experimental results show that the proposed approach outperforms both a baseline retrieval-augmented system with classical graph search and one using an RL agent trained with heuristic rewards, achieving a significant increase in NLP metrics and LLM-as-a-Judge scores. Our findings demonstrate that deep integration of LLMs into the RL training loop yields more effective and semantically aligned retrieval policies for complex reasoning tasks.
• 中文摘要 大型语言模型与强化学习的整合为复杂的顺序决策任务（如知识图谱上的多跳问答）提供了有前景的方向。虽然大型语言模型提供强大的语义理解和推理能力，但在结构化环境中常常难以实现精确的多步导航。相反，强化学习代理在学习导航策略方面表现出色，但因设计语义有意义的奖励函数困难而受限。本文引入了一种新颖的混合方法，LLM在训练过程中通过提供密集、语义依据的奖励信号，作为强化学习代理的导师。强化学习代理基于基于Transformer的策略，并接受群相对策略优化训练，学习如何导航知识图谱以获取相关信息。LLM评估代理所经过路径的语义相关性，生成奖励信号，使智能体的行为与查询中表达的信息需求保持一致。我们将该带语义评估奖励的图导航检索代理（RANGER）集成到完整的问答流程中，并在开放问答数据集上进行评估。实验结果显示，所提方法优于采用经典图搜索的基线检索增强系统和使用启发式奖励训练的强化学习代理，显著提升了自然语言处理指标和LLM作为评判的得分。我们的发现表明，将LLM深度集成到强化学习训练循环中，能为复杂推理任务带来更有效且语义对齐的检索策略。

From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

从强化学习论文到框架代码：七个开源培训框架的纳入与评审

• Authors: Taiheng Pan
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.28027
• Pdf link: https://arxiv.org/pdf/2610.28027
• Abstract Practitioners use reinforcement learning methods for language models through training frameworks, so a method that a framework includes is one configuration change away for its users. We measure how often this happens. We trace seven open-source frameworks for RL post-training through their commit histories and link them to 6,939 screened arXiv papers from January 2025 to June 2026. Of 1,598 papers that propose a general RL training method, 37 reached at least one framework. Across five quarterly cohorts of papers, the number included within 180 days stayed between 5 and 7 while the cohorts grew from 72 to 386 papers. Compared with papers from the same quarters, included papers more often trained policies of 30B parameters or more, at 41 against 10 percent, and more often changed a coefficient, clip, mask or weight in the existing loss, at 29 against 9 percent. Both gaps hold without papers from the companies that maintain the frameworks, while code release was equally common in both groups. The frameworks' issues and pull requests show that few methods are turned away in public. More than nine in ten of the general methods no framework included have no thread there that cites or names them, and many proposals that do appear stay open without a reply. Two further records, the frameworks' issues and the technical reports of 13 large open-weight models, show that users rarely report training failures in public and that the reports state 8 of the 13 widely included methods. We release the data as a tracker that links training failures to framework implementations and lists each framework's contribution rules and earlier proposals.
• 中文摘要 实践者通过训练框架对语言模型使用强化学习方法，因此框架中包含的方法对用户来说只需一次配置更改即可。我们测量了这种过程发生的频率。我们通过提交历史追踪了七个开源的强化学习训练后框架，并将其与2025年1月至2026年6月间筛选的6,939篇arXiv论文联系起来。在1,598篇提出通用强化学习训练方法的论文中，有37篇至少达到了一个框架。在五个季度一次的论文队列中，180天内纳入的论文数量保持在5至7篇之间，而队列数量从72篇增长到386篇。与同期论文相比，纳入的论文更常训练30B及以上的策略，比例为41%对10%，且更频繁地改变现有流失中的系数、夹子、掩码或权重，为29%对9%。没有维护框架的公司提交论文，这两个空白都存在，而代码发布在两组中同样普遍。框架的问题和拉取请求显示，公开拒绝的方法很少。超过九成未包含框架的通用方法没有相关线索引用或命名，许多出现的提案也未获回复。另外两条记录——框架问题和13个大型开放权重模型的技术报告显示，用户很少公开报告训练失败，报告中列出了13种广泛包含的方法中的8种。我们将数据作为跟踪器发布，将训练失败与框架实现关联，列出每个框架的贡献规则和早期提案。

LoCA: Local Credit Assignment for Reinforcement Learning of Autoregressive Flow-Matching TTS

LoCA：用于强化学习自回归流匹配TTS的局部学分赋值

• Authors: Jie Liu, Junyu Zhou, Junhao Zhong, WeiSen Tang, Rukai Cai, Haoyue Zhan, Ruiqi Xing, Ling Long
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.28035
• Pdf link: https://arxiv.org/pdf/2610.28035
• Abstract Reinforcement learning can improve text-to-speech synthesis through feedback on generated speech. However, sentence-level rewards do not identify local errors, while signed regression updates can lead to excessive fitting or repulsion of sampled outputs. We propose LoCA (Local Credit Assignment) for autoregressive flow-matching text-to-speech. LoCA uses ASR-based text alignment to locate content errors in generated speech and redistribute sentence-level advantages across acoustic frames. To improve training stability, we introduce reference-relative positive and negative thresholds that limit continued fitting and repulsion. Experiments on VoxCPM2 show that LoCA reduces sample-weighted content error on a multilingual CV3 evaluation subset by 20.5% and 14.3% relative to the pretrained backbone and GROW, respectively, while improving speaker similarity. LoCA also improves content accuracy on the Seed-TTS-Eval hard subset. Ablation results support the contributions of local credit and threshold-based regression control.
• 中文摘要 强化学习可以通过对生成语音的反馈提升文本到语音合成。然而，句子级奖励无法识别局部错误，而符号回归更新可能导致采样输出的过度拟合或排斥。我们提出了LoCA（局部信用分配）用于自回归流匹配文本转语音。LoCA利用基于ASR的文本对齐定位生成语音中的内容错误，并将句子层面优势重新分布到声学框架中。为提升训练稳定性，我们引入了参考相对的正负阈值，限制持续拟合和排斥。VoxCPM2上的实验显示，LoCA在多语言CV3评估子集上相较预训练骨干和GROW分别减少了20.5%和14.3%的样本加权内容误差，同时提高了说话者的相似度。LoCA还提高了Seed-TTS-Eval硬子集的内容准确性。消融结果支持局部信用和基于阈值的回归控制的贡献。

TrajEvolve: Trajectory Evolution for Reinforcement Learning with Hindsight Credit Assignment

TrajEvolve：带回顾性强化学习的轨迹演变学分作业

• Authors: Yi Wen, Hao Chen, Wanyu Wang, Maolin Wang, Pengyue Jia, Derong Xu, Huize Tan, Yingyi Zhang, Wenlin Zhang, Weihong Luo, Xiku Du, Xiangyu Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.28060
• Pdf link: https://arxiv.org/pdf/2610.28060
• Abstract The Reinforcement Learning (RL) paradigm has achieved remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, when applied to long-horizon tasks, it faces severe challenges: i) Due to the high task complexity, the model struggles to sample enough successful trajectories, resulting in slow convergence. ii) Relying solely on sparse final rewards causes all steps to be updated indiscriminately, encouraging redundant step generation. When both the quantity and quality of successful trajectories become difficult to guarantee, the performance is severely constrained. This dilemma stems from the limitations: existing RL paradigms rely entirely on the reasoning capability of base models, lacking the ability to actively optimize trajectories. To address the above issues, we propose a novel trajectory evolution paradigm with hindsight credit assignment, termed TrajEvolve. TrajEvolve not only relies on the trajectory inferred by the model but also generates ``evolutionary trajectories'' by refining low-quality steps in historical trajectories, thereby exposing the policy to high-quality compositions that lie within the base model's support but are rarely sampled within practical training budgets. Furthermore, TrajEvolve reformulates step-level continuous credit assignment as a binary classification task, whether each step is necessary for task completion. This simplification not only reduces the learning difficulty but also enables step labels to be obtained through counterfactual verification, achieving precise identification of redundant steps. Extensive experiments on two datasets demonstrate the superiority of the proposed method.
• 中文摘要 强化学习（RL）范式在提升大型语言模型（LLMs）推理能力方面取得了显著成功。然而，应用于长期任务时，它面临严峻挑战：i） 由于任务复杂度高，模型难以采样足够成功轨迹，导致收敛缓慢。ii） 仅依赖稀疏的最终奖励会导致所有步骤被无差别地更新，助长冗余步骤生成。当成功轨迹的数量和质量难以保证时，性能将受到严重限制。这一困境源于局限性：现有强化学习范式完全依赖基础模型的推理能力，缺乏主动优化轨迹的能力。为解决上述问题，我们提出了一种带有事后诸葛亮赋值的新型轨迹演化范式，称为TrajEvolve。TrajEvolve不仅依赖模型推断的轨迹，还通过细化历史轨迹中的低质量步骤生成“进化轨迹”，从而使策略暴露于基础模型支持范围内但很少在实际训练预算中采样的高质量组合。此外，TrajEvolve将步骤级连续学分分配重新表述为二元分类任务，判断每一步是否对任务完成必要。这种简化不仅降低了学习难度，还使得通过反事实验证获得步骤标签，实现了冗余步骤的精确识别。对两个数据集的广泛实验证明了该方法的优越性。

Training the Whole Agent, Not Just the Model

训练整个代理，而不仅仅是模型

• Authors: Lingxiao Tang, Wei Tao, Yiwen Guo, Lingfeng Bao
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.28069
• Pdf link: https://arxiv.org/pdf/2610.28069
• Abstract Training coding agents typically updates the policy while keeping the harness fixed. During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns. We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. The policy trains with the adapted harness, and its new trajectories guide further adaptation. With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. Gains from mechanism-level skills persist after their removal at evaluation. Re-evolving tools and hooks as the policy changes outperforms freezing early versions. Removing them at deployment, in contrast, costs up to 7.7 percentage points. Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.
• 中文摘要 训练编码代理通常在保持束缚固定的同时更新策略。在培训过程中，轨迹揭示了缺陷的修复策略和交互问题，这些问题因任务而异，并随着策略学习而变化。我们引入了HarnessAdapt-RL，它在强化学习过程中训练策略并调整其束缚。对于有缺陷的修复策略，固定教师生成任务特定的机制级技能。对于跨任务反复出现的交互问题，一个独立的教师模型会演化工具和钩子，只有当每次变更提升观察到的通过率或观察到的可训练率而不降低任何一方时才保留。该策略与适应后的约束框架进行训练，其新轨迹引导进一步适应。基于9B基础模型，HarnessAdapt-RL在SWE-bench Verified、Multilingual和Pro版本上相比GRPO提升pass@1分别提升9.8、12.6和6.4个百分点，并且在使用人工设计的OpenHands框架时优于GRPO。机制级技能的提升在评估时被移除后依然存在。随着策略变化，重新演进工具和钩子的效果优于冻结早期版本。相比之下，部署时移除它们的成本高达7.7个百分点。因此，训练会产生模型-工具与工具和钩子对：训练策略和进化工具及钩子，技能在部署时被移除。

Interruption-Aware Recurrent Reinforcement Learning for UAS Navigation Under Temporary Overrides

在临时覆盖下，针对无人机导航的中断感知重复强化学习

• Authors: Christopher Conrad, Deepak Panda, Weisi Guo, Antonios Tsourdos
• Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.28121
• Pdf link: https://arxiv.org/pdf/2610.28121
• Abstract Safe autonomous navigation of UASs requires tactical deconfliction to address conflicts that strategic coordination cannot prevent. This task is challenging under partial observability, when sensor or traffic data are missing, delayed, or degraded. Geometry-based methods can offer separation assurances but face limitations in dense encounters. Recurrent reinforcement learning has therefore been proposed to learn navigation policies that integrate observation-action histories to optimise long-term safety-efficiency trade-offs. However, operational safeguards may require a remote pilot, onboard detect-and-avoid system, or runtime assurance mechanism to temporarily override the learned policy. A research gap remains in understanding how these interruptions affect recurrent policies when the agent regains authority. Benchmark evidence shows that interruptions make task variables less decodable from the recurrent hidden state and increase post-interruption collision risk. The findings suggest that altered recurrent context prolongs post-handback degradation through an action-observation feedback loop, in which changed actions alter the flight path and subsequent recurrent updates. To address this vulnerability, we propose three interruption-aware architectures that condition recurrent processing on interruption context through separate feature streams, dedicated interruption memory, or gated environmental features. Across simulations, all three architectures preserve more decodable task information than a standard recurrent policy. Moreover, the strongest architecture reduces post-interruption collisions by 74% and interruption-induced mission failures by 69% relative to the recurrent baseline, without degrading interruption-free performance. These results show that safe interruption does not guarantee safe handback, and support interruption-aware recurrent design for resilient UAS autonomy.
• 中文摘要 无人机的安全自主导航需要战术性冲突消除，以解决战略协调无法避免的冲突。在部分可观测性下，当传感器或交通数据缺失、延迟或退化时，这一任务尤为具有挑战性。基于几何的方法可以提供分离保证，但在密集遭遇中存在局限性。因此，被提出通过循环强化学习来学习整合观察-行动历史的导航策略，以优化长期的安全效率权衡。然而，操作保障措施可能需要远程飞行员、机载检测与避让系统或运行时保证机制来暂时覆盖已学到的策略。研究中仍有一个空白，目前尚不清楚这些中断如何影响代理恢复权限时的重复策略。基准证据表明，中断使任务变量难以从重复隐藏状态中解码，并增加中断后碰撞风险。研究结果表明，改变的循环上下文通过动作-观察反馈循环延长回传后退化，该循环中改变动作改变飞行路径及后续的重复更新。为解决该漏洞，我们提出了三种中断感知架构，通过独立的功能流、专用中断内存或门控环境特征来对中断上下文进行条件反射处理。在模拟中，这三种架构都比标准重复策略保留更多可解码的任务信息。此外，最强架构相较于重复基线，将中断后碰撞减少74%，中断引发的任务失败减少69%，同时不降低无中断性能。这些结果表明，安全中断并不保证安全回传，支持中断感知的循环设计以实现韧性无人机自主性。

Reframing Object-Centric Reinforcement Learning: From Absolute to Relative Representations

重构以对象为中心的强化学习：从绝对到相对表征

• Authors: Cedric Derstroff, Jannis Blüml, Janosch Moos, Michael Girstl, Zihan Ye, Debora Clever, Kristian Kersting
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.28197
• Pdf link: https://arxiv.org/pdf/2610.28197
• Abstract Object-centric reinforcement learning agents typically receive each object's position in a fixed global frame, even though their decisions usually depend on where objects are relative to one another. These absolute views often fail to capture the local context that is most relevant to the agent's behavior. As a result, an agent must relearn the same response at every location where a configuration can occur, and its performance collapses when the scene is merely shifted. In this work, we propose a shift toward Object-Relative Coordinates fixed to Anchor (ORCA), where entities are described in a relative frame of reference. To operationalize this idea, we introduce Object-Relative inductive-Bias Transformers (ORBiT), a transformer-based architecture for processing anchor-relative inputs in reinforcement learning. Our results suggest these relative representations improve performance for both MLP and transformer agents. Furthermore, we demonstrate that ORBiT outperforms the pixel-based PPO baseline, achieving an aggregate HNS IQM greater than 1.0 over 13 Atari games. ORBiT learns the game of Foosball from scratch, dominating all baselines. In our controlled Tag environment, we further show that ORCA extends to rotations, that ORBiT generalizes to unseen numbers of distractors, and that these gains require a decision-relevant anchor.
• 中文摘要 以对象为中心的强化学习代理通常会接收每个对象在固定全局框架中的位置，尽管其决策通常取决于物体之间的相对位置。这些绝对视图往往无法捕捉与代理行为最相关的局部上下文。因此，代理必须在每个配置可能发生的位置重新学习相同的响应，且仅仅移动场景时，其性能就会崩溃。本研究提出向固定对象相对坐标（ORCA）转变，即在相对参考系中描述实体。为实现这一理念，我们引入了对象相对归纳偏置变换器（ORBiT），这是一种基于变换器的架构，用于强化学习中处理锚点相对输入。我们的结果表明，这些相对表示能够提升MLP和变换器代理的性能。此外，我们证明ORBiT在13款Atari游戏中表现优于基于像素的PPO基线，实现了HNS IQM的总值大于1.0。ORBiT从零学习桌上足球，主导所有基线。在我们受控的标签环境中，我们进一步证明ORCA扩展到旋转，ORBiT推广到看不见数量的干扰因素，这些提升需要决策相关的锚点。

Reward Misspecification Degrades Chain-of-Thought Monitorability

奖励错误指定降低了思维链的可监控性

• Authors: Tianyi Lorena Yan, John Hewitt
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.28303
• Pdf link: https://arxiv.org/pdf/2610.28303
• Abstract Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at this https URL.
• 中文摘要 监控语言模型（LM）的思维链（CoT）是检测和防止错误行为以保障人工智能安全的关键策略。因此，识别导致无法监控思维链的强化学习（RL）实践至关重要。其中一种做法可能是不可避免的：奖励错误指定。随着强化学习环境日益复杂，完美验证变得不可行，模型因行为与真实任务意图不同而获得奖励。我们证明，仅奖励错误指定本身就已经降低可监测性。我们创建了一个测试平台，包含与任务意图对齐的真实奖励和偏向不同行为的错误奖励，从而通过训练诱发奖励黑客行为并可验证地标记。在三种模型和三个任务中，CoT可监控性保持较低，模型被黑客攻击率上升：CoT不断推理任务意图，而输出违反意图并获得奖励。为了减少在不先了解真实奖励或黑客的情况下无法监控的情况，我们收集了不同训练步骤中多样化奖励的发布，并使用学习管理推断奖励描述。然后我们将描述注入输入，使模型能够将奖励口头表达。这种奖励描述训练在所有模型和任务中，根据指标平均提升了24%或42%的绝对可监控性，并产生能够明确推理错误奖励的CoT。代码可在此 https URL 访问。

Reinforced Iterative Learning Control for elastic joint arms: Guaranteed cross-trajectory generalization under model uncertainties

弹性关节臂的强化迭代学习控制：模型不确定性下的保证交叉轨迹推广

• Authors: Yuri De Santis, Michele Pierallini, Franco Angelini, Manolo Garabini
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.28329
• Pdf link: https://arxiv.org/pdf/2610.28329
• Abstract Trajectory tracking for compliant robots is challenging due to nonlinear dynamics and modeling uncertainties, often requiring learning-based methods to avoid explicit parameter identification. Iterative Learning Control (ILC) ensures theoretically high precision and preserves the passive compliance of the robot. However, traditional ILC requires a separate learning phase for each new trajectory. Conversely, Reinforcement Learning (RL) generalizes to new trajectories, but practically suffers from high sample complexity and limited tracking precision. This work proposes Reinforced Iterative Learning Control (RILC) that combines the strengths of both approaches to achieve fast adaptation and guaranteed tracking convergence across diverse trajectories. By establishing a concurrent coupling between the two controllers, RILC overcomes the high initial errors and slow adaptation of traditional ILC, enabling high-performance tracking from the very first trials. In this architecture, ILC provides high-quality samples, while RL contributes as a generalizing best-effort policy across iterations. Additionally, a theoretical proof is presented to show that the RL component preserves ILC stability and, under specific conditions, can also reduce the error bound asymptotically. The efficacy of this framework is demonstrated through simulation and experiments on an underactuated 4-degree-of-freedom serial elastic robot using different trajectories and modeling mismatches.
• 中文摘要 对于符合标准的机器人进行轨迹跟踪具有挑战性，因为其存在非线性动力学和建模不确定性，通常需要基于学习的方法以避免显式参数识别。迭代学习控制（ILC）确保理论上的高精度，并保持机器人的被动顺应性。然而，传统ILC需要为每个新轨迹单独学习阶段。相反，强化学习（RL）可以推广到新的轨迹，但实际存在高样本复杂性和有限的跟踪精度。本研究提出了强化迭代学习控制（RILC），结合两种方法的优势，实现快速适应和保证跨多轨迹的跟踪收敛。通过建立两个控制器之间的并发耦合，RILC克服了传统ILC的高初始误差和缓慢的适应，实现了从最初试验起就实现高性能跟踪。在该架构中，ILC提供高质量样本，而RL作为跨迭代的推广性尽力而为策略贡献力量。此外，还提出了理论证明，证明RL组件保持ILC稳定性，并在特定条件下可渐近地减少误差界限。该框架的有效性通过在欠驱动的4自由度串联弹性机器人上使用不同轨迹和建模错配的模拟和实验得到验证。

Fast Motion Retargeting Pipeline for Scalable Humanoid Robot Learning

可扩展类人机器人学习的快速运动重定向流水线

• Authors: Daniel Marew, Sangjun Park, Nisal Perera, Donghyun Kim
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.28470
• Pdf link: https://arxiv.org/pdf/2610.28470
• Abstract While large-scale human motion capture datasets provide a rich source of demonstrations for humanoid robot learning, retargeting the diverse motions to different humanoid robots remains a nontrivial challenge. Existing kinodynamic trajectory optimization pipelines are computationally expensive and often require manual adjustments per motion. We present FRDV (Fast Retargeting with Dynamics Validation), a scalable framework consisting of rapid constrained kinematic retargeting and an additional dynamics validation pipeline. Its kinematic stage integrates a custom Sequential Quadratic Programming solver, Real-Time Iteration, and parallel chunk-based processing, achieving up to 9,156 frames per second, approximately up to 136 times faster than GMR and ProtoMotions, while robot-specific constraints are incorporated into the optimization to reduce common retargeting artifacts, such as ground penetration and self-collision. For motions requiring additional verification, a contact implicit iLQR controller is utilized to test the retargeted reference under full rigid body dynamics. This dynamics-based validation identifies dynamically infeasible motions and refines contact-related artifacts such as foot floating and foot skating. We evaluate retargeting quality by training motion tracking reinforcement learning policies using several state-of-the-art pipelines and comparing their tracking fidelity and success rates. Furthermore, sim-to-real transfer is demonstrated on a Unitree G1 humanoid. Videos and code are available at: this https URL.
• 中文摘要 虽然大规模人体动作捕捉数据集为人形机器人学习提供了丰富的演示资源，但将多样化动作重新定位到不同类人机器人仍是一项非凡的挑战。现有的运动动力学轨迹优化流程计算量大，且通常每次动作都需要手动调整。我们介绍了FRDV（带动力验证的快速重定向），这是一个可扩展框架，包含快速约束的运动学重定向和额外的动力学验证流水线。其运动学阶段集成了定制的顺序二次规划求解器、实时迭代和基于块的并行处理，最高可达9,156帧每秒，约比GMR和ProtoMotions快136倍，同时在优化中融入了机器人专用约束，以减少常见的重定向伪影，如地面穿透和自我碰撞。对于需要额外验证的运动，使用接触隐式iLQR控制器在全刚体动力学下测试重定向参考。这种基于动力学的验证识别了动态不可行的运动，并优化了与接触相关的伪影，如脚漂和脚滑。我们通过使用多个最先进的流水线训练运动跟踪强化学习策略，并比较其跟踪精度和成功率，评估重定向质量。此外，模拟到真实的传输在Unitree G1人形机器人上进行了演示。视频和代码可在以下链接获取。

Answering in the User's Variety: Calibrated Accommodation Pairs for Post-Training

用用户多样化回答：为培训后校准的住宿对

• Authors: Zhizhuo Kou, Heming Li, Jian Yang, Junyu Luo, Jingshu Peng, Sirui Han, Yike Guo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.28474
• Pdf link: https://arxiv.org/pdf/2610.28474
• Abstract People adapt how they talk to the person in front of them. However, chat models tend to respond in Standard English regardless of the dialect used by the user. A uniform answer can hide whether training treats these users alike, and output audits cannot tell as well. We trace how each post-training stage, from supervised fine-tuning (SFT) to preference optimization (DPO) and reinforcement learning, changes a model's tendency to match the user, and read what each update prefers from two content-matched replies, one matching the user and one not. Across 10 open model lineages, post-training standardizes replies to every informal cue, yet after adjusting for how likely each reply already was, the DPO update still favours matching British and Australian spelling over AAE and Singlish on 10 lineages, and models still produce AAE features when asked. We then introduce calibrated accommodation pairs (CAP), preference data that reward matching users' dialect density while preserving Standard English for Standard-English users. On a fully open model, CAP raises convergence on features it was not trained on without a detectable increase in dialect for Standard-English users or a cost in instruction following. In contrast, one-sided pairs reach a similar gain only by overshooting the user and imposing dialect. Our findings show that uniform replies can hide selective updates and audits should read the update as well as the output. Code: this https URL.
• 中文摘要 人们会调整与面前的人交流的方式。然而，聊天模型往往会用标准英语回应，无论用户使用何种方言。统一的回答可以隐藏训练是否对这些用户一视同仁，输出审计也无法判断。我们追踪了每个训练后阶段，从监督微调（SFT）到偏好优化（DPO）再到强化学习，如何改变模型匹配用户的倾向，并从两个内容匹配的回复中读取每个更新的偏好，一个匹配用户，一个不匹配。在10个开放模型谱系中，训练后标准化了对每个非正式提示的回复，但在调整了每个回复的概率后，DPO更新在10条谱系中仍偏好匹配英式和澳大利亚拼写而非AAE和新加坡语，模型在被问及时仍能提供AAE特征。随后引入校准适应对（CAP），即奖励匹配用户方言密度的偏好数据，同时保持标准英语对标准英语用户的标准英语。在完全开放模型中，CAP在未训练的特征上提升收敛率，且标准英语用户方言可检测到增加或指令跟随成本。相比之下，单侧对仅通过超越用户并强加方言来获得类似收益。我们的发现表明，统一回复可以隐藏选择性更新，审计应同时读取更新内容和输出。代码：此https URL。

SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model

超级模仿者：用一个可微分的世界模型扩展人形全身控制

• Authors: Qiayuan Liao, Yiyang Shao, Junfeng Long, Kevin Zakka, Roberto Horowitz, Pieter Abbeel, Koushil Sreenath
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.28480
• Pdf link: https://arxiv.org/pdf/2610.28480
• Abstract Tracking large human-motion datasets has become the dominant route to general, expressive humanoid whole-body control, but scaling across many motions is expensive. Current pipelines use model-free reinforcement learning, either merging per-motion specialists through lossy distillation, or brute-forcing a single policy with cluster-scale compute. SuperMimic amortizes learning across all motions with one differentiable world model of their shared dynamics, fit from logged transitions of a standard rigid-body simulator. The world model and the tracking policy train together in one loop, and backpropagation-through-time over short imagined rollouts gives the policy analytic gradients. Fast imagined rollouts and analytic gradients make policy learning efficient. In 3 hours on a single GPU, SuperMimic surpasses the return of a state-of-the-art motion tracker baseline trained for 7 days on 128 GPUs. It reduces GPU-hours by more than $1000\times$ and fails on only 25 of 129,514 benchmark motions, versus 179 failures for the baseline. After 9.8 hours on two GPUs, SuperMimic achieves zero failures. From 1 to 264 hours of motion, the world model's prediction error stays stable, and GPU-hours to a target return grow slowly. Our policy transfers to a physical Unitree G1 across diverse and acrobatic motions. SuperMimic removes compute as the bottleneck to scaling humanoid whole-body control, making it practical to measure motion-tracking scaling laws.
• 中文摘要 追踪大型人体运动数据集已成为实现通用、富有表现力的人形全身控制的主流途径，但跨多动作进行扩展成本高昂。当前流水线采用无模型强化学习，要么通过有损蒸馏合并每动作专家，要么用集群级计算暴力破解单一策略。SuperMimic通过一个可微世界模型摊销所有运动的学习，模型基于标准刚体模拟器的记录转换拟合共享动态。世界模型和跟踪策略在一个循环中共同训练，通过短暂想象的滚动反向传播，生成策略分析梯度。快速的想象展开和分析梯度使策略学习高效。在单一GPU上3小时内，SuperMimic超越了在128个GPU上训练7天的先进运动追踪基线。它将GPU小时数减少超过1000美元，且在129,514次基准动作中仅有25次失败，而基线测试中仅有179次失败。在两块GPU上运行9.8小时后，SuperMimic实现了零失败。从1小时到264小时的运动时间，世界模型的预测误差保持稳定，GPU小时数到目标回波时间缓慢增长。我们的策略在多样且具杂技性的动作中转移到物理Unitree G1。SuperMimic消除了计算作为人形全身控制缩放瓶颈的限制，使得测量运动追踪尺度定律变得切实可行。

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

离散数据的连续时间策略梯度：收敛分析与无模型算法

• Authors: Martin Hernandez, Xinyan Qing, Mo Zhou, Yuhua Zhu
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.28580
• Pdf link: https://arxiv.org/pdf/2610.28580
• Abstract Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.
• 中文摘要 连续时间强化学习通过随机微分方程驱动的未知动态，利用离散时间收集的观测数据，寻求最优策略。我们开发了一个策略梯度框架，结合了严谨的收敛分析与离散数据的学习。首先建立性能差引理，推导随机化和参数化策略的策略梯度公式。随后，我们证明熵正则化策略流的指数收敛至正则化最优值。对于离散参数化更新，我们推导出一个收敛界限，明确区分批判误差、策略回归误差和学习步长的贡献。为实现未知动态下的学习，我们将该框架与PhiBE集成，PhiBE从离散转移数据构建连续时间代理生成器。由此产生的开策略和非策略模型无演员-批判算法保持了控制问题的连续时间结构。在熵正则化线性二次调节器和非线性轮式机器人上的数值实验展示了所提方法在已知和数据驱动动力学条件下的性能和稳定性。

AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics

AgenticRoboRL：将编码代理后培训带入智能机器人领域

• Authors: Xinxin Liu, Bingxin Xu, Wenrui Bao, Ming Li, Wentao Bao, Chen Chen, Yuzhang Shang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.28589
• Pdf link: https://arxiv.org/pdf/2610.28589
• Abstract Agentic robotics systems combine planners with robot harnesses to perform complex manipulation tasks. Recent work improves these systems through better harnesses and external memory, but relies on powerful closed-source models as planners and does not improve the planners' own capabilities. In contrast, coding agents have improved substantially by post-training the model itself. We therefore investigate whether post-training can also improve robot planners. However, applying it across robot harnesses requires accommodating different execution mechanisms, including VLA skill invocation and control-code generation. To this end, we introduce AgenticRoboRL, a framework for post-training robot planners across different harnesses. The training recipe combines supervised fine-tuning with multi-turn reinforcement learning. To accommodate different execution mechanisms, harness-specific adapters connect recorded planner interactions with task outcomes, enabling a common RL training pipeline while keeping execution components fixed. On LIBERO-PRO, AgenticRoboRL raises the success rate of a 9B-parameter planner from 18.1% to 71.0% without external memory, approaching the 72.1% achieved by Codex (GPT-5.5) with external memory. These results demonstrate post-training as an effective approach for improving robot planners. We will release the framework and trained planner checkpoints.
• 中文摘要 智能机器人系统将规划器与机器人安全带结合，执行复杂的操作任务。近期工作通过改进工具和外部存储改进了这些系统，但依赖强大的闭源模型作为规划工具，并未提升规划者自身能力。相比之下，编码代理通过对模型本身进行后训练已显著提升。因此，我们探讨后训练是否也能提升机器人规划器。然而，将其应用于机器人工具框架需要适应不同的执行机制，包括VLA技能调用和控制代码生成。为此，我们引入了AgenticRoboRL，这是一个适用于不同工具后训练机器人规划者的框架。该训练方案结合了监督微调与多回合强化学习。为适应不同的执行机制，基于规范的适配器将录制的规划器交互与任务结果连接起来，实现统一的强化学习流水线，同时保持执行组件的固定性。在LIBERO-PRO上，AgenticRoboRL将9B参数规划器在无外部存储情况下的成功率从18.1%提升至71.0%，接近Codex（GPT-5.5）带外部存储的72.1%。这些结果表明后训练是提升机器人规划器的有效方法。我们将发布框架和训练后的规划检查点。

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

通过混合灵巧模仿学习适应性灵巧技能

• Authors: Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye, Sungjae Park, Umar Iqbal, Katerina Fragkiadaki, Shubham Tulsiani
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28628
• Pdf link: https://arxiv.org/pdf/2610.28628
• Abstract Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: this https URL.
• 中文摘要 灵巧的多指手有望解锁功能性抓握、手部重新定向和动态切换等人类设计世界中所假设的多样技能，但现有学习方案迫使其权衡。目标达成强化学习可推广到任意物体姿势，但由于缺乏人类先验，最终收敛为不自然且不可靠的手指策略;动作追踪通过单一片段再现类人行为，但不能偏离该片段。我们介绍AdaDex，一个目标条件化的强化学习框架，将几秒钟的手-物体轨迹---从动作捕捉数据或基于野外RGB（D）视频的HOI重建---转变为可适应的灵巧技能。AdaDex 的核心是混合灵巧模仿：共享策略训练于两个大规模平行环境群体：运动跟踪环境采用密集运动跟踪目标，从人类运动中植入人类先验;以及泛化环境，初始状态和目标噪声化，物体成功奖励稀疏，迫使技能远离参考。从每项技能的人类动作秒数，策略可学习多样化且灵活的灵巧技能，如功能抓取、物体定位、手指步态的手部重新定向，以及投掷接球等动态行为。我们的方法在泛化和功能上优于基线，并展示了模拟器间及现实世界的迁移。项目页面：此 https URL。

Keyword: diffusion policy

CLOSER: Evidence-Aware Active Visual Verification for Embodied Object-Goal Navigation

结尾：具身对象-目标导航的证据感知主动视觉验证

• Authors: Pengying Wu, Mingye Jiang, Yunze Hu, Zhejia Wen, Kangjie Zhou, Zongru Yang, Zhe Peng, Chang Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.28476
• Pdf link: https://arxiv.org/pdf/2610.28476
• Abstract Object-goal navigation (ObjectNav) requires an embodied agent to find an instance of a specified category in an unseen environment and stop near it. The agent typically relies on an object detector but must judge online from a limited egocentric video stream, sometimes a single frame, whether a detected candidate is the target. Detectors often confidently mistake similar objects, such as a sofa for a bed, and stopping at a false positive fails the episode. Prevailing systems also approach candidates greedily, leaving them occluded or partially observed. We present CLOSER, an evidence-aware active verification framework that treats stopping as closed-loop evidence acquisition, coupling vision-language model (VLM) verification with active perception. CLOSER fine-tunes an open-source VLM to label each view of a candidate tracked across frames as target, non-target, or uncertain, selects complementary views with an embedding-based diversity gate, and fuses their per-view probabilities with a lightweight network. If the fused judgment remains uncertain, a diffusion policy drives the agent toward the candidate using only the tracked mask and depth until the decision becomes decisive. CLOSER runs locally and plugs into existing ObjectNav systems without retraining them. On Gibson and Habitat-Matterport 3D, it raises average success rate by 3.6 points and success weighted by path length by 2.0 points, showing that verifying evidence before committing improves embodied navigation.
• 中文摘要 对象-目标导航（ObjectNav）要求具身代理在看不见的环境中找到指定类别的实例并停在附近。代理通常依赖对象检测器，但必须通过有限的自我中心视频流（有时仅一帧）在线判断检测到的候选是否为目标。检测器常常自信地将类似物体误认为床，如沙发，误报即失败。主流系统也会贪婪地接近候选对象，使其被遮蔽或部分观察。我们提出了CLOSER，一种证据感知主动验证框架，将停止视为闭环证据获取，将视觉语言模型（VLM）验证与主动感知结合。CLOSER对开源VLM进行微调，将跨帧追踪候选的每个视图标记为目标、非目标或不确定，并用嵌入多样性门选择互补视图，并将它们的每视图概率与轻量级网络融合。如果融合判断仍不确定，扩散策略将引导智能体仅使用追踪的掩码和深度向候选者移动，直到决策变得决定性。CLOSER本地运行，直接插入现有的ObjectNav系统，无需重新训练。在Gibson和Habitat-Matterport 3D上，平均成功率提高了3.6分，按路径长度加权成功率提升2.0分，显示在承诺前验证证据能改善具体导航。