生成时间: 2026-10-20 21:29:03 (UTC+8); Arxiv 发布时间: 2026-10-20 20:00 EDT (2026-10-21 08:00 UTC+8)

今天共有 95 篇相关文章

Keyword: reinforcement learning

Web Agent Agentic Reinforcement Learning Decision Model Under Multi-Cost and Failure Risk

Web代理代理智能强化学习在多重成本和失败风险下的决策模型

• Authors: Qianli Ma, Limengxi Yue, Shuyang Xu, Yanpei Shi, Hongrui Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23857
• Pdf link: https://arxiv.org/pdf/2610.23857
• Abstract Intelligent agent interactions in real-world web environments are commonly constrained by request budgets, time delays, anti-crawling restrictions, and operational failure risks. Strategies solely optimizing task success rates often exhibit unusable phenomena such as "high success but high cost" or "low risk but conservative failure." This paper proposes a constrained agentic reinforcement learning model for web agents, unifying page access, search requests, and external API calls into a unified long-term decision-making framework with associated costs. It simultaneously incorporates cost budget constraints and tail risk control into the optimization objective: constructing a multidimensional cost vector comprising cumulative request count, total latency, and failure penalties to achieve budget compliance via Lagrange dual updates, while employing a CVaR risk term to suppress excessive exploration of high-failure-probability paths, thereby achieving an adaptive balance among "completion rate, cost, and risk." Experiments were conducted across 30-70 site/page templates and 800-1,500 end-to-end web tasks, including information extraction, price comparison, form submission, and cross-page navigation. Interaction sequences spanned 20-120 steps with tool scales of 30-200. Performance was benchmarked against unconstrained RL, budget-constrained RL, and rule-based/scripted web agents, quantifying task completion rates, cost-per-success, failure rates, and policy stability. Results demonstrate that at equivalent completion rates, our method reduces C-PS by 22%-31% and lowers failure rates by 18%-26% under high failure penalties. Under fixed budgets, task completion rates increase by 10%-16%, highlighting the necessity and effectiveness of constraint modeling for practical web agent deployment.
• 中文摘要 现实世界网络环境中的智能代理交互通常受请求预算、时间延迟、反爬取限制和运营失败风险的限制。仅优化任务成功率的策略常常出现“高成功但高成本”或“低风险但保守失败”等不可用现象。本文提出了一个受限的代理强化学习模型，将页面访问、搜索请求和外部API调用统一为一个统一的长期决策框架，并承担相关成本。该模型同时将成本预算约束和尾部风险控制纳入优化目标：构建一个多维成本向量，包括累计请求数、总延迟和失败惩罚，通过拉格朗日双重更新实现预算合规，同时采用CVaR风险项抑制对高失败概率路径的过度探索，从而实现“完成率、成本和风险”之间的自适应平衡。实验涵盖30-70个网站/页面模板和800-1,500个端到端网页任务，包括信息提取、价格比较、表单提交和跨页导航。交互序列涵盖20-120步，工具规模为30-200。性能与无约束的强化学习、预算受限的强化学习以及基于规则/脚本化的网页代理进行了基准测试，量化任务完成率、每次成功成本、失败率和策略稳定性。结果表明，在相同完成率下，我们的方法在高失败惩罚下可降低22%-31%的C-PS，降低18%-26%的失败率。在固定预算下，任务完成率提升10%-16%，凸显约束建模对实际部署Web代理的必要性和有效性。

Learning Whole-Body Kinetic Coordination for Wheeled-Quadruped Long-Range Throwing

学习轮式四足远距离投掷的全身动能协调

• Authors: Qinglian Wu, Yidian Hu, Weiqi Fan, Yutian Zhang, Siyuan Ma, Jiaqi Zhai, Wei Dong, Li Xu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.23883
• Pdf link: https://arxiv.org/pdf/2610.23883
• Abstract Long-range throwing on unstructured terrain is a key capability for mobile robots in delivery and retrieval tasks, but it requires high release velocity, coordinated whole-body motion, and robustness to terrain-induced disturbances. We present a whole-body kinetic-coordination framework for a wheeled-quadruped manipulator that combines kinetic-chain arm modeling with SQP-based trajectory optimization and a reference-guided reinforcement learning policy for coordinated base motion. The manipulator is first reduced to a kinetic-chain model and optimized under kinematic and dynamic constraints to produce a high-speed throwing reference. On a stationary base, the optimized arm trajectory reaches a release speed of 12.75 m/s and a throwing distance of 16.68 m. When whole-body coordination is enabled, the system further increases the release speed to 16.64 m/s and the throwing distance to 28.25 m. We then integrate the mobility of the wheeled-quadruped platform and validate the complete policy through hardware experiments on flat ground, grass, slopes, and rough roads. The results show that kinetic-chain optimization and whole-body coordination together enable robust long-range throwing across diverse unstructured terrains.
• 中文摘要 在无结构地形上的远程投掷是移动机器人在投递和回收任务中的关键能力，但需要高释放速度、协调的全身运动以及对地形干扰的韧性。我们提出了一个全身动能协调框架，适用于轮式四足操作手，结合了动能链臂建模、基于SQP的轨迹优化以及参考引导的强化学习策略以实现协调基座运动。首先将机械臂简化为动能链模型，并在运动学和动力学约束下优化，以生成高速投掷参考。在静止基座上，优化的臂部轨迹释放速度达到12.75米/秒，投掷距离为16.68米。当启用全身协调时，系统进一步将释放速度提升至16.64米/秒，投掷距离提升至28.25米。随后，我们整合了轮式四足平台的机动性，并通过在平坦地面、草地、坡地和崎岖道路上的硬件实验验证了完整策略。结果表明，动能链优化与全身协调相结合，能够实现在多样非结构化地形上的强健远程投掷。

miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning

miniTE：诊断并弥合原生NVFP4强化学习中的执行差距

• Authors: Zhen Li, Mingfa Feng, Yiming Zhang, Yang Yu, Changhai Zhou, Pengxiang Li, Dongkuan Wu, Congkai Xie, Hongxia Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.23928
• Pdf link: https://arxiv.org/pdf/2610.23928
• Abstract Native NVFP4 promises to accelerate reinforcement learning (RL) for large language models (LLMs), but achieving stable learning across separate training and sampling engines remains challenging. We show that matching model weights and numerical formats is insufficient to ensure consistent policy evaluation, highlighting the importance of cross-engine numerical execution. We introduce miniTE, a lightweight framework that combines execution diagnostics, cross-engine numerical alignment, and efficient native W4A4 (4-bit weights and activations) GEMM computation for LLM RL. It improves numerical execution consistency without modifying the RL objective, while reducing implementation overhead to realize the benefits of native low-bit computation. Long-horizon RL experiments on Qwen3-30B-A3B demonstrate stable training and benchmark performance close to BF16, outperforming naive Transformer Engine (TE) NVFP4 in evaluation quality. On single NVIDIA B300 GPU node, the optimized learner reaches 33,540 tokens/s, yielding 1.39x and 1.47x the throughput of BF16 and TE NVFP4, respectively, for complete learner updates. We released the framework, kernels, diagnostics, and training configurations to make native low-bit RL easier.
• 中文摘要 原生NVFP4有望加速大型语言模型（LLM）的强化学习（RL），但实现跨不同训练和采样引擎的稳定学习仍具挑战性。我们表明，匹配模型权重和数值格式不足以确保策略评估的一致性，凸显了跨引擎数值执行的重要性。我们介绍miniTE，一个轻量级框架，结合了执行诊断、跨引擎数值对齐和高效的原生W4A4（4位权重和激活）GEMM计算，用于LLM RL。它在不修改RL目标的情况下提升了数值执行一致性，同时降低实现开销，实现原生低位计算的优势。在Qwen3-30B-A3B上的长期RL实验显示出接近BF16的稳定训练和基准性能，在评估质量上优于初级变换器引擎（TE）NVFP4。在单个NVIDIA B300 GPU节点上，优化学习器可达33,540令牌/秒，分别实现BF16和TE NVFP4的1.39倍和1.47倍吞吐量，实现学习器的完整更新。我们发布了框架、内核、诊断和训练配置，使原生低位强化学习更为便捷。

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

数据如何在智能强化学习中扩展：定律与综合策略

• Authors: Bowei He, Yankai Chen, Xiaokun Zhang, Changjiang Han, Ye Yuan, Chong Li, Xue Liu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.23946
• Pdf link: https://arxiv.org/pdf/2610.23946
• Abstract Pretraining scaling laws treat training data as a one-dimensional quantity: a token count. Agentic reinforcement learning (RL) inherits this scalar abstraction, but modern synthesis pipelines now generate \textit{tasks}, \textit{environments}, and \textit{trajectories} independently and at very different costs, turning data into a structured design space. This raises a question pretraining never had to answer: given a synthesis budget, \textit{which axis} should be scaled? We address this question end-to-end with three tightly coupled contributions. First, we decompose agentic data into three cardinality axes (task, environment, trajectory), plus a reward-density control variable, and fit per-axis scaling relations on different large language models (LLMs) across math+code, SWE, web, and UI/tool application domains. Across all model sizes, model families, domains, and RL algorithms (PPO and GRPO) we measured, the scaling exponents order as environment $>$ task $>$ trajectory. When environment diversity is decomposed into world and environment-structure diversity, its advantage is mainly driven by the former, while the latter is just comparable to the task diversity. Second, we use the fitted laws as a common ruler to price synthesis methods, capturing each method's efficiency as a per-axis discount factor and identifying critical synthetic-to-real ratios at which collapse begins; these ratios differ by an order of magnitude across axes. Third, we cast budget allocation as a constrained optimization under our laws and discounts, deriving a Chinchilla-style compute-optimal recipe and validating it at held-out scale. The empirical results demonstrate that following the recipe yields up to $1.7\times$ compute efficiency over the dominant ``scale-the-trajectories'' practice.
• 中文摘要 预训练尺度定律将训练数据视为一维量：一个令牌计数。代理强化学习（RL）继承了这种标量抽象，但现代综合流程现在独立生成\textit{tasks}、\textit{environments}和\textit{trajectories}，成本截然不同，将数据转化为结构化设计空间。这引出了一个预训练从未需要回答的问题：给定综合预算，\textit{哪个轴}应该被缩放？我们通过三个紧密耦合的贡献从头到尾回答这个问题。首先，我们将智能数据分解为三个基数轴（任务、环境、轨迹），加上一个奖励-密度控制变量，并在数学+代码、软件软件、网页和UI/工具应用领域的不同大型语言模型（LLM）上拟合每轴的缩放关系。在我们测量的所有模型大小、模型族、域和强化学习算法（PPO和GRPO）中，缩放指数的顺序为环境$>$任务$>$轨迹。当环境多样性分解为世界多样性和环境结构多样性时，其优势主要由前者驱动，而后者仅可与任务多样性比较。其次，我们使用拟合定律作为定价综合方法的通用尺，捕捉每种方法的效率作为每轴折现因子，并识别崩溃开始的关键合成与实际比率;这些比率在各轴上相差一个数量级。第三，我们将预算分配视为在定律和折扣下有约束的优化，推导出类似Chinchilla的计算最优配方，并在实际规模下验证。实证结果表明，按照配方操作，计算效率可高达1.7美元乘以$，相较于主流的“轨迹尺度”做法。

Compaction-Aware Reinforcement Learning For Ascend Triton Kernel Generation

针对Ascend Triton内核生成的压缩感知强化学习

• Authors: Chuansai Zhou, Yanlin Liu, Bingning Huang, Shidi Xi, Chao Huang, Boxing Chen, Mingming Zhu, Guanghuan Fang, Ben Feng, Zongzuo Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.23950
• Pdf link: https://arxiv.org/pdf/2610.23950
• Abstract Automatically generating high-performance kernels with LLM-based agents promises substantial gains in development efficiency, but two obstacles remain. First, while increasing multi-turn interaction with execution environments is crucial for refining generation quality, continuous context accumulation rapidly exceeds the model's effective context window. Second, training long-horizon tasks via GRPO inherently incurs sparse rewards due to its reliance on terminal outcome evaluation, severely hindering step-level credit assignment; To address these obstacles, we propose ATRL (Ascend Triton Reinforcement Learning), a post-training method that refines GRPO into a compaction-aware, segment-level objective for long-horizon kernel generation. We also design a lightweight agent harness providing cascaded compilation, correctness, and profiling feedback, alongside an Ascend-specific knowledge base injected during agentic RL training and evaluation to compensate for the base model's missing platform expertise. Evaluations on NPUKernelBench and KernelBench, both executed on Ascend hardware, show that ATRL consistently surpasses all baselines, including GRPO, in both correctness and speedup.
• 中文摘要 自动生成基于LLM的智能体高性能内核有望显著提升开发效率，但仍存在两个障碍。首先，虽然增加与执行环境的多回合交互对于提升生成质量至关重要，但持续上下文积累会迅速超过模型的有效上下文窗口。其次，通过GRPO训练长视野任务，由于依赖终端结果评估，奖励稀疏，严重阻碍了步级学分分配;为解决这些障碍，我们提出了ATRL（Ascend Triton Reinforcement Learning），这是一种训练后方法，将GRPO精炼为具压缩感知、段级目标，用于长视野内核生成。我们还设计了轻量级代理工具束，提供级联编译、正确性和剖析反馈，并在智能强化学习训练和评估中注入Ascend专用知识库，以弥补基础模型缺乏的平台专业知识。在Ascend硬件上执行的NPUKernelBench和KernelBench评估显示，ATRL在正确性和加速率上始终超越所有基线，包括GRPO。

RL Teaches VLMs Where to Attend but Not How to Perceive

强化学习教会VLM该去哪里参加，但不教他们如何感知

• Authors: Ziqian Zhang, Junjie Liao, Zhining Zhang, Wentao Zhu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23983
• Pdf link: https://arxiv.org/pdf/2610.23983
• Abstract Although reinforcement learning (RL) effectively trains Vision-Language Model (VLM) agents, the mechanisms behind these capability gains are poorly understood. Across four diverse agent environments, we investigate whether the task-success gains from multi-turn RL are accompanied by improved grounded visual question answering about the environment. To assess this, we design environment-specific grounded VQA tasks that reveal improvements in grounded visual understanding. Representational probing shows that vision-encoder probe accuracy remains relatively stable, while LLM-layer probe accuracy improves. Attention analysis further reveals that LLM layers pay more attention to visual tokens during RL training. To test whether increased attention to visual tokens contributes to VQA gains, we introduce a lightweight plug-and-play intervention using a fixed attention bias. Without updating model parameters or computing gradients, this intervention partially restores VQA gains on the untrained model. These results reveal better utilization of visual information as a key mechanism behind the gains from RL.
• 中文摘要 尽管强化学习（RL）有效训练了视觉语言模型（VLM）代理，但这些能力提升背后的机制尚不清楚。在四种不同的代理环境中，我们研究了多回合强化学习带来的任务成功提升是否伴随着对环境的更强的接地视觉问题回答。为此，我们设计了环境特定且基于环境的VQA任务，揭示了基于视觉理解的改进。表征探测显示，视觉编码器探针的准确性相对稳定，而LLM层探针的准确性有所提升。注意力分析进一步揭示，LLM层在强化学习训练中更关注视觉符号。为测试对视觉符号的关注增加是否有助于VQA的提升，我们引入了一种采用固定注意力偏差的轻量级即插即用干预。该干预无需更新模型参数或计算梯度，部分恢复了未训练模型上的VQA收益。这些结果表明，视觉信息作为强化学习带来收益的关键机制，是更好的利用。

Weakly Supervised Discovery of Motion Styles for Compositional Robot Control

弱监督发现构图机器人控制运动风格

• Authors: Heedo Woo, Antonio Marino, Eduardo Sebastián, Amanda Prorok
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24008
• Pdf link: https://arxiv.org/pdf/2610.24008
• Abstract When a robot possesses more degrees of freedom than a task requires, multiple trajectories can achieve the same objective using distinct motion styles. Compositional robot control aims to preserve a selected style while simultaneously executing an independent task command, effectively balancing these potentially competing objectives. In this paper, we present CODES, a framework for compositional robot control that discovers a coherent style repertoire from diverse recordings and composes it with independent commands. First, motion retargeting maps the recordings to motions executable by the target robot; next, weakly supervised consensus encoding and density-based clustering identify styles within and across recordings without a prescribed count; finally, reinforcement learning trains a policy to compose the discovered styles with independent commands. We evaluate CODES on a Unitree Go2 quadruped across two simulators and physical hardware. Compared to baselines, CODES significantly improves kinematic coherence. Furthermore, our controller successfully tracks velocity commands that generalize beyond the support of the recorded data, exhibiting robust and distinguishable style characteristics during both cross-simulator and real-world hardware deployments. Project website: this https URL
• 中文摘要 当机器人拥有比任务所需的自由度更多时，多种轨迹可以通过不同的运动风格实现同一目标。合成型机器人控制旨在保持所选风格，同时执行独立任务指令，有效平衡这些潜在的竞争目标。本文提出了CODES，一种组合机器人控制框架，它从不同录音中发现连贯的风格库，并用独立命令组成。首先，运动重定向将录制映射到目标机器人可执行的动作;其次，弱监督共识编码和基于密度的聚类识别录音内及跨录音的风格，无需规定计数;最后，强化学习训练策略，用独立指令组合发现的风格。我们在两个模拟器和物理硬件上四足的Unitree Go2上评估代码。与基线相比，CODES显著提升了运动学的相干性。此外，我们的控制器成功追踪速度命令，这些命令的泛化超出记录数据的支持，在跨模拟器和实际硬件部署中都展现出稳健且易于区分的风格特征。项目网站：此 https URL

Beyond Explicit Intent: Discovering and Shaping Artistic Intent through Human-AI Co-Creation

超越明确意图：通过人机共创发现与塑造艺术意图

• Authors: Renye Yan, Jialun Zhong, Jikang Cheng, Yi Gao, Yanzeng Li, Ling Liang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.24010
• Pdf link: https://arxiv.org/pdf/2610.24010
• Abstract Generative AI is becoming an important medium for artistic creation, yet existing interactions often assume that creators can specify intentions in advance and accurately communicate them through text prompts. Artistic intentions, however, may be vague, latent, or recognizable only after seeing concrete visual possibilities. We propose a human-AI co-creation paradigm beyond explicit intent expression. By expanding model exploration, we expose artists to diverse visual possibilities, allowing them to identify previously unarticulated preferences through observation, comparison, and selection. We then integrate artists' curatorial judgments into reinforcement learning, assigning different learning strengths based on preferences over generation trajectories, so recognized directions are more strongly reinforced and the model gradually adapts to individual aesthetics. This creates a continuous "explore-identify-shape-re-explore" loop in which artistic intent is progressively discovered, externalized, and shaped. Through three user studies, we examine latent-preference discovery, personalized preference alignment, and the externalization of vague, difficult-to-express creative intentions.
• 中文摘要 生成式人工智能正成为艺术创作的重要媒介，但现有互动往往假设创作者可以提前指定意图并通过文本提示准确传达。然而，艺术意图可能模糊、潜在，或只有在看到具体视觉可能性后才能被识别。我们提出了超越明确意图表达的人机共创范式。通过扩展模型探索，我们让艺术家接触到多样的视觉可能性，使他们通过观察、比较和选择识别此前未曾表达过的偏好。随后，我们将艺术家的策展判断整合进强化学习中，根据世代轨迹的偏好分配不同的学习优势，使识别的方向得到更强的强化，模型逐渐适应个体美学。这形成了一个连续的“探索-识别-形状-再探索”循环，艺术意图逐步被发现、外化和塑造。通过三项用户研究，我们考察了潜在偏好的发现、个性化偏好对齐以及模糊、难以表达的创造意图的外化。

Alliance of Stochastic Routing and Reinforcement Fusion for Multi-Model Reasoning

随机路由与强化融合联盟用于多模型推理

• Authors: Selim Furkan Tekin, Ling Liu
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.24054
• Pdf link: https://arxiv.org/pdf/2610.24054
• Abstract This paper studies dynamic model routing under uncertainty in multimodal settings, where a system must select and potentially compose multiple models to answer a query. Standard reinforcement learning approaches for routing often suffer from premature convergence to suboptimal models due to noisy rewards and insufficient exploration. We propose a unified vision-language framework (VL-Scout) for efficient multimodal and multi-step reasoning with turn-based reinforcement fusion. First, VL-Scout combines Bayesian priors, policy gradient optimization, and sequential decision-making to enable structured exploration and adaptive inference. A context-dependent Dirichlet belief model allows for epistemic uncertainty over model quality and is updated online based on observed rewards and context-specific evidence, enabling Thompson-style exploration while serving as a prior for regularization. Second, a routing policy is trained via a KL-regularized REINFORCE objective by maximizing a reward-weighted evidence lower bound, where the policy acts as an amortized posterior, and the Dirichlet belief serves as a prior. Third, we introduce a fusion policy trained with Group Relative Policy Optimization (GRPO) or Reinforcement Learning with Verifiable Rewards (RLVR), which operates over intermediate model rationales and selects between two actions: CONTINUE or ANSWER, terminating with a generated output. We evaluate VL-Scout on four representative multi-modal benchmarks (MMMU, MMMU-Pro, OKVQA, and RealWorldQA), compared against 11 representative routing and policy optimization baselines. Our method provides consistent performance improvements while reducing premature policy collapse. These results highlight the importance of integrating uncertainty-aware priors with both amortized policy learning and sequential fusion for robust and efficient model selection in discrete action spaces
• 中文摘要 本文研究多模态环境中的动态模型路由，系统必须选择并可能组合多个模型以回答查询。标准的强化学习路由方法常因奖励噪声和探索不足而过早收敛到次优模型。我们提出一个统一视觉语言框架（VL-Scout），用于高效的多模态和多步推理，并结合回合制强化融合。首先，VL-Scout结合贝叶斯先验、策略梯度优化和顺序决策，实现结构化探索和自适应推断。上下文依赖的狄利克雷信念模型允许对模型质量存在认知不确定性，并根据观察到的奖励和上下文特定证据在线更新，支持汤普森式探索，同时作为正则化的先验。其次，路由策略通过最大化奖励加权证据下界，最大化KL正则化的REINFORCE目标进行训练，其中策略作为摊销后验，Dirichlet信念作为先验。第三，我们引入一种融合策略，采用Group Relative Policy Optimization（GRPO）或可验证奖励强化学习（RLVR）训练，基于中间模型理据，选择两种动作：继续或ANSWER，最终以生成输出结束。我们在四个代表性多模态基准测试（MMMU、MMMU-Pro、OKVQA和RealWorldQA）上评估VL-Scout，并与11个具代表性的路由和策略优化基线进行比较。我们的方法在减少政策过早崩溃的同时，提供了持续的性能提升。这些结果凸显了将不确定性感知先验与摊销策略学习和顺序融合整合的重要性，以便在离散行动空间中实现稳健高效的模型选择

Data-Driven Resource Aggregation for Scalable Digital Twin and Simulation Model Generation

数据驱动资源聚合，用于可扩展的数字孪生和仿真模型生成

• Authors: Yucheng Wu, Alp Akcay
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24126
• Pdf link: https://arxiv.org/pdf/2610.24126
• Abstract As industrial digitization accelerates, the use of digital twins has become widespread. However, modern manufacturing systems are highly dynamic and frequently reconfigured, making manual model development and updates increasingly difficult. Recent research on automated simulation model generation shows that historical data such as event logs can yield simulation models automatically. However, modeling every resource at full granularity leads to unnecessary complexity and computational cost. We propose a method for automatically selecting and aggregating resources to simplify simulation modeling. Our method uses ideas from inverse reinforcement learning to learn behavioral value representations of resources from production event logs. This process identifies and aggregates resources with similar functional contributions. Experiments on manufacturing systems, using throughput and system time as performance measures, demonstrate that value-based resource aggregation performs well when the goal is to substantially reduce complexity while preserving performance accuracy.
• 中文摘要 随着工业数字化加速，数字孪生的应用变得广泛。然而，现代制造系统高度动态且频繁重构，使得手工模型开发和更新变得越来越困难。最新关于自动仿真模型生成的研究表明，历史数据如事件日志可以自动生成仿真模型。然而，对每个资源进行全粒度建模会导致不必要的复杂性和计算成本。我们提出了一种自动选择和聚合资源的方法，以简化仿真建模。我们的方法利用逆强化学习的理念，从生产事件日志中学习资源的行为价值表示。该过程识别并汇总具有相似功能贡献的资源。在制造系统上以吞吐量和系统时间作为性能衡量标准的实验表明，基于价值的资源聚合在旨在显著降低复杂性且保持性能准确性时表现良好。

Continual Test-Time Learning with Evolution Strategies

持续测试时间学习与进化策略

• Authors: Guangya Hao, Yiting Duan, Longyan Tan, Yunbo Long, Hao Wu, Ge Zheng, Alexandra Brintrup, Zhuokai Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24137
• Pdf link: https://arxiv.org/pdf/2610.24137
• Abstract Large language models increasingly operate in open-ended environments, where they are expected to continually adapt to evolving tasks and distributions, often without access to external supervision. Existing continual learning approaches rely on external supervision, such as labeled data, demonstrations, or reward signals, which can be costly or unavailable in practice. Test-time adaptation removes this dependence by learning from self-generated supervision, but existing methods derive that supervision from the single policy being updated, so once the policy drifts, its own learning signal drifts with it, leading to policy collapse, degraded reasoning, and catastrophic forgetting. In this paper, we propose \textbf{Evolution Strategy Reinforcement Learning (ESRL)}, a continual test-time learning framework that draws its self-generated supervision from a population of policies instead of the single policy being trained. ESRL perturbs an anchor policy in parameter space to form a population of nearby policies, samples one trajectory from each, and aggregates their answers into label-free group-relative learning signals for the central student. The anchor is held fixed within each block and refreshed from the student at block boundaries, so the population stays centered on a stable yet current policy and continues to supply diverse and reliable supervision as the student moves. In single-task test-time adaptation across mathematical reasoning, general knowledge, and medical domains, ESRL improves average accuracy by up to 31.7% relative to the base model and 16.6% relative to the strongest test-time adaptation baseline. Notably, ESRL substantially reduces catastrophic forgetting and achieves the strongest final performance in continual learning.
• 中文摘要 大型语言模型越来越多地在开放式环境中运行，期望它们不断适应不断演变的任务和分布，且往往无法获得外部监督。现有的持续学习方法依赖外部监督，如标记数据、演示或奖励信号，这些在实际中可能成本高昂或难以获得。测试时间适应通过自生成监督来消除这种依赖，但现有方法从更新的单一策略中获得监督，因此一旦策略漂移，其自身学习信号也会随之漂移，导致策略崩溃、推理退化和灾难性遗忘。本文提出 \textbf{进化策略强化学习（ESRL）}，一种持续测试时学习框架，其自我生成的监督来自一组策略，而非单一策略。ESRL扰动参数空间中的锚点策略，形成邻近策略的总体，从每个策略中抽取一条轨迹，并将其答案汇聚为无标签的组别相对学习信号，供中央学生使用。锚点固定在每个区块内，并在区块边界从学生处刷新，使总体保持稳定且最新的策略中心，并在学生移动过程中持续提供多样且可靠的指导。在数学推理、常识和医学领域的单任务测试时间适应中，ESRL相较基础模型提升平均准确率最高达31.7%，相较于最强测试时间适应基线提升16.6%。值得注意的是，ESRL显著减少灾难性遗忘，并在持续学习中实现最强的最终表现。

Training-Free Prompt Selection for Efficient Reinforcement Learning With Partial Rollouts

无需培训的提示选择，实现部分推广的高效强化学习

• Authors: Lei Jiang, Honghui Bao, Yue Zhou, Zhengyao Gu, Langzhou He, Natalie Parde
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24229
• Pdf link: https://arxiv.org/pdf/2610.24229
• Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as a prominent paradigm for training reasoning models, but its computational cost makes efficient data selection increasingly important. Since training examples contribute unequally to learning, selecting a small, informative subset can offer a more efficient alternative to using the entire data pool. The training-free selection method estimates prompt value with hidden states from complete greedy rollouts. However, in our datasets, $26$--$32\%$ of these rollouts fail to terminate within the generation budget. We introduce LIFT, a training-free data selector that identifies impactful prompts using only partial reasoning rollouts. At its core, LIFT uses Partial Rollout Context Lift (PRCL), a score inspired by pointwise $\mathcal{V}$-information that accumulates token-level log-likelihood ratios between a question-conditioned context and a null context with the question removed. This enables prompt scoring from a short rollout prefix rather than a complete trajectory. Across Qwen2.5-Math-1.5B and DeepSeek-R1-Distill-1.5B, prefixes retaining at least 90\% and 95\% of the full-rollout PRCL score, respectively, require only 26\% and 4.7\% of the tokens generated by full rollouts. Compared with SHIFT, LIFT reduces selection cost by 3.7--9.1$\times$ while achieving competitive downstream RLVR performance, with the highest six-benchmark average in three of six settings.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为训练推理模型的重要范式，但其计算成本使得高效数据选择变得越来越重要。由于训练示例对学习贡献不均，选择一个小而信息丰富的子集可以为使用整个数据池提供更高效的替代方案。无训练选择方法通过完全贪婪的推举估算带有隐藏状态的提示价值。然而，在我们的数据集中，有26美元至32%美元的这些推展未能在生成预算内终止。我们引入了LIFT，一种无需训练的数据选择器，仅通过部分推理推理推测识别有影响力的提示。LIFT的核心使用部分推出上下文提升（PRCL），该评分基于点数的$\mathcal{V}$信息，累积问题条件上下文与消除问题上下文的代币级对数似然比。这使得从短前置前缀而非完整轨迹中获得提示评分成为可能。在Qwen2.5-Math-1.5B和DeepSeek-R1-Distill-1.5B中，前缀保留至少90%和95%的完整推销PRCL分数，但仅需26%和4.7%的代币。与SHIFT相比，LIFT降低了3.7-9.1$\times$的选择成本，同时实现了竞争性的下游RLVR表现，在六个设置中有三个设置中拥有最高的六个基准平均值。

Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

通过自适应回放保持强化学习中的推理多样性，并提供可验证的奖励

• Authors: Wen-Zhi Li, Sainyam Galhotra, Claire Cardie
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24236
• Pdf link: https://arxiv.org/pdf/2610.24236
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capabilities of large language models, but often reduces the diversity of their reasoning behaviors. We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively suppress semantically distinct but equally rewarded reasoning modes. Once a mode becomes sufficiently under-sampled, standard on-policy RLVR rarely recovers it. Motivated by this observation, we propose TAR, a Targeted Adaptive Replay framework that preserves diverse correct reasoning behaviors already present in the base policy. TAR constructs a compact memory bank of diverse base-model solutions and selectively replays them through an adaptive mechanism that activates only when stored behaviors are being suppressed during RLVR. Across six mathematical reasoning benchmarks and two Qwen2.5 model scales, TAR consistently achieves a better balance between reasoning performance and diversity preservation than existing diversity-preserving post-training methods, while substantially reducing the forgetting of correct base-model solutions. Code is available at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了大型语言模型的推理能力，但往往减少了其推理行为的多样性。我们从有限样本学习动态的角度研究这一现象，表明有限的策略内推广可以逐步抑制语义上不同但同样获得奖励的推理模式。一旦模式采样不足，标准的策略内RLVR很少恢复该模式。基于这一观察，我们提出了TAR，一种目标自适应回放框架，保留基础策略中已存在的多样化正确推理行为。TAR构建了一个紧凑的多样基础模型解记忆库，并通过一种自适应机制选择性重放，该机制仅在RLVR期间抑制存储的行为时激活。在六个数学推理基准测试和两个Qwen2.5模型量表中，TAR始终比现有的多样性保存后方法在推理性能与多样性保持之间取得更好的平衡，同时大幅减少了对正确基础模型解的遗忘。代码可在此 https URL 获取。

Towards Reinforcement Learning for Space Robotics: Fast Training and Reliable Real-World Transfer

迈向空间机器人强化学习：快速训练与可靠的现实世界迁移

• Authors: Abhishek Naik, Michael Wu, Michael O'Sullivan, Colin Bellinger, Yunli Wang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.24241
• Pdf link: https://arxiv.org/pdf/2610.24241
• Abstract This work demonstrates the use of a fast simulation-based training framework to learn control policies using reinforcement learning. The hardware requirements are minimal and the simulation-learned policies transfer reliably to the real world. With this setup, we show that a quadruped robot can learn to traverse diverse real-world terrains reliably---including in a lunar-analog facility---after less than 20 minutes of training on a single consumer-grade GPU. These results provide a glimpse of a future where reinforcement learning is a crucial tool in developing more capable and robust space robotic systems. Video demonstrations are available at this https URL.
• 中文摘要 本研究展示了利用快速仿真训练框架通过强化学习学习控制策略。硬件需求极低，且仿真学到的策略能够可靠地转移到现实世界。通过这种配置，我们证明了四足机器人可以在单个消费级GPU上训练不到20分钟后，可靠地学习穿越多样的真实地形---包括月球模拟设施---。这些结果为强化学习成为开发更强大、更稳健空间机器人系统的关键工具提供了未来一瞥。视频演示可在此 https 网址观看。

Probing Ambiguity: Adversarial Intent Resolution for Robust Text-to-SQL

探究歧义：强健文本转SQL的对抗意图解决

• Authors: Youbing Yin, Xujun Peng, Muhammad S. Uddin, Erin Babinsky, Daben Liu, Anoop Kumar
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24248
• Pdf link: https://arxiv.org/pdf/2610.24248
• Abstract Text-to-SQL systems typically assume a one-to-one mapping between natural-language queries and correct SQL translations. In practice, however, many queries are inherently ambiguous, admitting multiple valid interpretations that yield distinct outputs. We introduce Adversarial Intent Resolution (AIR), a multi-agent framework that actively probes and resolves such ambiguity without any task-specific training. AIR first constructs competing interpretations, verifies through SQL execution that they produce genuinely different results, and generates targeted clarification questions grounded in the observed data differences. Evaluating on leading benchmarks for ambiguous questions, AIR demonstrates strong performance, efficiency, and robust cross-benchmark transferability. On the AMBROSIA benchmark under schema-only context, a realistic production setting where exposing the full database is infeasible, AIR outperforms IntentRL, the state-of-the-art reinforcement learning model, across all metrics (e.g., F1 0.679 vs. 0.618). Crucially, AIR exhibits only ~7% performance degradation without the full database, compared to ~20% for IntentRL. On the AmbiQT benchmark, where ambiguity arises from schema design rather than linguistic structure, AIR achieves higher F1 under both context settings without any benchmark-specific adaptation.
• 中文摘要 文本转SQL系统通常假设自然语言查询与正确SQL翻译之间有一对一的映射。然而，在实际操作中，许多查询本质上存在歧义，允许多种有效解释，从而产生不同的输出。我们引入了对抗意图解析（AIR），这是一个多代理框架，能够主动探究并解决这种歧义，无需任何任务特定的培训。AIR首先构建竞争解释，通过SQL执行验证它们产生真实不同的结果，并基于观察到的数据差异生成针对性的澄清问题。通过评估歧义问题的领先基准测试，AIR展现出强大的性能、效率和强大的跨基准测试可迁移性。在AMBROSIA基准测试中，仅在模式上下文下（即暴露完整数据库不可行的现实生产环境），AIR在所有指标上均优于最先进的强化学习模型IntentRL（例如F1为0.679对比0.618）。关键是，AIR在不使用完整数据库时性能下降仅为~7%，而IntentRL为~20%。在AmbiQT基准测试中，模糊性源于模式设计而非语言结构，AIR在两种上下文设置下均实现更高的F1，且未做任何基准特有的调整。

Blackbox Performance Analysis of LLM-based Network Control

基于LLM的网络控制的黑箱性能分析

• Authors: Mahdi Alizadeh, Pooria Namyar, Arvin Ghavidel, Ramesh Govindan
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24348
• Pdf link: https://arxiv.org/pdf/2610.24348
• Abstract LLMs are becoming an indispensable part of network control systems due to their impressive reasoning. Before such LLM-enabled systems can be deployed, it is imperative to develop tools that can audit their performance on previously unseen inputs. However, auditing these systems is challenging due to the inherent opacity of LLMs. LENS is the first step towards black-box performance analysis of LLM-enabled network control systems. It uses novel methods to extend deep reinforcement learning to guide the search for performance gaps and inputs that make LLMs struggle. LENS finds large gaps for LLM-based traffic engineering and GPU allocation systems, and even matches existing white-box analyzers designed for hand-crafted TE heuristics.
• 中文摘要 由于其卓越的推理能力，LLM正成为网络控制系统不可或缺的一部分。在部署此类LLM支持系统之前，必须开发能够审计其对此前未见输入性能的工具。然而，由于LLM本身具有不透明性，审计这些系统具有挑战性。LENS是迈向LLM驱动网络控制系统黑箱性能分析的第一步。它采用新颖方法扩展深度强化学习，指导寻找性能差距和导致LLM难以使用的输入。LENS发现基于LLM的流量工程和GPU分配系统存在巨大缺口，甚至匹配了为手工TE启发式设计的现有白盒分析仪。

Gen2Dex: Learning Dexterous Manipulation from Simulation-Initialized Generated Video Demonstrations

Gen2Dex：从模拟初始生成视频演示中学习灵巧操作

• Authors: Xiaowen Qiu, Yian Wang, Haoxiang You, Chuang Gan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.24392
• Pdf link: https://arxiv.org/pdf/2610.24392
• Abstract Human video demonstrations provide rich manipulation strategies for dexterous manipulation learning. We study a setting in which a fully specified simulated scene is used to generate videos that specify the demonstrated manipulation behavior. This setup provides access to scene information such as object geometry, articulation, and camera parameters, while allowing direct control over task and scene configuration. However, generated videos may contain geometric distortions, inaccurate contacts, and other physical inconsistencies, making the recovered motion difficult to execute directly on a robot. We present an end-to-end framework for converting such simulation-grounded videos into physically executable dexterous robot demonstrations. Our perception pipeline leverages the known scene through model-based object tracking with dense foundation-model features and depth-based hand-object alignment. The recovered human motion is then retargeted and physically grounded using residual reinforcement learning, where a backward curriculum and a cache of physically valid reset states enable the policy to correct imperfect video-derived references while preserving the demonstrated manipulation strategy. On clean ARCTIC demonstrations, our method achieves competitive performance with prior dexterous retargeting approaches and shows greater robustness when applied to references extracted from generated videos. We further demonstrate diverse rigid and articulated manipulation tasks in simulation and successful open-loop transfer of the optimized trajectories to a real dexterous robot.
• 中文摘要 人类视频演示为灵巧操作学习提供了丰富的操作策略。我们研究了一个环境，在该环境中使用完全指定的模拟场景生成指定演示操作行为的视频。该设置提供了对物体几何、关节和摄像机参数等场景信息的访问，同时允许对任务和场景配置的直接控制。然而，生成的视频可能包含几何畸变、不准确的接触及其他物理不一致，使得恢复的运动难以直接在机器人上执行。我们提出了一个端到端框架，将此类基于模拟的视频转换为可执行的灵巧机器人演示。我们的感知流程利用基于模型的物体跟踪，结合密集的基础模型特征和基于深度的手部-物体对齐，利用已知场景。恢复的人体运动随后通过残余强化学习进行重新定向和物理基础化，利用倒向课程和物理有效的重置状态缓存，使策略能够纠正不完美的视频衍生参考，同时保持已展示的操作策略。在干净的ARCTIC演示中，我们的方法在与先前灵巧重定向方法竞争性能上达到竞争力，且在应用于生成视频中提取的参考时表现出更高的鲁棒性。我们还进一步展示了模拟中多样的刚性和关节操作任务，并成功将优化轨迹开环传输到真实灵巧机器人。

Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

通过离线数据测量赋能，以实现内在动机探索

• Authors: Ishir Garg, Vivek Myers, Qiyang Li, Catherine Ji, Benjamin Eysenbach, Sergey Levine
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24461
• Pdf link: https://arxiv.org/pdf/2610.24461
• Abstract Large, unlabeled datasets of prior experience have become increasingly available for agentic tasks in domains such as robotics and coding. A common recipe for these tasks is to train a policy using the prior data, and then further improve it with online rollouts. But while much work has studied how to use both the offline and online data for policy extraction, far less work has been done to understand how the prior data can be used to efficiently \textit{explore} during the online learning phase. Previous work has attempted to formalize this idea by defining the \textit{empowerment}, a measure of how many future states an agent can realize; states with higher empowerment require more exploration as there are more futures to explore. However, existing methods for estimating empowerment require online policy rollouts in the environment. We propose the first method for estimating empowerment from unlabeled prior data, without any environment interaction. By jointly learning an empowerment-maximizing policy together with its discounted state-occupancy measure, our estimator can be trained on unlabeled trajectories generated from any unknown policy. We demonstrate that our empowerment estimator qualitatively gives intuitive empowerment measures and show that the pretrained estimator can be used to improve exploration in the online goal-conditioned reinforcement learning setting. Code is available at \url{this https URL}.
• 中文摘要 大型未标记的先前经验数据集越来越多地可用于机器人和编码等领域的代理任务。这些任务的常见做法是利用先前数据训练策略，然后通过在线推广进一步改进。尽管大量研究如何利用线下和在线数据进行策略提取，但对如何利用先前数据在在线学习阶段高效\textit{explore}的研究却远少。以往的研究试图通过定义\textit{赋能}来形式化这一思想，衡量智能体能实现多少未来状态;赋权程度越高的状态需要更多探索，因为未来可探索更多。然而，现有赋权估计方法需要在环境中进行在线政策推广。我们提出第一种从未标记的先前数据估算赋能的方法，无需环境交互。通过联合学习赋权最大化策略及其折现状态占有度量，我们的估计器可以训练于任意未知策略生成的未标记轨迹。我们证明赋权估计器定性地给出直观赋权指标，并展示了预训练估计器可用于提升在线目标条件强化学习环境中的探索。代码可在 \url{this https URL} 获取。

ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning

ProbePO：稀疏探针，重新分配无批评者LLM强化学习的功劳

• Authors: Diyi Hu, Ruizhong Qiu, Lin Qiu, Hanghang Tong, Hanqing Zeng
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24515
• Pdf link: https://arxiv.org/pdf/2610.24515
• Abstract Reinforcement learning from verifiable rewards assigns one terminal reward to an entire reasoning trajectory, obscuring how individual reasoning steps contributed to the outcome. GRPO broadcasts a group-normalized advantage to every response token, while finer-grained approaches often rely on dense auxiliary rollouts or branching schemes that can also alter the data used for policy updates. We cast the missing middle as budgeted measurement: with a limited rollout budget, which partial solutions should be evaluated, and how should their value estimates refine credit? ProbePO selects partial-solution states with a reward-free uncertainty score and estimates their expected verifier rewards under a capped continuation budget. ProbePO redistributes, rather than replaces, GRPO's trajectory-level advantage across tokens while preserving each trajectory's mean token-level advantage; probe continuations serve only as measurements. Across three DeepSeekMath and Qwen-family models spanning 1.5B-7B parameters, ProbePO consistently outperforms GRPO and achieves the highest accuracy among the four evaluated methods in 16 of 18 model-benchmark comparisons. Against dense-probing VinePPO on DeepSeekMath-7B, ProbePO achieves higher accuracy on all six benchmarks with 76% less end-to-end training time.
• 中文摘要 从可验证奖励中进行强化学习，将一个终端奖励分配给整个推理轨迹，从而掩盖了各个推理步骤对结果的贡献。GRPO向每个响应代币广播群体归一化优势，而更细粒度的方法通常依赖密集的辅助展开或分支方案，这些方案也可能改变用于策略更新的数据。我们将缺失的中间部分归为预算测量：在有限的推广预算下，应评估哪些部分解？其价值估计应如何细化信用？ProbePO选择具有无奖励不确定性分数的部分解状态，并在有上限的延续预算下估算其预期验证者奖励。ProbePO在不同代币间重新分配而非替代GRPO的轨迹级优势，同时保持每个轨迹的平均代币层级优势;探针延续仅作为测量。在三个涵盖1.5B-7B参数的DeepSeekMath和Qwen家族模型中，ProbePO在18次模型-基准比较中16次中，持续优于GRPO，并获得4种方法中最高的准确率。在DeepSeekMath-7B上面对密集探测VinePPO，ProbePO在所有六个基准测试中均实现更高的准确率，且端到端训练时间减少76%。

Axiomatic World Modeling for Physics Reasoning

公理化世界建模用于物理推理

• Authors: Xinye Yang, Zhenyang Liu, Yuxuan Wang, Yuanyuan Lei
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24519
• Pdf link: https://arxiv.org/pdf/2610.24519
• Abstract Recent research-level physics-reasoning benchmarks for LLMs reveal a failure mode that lies not in knowledge or execution, but in modeling the world being reasoned about. Models often silently alter given quantities, fabricate laws, or inject unstated assumptions to force a problem into familiar patterns, and then reason inside this corrupted world. We define this premise-level divergence as modeling drift: the gap between a model's internal axiomatic world model (AWM) and the true physical world specified in the problem. Although RLVR and PRM have yielded substantial gains in mathematics and code, their reward shapes do not directly supervise world modeling in axiomatic domains such as physics. We propose Reinforcement Learning with Axiomatic World Modeling (RLAWM), which optimizes the policy by penalizing divergence between its inferred axiomatic world model and the true physics world. RLAWM probes physics reasoning at two hierarchical levels (modeling and solving), each scored by a physically grounded reward. The modeling reward targets abstract knowledge formulation, evaluating whether the model captures the true axiomatic world before attempting calculation. The solving reward targets instantiated knowledge reasoning, checking whether the model's concrete derivation remains consistent with the axiomatic world. This two-level structure separates what world the model believes it is reasoning in from how it reasons within that world. The framework operates exclusively during training, with no extra cost during inference. RLAWM improves performance by up to +12.2 points on PhysReason and +5.2 on PHYSICS over the baseline. It also generalizes zero-shot across difficulty, source, and domain, with a 2.45x gain and +10 points over the strongest baseline on olympiad-level physics and math.
• 中文摘要 近期研究级的大型语言模型物理推理基准显示，失败模式不在于知识或执行，而在于对被推理世界的建模。模型常常默默改变给定量、捏造定律，或注入未明说的假设，将问题强行引入熟悉模式，然后在这个腐败世界中进行推理。我们将这种前提层面的发散定义为建模漂移：模型内部公理化世界模型（AWM）与问题中指定的真实物理世界之间的差距。尽管RLVR和PRM在数学和代码方面取得了显著进步，但它们的奖励形态并不直接指导物理等公理化领域的世界建模。我们提出了基于公理化世界建模的强化学习（RLAWM），通过惩罚推断公理化世界模型与真实物理世界之间的发散来优化该策略。RLAWM在两个层级（建模和求解）探讨物理推理，每个层级都由物理基础奖励评分。建模奖励针对抽象知识表述，评估模型是否捕捉了真实的公理世界，然后再尝试计算。求解奖励针对实例化的知识推理，检查模型的具体推导是否与公理化世界保持一致。该两层结构区分了模型认为其推理的世界中与其在该世界中的推理方式。该框架仅在训练期间运行，推理过程中不额外付费。RLAWM在PhysReason上比基线提升性能最高+12.2点，PHYSICS提升+5.2分。它还将零射击在难度、来源和领域上推广，在奥林匹克级物理和数学中提升2.45倍，较最强基线提升+10分。

Solving a Giant Rubik's Cube with a Quadruped Robot

用四足机器人解决一个巨大的魔方

• Authors: Lukas Amiet, Gabriel Sánchez-Blanco Repgen, Mayank Mittal, Victor Klemm, Andrei Cramariuc, Marco Hutter, Cesar Cadena
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.24610
• Pdf link: https://arxiv.org/pdf/2610.24610
• Abstract Pedipulating a freestanding articulated object requires controlling its pose and internal configuration while maintaining balance. We demonstrate this capability by autonomously solving a giant Rubik's Cube with a quadruped using only its feet. The task demands repeated precise interactions, since unintended layer motion can invalidate subsequent operations. We develop an onboard RGB-D perception pipeline that estimates the cube's metric pose and its full color state from partial views acquired through locomotion and flipping. A planner converts the computed solve into robot-executable skill sequences. Learned whole-body skills reorient the cube while preserving its layer configuration and turn its layers with one foot pinning the cube as the other pushes. We train these skills using reinforcement learning with privileged observations and distill them into policies that map onboard depth and proprioception to joint targets. The deployed policies tolerate variations in relative robot-cube placement and recover from external disturbances. The system completes three autonomous hardware solves, each about 30 steps over 8-9 minutes. These results show that combining onboard perception with coordinated pedipulation enables a quadruped to complete extended multi-step physical tasks. Additional material can be found at: this https URL
• 中文摘要 在独立可动物体上行走需要控制其姿态和内部配置，同时保持平衡。我们通过仅用脚自主解决一个四足动物的巨大魔方来展示这一能力。该任务需要反复精确的交互，因为非预期的层移动可能导致后续操作失效。我们开发了内置RGB-D感知流水线，通过移动和翻转获得的部分视角估计立方体的度规姿态和全色状态。规划器将计算出的解算转换为机器人可执行的技能序列。学习到的全身技能在保持层配置的同时重新定向魔方，并在一脚踩住立方体、另一脚推动时转动其层。我们通过强化学习和特权观察训练这些技能，并将其提炼成将船载深度和本体感觉映射到联合目标的策略。部署的策略容忍机器人与立方体相对位置的差异，并能从外部干扰中恢复。系统完成三次自主硬件求解，每次约30步，耗时8-9分钟。这些结果表明，将机载感知与协调的步行结合，使四足动物能够完成长时间的多步物理任务。更多信息可见：此 https URL

What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

强化学习对推理的贡献：基础模型不会做出的一些决策

• Authors: Siyu Zhang, Yuquan Lu, Yoshua Y. Li, Yifan Ding, Kun Zeng, Yu Huo, Yifu Guo, Rongxiang Weng, Xunliang Cai, Xiaoying Tang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24663
• Pdf link: https://arxiv.org/pdf/2610.24663
• Abstract Reinforcement learning with verifiable rewards (RLVR) substantially improves mathematical reasoning, but what it adds to the base model remains unclear. We investigate this through a sequence of generation-time interventions on Qwen3-4B-Base, measuring how much of its RL-trained counterpart's gain each intervention recovers on 120 AIME problems. Power sampling with the same 8,192-token output limit brings no gain beyond greedy decoding. Yet borrowing the RL model's token only where the two models' greedy choices differ, roughly 4% of generated positions, recovers 90.5% of the pass@1 gain, with similar recovery across three model pairs on MATH-500. We find that even a single opening token has a substantial effect: forcing the base model to begin with "Alright" recovers 35% of the RL gain. The opening alone is insufficient, however. Later replacements after the same opening add another 5.2 percentage points to pass@1, and the fixed rules we test for inserting reflective words such as "Wait" do not reproduce this benefit. Sparse replacement also retains much of the base model's diversity while staying close to the RL model at pass@1. At k=256, it solves 67.5% of the problems, significantly more than the RL model (61.4%) and a mixture of whole responses matched in pass@1 (61.45%). The comparison is at equal attempt counts rather than equal compute, since rl-replace consults both models at every decoding step. These results show that the opening can recover a meaningful part of the RL gain, but a larger contribution comes from later choices, which can be borrowed while retaining much of the base model's diversity. Code is available at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）显著提升了数学推理能力，但它对基础模型的贡献尚不明确。我们通过对Qwen3-4B-Base的一系列生成时间干预进行研究，测量每次干预在120个AIME问题上回收了多少强化学习训练对应物的收益。使用相同的8,192个令牌输出限制进行幂抽样，除了贪婪解码外没有额外收益。然而，仅在两模型贪婪选择不一致（约4%的生成仓位）借用RL模型的代币，能够回收90.5%的 pass@1收益，在MATH-500的三对模型中回收率相似。我们发现即使是单个开局代币也有显著效果：强制基础模型以“好的”开始，可回收35%的强化学习收益。然而，仅凭开局本身是不够的。同一开局后的替换会为pass@1增加5.2个百分点，而我们测试插入反映词（如“等待”）的固定规则并未再现这一优势。稀疏替换在保持pass@1时与强化学习模型相近的同时，保留了基础模型的大部分多样性。在k=256时，它解决了67.5%的问题，显著高于强化学习模型（61.4%），且整体响应与pass@1匹配的比例为61.45%。比较采用相等尝试次数而非计算次数相等，因为实行替换在每个译码步骤中都会参考两个模型。这些结果表明，开口可以恢复有意义的强化学习增益，但更大贡献来自后续选择，这些选择可以借用，同时保留基础模型的大部分多样性。代码可在此 https 网址获取。

From Views to Worlds: Active Exploration over 3D Worlds for Vision-Language Models

从视角到世界：视觉语言模型对三维世界的主动探索

• Authors: Qijian Tian, Jiayu Ying, Ke Fan, Lizhuang Ma, Xin Tan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.24676
• Pdf link: https://arxiv.org/pdf/2610.24676
• Abstract Vision-language models (VLMs) have achieved remarkable progress in 2D visual understanding tasks, yet the passive view-centric perception paradigm limits their capacity for 3D spatial reasoning. Such reasoning often requires actively acquiring spatial evidence beyond the currently visible views, such as camera poses and cross-view spatial relationships, to infer 3D relationships that are not explicitly represented in 2D images. However, existing methods either lack active exploration or merely provide additional views during inference, without explicitly acquiring such spatial evidence beyond 2D images. In this paper, we propose View-to-World (V2W), an active exploration framework that enables VLMs to actively explore 3D worlds and lifts VLMs from passive view-centric perception to active world-centric spatial reasoning. Given a spatial reasoning task with multi-view images, V2W first reconstructs an explicit 3D world with geometry, camera poses, and language-grounded semantics, and exposes it to VLMs through visual and linguistic interfaces. Through multi-turn interaction with these interfaces, VLMs iteratively acquire spatial evidence from the constructed 3D world, enabling active world-centric spatial reasoning. V2W can be integrated with existing VLMs in a training-free manner and can be further enhanced by optimizing the exploration policy via reinforcement learning. Experiments on spatial mental modeling benchmarks demonstrate that V2W substantially improves VLM baselines in the training-free setting and achieves state-of-the-art performance with the learned exploration policy, highlighting the importance of active exploration for spatial intelligence. Project page: \url{this https URL}.
• 中文摘要 视觉语言模型（VLMs）在二维视觉理解任务方面取得了显著进展，但被动视角中心感知范式限制了其三维空间推理能力。这种推理通常需要主动获取超越当前可见视角的空间证据，如相机姿势和交叉视角空间关系，以推断二维图像中未明确表示的三维关系。然而，现有方法要么缺乏主动探索，要么仅在推理时提供额外视角，而未明确获取超越二维图像的空间证据。本文提出“视世界”（View-to-World，V2W）这一主动探索框架，使VLM能够主动探索三维世界，并将VLM从被动视角感知提升为主动以世界为中心的空间推理。在给定一个多视图图像的空间推理任务时，V2W首先重构一个带有几何体、摄像机姿势和语言语义的显式三维世界，并通过视觉和语言接口将其暴露给VLM。通过与这些接口的多回合交互，VLM迭代地从建构的三维世界中获取空间证据，实现以世界为中心的主动空间推理。V2W可以无训练的方式与现有VLM集成，并通过强化学习优化探索策略进一步增强。空间心理建模基准测试的实验表明，V2W在无训练环境中显著提升了VLM基线，并凭借学习的探索策略实现了最先进的性能，凸显了主动探索对空间智能的重要性。项目页面：\url{this https URL}。

Learning Structured Locomotion References for Optimization-Based Whole-Body Control

学习结构化运动参考以实现基于优化的全身控制

• Authors: Ömer Tarık Banuş, Uluç Saranlı, M. Mert Ankaralı
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.24779
• Pdf link: https://arxiv.org/pdf/2610.24779
• Abstract Quadruped locomotion requires coordinating body motion,contact forces, foothold placement, and gait timing. We present a hierarchical controller in which reinforcement learning adapts these structured locomotion references while a gait scheduler, reactive footstep planner, and whole-body impulse controller retain responsibility for high-frequency execution. The model-based backbone can sustain a nominal trot independently, allowing the policy to focus on when and how to modify an already functional locomotion controller. We train the complete hierarchy in a vectorized controller-in-the-loop environment using a six-direction disturbance curriculum. Under a common simulator and evaluation protocol, the proposed controller achieves 77.29% mean survival across the tested disturbance grid, compared with 50.35% for the model-based reference controller. Training-time channel ablations identify adaptive foothold placement as the largest marginal contributor to grid survival and reveal direction-dependent effects of gait-rate adaptation. Together, these results demonstrate the viability of the proposed hierarchical controller architecture for integrating learned adaptation with optimization-based whole-body control through structured locomotion references.
• 中文摘要 四足行走需要协调身体运动、接触力、脚点定位和步态时机。我们提出了一个层级控制器，其中强化学习调整这些结构化的运动参考，而步态调度器、反应式脚步规划器和全身冲动控制器则负责高频执行。基于模型的骨干能够独立维持名义上的慢跑，使政策能够专注于何时以及如何修改已运行的运动控制器。我们在向量化控制环路环境中，使用六方向扰动课程训练完整的层级结构。在通用的模拟器和评估协议下，拟议的控制器在测试扰动网格中实现了77.29%的平均存活率，而基于模型的参考控制器为50.35%。训练时间通道消融确定了自适应立足点布置是网格存活的最大边际贡献因素，并揭示了步态速率适应的方向依赖效应。这些结果共同展示了所提出的分层控制器架构在通过结构化运动参考整合学习适应与基于优化的全身控制的可行性。

DiPLO: Self-distilled Diffusion Planning for Loco-manipulation

DiPLO：机车操控的自蒸馏扩散计划

• Authors: Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand, Dian Yu, Daniel Cremers, Cordelia Schmid, Majid Khadiv
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.24847
• Pdf link: https://arxiv.org/pdf/2610.24847
• Abstract Humanoid loco-manipulation is particularly challenging because it requires coordinating locomotion, balance, and object interaction, making it difficult to learn from limited demonstrations or from scratch through Reinforcement Learning (RL). We introduce DiPLO (self-distilled Diffusion Planning for Loco-manipulation), a framework that combines a self-distilled diffusion planner with a low-level whole-body tracking policy. Our planner is pretrained via supervised learning on motions derived from human-to-humanoid retargeting, while the tracker is trained via RL to execute them. DiPLO centers around two complementary post-training stages. First, we introduce Self-Distillation Diffusion Policy Optimisation (SD:DPO) to remove the planner's dependence on strong task conditioning, which we identify as a crucial ingredient for effective pretraining, as downstream fine-tuning does not recover comparable performance when it is omitted.. We then fine-tune this distilled model with Diffusion-Augmented Proximal Policy Optimization (DA:PPO), a maximum-entropy RL method for diffusion models, to improve task success and align generated plans with the frozen tracker's execution capabilities. Together, these stages enable motion generation through state-history and goal conditioning alone, enabling 99% task success in pick-carry-place tasks, compared to just 45% for the pretrained model. The resulting planner also generates motions for unseen object goals, with further evaluations on kicking and pushing/pulling tasks. Finally, we deploy our pipeline on the hardware, using our DA:PPO planner-tracker on a Unitree G1, achieving repeated successful executions without any real-world fine-tuning.
• 中文摘要 类人机车操作尤其具有挑战性，因为它需要协调运动、平衡和物体交互，这使得通过有限演示或通过强化学习（RL）从零开始学习变得困难。我们引入了DiPLE（自提炼式机车操控扩散规划），这是一个结合了自提炼扩散规划器和低层次全身跟踪策略的框架。我们的规划器通过监督学习预训练，针对人形生物重新定向产生的动作，而追踪器则通过强化学习训练以执行这些动作。DiPLO围绕两个互补的训练后阶段展开。首先，我们引入自蒸馏扩散策略优化（SD：DPO），以消除规划者对强任务条件的依赖，我们认为强任务条件是有效预训练的关键要素，因为下游微调若省略，无法恢复相当的性能。随后，我们用扩散增强近端策略优化（DA：PPO）微调该提炼模型，这是一种扩散模型的最大熵强化学习方法，以提升任务成功率，并将生成的计划与冻结跟踪器的执行能力对齐。这两个阶段共同实现了仅通过状态历史和目标条件生成运动，使得拾取-携带-放置任务的任务成功率达到99%，而预训练模型仅为45%。生成的规划器还生成未见物体目标的动作，并进一步评估踢动和推拉任务。最后，我们将流水线部署到硬件上，使用Unitree G1上的DA：PPO规划追踪器，实现多次成功执行，无需任何实际微调。

Learning to Program Retrievals: Composing Search Primitives via Reinforcement Learning

学习编程检索：通过强化学习组合搜索原语

• Authors: Kyuyoung Kim, Zilin Xiao, Zhiyuan Wang, Avinash Atreya, Ke Yang, Jinwoo Shin, Kecheng Hao
• Subjects: Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.24854
• Pdf link: https://arxiv.org/pdf/2610.24854
• Abstract Retrieval-augmented language models increasingly use multi-step search to solve knowledge-intensive tasks, yet existing systems typically expose retrieval through fixed pipelines or collections of atomic tools. We introduce Retrieval Program Learning (RPL), a programmable search framework that allows a policy to compose search primitives into executable programs with local state and control flow. This enables adaptive search that can retrieve broadly, process intermediate results, and selectively construct model context without tying retrieval breadth to context consumption. However, the resulting flexibility creates a difficult exploration problem: useful programs may require several coordinated decisions, while reinforcement learning often supplies only sparse feedback. To address this challenge, we propose program mutation, a grammar-constrained and execution-verified procedure that generates local variants of suboptimal programs sampled during RL, converts them into training trajectories, and distills them back into the policy. We evaluate RPL on open-domain question answering and corpus-level reasoning tasks, comparing it with a range of retrieval and reasoning baselines. RPL consistently improves open-domain QA performance across model sizes and in- and out-of-domain benchmarks, with particularly large gains on multi-hop tasks. Notably, a 4B model trained with RPL outperforms all 8B model baselines. On corpus-level reasoning, RPL achieves the highest answer F1 and document-coverage F1 for both model sizes while substantially reducing context consumption and retrieval turns. Our results show that programmable search expands the retrieval strategies that agents can learn, while program mutation provides an effective mechanism for exploring useful program structures beyond direct on-policy sampling.
• 中文摘要 检索增强语言模型越来越多地使用多步搜索来解决知识密集型任务，但现有系统通常通过固定流水线或原子工具集合来暴露检索。我们引入了检索程序学习（RPL），这是一种可编程搜索框架，允许策略将搜索原语组合成具有局部状态和控制流的可执行程序。这使得自适应搜索成为可能，能够广泛检索、处理中间结果，并选择性构建模型上下文，而不将检索广度与上下文消耗绑定。然而，这种灵活性带来了复杂的探索难题：有用的程序可能需要多个协调决策，而强化学习往往只能提供稀疏的反馈。为应对这一挑战，我们提出了程序变异，这是一种语法约束且执行验证的过程，生成在强化学习中采样的次优程序的局部变体，将其转换为训练轨迹，并再提炼回策略中。我们在开放领域问答和语料库层推理任务中评估RPL，并与一系列检索和推理基线进行比较。RPL在不同模型规模和域内外基准测试中持续提升开放域QA性能，尤其是在多跳任务上表现显著。值得注意的是，使用RPL训练的4B模型表现优于所有8B模型基线。在语料库层面推理中，RPL在两种模型大小下均获得最高的F1答案和文档覆盖率F1，同时显著减少上下文消耗和检索次数。我们的结果表明，可编程搜索扩展了代理可学习的检索策略，而程序突变则提供了探索超越直接策略抽样的有用程序结构的有效机制。

Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics

通过神经对偶批判器进行样本高效强化学习，采用通用效用

• Authors: Ankur Naskar, Vaneet Aggarwal
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.24880
• Pdf link: https://arxiv.org/pdf/2610.24880
• Abstract Reinforcement learning with general utilities optimizes nonlinear functionals of policy-induced occupancy measures. We develop a natural actor-critic method for concave RLGU objectives admitting an exact variational dual representation, with explicit constructions for a broad class of separable occupancy utilities. Rather than estimating the occupancy measure and then forming the policy-gradient pseudo-reward, our method learns this pseudo-reward directly as a function of the state-action pair using a neural critic. The critic is trained by projected stochastic updates, while multilevel Monte Carlo combines outputs across training horizons to control its bias at logarithmic expected cost. Our analysis separates conditional actor bias from estimator second moments and accounts explicitly for dual optimization, critic approximation, and neural linearization errors. Under policy-transfer, dual-stability, and local critic-regularity conditions, the method achieves $\widetilde{O}(T^{-1/2})$ global convergence above policy, critic-class, and finite-width approximation errors using $\widetilde{O}(T)$ expected environment transitions and reference samples. For a fixed critic representation, this yields $\widetilde{O}(\epsilon^{-2})$ sampling complexity above the approximation floor, covering infinite state-action spaces and occupancy utilities beyond fixed finite vectors of returns.
• 中文摘要 使用通用效用的强化学习优化策略诱导占用度量的非线性泛函。我们开发了一种自然的演员-批评者方法，用于凹 RLGU 目标，允许精确的变分对偶表示，并明确构造了一类可分离占用效用。我们不先估计占用度量再形成策略梯度伪奖励，而是通过神经批评者直接学习该伪奖励作为状态-动作对函数。批评者通过预测随机更新训练，而多级蒙特卡洛将跨训练视野的输出结合，以对数期望成本控制其偏置。我们的分析将条件演员偏置与估计器第二矩分离，明确考虑对偶优化、批判者近似和神经线性化误差。在策略转移、双重稳定性和局部批判者正则性条件下，该方法通过 $\widetilde{O}（T^{-1/2}）$ 在策略、批判类和有限宽度近似误差上实现 $\widetilde{O}（T）$ 的全局收敛，使用期望环境转变和参考样本。对于固定批判表示，这会产生 $\widetilde{O}（\epsilon^{-2}）$ 在近似底以上的采样复杂度，涵盖无限状态-行动空间和超出固定有限收益向量的占用效用。

HybridResearcher: Bridging the Structured-Unstructured Boundary for Deep Research Agents

HybridResearcher：跨越深度研究代理的结构化与非结构化界限

• Authors: Soyoung Yoon, Peiran Xu, Han Wang, Ruofan Wu, Xiaodong Yu, Feng Yan, Mohit Bansal, Debiao Li, Seung-Won Hwang, Yuxiong He, Zhewei Yao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24899
• Pdf link: https://arxiv.org/pdf/2610.24899
• Abstract While LLM agents are increasingly capable of gathering and synthesizing evidence from a single homogeneous source, such as structured tables or unstructured data, real-world analytical questions frequently span heterogeneous sources. For example, answering one enterprise question may require both a result computed over an internal database and evidence found on the web. Recent evaluations on benchmarks like HybridDeepResearch show that even frontier agents struggle when evidence must cross this boundary. While these benchmarks have exposed the limitation, methodologies to explicitly teach cross-source reasoning remain absent. To address this, we introduce HybridResearcher, a 4B agent trained to bridge evidence across the structured and unstructured boundary. To build it, we synthesize cross-source bridge trajectories by starting with validated single-source tasks-such as existing text-to-SQL or web search queries-and fabricating the missing source around a bridge entity drawn from the seed's own output. We retain a sample only when strict construction gates confirm that neither side alone suffices, ensuring the data requires true cross-source interdependence. Finally, we train on this curated dataset using supervised fine-tuning followed by reinforcement learning. Evaluated on the existing HybridDeepResearch benchmark, HybridResearcher improves over our base model, Qwen3.5-4B, by 30.0 points pass@8 and surpasses the far larger Qwen3.5-397B-A17B by 8.3 points, while also improving over its base model on established, held-out single-source benchmarks (BrowseComp and BrowseComp-Plus), demonstrating that this complex cross-source reasoning can be taught directly to small models without triggering catastrophic forgetting.
• 中文摘要 虽然LLM代理越来越能从单一同质来源（如结构化表或非结构化数据）收集和综合证据，但现实世界的分析问题往往跨越异构来源。例如，回答一个企业问题可能需要通过内部数据库计算结果和网络上的证据。最近对HybridDeepResearch等基准测试的评估显示，即使是前沿代理在证据必须跨越这一边界时也会感到困难。尽管这些基准测试揭示了这一局限性，但明确教授跨源推理的方法仍然缺失。为此，我们引入了HybridResearcher，一款4B智能体，经过训练，能够跨越结构化与非结构化边界连接证据。构建过程中，我们从经过验证的单源任务（如现有文本转SQL或网页搜索查询）出发，综合跨源桥梁轨迹，并围绕从种子自身输出提取的桥接实体构建缺失源代码。只有当严格的构造门确认单靠任何一方都不够时，我们才保留样本，确保数据需要真正的跨源相互依赖。最后，我们通过监督微调和强化学习训练该精心策划的数据集。基于现有的HybridDeepResearch基准测试，HybridResearcher比我们的基础模型Qwen3.5-4B提升了30.0个百分点pass@8，并且比规模更大的Qwen3.5-397B-A17B高出8.3个百分点，同时在已建立的单一来源基准测试（BrowseComp和BrowseComp-Plus）上也优于其基础模型，证明这种复杂的跨源推理可以直接教给小模型而不会引发灾难性的遗忘。

Robust Direct Preference Optimization under Preference Label-Flip Attacks

在偏好标签翻转攻击下的鲁棒直接偏好优化

• Authors: Chenye Yang, Weiyu Xu, Lifeng Lai
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.24945
• Pdf link: https://arxiv.org/pdf/2610.24945
• Abstract Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-$K$ Robust-DPO, which removes the $K$ highest-impact pairs, and Soft-Trim-$K$ Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most $K$ flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.
• 中文摘要 直接偏好优化（DPO）被广泛用于离线强化学习。然而，近期研究表明，DPO容易被对抗性偏好标签翻转攻击操控，这些攻击专门设计用来误导学习者走向攻击者选择的目标策略。为解决这一问题，我们设计了能够防御此类攻击的稳健DPO方法。基于可称标签翻转诱导对数线性DPO参数无关梯度偏移的性质，我们提出了Trim-$K$ Robust-DPO，去除$K$影响最高的对，以及Soft-Trim-$K$ Robust-DPO，使其平滑地降权。我们对所有最多反转不超过$K$的攻击集建立了均匀的确定性梯度扰动界限，明确区分防御引起的偏见与幸存的攻击效应。在有界区域假设下，这些界限保证时间一致参数和策略偏差。我们进一步量化二次正则化如何改善曲率并引入额外偏差。斯坦福人类偏好、HH-RLHF和UltraFeedback-Binarized实验验证了我们的分析。

CasePlay: Self-Play Reinforcement Learning from Case Reports for Medical Reasoning

案例游戏：自学强化医学推理案例报告

• Authors: Hao Wang, Zihan Wang, Kai Wu, Jian Yao, Yiwen Ye, Yongcan Luo, Dapeng Wu, Jiale Chen, Haihua Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24952
• Pdf link: https://arxiv.org/pdf/2610.24952
• Abstract Reinforcement learning (RL) is a promising way to improve the clinical reasoning abilities of large language models (LLMs), but current pipelines often rely on expert-labeled questions or costly synthetic-data generation. Meanwhile, the medical literature offers a wealth of high-quality unlabeled documents, including clinical case reports that trace the full clinical course from initial presentation to final treatment decisions. We study whether document-grounded self-play can convert these reports into effective RL training signals. We identify two challenges that limit existing self-play methods in this setting: knowledge-point fixation}, where the proposer repeatedly asks about a narrow set of salient clues in a source report, and difficulty-quality mismatch, where appropriately difficult questions may still be poorly formed or shallow, limiting their value for reasoner training. To address these challenges, we propose CasePlay, a self-play framework with two key components: (a) a Knowledge-Conditioned Proposer, which anchors question generation to case-specific knowledge points so that questions draw on a broader range of evidence from the source report; and (b) a Rubric-Judge Reasoner, which augments answer-accuracy feedback with rubric-based quality feedback on clue integration, reasoning depth, and option quality, guiding the proposer toward reasoning-intensive questions that better support reasoner training. Extensive experiments across seven medically related benchmarks demonstrate that CasePlay outperforms existing self-play baselines and improves the SFT warm-start model by 3.1 points on average, highlighted by a 6.3-point gain on LiveClin. This approach offers an effective path toward continually improving medical reasoning ability through self-play RL.
• 中文摘要 强化学习（RL）是提升大型语言模型（LLM）临床推理能力的有前景方式，但当前的流程往往依赖专家标注的问题或昂贵的合成数据生成。与此同时，医学文献提供了大量高质量的无标签文档，包括追踪从初次呈现到最终治疗决策的完整临床案例报告。我们研究基于文档的自我游戏是否能将这些报告转化为有效的强化学习训练信号。我们识别出限制现有自我游戏方法的两个挑战：知识点固化，即提案者反复询问源报告中有限的显著线索;以及难度质量不匹配，即适当难度问题可能仍不完整或浅显，限制了其对推理者培训的价值。为应对这些挑战，我们提出了CasePlay，一种具有两个关键组成部分的自我游戏框架：（a）知识条件提出器，将问题生成锚定于案例特定知识点，使问题能够基于来源报告中更广泛的证据;（b）评分标准评判推理器，通过基于评分标准的线索整合、推理深度和选项质量的质量反馈，增强答案准确性反馈，引导提案者提出更支持推理者培训的推理密集型问题。涵盖七个医学相关基准的广泛实验表明，CasePlay优于现有的自我对弈基线，并平均提升SFT热启模型3.1分，LiveClin提升6.3分为显著。该方法为通过自我扮演强化学习持续提升医学推理能力提供了有效途径。

CoworkEnv: Synthesizing Training Environments for Long-Horizon Cowork Agents

CoworkEnv：为长期协作代理综合训练环境

• Authors: Zhongwei Xie, Yiming Du, Jiaxin Bai, Baixuan Xu, Chunyang Li, Xiyu Ren, Wei Fan, Tianshi Zheng, Weizhi Du, Jingqiao Yu, Xingsu Li, Zhaowei Wang, Yangqiu Song
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.24960
• Pdf link: https://arxiv.org/pdf/2610.24960
• Abstract AI coworkers must coordinate tools, communicate across turns, and act on changing workspace state while respecting workflow constraints. Constructing training experience for these capabilities requires coherent requests, service records, tool interfaces, and grading checks. We present CoworkEnv, a capability-guided framework for synthesizing executable training environments for AI coworkers. Workflow requirements guide task construction, and programmatic validation with feedback repair checks generated specifications and grading behavior. Teacher executions yield curated demonstrations for supervised fine-tuning (SFT), while student interactions provide rollouts and rewards for reinforcement learning (RL). Using these resources, we train compact PocketWorker models. For Qwen3-VL-8B-Instruct, SFT raises the mean score from 46.6 to 64.1 on Claw-Eval, including a 31.9-point gain on Multi-turn tasks. An SFT-only extension to ClawMark demonstrates applicability across coworker benchmarks. We further compare RL training dynamics across configurations that vary reward processing and credit assignment. Improving training rewards provide preliminary evidence that the constructed environments support policy optimization.
• 中文摘要 AI同事必须协调工具，跨轮次沟通，并在尊重工作流约束的前提下对工作区状态变化采取行动。构建这些能力的培训体验需要连贯的请求、服务记录、工具界面和评分检查。我们介绍CoworkEnv，一个能力引导框架，用于综合AI同事可执行的训练环境。工作流需求指导任务构建，程序验证通过反馈修复检查生成规格和评分行为。教师执行产生精选演示用于监督微调（SFT），而学生互动则为强化学习（RL）提供推广和奖励。利用这些资源，我们训练紧凑的PocketWorker模型。对于Qwen3-VL-8B-Instruct，SFT将Claw-Eval的平均得分从46.6提升至64.1，其中多回合任务提升31.9分。仅SFT的ClawMark扩展展示了在同事基准中的适用性。我们还进一步比较了不同配置中的强化学习训练动态，这些配置涵盖了奖励处理和学分分配。改进训练奖励提供了初步证据，表明构建环境支持策略优化。

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

跨越时空的视觉思维：视频分割的扎实推理

• Authors: Junwei Wen, Deshui Miao, Guangming Lu, Ming-Hsuan Yang, Xin Li, Wenjie Pei
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.25090
• Pdf link: https://arxiv.org/pdf/2610.25090
• Abstract Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a \emph{video reference gap} between linguistic inference and the evidence distributed across frames. We introduce \textbf{Temporal Chain of Video Primitives (TCoVP)}, a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released. Project page: \url{this https URL}
• 中文摘要 视频推理切割（VRS）需要一个模型识别隐式描述的目标，并预测其像素级掩码轨迹，同时解析决定答案的空间关系和时间事件。纯文本思维链（CoT）可以在不明确说明底层视觉引用的情况下解释目标选择，从而在语言推断与分布于帧中的证据之间留下\emph{视频引用空白}。我们引入了\textbf{视频原语时序链（TCoVP）}，这是一种基于时空的视频CoT形式，将查询相关观察组织成时间顺序的推理链。每个原语将一个实体引用与时间支持和时间戳空间证据关联起来，链条将这些引用与语言交织，指导推理哪个实体相关、查询事件何时成立以及证据出现在哪里。选定的目标原语随后提供框提示和视频掩码预测的时间支持。我们构建TCoVP-SFT-15k和TCoVP-RL-13k，通过监督微调和链对齐强化学习该表示。跨越五个VRS基准测试和三个模型骨干的广泛实验展示了TCoVP的强大性能，凸显了在推理过程中组织空间和时间证据的有效性。所有模型和代码将公开发布。项目页面：\url{此 https URL}

Conformal Prediction-Enabled Sequence-level Policy Optimization for Reasoning Language Models

推理语言模型的共形预测支持序列级策略优化

• Authors: Hui Yang, Wan Tian, Tao Ren, Zhongyi Li, Yijie Peng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25092
• Pdf link: https://arxiv.org/pdf/2610.25092
• Abstract Reinforcement learning improves the reasoning capability of large language models through verifiable rewards for mathematical and coding tasks. In token-level methods such as GRPO, clipping acts on individual token ratios while the reward and advantage apply to a complete response. Recent sequence-level methods align these units but still use fixed clipping bands, whose tail probabilities change as the score distribution evolves. We propose Conformal Prediction-Enabled Sequence-level Policy Optimization (CSPO), which adaptively calibrates sequence-level clipping bounds using recent training statistics. The resulting band targets a specified marginal tail budget and adapts to policy drift; calibration conditional on response length requires additional assumptions. We bound the probability that a fresh score falls outside the band by the target miscoverage level plus explicit CDF discrepancy and drift terms. Under additional moment conditions, this bound also controls the objective and gradient error relative to the unclipped GSPO power surrogate. Experiments with the Qwen3-1.7B series show improvements over GSPO on difficult mathematical reasoning benchmarks and ALFWorld, with clipping endpoints that adapt during training.
• 中文摘要 强化学习通过对数学和编码任务的可验证奖励，提升大型语言模型的推理能力。在像GRPO这样的令牌级方法中，裁剪作用于单个令牌比率，而奖励和优势则应用于完整响应。近期的序列级方法对齐这些单元，但仍使用固定的剪裁带，其尾部概率随着分数分布变化而变化。我们提出了共形预测赋能序列级策略优化（CSPO），利用近期训练统计量自适应校准序列层级裁剪边界。所得带针对指定的边际尾部预算并适应策略漂移;校准基于响应长度需要额外假设。我们将新得分落在区间外的概率限制为目标误差水平加上显式的CDF差异和漂移项。在额外的矩条件下，该束缚还控制了相对于未截波GSPO功率代理的目标误差和梯度误差。对Qwen3-1.7B系列的实验显示，在复杂的数学推理基准和ALFWorld上，截波端点在训练过程中会适应，优于GSPO。

Self-organizing reinforcement learning: a swarm approaches the explore-exploit optimum

自组织强化学习：群体方法探索-利用最优

• Authors: Galen J. Wilkerson
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Social and Information Networks (cs.SI); Mathematical Physics (math-ph); Adaptation and Self-Organizing Systems (nlin.AO); Biological Physics (physics.bio-ph)
• Arxiv link: https://arxiv.org/abs/2610.25113
• Pdf link: https://arxiv.org/pdf/2610.25113
• Abstract The explore/exploit trade-off is the core problem of reinforcement learning (RL). We show that a minimal foraging swarm approaches its optimum - the ideal-free / Nash allocation of a multi-agent resource-sharing game (an equilibrium, not a return maximum) - with no learner and no reward signal. Agents explore by heavy-tailed (Levy) flights, recruit neighbours to a discovery through a threshold cascade on their proximity network, and disperse as a patch crowds; this cascade cycle balances exploration and exploitation and drives foragers toward the ideal free distribution. The swarm attains the shape of this optimum (committed foragers track richness with slope 0.94) but not its exact equilibrium - it stays measurably exploitable - reaching it dynamically (~150 steps) through the coupling of recruitment to movement: remove the coupling and matching collapses (0.94 -> 0.08). The RL sits at the level of the problem and the allocation reached, not the micro-rule: no step evaluates payoff (reward-free by construction, though the crowd signal is informationally tied to payoff), and the swarm reaches the allocation without learning it (quality plateaus and re-settles after resources move, no cumulative gain). Against a trained baseline the comparison is two-sided: in an easier, disembodied game a learner is more precise, while in the same embodied environment the reward-free swarm reaches ~65% of a validated oracle ceiling at zero training and leads until trained PPO and centralised-critic (CTDE) learners overtake it (~74-75%, 16 seeds, all above the swarm mean) after ~4,000 of 15,000 training steps - its advantage is in budget, not asymptote. In this one setting we read it not as beating RL but as a strong zero-shot, reward-free operating point: a demonstration that the self-organizing physics of a minimal swarm can carry a collective close to the optimum RL seeks, with none of RL's machinery.
• 中文摘要 探索/利用权衡是强化学习（RL）的核心问题。我们证明，最小采集群体接近其最优值——多智能体资源共享博弈的理想自由/纳什分配（均衡，而非回报最大值）——且没有学习者和奖励信号。代理通过重尾（Levy）飞行探索，通过其邻近网络的阈值级联招募邻居参与发现，并随着补丁群聚而分散;这一级联循环平衡了探索与利用，推动采集者朝向理想的自由分布方向发展。群体达到该最优的形状（承诺采集者跟踪的富度为斜率0.94），但未达到精确平衡——它保持可测量的可利用性——通过招募与移动的耦合动态达到（~150步）：移除耦合和匹配崩溃（0.94 -> 0.08）。强化学习处于问题和已分配的水平，而非微观规则：没有步骤评估收益（结构上无奖励，尽管群体信号与收益信息相关），群体在未学习分配的情况下达到分配（质量停滞后资源移动后重新稳定，无累计收益）。在训练过的基线下，比较是双向的：在更简单、无实体的博弈中，学习者更精准;而在同一具身环境中，无奖励群体在零训练时达到验证预言机上限的~65%，并领先，直到训练有素的PPO和中心化批评（CTDE）学习者在15,000步训练中~4,000步后超过它（~74-75%，16个种子，均高于群体平均值）——它的优势在于预算，而非渐近线。在这个设定中，我们不将其解读为击败强化学习，而是一个强而有力的零机会、无奖励的操作点：证明最小群体的自组织物理可以让集体接近强化学习所追求的最优，而无需任何强化学习的机制。

ClinSDT: LLM-Encoded Clinical Semantic Guidance for Sepsis Treatment via Decision Transformer

ClinSDT：通过决策转换器实现败血症治疗的LLM编码临床语义指导

• Authors: Yanyan Wu, Guanyu Zhu, Changbin Zhao, Jin Zhang, Dexu Yu, Tianmin Sun, Jining Luan, Chunxiao Li, Hanwen Du, Youhua Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25121
• Pdf link: https://arxiv.org/pdf/2610.25121
• Abstract Offline sepsis treatment aims to learn treatment policies from retrospective ICU trajectories. Decision Transformer-based methods provide a natural sequence modeling framework for this task, but they mainly rely on raw physiological states and historical clinician actions. Such raw trajectory tokens may not explicitly capture clinical semantics that clinicians use in practice, such as patient severity, recent treatment response, and current risk priorities. To address this limitation, we propose ClinSDT, an LLM-assisted Decision Transformer framework for offline sepsis treatment. ClinSDT uses a frozen LLM as a semantic bridge rather than a treatment decision maker. It constructs three clinical semantic views from ICU trajectories, namely Profile, Context, and Advisory, and inserts their encoded representations as independent tokens into a Decision Transformer policy. To support policy refinement and rollout, ClinSDT further trains a Treatment-Response Forecaster to estimate the next physiological state, SAPS-II change, and predictive uncertainty from state-action inputs. In the second training stage, the forecaster screens local counterfactual treatment candidates, and only low-uncertainty candidates with lower predicted SAPS-II increase are distilled into the policy after semantic regeneration. During inference, ClinSDT performs closed-loop treatment rollout by refreshing semantic inputs from the currently available patient state. Experiments on MIMIC-III and MIMIC-IV show that ClinSDT achieves consistent relative improvements over representative imitation learning, offline reinforcement learning, and Decision Transformer baselines under the same rollout protocol. Code and data are available at this https URL.
• 中文摘要 离线败血症治疗旨在从回顾性ICU轨迹中学习治疗策略。基于决策转换器的方法为该任务提供了自然的序列建模框架，但主要依赖原始生理状态和临床医生的历史行为。这些原始轨迹标记可能无法明确捕捉临床医生在实践中使用的临床语义，如患者严重程度、近期治疗反应和当前风险优先级。为解决这一限制，我们提出了ClinSDT，一种基于LLM辅助的决策转换器框架，用于离线败血症治疗。ClinSDT使用冻结的LLM作为语义桥梁，而非治疗决策者。它从ICU轨迹构建了三种临床语义视图，即Profile、Context和Advisory，并将其编码表示作为独立token插入决策转换器策略中。为支持策略细化和推广，ClinSDT 进一步训练治疗-反应预测器，估计下一生理状态、SAPS-II 变化及状态-行动输入的预测不确定性。第二阶段，预测器筛选局部反事实治疗候选，只有低确定性且 SAPS-II 增长预测较低的候选者在语义再生后被提炼入策略。在推断过程中，ClinSDT 通过刷新当前患者状态的语义输入，实现闭环治疗推广。MIMIC-III 和 MIMIC-IV 实验显示，ClinSDT 在同一推广协议下相较代表性模仿学习、离线强化学习和决策变换器基线实现了持续的相对改进。代码和数据可在此 https URL 访问。

TIRE: Top-Down Formula Instantiation with Reliable Execution for Physics Problem Synthesis

TIRE：俯视化公式实例化，可靠执行物理问题综合

• Authors: Xiaoyu Wu, Zhaoxin Yu, Xinglin Xiao, Nan Xu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25134
• Pdf link: https://arxiv.org/pdf/2610.25134
• Abstract Advancing physics reasoning in large language models requires scalable training data that connect physical laws, concrete problem scenarios, and verifiable numerical answers. Expert-designed problems offer top-down control over physical laws, conditions, and scenarios, but require costly human effort. LLM-based synthesis from texts scales efficiently, but model-generated answers and assessments leave supervision vulnerable to hallucinations. We introduce TIRE, a framework for Top-Down Formula Instantiation with Reliable Execution that brings knowledge-driven problem construction to large-scale data generation. Top-down instantiation uses reusable physical formulas and their applicability conditions to control the physical relations and numerical configurations of generated problems. Reliable execution establishes reference answers before problem generation, providing a common numerical foundation for problem construction and teacher distillation. Together, these mechanisms combine control over problem content with execution-grounded supervision for both supervised fine-tuning and reinforcement learning. Using TIRE, we construct TIRE-19k, spanning nine subfields of physics. Across 3 backbone LLMs and 4 benchmarks, TIRE-19k outperforms 5 representative datasets with equal amounts of SFT data, exceeding the strongest baseline by 2.86 percentage points on average. Further GRPO raises Qwen3-8B's average accuracy by 1.52 percentage points over its TIRE-19k SFT checkpoint. These results demonstrate the value of generating physics training data from explicit physical knowledge with reliable execution.
• 中文摘要 在大型语言模型中推进物理推理需要可扩展的训练数据，将物理定律、具体问题情景和可验证的数值答案连接起来。专家设计的问题提供自上而下对物理定律、条件和情景的控制，但需要付出代价的人工劳动力。基于LLM的文本综合能够高效扩展，但模型生成的答案和评估使监督容易产生幻觉。我们介绍了TIRE，一个自上而下公式实例化与可靠执行框架，将知识驱动的问题构建带入大规模数据生成。自上而下实例化利用可复用的物理公式及其适用条件来控制生成问题的物理关系和数值配置。可靠执行在问题生成前建立参考答案，为问题构建和教师提炼提供共同的数值基础。这些机制结合了对问题内容的控制与基于执行的监督，用于监督式微调和强化学习。利用TIRE，我们构建了跨越物理九个子领域的TIRE-19k。在3个骨干大型语言模型和4个基准测试中，TIRE-19k优于5个具有相同SFT数据的代表性数据集，平均比最强基线高出2.86个百分点。进一步GRPO使Qwen3-8B的平均准确率比TIRE-19k SFT检查点提高了1.52个百分点。这些结果展示了从显式物理知识生成物理训练数据并可靠执行的价值。

BatchParScale: Sparse Batch-Mixing for Sequence Models

BatchParScale：序列模型的稀疏批量混合

• Authors: Hans Peter Lyngsøe Raaschou-jensen (University of Copenhagen), M. Onurcan Kaya (Technical University of Denmark), Desmond Elliott (University of Copenhagen)
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25254
• Pdf link: https://arxiv.org/pdf/2610.25254
• Abstract Recent test-time scaling methods improve model performance by increasing inference-time compute. One practical form of this is parallel scaling, where a model samples multiple independent reasoning streams and aggregates their answers using methods like voting, reranking, or verification. While effective, this approach leaves the streams uncoordinated, and a discovery made in one rollout cannot influence any other. Multi-Sequence Verifiers jointly score candidate answers after generation, but the reasoning streams remain independent during inference. ParScale takes a step toward model-internal parallel scaling by learning to aggregate multiple forward passes, but its streams still do not exchange information while processing a sequence. We introduce BatchParScale, an attention mechanism that allows parallel streams to exchange information through a small set of communication positions. We evaluate BatchParScale on a new Parallel Needle-in-a-Haystack retrieval benchmark and in pretraining, fine-tuning, and cooperative multi-agent reinforcement learning. Across these settings, BatchParScale outperforms ParScale on language-model loss and remains competitive on coordination-sensitive MARL tasks. Masked pretraining runs beat an unmasked single-sequence baseline despite local sequence corruption, and LoRA fine-tuning shows the mechanism can be added to a pretrained Transformer. At the same time, fact retrieval performance for pretrained and finetuned BatchParScale models remains unclear.
• 中文摘要 近期的测试时间缩放方法通过增加推理时间计算来提升模型性能。其中一种实用形式是并行缩放，模型采样多个独立推理流，并通过投票、重新排序或验证等方法汇总其答案。虽然有效，但这种方法使流无法协调，且一次部署中的发现不会影响其他推理。多序列验证器在生成后共同对候选答案进行评分，但推理流在推断过程中保持独立。ParScale通过学习汇总多次前向传递，迈向模型内部并行扩展，但其流在处理序列时仍不交换信息。我们引入了BatchParScale，一种注意力机制，允许并行流通过一小部分通信位置交换信息。我们在新的“大海捞针”并行检索基准测试以及预训练、微调和合作多智能体强化学习中评估了BatchParScale。在这些设置下，BatchParScale在语言模型丢失方面表现优于ParScale，并在协调敏感的MARL任务中保持竞争力。掩盖预训练运行在无掩蔽单序列基线中表现优于局部序列损坏，LoRA微调显示该机制可添加到预训练的Transformer中。同时，预训练和微调的BatchParScale模型的事实检索性能仍不明确。

EG-GRPO: Evidence-Guided Step-level Credit Assignment for Deep Search Agents

EG-GRPO：深度检索代理的循证步骤级学分作业

• Authors: Sijia Luo, Mengjie Gao, Yuxiao He, Mengshu Sun, Cuiping Li, Hong Chen, Jing Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25269
• Pdf link: https://arxiv.org/pdf/2610.25269
• Abstract LLM-based search agents demonstrate strong capabilities in long-horizon information seeking and reasoning, typically trained via reinforcement learning. However, existing approaches predominantly rely on trajectory-level outcome rewards, providing limited supervision for complex search and reasoning processes. Although step-level credit assignment offers a promising solution, obtaining reliable and task-grounded supervision for open-ended search remains challenging. In this paper, we propose Evidence-Guided Group Relative Policy Optimization (EG-GRPO), a framework that derives step-level supervision from the evidence annotations available in synthetic deep search data. EG-GRPO identifies the first supporting citation for each required evidence claim and uses the newly acquired evidence to construct step-level process rewards. By representing the accumulated evidence as an evolving knowledge state, EG-GRPO further performs state-conditioned relative advantage estimation and combines the resulting step-level signal with the standard trajectory-level outcome advantage. Experiments on five challenging deep search benchmarks show that EG-GRPO generally outperforms standard GRPO and existing reward-enhanced and step-level RL baselines across different model scales. These results demonstrate that task-grounded evidence annotations can provide effective process supervision for credit assignment in deep search RL.
• 中文摘要 基于LLM的搜索代理在长期信息搜索和推理方面表现出强大能力，通常通过强化学习训练。然而，现有方法主要依赖轨迹级结果奖励，对复杂的搜索和推理过程提供了有限的监督。尽管阶级学分赋赋提供了有前景的解决方案，但获得可靠且基于任务的开放式搜索监督仍然具有挑战性。本文提出证据引导群体相对策略优化（EG-GRPO），该框架基于合成深度搜索数据中的证据注释，推导出步骤级监督。EG-GRPO识别每个必要证据主张的首个支持引用，并利用新获得的证据构建步骤级过程奖励。通过将累积证据表示为不断演变的知识状态，EG-GRPO进一步执行状态条件相对优势估计，并将所得的步级信号与标准轨迹级结果优势结合起来。五个具有挑战性的深度搜索基准测试的实验表明，EG-GRPO在不同模型尺度下通常优于标准GRPO及现有的奖励增强和步级强化学习基线。这些结果表明，基于任务的证据注释能够为深度搜索强化学习中的学分分配提供有效的过程监督。

Outcome-Anchored Credit Reallocation for Agentic Knowledge Graph Question Answering

结果锚定学分重新分配用于智能知识图谱问答

• Authors: Mingyuan Li, Yucheng Li, Xiaxia Wang, Gong Cheng
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25384
• Pdf link: https://arxiv.org/pdf/2610.25384
• Abstract Agentic knowledge graph question answering (KGQA) requires large language models (LLMs) to solve complex questions through multi-turn interactions with a knowledge graph (KG). Outcome-supervised reinforcement learning derives verifiable outcome rewards from final answers but uniformly assigns each rollout's outcome-level advantage to all policy-generated tokens, creating a trajectory-to-operation mismatch since KG operations within a rollout can have varying effects on the search process and final answer. To address this, we propose Outcome-Anchored Credit Reallocation (OACR), which employs a training-only KG-state outcome model to estimate outcome-reward changes between adjacent KG search states and converts them into operation credit, from which token-wise weights are derived. These multiplicative weights reallocate token-level update magnitudes while preserving the sign of the outcome-level advantage at every generated token. Across three benchmarks, OACR achieves stronger overall performance than existing KGQA methods while using fewer inference-time LLM calls and tokens.
• 中文摘要 代理知识图解答（KGQA）需要大型语言模型（LLMs）通过与知识图谱（KG）的多回合交互来解决复杂问题。结果监督强化学习从最终答案中推导出可验证的结果奖励，但统一地将每次部署的结果层级优势分配给所有策略生成的代币，导致轨迹与操作不匹配，因为推广中的KG操作对搜索过程和最终答案的影响各不相同。为解决这个问题，我们提出了结果锚定信用重新分配（OACR），它采用仅训练的KG状态结果模型来估计相邻KG搜索状态之间的结果奖励变化，并将其转换为操作积分，从中推导出按代币的权重。这些乘法权重重新分配了代币级更新幅度，同时保持每个生成代币的成果级优势符号。在三个基准测试中，OACR在使用更少推理时间的LLM调用和代币的情况下，整体性能优于现有的KGQA方法。

From Unverifiable Domain Text to Cross-Lingual RLVR via Code-Switched Task Synthesis

从无法验证的领域文本到通过代码切换任务综合实现的跨语言RLVR

• Authors: Cheonbok Park, Jihwan Kim, Shinyoung Joo, Joosung Lee, Hwiyeol Jo, Jeonghoon Kim, Jaegul Choo
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.25399
• Pdf link: https://arxiv.org/pdf/2610.25399
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) can improve reasoning in large language models, but specialized non-English domains expose a data mismatch. Raw English domain text is often available, while target-language RLVR tasks with verifiable rewards are scarce. We introduce code-switched task synthesis for cross-lingual RLVR, a data-centric approach that derives target-language and code-switched RLVR task variants from otherwise unverifiable English domain text. Using fill-in-the-middle multiple-choice QA as a seed task format, we synthesize automatically verifiable tasks from domain passages and reshape them into cross-lingual variants. These variants preserve the same evidence-answer relation but change how English and target-language evidence are presented during RLVR. On Korean medical benchmarks and cybersecurity benchmarks across four target languages, code-switched training outperforms matched monolingual and bilingual-mixture baselines on average target-language performance while maintaining comparable English performance. Additional analyses provide evidence of better target-language input understanding and stronger cross-lingual alignment in both model outputs and internal representations. These results suggest that code-switched training helps target-language inputs access domain reasoning learned more strongly in English.
• 中文摘要 带可验证奖励的强化学习（RLVR）可以提升大型语言模型中的推理能力，但专门的非英语领域会暴露数据不匹配。原始英语领域文本通常可用，而带有可验证奖励的目标语言RLVR任务较为稀缺。我们引入了针对跨语言RLVR的代码切换任务综合，这是一种以数据为中心的方法，从原本无法验证的英语领域文本中推导出目标语言和代码切换的RLVR任务变体。使用中间填充多项选择题质询作为种子任务格式，我们从领域段落中自动合成可验证任务，并将其重塑为跨语言变体。这些变体保持相同的证据-答案关系，但改变了RLVR中英语和目标语言证据的呈现方式。在韩国医学基准和网络安全基准中，针对四种目标语言，语码切换训练在平均目标语言表现上优于匹配的单语和双语混合基线，同时保持了相当的英语表现。额外分析显示，目标语言输入理解力更好，模型输出和内部表征的跨语言对应性增强。这些结果表明，语码切换训练有助于目标语言输入更强地理解以英语学习的领域推理。

Adaptive Cyber-Defense for Evolving Attacker Capabilities using Reinforcement Learning and Behavior Trees

利用强化学习和行为树实现攻击者能力演进的自适应网络防御

• Authors: Nicholas Potteiger, Ankita Samaddar, Xenofon Koutsoukos
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2610.25420
• Pdf link: https://arxiv.org/pdf/2610.25420
• Abstract Autonomous cyber-defense agents trained using reinforcement learning (RL) are increasingly adopted for defending enterprise computer networks. However, existing approaches may become brittle as attacker capabilities evolve and enable new attack strategies. In this work, we study the problem of adapting defender policies learned in CAGE Challenge Scenario 2 (CC2), a benchmark enterprise network defense scenario, to an extended network scenario with new attacker capabilities represented by new attacker actions. Effective attack strategies that leverage new actions are initially unknown, and therefore attacker policies must be learned before defender adaptation can occur. To address this challenge, we develop adversarial RL (ARL) with behavior trees (ARLBT), which uses reward behavior trees (RBTs) to define progression-based reward functions and condition attacker policies on RBT execution. The proposed RBT incorporates the new attacker actions into structured attack strategies while leveraging parallel execution to generate rewards across multiple network nodes simultaneously. We evaluate ARLBT for learning attacker policies against two baseline defender policies across network scenarios with different sets of new attacker actions and compare against multiple attacker baselines. The results demonstrate that ARLBT improves attacker success compared to ARL using the original CC2 reward function and that conditioning policies on RBT execution can improve sample efficiency. We further show that defender policies can be adapted against the learned attackers, improving robustness to attacks that leverage new attacker actions.
• 中文摘要 通过强化学习（RL）训练的自主网络防御智能体越来越多地被用于防御企业计算机网络。然而，随着攻击者能力的演变和新攻击策略的启用，现有方法可能会变得脆弱。本研究研究如何将CAGE挑战情景2（CC2）中学到的防御策略——一个基准企业网络防御场景——适应到具有新攻击者能力的扩展网络场景中的问题。有效的攻击策略利用新动作的策略最初未知，因此必须先学习攻击者策略，防御者才能适应。为应对这一挑战，我们开发了带有行为树的对抗性RL（ARL），利用奖励行为树（RBT）定义基于进展的奖励函数，并以RBT执行为攻击者策略的条件。所提议的RBT将新的攻击者动作纳入结构化攻击策略，同时利用并行执行在多个网络节点间同时生成奖励。我们评估ARLBT在不同新攻击者行动集的网络场景中，结合两种基线防御策略学习，并与多个攻击者基线进行比较。结果表明，ARLBT相比使用原始CC2奖励函数的ARL提高了攻击者的成功率，且基于RBT执行的策略条件可以提升样本效率。我们还进一步证明，防御者策略可以针对已学习的攻击者进行调整，提升对利用新攻击者行动的攻击的鲁棒性。

RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation

稀有：基于自适应规规演化的强化学习，实现开放式生成

• Authors: Xueyun Tian, Bingbing Xu, Chuyue Li, Jiaming Liu, Chao Zhang, Weijia Feng, MingHua Ma, Yu Zhao, Jingchu Wang, Ruihua Huang, Yuanzhuo Wang, Huawei Shen
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25434
• Pdf link: https://arxiv.org/pdf/2610.25434
• Abstract Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desirable qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.
• 中文摘要 可验证奖励推动了数学和代码强化学习的发展。然而，许多有价值的开放式任务共享三个特性：没有自动验证整体质量、需共同满足多个要求，以及对每个查询有多个有效回答。因此，培训需要对不同有效回答的需求满足度进行反馈。基于评分标准的强化学习通过汇总明确标准的分数来提供这种反馈。随着学习进展不均匀，一些标准会完全满足并失去辨别力，而另一些则保持信息性。动态方法修订共享评分标准或选择不断演变的标准以恢复歧视，但也可能改变有用的标准。为解决这个问题，我们提出了带自适应评分标准演变的强化学习（RARE）。核心思想是利用当前的政策反馈来指导哪些标准会变更，哪些标准会被替换来评估哪些。样本级更新用剩余质量差异的检查取代当前输出中完全满足的标准，保留所有其他标准和评分。在该规则中，我们将奖励理想品质与惩罚观察到的缺陷进行比较。视频脚本生成体现了这些特性，支持内容制作。其对在约束条件下连贯可执行叙事的需求使其成为一个具有挑战性的测试平台。针对缺陷在三种视频类型中表现最佳，平均比匹配静态评分标准提升3.82分。模型和人类偏好支持这些提升。改进也延伸至写作和通用生成基准。

Dynamics-Aligned Representation Learning under Induced Partial Observability

在诱导部分可观测性下动态对齐表示学习

• Authors: SM Mazharul Islam, Manfred Huber
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25511
• Pdf link: https://arxiv.org/pdf/2610.25511
• Abstract Model-based reinforcement learning (MBRL) has enabled sample-efficient learning of continuous control policies from high-dimensional observations through learned latent dynamics models. These models are typically trained with observation reconstruction objectives, which provide strong supervision but can bias representations toward visual fidelity leading to an undertrained prior and unreliable long-horizon predictions. We propose a stochastic gating mechanism that induces controlled partial observability during training by sampling a spatial binary mask from a Beta--Bernoulli random field and applying it to the raw observation. The gate probability is resampled at each time step, exposing the encoder to conditions ranging from near-total occlusion to near-full visibility. Integration of our method requires minimal architectural changes from existing dynamics models: the gated observation and the binary mask are channel-wise concatenated and passed to the encoder as a single augmented input, with no modifications to the broader training pipeline. We evaluate the proposed approach on a set of pixel-based continuous control tasks selected from the DeepMind Control Suite. Using model predictive control for evaluation, which relies solely on open-loop model rollouts, we show that stochastic gating improves planning performance and data efficiency across tasks. Further analysis demonstrates that performance is robust to the spatial scale of the mask, and a balanced observability during training outperforms configurations biased toward either sparser or denser observations.
• 中文摘要 基于模型的强化学习（MBRL）使得通过学习的潜在动力学模型，从高维观测中高效地学习连续控制策略。这些模型通常以观测重建目标训练，提供强有力的监督，但可能使表征偏向视觉忠实度，导致预先预测不足且不可靠。我们提出了一种随机门控机制，通过从Beta-伯努利随机场采样空间二进制掩码并应用于原始观测，诱导训练过程中受控的部分可观测性。每个时间步对门概率进行重采样，使编码器暴露于从近乎完全遮挡到近乎全能见的各种条件。我们方法的集成只需对现有动力学模型进行最小的架构调整：门控观测和二元掩码通过通道串接，作为单一增强输入传递给编码器，不修改更广泛的训练流水线。我们在DeepMind控制套件中选定的基于像素的连续控制任务中评估了该方法。利用模型预测控制进行评估，该控制仅依赖开环模型展开，我们证明随机门控提升了任务间的规划性能和数据效率。进一步分析表明，性能对掩模的空间尺度具有鲁棒性，且在训练期间保持平衡的可观测性优于偏向稀疏或密集观测的配置。

Reinforcement learning on the EPRI AI-ccelerating Unit Commitment benchmark: diagnosing over-commitment and guiding MILP with learned policies

基于EPRI AI加速单元承诺基准的强化学习：诊断过度承诺并引导MILP通过已学政策

• Authors: Maharani Devira Pramita, Aidan O'Sullivan
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.25544
• Pdf link: https://arxiv.org/pdf/2610.25544
• Abstract Reinforcement learning (RL) is increasingly proposed as a fast solver for unit commitment (UC), but standalone agents remain far more expensive than mixed-integer linear programming (MILP). In the first study of RL on the 51-generator, 72-hour benchmark of the Electric Power Research Institute (EPRI) AI-ccelerating Unit Commitment competition, we trace this gap to the agents' own exploration, close most of it, and show that the resulting policy can accelerate an exact solver. In every configuration tested, Proximal Policy Optimisation (PPO) and an action-branching Deep Q-Network (DQN) trained from scratch committed about twice the units of a thermal-only MILP optimum, at 39-64% higher cost, whereas a merit-order rule came within 1.6-7% (10-24% when unserved energy is priced). When each unit's decision is sampled independently, spare units insure against random de-commitments. Consistent with this, injecting exploration noise during fine-tuning enlarged a lean policy's fleet, a reward without shaping terms did not remove the over-commitment, and under their training noise the agents outperformed the rule subjected to comparable noise. Imitating the MILP solutions and fine-tuning conservatively brought PPO within 2.6% of the optimum, and these policies had the lowest penalised cost of any evaluated (about 5%). Fixing the fine-tuned policy's confident decisions made the MILP 3.8-8.5 times faster within 0.10-0.25% of the best solution, with a smaller worst case than an imitation-only policy fixing as many variables (1.1% against 8.9%). Because dispatch method and benchmark scope alone shift the reported optimality ratio by up to 3.6% and 9%, we propose a reporting protocol.
• 中文摘要 强化学习（RL）越来越被提议作为单位承诺（UC）的快速求解器，但独立代理的成本仍远高于混合整线性规划（MILP）。在电力研究院（EPRI）51个发电机、72小时的AI加速单元承诺竞赛中，我们将这一差距追溯到代理自身的探索，缩小了大部分差距，并证明最终策略能够加速精确求解器。在所有测试的配置中，近端策略优化（PPO）和从零训练的动作分支深度Q网络（DQN）贡献的单位约是仅热量MILP最优值的两倍，成本高出39-64%，而优选顺序规则则在1.6-7%内（未服务能源定价后为10-24%）。当每个单元的决策独立采样时，备用单元会为随机取消承诺投保。与此一致，在微调过程中注入探索噪声会扩大精益策略的车队，没有整形条款的奖励并未消除过度承诺，且在其训练噪声下，代理表现优于受类似噪声影响的规则。通过模拟MILP解决方案并保守微调，使PPO距离最优值仅剩2.6%，这些政策的惩罚成本最低（约5%）。修正微调策略的信心决策使MILP在最优解0.10%至0.25%内速度快3.8-8.5倍，且最坏情况比仅模拟的策略修正相同变量（1.1%对8.9%）更小。由于仅调度方法和基准范围就会导致报告的最优性比率增加最多3.6%和9%，我们提出了一种报告协议。

Zephyrus-R1: Agentic Reinforcement Learning for Weather Reporting

Zephyrus-R1：气象报告中的能动强化学习

• Authors: Fengfei Yu, Marshall Fisher, Isadora White, Sumanth Varambally, Yasaman Jafari, Shivam Yadav, Samuel M. Bartlett, Shawn Roj, Benjamin J. Moore, Michael D. Warner, Luyu Han, Zhirui Xia, Duncan Watson-Parris, Taylor Berg-Kirkpatrick, Yian Ma, Rose Yu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25625
• Pdf link: https://arxiv.org/pdf/2610.25625
• Abstract Reinforcement learning (RL) has emerged as a key method for training AI models in scientific applications, but RL remains difficult when outputs cannot easily be verified against a known ground-truth answer. Natural-language weather forecast reports are one such domain: they translate atmospheric data into assessments of weather evolution, hazards, and uncertainty for both the general public and key decision-makers. Unlike numerical weather-analysis questions, they admit many valid outputs, which limits deterministic verification. To address this, we develop Zephyrus-R1, a practical framework for RL on scientific weather forecast reporting. We focus on three key components of the agentic RL pipeline: reward design, curriculum learning, and an analysis of human-expert alignment. To encourage both report legibility and factual accuracy, we design a reward function that combines a meteorological rubric with bidirectional claim entailment, both evaluated against National Weather Service reference reports. Our framework optimizes this reward in a simulated tool-calling environment, enabling vision-language agents to gather and analyze forecast evidence across regions and lead times before producing a report. We use annotations from three meteorologists to audit the claim labels produced by our LLM judges. Finally, a curriculum helps small models first develop tool use and scientific reasoning through RL on verifiable weather-analysis questions, then optimize the open-ended report reward. After training Qwen3.5-4B with Zephyrus-R1, we improve the mean across six judge-metric scores by 26% over the base model and achieve results competitive with frontier open-weight models. For Qwen3.8-27B, we improve the same mean by 13% over the base model, by 6.7% over Kimi-K3, and by 2.7% over GPT-5.6-Luna.
• 中文摘要 强化学习（RL）已成为训练科学应用中AI模型的关键方法，但当输出无法与已知的真实答案进行验证时，强化学习仍然困难。自然语言天气预报报告就是其中一个领域：它们将大气数据转化为对天气演变、危害和不确定性的评估，面向公众和关键决策者。与数值天气分析问题不同，它们允许许多有效输出，限制了确定性验证。为此，我们开发了Zephyrus-R1，这是一个用于科学天气预报报告的强化学习实用框架。我们聚焦于能动强化学习流程的三个关键组成部分：奖励设计、课程学习和人与专家对齐的分析。为了促进报告的可读性和事实准确性，我们设计了一个奖励函数，结合气象规律和双向索赔承包，均基于国家气象局参考报告进行评估。我们的框架在模拟工具调用环境中优化了这一奖励，使视觉语言代理能够在生成报告前收集和分析跨区域和前置时间的预报证据。我们使用三位气象学家的注释来审计LLM评审产生的索赔标签。最后，课程帮助小模型先通过强化学习在可验证的天气分析问题上发展工具使用和科学推理，然后优化开放式报告奖励。在用Zephyrus-R1训练Qwen3.5-4B后，我们在六个评判指标得分中平均值较基础模型提升了26%，并取得了与前沿开放权重模型竞争的结果。对于Qwen3.8-27B，我们在同一个平均值上比基础模型提升了13%，比Kimi-K3提升了6.7%，比GPT-5.6-Luna提高了2.7%。

Humanoid SRL: Augmenting Humanoids with Supernumerary Limbs for Coordinated Loco-Manipulation

类人生物SRL：增强拥有多余肢体以实现协调操作的人形生物

• Authors: Fukang Liu, Feiyang Wu, Paul Barsa, Jaehwi Jang, Akshat Jain, Zhigen Zhao, Yipu Chen, Zhaoyuan Gu, Nikhil Vemuri, Anshul Chandaliya, Kaijie Zhu, Sean Lublinsky, Matthew Zhu, Tony G. Chen, Ye Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.25628
• Pdf link: https://arxiv.org/pdf/2610.25628
• Abstract Humanoid robots are envisioned to perform diverse human-level tasks through their anthropomorphic morphology. Their dual-arm configuration enables a broad range of everyday actions. Yet, as in humans, this morphology can constrain loco-manipulation when both arms are occupied or additional contacts are required beyond their reach. Supernumerary robotic limbs (SRLs) offer a way to extend these capabilities, but their integration requires the added limb to be coordinated with whole-body motion and balance. In this study, we develop and integrate a supernumerary arm into a humanoid platform to investigate how morphological augmentation reshapes its feasible action space. Through quantitative analysis and hardware experiments across two configurations, we show how the additional limb expands whole-body interaction capabilities by enabling manipulation and support to be coordinated beyond the constraints of the native dual-arm configuration. To control this augmented morphology, we leverage model-based trajectory optimization (TO) to generate dynamically feasible motions for goal-conditioned reinforcement learning (RL) controllers. The controllers are trained in simulation and transferred to hardware in a zero-shot manner, enabling coordinated whole-body skills for contact-rich tri-arm loco-manipulation. We further show vision-guided autonomous execution on selected tasks using diffusion policies trained from hardware demonstrations. Our findings suggest that humanoid morphology need not be restricted to human-like forms, but can be reimagined as an extensible design space for robotic augmentation.
• 中文摘要 类人机器人被设想通过其拟人形态来执行多种人类级别任务。其双臂结构使得日常动作范围广泛。然而，与人类类似，当双臂同时被占据或需要超出其够及范围的额外接触时，这种形态可能限制了运动操作。多余机械臂（SRL）为扩展这些能力提供了一种途径，但其集成要求额外肢体必须与全身运动和平衡协调。本研究将多余臂开发并集成到类人平台中，探讨形态增强如何重塑其可行的动作空间。通过定量分析和两种配置的硬件实验，我们展示了额外肢体如何通过协调操作和支撑，超越原生双臂配置的限制，扩展全身交互能力。为控制这种增强形态，我们利用基于模型的轨迹优化（TO）为目标条件强化学习（RL）控制器生成动态可行的运动。控制器通过模拟训练，并以零射击方式转移到硬件，实现全体协调技能，实现接触丰富的三臂机车操作。我们还展示了利用硬件演示训练的扩散策略，在选定任务中实现视觉引导自主执行。我们的发现表明，类人形态不必局限于类人形态，而可重新构想为机器人增强的可扩展设计空间。

Think, Compress, Answer: Reinforcement Learning for Multimodal Grounding via Reasoning Compression

思考、压缩、回答：通过推理压缩实现多模态基础的强化学习

• Authors: Jinbae Seo, Hyeongjun Kwon, Jiyoung Lee, Hyunsung Jang, Namkoo Ha, Jungin Park, Kwanghoon Sohn
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25640
• Pdf link: https://arxiv.org/pdf/2610.25640
• Abstract Extended reasoning has substantially improved multimodal large language models (MLLMs) on tasks requiring compositional and relational inference. However, multimodal grounding imposes a distinct requirement: reasoning must ultimately converge on a target-specific compact representation that supports precise localization. As reasoning trajectories grow longer, they often expand into broad scene narratives and peripheral relational details, which can dilute grounded perceptual evidence and lead to ambiguous or incorrect predictions. Motivated by the information bottleneck principle, we propose Think-Compress-Answer (TCA), a reinforcement learning framework that decomposes multimodal grounding into three phases: free-form reasoning, target-aware context compression, and answer generation conditioned on the compressed context. TCA is optimized through a multi-stage Group Relative Policy Optimization (GRPO) with progressive rewards that establish the structured output protocol, minimize reasoning-specific information while preserving target-relevant content, and directly improve localization accuracy. This staged design enables MLLMs to retain the benefits of extended reasoning while distilling it into the minimal context required for grounding. Experiments across six multimodal grounding benchmarks show that TCA attains substantial zero-shot performance gains, demonstrating the effectiveness of reasoning compression.
• 中文摘要 扩展推理在需要组合和关系推理的任务上，显著提升了多模态大型语言模型（MLLM）。然而，多模态基础提出了一个特殊要求：推理最终必须收敛到支持精确定位的目标特定紧凑表示。随着推理轨迹的延长，它们往往扩展为广泛的场景叙事和边缘关系细节，这可能稀释扎根的感知证据，导致模糊或错误的预测。受信息瓶颈原则启发，我们提出了Think-Compress-Answer（TCA）强化学习框架，将多模态基础分解为三个阶段：自由形式推理、目标感知上下文压缩和基于压缩上下文的答案生成。TCA通过多阶段的组相对策略优化（GRPO）进行优化，并通过渐进奖励建立结构化输出协议，最小化推理特定信息，同时保留目标相关内容，并直接提升定位准确性。这种分阶段设计使MLLM能够保留扩展推理的优势，同时将其提炼到对接地所需的最小上下文中。六个多模态接地基准测试的实验显示，TCA实现了显著的零射击性能提升，展示了推理压缩的有效性。

OPDArena: Understanding Learning and Forgetting in On-Policy Distillation

OPDArena：理解政策提炼中的学习与遗忘

• Authors: Yuanze Xu, Tianzhu Ye, Hui Sun, Wenhui Wang, Li Dong, Furu Wei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25651
• Pdf link: https://arxiv.org/pdf/2610.25651
• Abstract On-policy distillation (OPD) is increasingly used for language model post-training, yet how teacher supervision shapes learning and forgetting remains unclear. We present OPDArena, a framework comparing supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), and OPD across five pretrained models and diverse teacher sources. It jointly evaluates mathematical learning, general capability retention, and predictive distribution changes. Teacher choice strongly shapes retention: students of RLVR-trained teachers improve on the target task while retaining prior capabilities, whereas students of SFT-trained teachers incur greater capability loss. Across post-training methods, out-of-domain (OOD) distribution change shows a more consistent positive association with forgetting than in-domain change. To understand how teacher choice affects these changes, we examine the additional information teachers provide relative to the student's pretrained model. Controlled teacher comparisons show that the task-specific preferences teachers acquire through post-training make student updates more concentrated in-domain.
• 中文摘要 政策提炼（OPD）越来越多地被用于语言模型的培训后，但教师监督如何影响学习和遗忘仍不明确。我们提出了OPDArena，这是一个比较监督微调（SFT）、可验证奖励强化学习（RLVR）和OPD的框架，涵盖五个预训练模型和多样化教师来源。该框架联合评估数学学习、一般能力保持率和预测分布变化。教师选择对留任有很大影响：RLVR培训教师的学生在保持既有能力的同时提升目标任务，而SFT培训教师的学生能力损失更大。在培训后方法中，域外（OOD）分布变化与遗忘的正相关性比域内变化更为一致。为理解教师选择如何影响这些变化，我们考察教师相对于学生预训练模型提供的额外信息。受控教师比较显示，教师通过培训后获得的任务特定偏好使学生更新在领域内更加集中。

ToolOPD: Credit-Aware On-Policy Distillation for Tool-Use Agents

ToolOPD：针对工具使用代理的信用感知政策提炼

• Authors: Chu Zhao, Lei Tang, Yanming Wang, Jianzhe Zhao, Guibing Guo, Zhengzong Chen, Yuanyuan Zhao, Fei Huang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25693
• Pdf link: https://arxiv.org/pdf/2610.25693
• Abstract On-policy distillation (OPD) enhances the capabilities of small language agents by providing dense, token-level supervision from a teacher model on student-generated trajectories. However, standard OPD implicitly assumes that credit is uniformly distributed across long-horizon trajectories, assigning the same strength of teacher supervision to different actions. Through credit-distribution analysis and outcome-reversal counterfactual experiments, we find that trajectory credit is concentrated on a small number of critical actions. Moreover, even when the trajectory is held fixed, reversing only the final outcome substantially changes the credit assigned to multiple interaction turns. These findings show that the uniform-credit assumption fails to identify critical decisions and ultimately degrades the model's reasoning accuracy. To address these challenges, we propose ToolOPD, an outcome-aware on-policy distillation framework designed for long-horizon tool use. ToolOPD first summarizes each completed trajectory and its outcome into a reusable hindsight skill. It then measures how this skill changes the teacher's probability of each student action token and converts the change into outcome-aware credit. The resulting credit preserves supervision that supports successful decisions or discourages failed ones, while suppressing irrelevant and conflicting signals. ToolOPD uses these credits to reweight the action-level distillation loss and jointly optimizes it with reinforcement learning. Extensive experiments across multiple long-horizon tool interaction benchmarks demonstrate that ToolOPD significantly outperforms existing reinforcement learning and self-distillation baselines.
• 中文摘要 政策提纯（OPD）通过提供教师模型对学生生成轨迹的密集、代币级监督，增强小型语言代理的能力。然而，标准OPD隐含假设功劳在长视野轨迹上均匀分布，将教师监督的强度分配给不同行动。通过学分分配分析和结果反事实实验，我们发现轨迹功劳集中在少数关键行动上。此外，即使轨迹保持固定，仅逆转最终结果会显著改变多交互回合的功劳。这些发现表明，统一功劳假设未能识别关键决策，最终降低模型的推理准确性。为应对这些挑战，我们提出了ToolOPD，一种面向长期视野工具的目标感知策略提炼框架。ToolOPD首先将每个完成的轨迹及其结果总结为可重复使用的事后诸葛亮技能。然后衡量该技能如何改变教师对每个学生行动标记的概率，并将变化转化为结果感知的学分。所得学分保留了支持成功决策或阻止失败决策的监督，同时抑制无关且相互矛盾的信号。ToolOPD利用这些学分重新加权动作级蒸馏损失，并与强化学习共同优化。在多个长期视野工具交互基准中的大量实验表明，ToolOPD显著优于现有的强化学习和自我蒸馏基线。

Validation of Human Motor Development Scales with Humanoid Robots

人形机器人对人体运动发展尺度的验证

• Authors: Fanxing Meng, Jing Xiao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.25701
• Pdf link: https://arxiv.org/pdf/2610.25701
• Abstract We investigate how humanoid robots can learn a series of motor skills defined by human infant motor development scales, and we evaluate which scales are comprehensive enough to serve as self-contained guides for developing gross motor skills up to standing and walking. We represent each motor development scale as a developmental graph that guides the training of a humanoid robot. We then apply this graph within a reinforcement learning framework so that the robot learns skills following the natural phases of human motor development described by the scale. Our work demonstrates that graphs derived from three out of four popular pediatric motor scales can successfully guide humanoid robots to develop these skills. The fourth scale misses some intermediate milestones that the other scales have, which causes some failures. After adding a missing milestone in its graph, it also enables the robot to develop those skills successfully.
• 中文摘要 我们研究类人机器人如何学习由人类婴儿运动发展量表定义的一系列运动技能，并评估哪些量表足够全面，可以作为发展大动作技能（直到站立和行走）的自成一体的指导。我们将每个运动发展量表表示为一个指导人形机器人训练的发展图。然后我们将该图应用于强化学习框架，使机器人按照量表描述的人类运动发展自然阶段学习技能。我们的研究表明，基于四个常见儿童运动量表中的三个的图表可以成功指导类人机器人发展这些技能。第四个量表遗漏了其他量表中的一些中间里程碑，导致一些失败。在图中添加缺失的里程碑后，机器人也能成功发展这些技能。

Predictive Bandit Curriculum Reinforcement Learning via Gradient-Based Sample Group Discovery

通过梯度样本组发现预测性强化课程强化学习

• Authors: Liang Xu, Shufan Shen, Zhongni Hou, Chao Bi, Yufei Zhang, Wei Lin, Guojun Yin, Shuhui Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25704
• Pdf link: https://arxiv.org/pdf/2610.25704
• Abstract Curriculum learning, which organizes training samples according to their learning priority, has been widely adopted in reinforcement learning to improve training efficiency. However, existing methods commonly construct curriculum units using predefined data properties and estimate their utility by simply aggregating a single proxy signal from recent training this http URL designs may group samples with different optimization effects and fail to anticipate how their utility evolves in relation to the policy. We propose Predictive Gradient-based Bandit (PGB) Curriculum, which first discovers optimization-aware learning units by clustering projected sample-level gradients. PGB then employs a lightweight temporal predictor to forecast the future utility of each unit and converts the predictions into a Boltzmann sampling distribution for adaptive data allocation. Experiments across multiple model scales and reasoning benchmarks show that PGB consistently outperforms standard GRPO, difficulty-based curricula, and history-based scheduling strategies, demonstrating improved data utilization and training efficiency.
• 中文摘要 课程学习（Curriculum learning）是根据学习优先级组织训练样本，已被强化学习广泛采用以提高训练效率。然而，现有方法通常利用预定义的数据属性构建课程单元，并通过简单聚合近期训练中的单个代理信号来估算其效用。http URL 设计可能将具有不同优化效果的样本分组，无法预见其效用如何随策略演变。我们提出了基于预测梯度的Bandit（PGB）课程，首先通过聚类预测的样本级梯度来发现优化感知学习单元。PGB随后使用轻量级时间预测器预测每个单元的未来效用，并将预测转换为玻尔兹曼抽样分布以实现自适应数据分配。跨多个模型尺度和推理基准的实验显示，PGB始终优于标准GRPO、基于难度的课程和基于历史的调度策略，显示出数据利用率和训练效率的提升。

Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines

非策略政策梯度的剖析：重要性抽样、基层正则化与基线

• Authors: Haoqun Cao, Yurun Yuan, Tengyang Xie
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25725
• Pdf link: https://arxiv.org/pdf/2610.25725
• Abstract Reinforcement learning has become a central tool for large language model (LLM) post-training, where policy gradient methods are routinely deployed off-policy, even though vanilla policy gradient assumes on-policy sampling. We study when such off-policy deployment is justified, and what roles its core ingredients, including importance sampling, KL regularization, and baselines, play in handling the resulting distribution shift. We show that for a general class of off-policy policy gradient updates, consistency, in the sense that the optimal policy remains a stationary point, requires one of two structures at the optimal policy: trajectory-level importance weighting, which suffers from the well-known curse of horizon, or KL regularization with a baseline equal to the optimal soft value, instantiated by the grouped mean. Focusing on the KL route, we identify it with the group-relative policy gradient (GRPG) update, establish an equivalence to a trajectory-level Bellman residual minimization objective, and obtain a finite-sample regret bound under all-policy coverage. This also gives a theoretical account of a variant of the empirically successful GRPO objective. We further uncover a hidden role of baselines in the off-policy setting: a constant shift in the baseline implements pessimism, the standard remedy in offline RL for partial data coverage. We instantiate this via the asymmetric REINFORCE objective, and connect it to the game-theoretic offline RL framework, showing a regret bound with single-policy coverage. Together, these results offer a unified theoretical view of off-policy policy gradient in LLM post-training and connect it to classical ideas in offline RL.
• 中文摘要 强化学习已成为大型语言模型（LLM）后训练的核心工具，策略梯度方法通常在非策略时部署，尽管普通策略梯度假设在策略中采样。我们研究何时合理部署此类非策略部署，以及其核心要素（包括重要性采样、KL正则化和基线）在处理由此产生的分布变化中扮演什么角色。我们证明，对于一类非策略策略梯度更新，一致性（即最优策略保持平稳点）需要两种最优策略结构之一：轨迹级重要性加权，这受到众所周知的地平线诅咒;或带有基线等于最优软值的KL正则化，由分组均值实例化。聚焦于KL路径，我们将其与组相对策略梯度（GRPG）更新相关联，建立轨迹级Bellman残差最小化目标的等价性，并获得在全政策覆盖下有限样本遗憾的界限。这也给出了对经验上成功的GRPO目标变体的理论解释。我们进一步揭示了非策略环境中基线的隐藏作用：基线的持续变化实现了悲观主义，这是离线RL中部分数据覆盖的标准解决方法。我们通过非对称REINFORCE目标实现这一目标，并将其与博弈论离线强化学习框架连接，展示了与单策略覆盖的遗憾绑定。综合来看，这些结果为LLM训练后非策略策略梯度提供了统一的理论视角，并将其与离线强化学习中的经典思想联系起来。

MAPS: A Multi-Agent Personalized Script-Driven Video Advertising System Enhanced by Reinforcement Learning

MAPS：一个多代理个性化脚本驱动视频广告系统，通过强化学习增强

• Authors: Zhongyu Wang, Chen Zuo, Qingxuan Sun, Jiayang Qiao, Qiuhao Qian, Songyan Cui, Guocheng Chen
• Subjects: Subjects: Information Retrieval (cs.IR); Computer Vision and Pattern Recognition (cs.CV); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.25748
• Pdf link: https://arxiv.org/pdf/2610.25748
• Abstract With the rapid growth of online video consumption, video advertising has become a dominant component of digital advertising. Most personalized video advertising systems follow a retrieval-based paradigm, which matches ad content with user preferences in a feature space and recommends the most similar candidates. However, since the candidate creatives are pre-produced, this paradigm limits the granularity of personalization and leads to performance disparities across diverse user groups. To address this limitation, we propose a multi-agent personalized script-driven video advertising system (MAPS). We design four collaborative agents that form a closed-loop pipeline of user profile modeling, script generation, video synthesis, and feedback-driven iterative refinement. To train the Scripter Agent, we develop a reinforcement learning framework equipped with a multi-dimensional reward function, which steers the agent toward producing scripts that simultaneously align with user content preferences and target product information. Extensive offline experiments on representative datasets demonstrate that our approach achieves encouraging performance in generating and integrating personalized creatives. Online A/B experiments in a specific scenario in the Kuaishou advertising system show that MAPS improves CTR by 2.87%, CVR by 2.74%, and ARPU by 9.85% over the generative baseline.
• 中文摘要 随着在线视频消费的快速增长，视频广告已成为数字广告的主导组成部分。大多数个性化视频广告系统遵循基于检索的范式，将广告内容与用户偏好匹配，并在功能空间中推荐最相似的候选产品。然而，由于候选创意是预先制作的，这种范式限制了个性化的细致度，导致不同用户群体间的性能差异。为解决这一限制，我们提出了一种多代理个性化脚本驱动视频广告系统（MAPS）。我们设计了四个协作代理，形成一个闭环流水线，涵盖用户画像建模、脚本生成、视频综合和反馈驱动的迭代优化。为了训练脚本代理，我们开发了一个配备多维奖励函数的强化学习框架，引导代理生成既符合用户内容偏好又符合目标产品信息的脚本。在代表性数据集上的大量离线实验表明，我们的方法在生成和整合个性化创意方面取得了令人鼓舞的表现。在快兽广告系统的特定场景下，在线A/B实验显示，MAPS相比生成基线提升了2.87%的点击率，2.74%的广告平均回报率（CVR）提升了2.74%，广告平均收入提升了9.85%。

Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory

少记，适应更久：带重复总结记忆的有界状态情境强化学习

• Authors: Jiawen Wang, Cheng Xue, Peng Zhang, Jochen Renz
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25758
• Pdf link: https://arxiv.org/pdf/2610.25758
• Abstract In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay $O(1)$ in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.
• 中文摘要 上下文强化学习（ICRL）使智能体能够适应新任务而无需参数更新，依靠内存来保留从经验中学到的内容。对于持续运行的智能体，现有使用内存的方法在使用有界成本与访问过去经验之间权衡。使用成长记忆的方法，如全历史关注和累积摘要，会产生存储和每次决策的成本，且成本随着任务长度增加而增加。固定大小的记忆限制存储但限制访问：滑动窗口丢弃旧步骤，而循环网络则将近期互动压缩为隐藏状态。我们引入了循环总结记忆（RSM），这是一种Transformer策略，专门关注近期交互的短暂工作缓冲区，并携带固定容量的早期经验摘要。当缓冲区满满时，RSM会从之前的内容和缓冲区重写摘要，然后清除缓冲区，使部署内存和每次决策成本保持$O（1）美元任务长度。在长期部分可观察任务中，冻结的RSM策略能维持其适应后的表现，达到训练时间的八倍，约占完整历史记忆的1%，提前于所有评估基线结束，而成长中的记忆则会恶化。当任务配置在运行中途发生变化时，RSM的重新适应能力优于增长和重复的记忆。清除摘要或阻止策略访问会失去其益处，而替换其他任务的摘要则会引导行为朝向该任务。综合来看，这些结果表明，我们的固定容量摘要能够在训练视野之外持续进行上下文适应。

UniJev: Separating Accuracy from Calibration in Trimodal System One Decision Models

UniJev：三峰系统一决策模型中准确性与校准的分离

• Authors: Jianing Wang, Youwen Jian, Derong Wang, Jinyang Yu, Qingyi Si, Chuanyu Qin, Yanghua Xiao, Hongcheng Guo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.25784
• Pdf link: https://arxiv.org/pdf/2610.25784
• Abstract System One models produce probabilistic judgments that software can consume directly, making reliable confidence as important as answer accuracy. Yet joint training across text, images, and video remains underexplored, and the contributions of supervised fine-tuning (SFT) and reinforcement learning for calibrated decisions (RLCD) are difficult to separate. We introduce UniJev, a trimodal framework that scores candidate answers with a shared head under joint supervised training across text, images, and video. Supervised decision models improve accuracy over their backbone readouts across all three modalities, while the existing RLCD algorithms change accuracy little and have inconsistent effects on visual calibration across recipes. These findings motivate a separate supervised objective for confidence calibration. We introduce Cora (Confidence--Outcome Reliability Alignment), which groups current predictions by confidence and optimizes each prediction's confidence toward the empirical accuracy of its bin. At both 2B and 9B, Cora lowers image and video expected calibration error (ECE) while preserving accuracy, without a reinforcement stage. We release our code at this https URL.
• 中文摘要 System One模型产生概率判断，软件可以直接采纳，使得可靠置信度与答案准确性同等重要。然而，跨文本、图像和视频的联合训练仍未被充分探索，监督微调（SFT）和校准决策强化学习（RLCD）的贡献难以区分。我们引入了UniJev，一种三模框架，在联合监督训练下对候选答案进行评分，涵盖文本、图像和视频。监督决策模型在三种模态中提升了骨干读数的准确性，而现有RLCD算法对准确性变化不大，且对不同配方的视觉校准影响不一致。这些发现促使设立一个独立的监督置信度校准目标。我们引入Cora（置信度-结果信度对齐），按置信度分组当前预测，并优化每个预测对其箱的实证准确性置信度。在2B和9B，Cora在保持准确性的同时降低图像和视频预期校准误差（ECE），无需强化阶段。我们将代码发布至https URL。

Long-Horizon Multi-Object Rearrangement via Distilled Tree Search over Continuous Actions

通过连续动作上的提炼树搜索实现长视界多对象重排

• Authors: Kallol Saha, Yining Hong, Amber Li, Yejin Choi, Maxim Likhachev, David Held
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.25785
• Pdf link: https://arxiv.org/pdf/2610.25785
• Abstract Despite recent advances in policy learning, long-horizon manipulation remains difficult because learned policies must avoid compounding errors while preserving future feasibility. While search-based planning can explicitly reason over such future consequences, it quickly becomes expensive in high-dimensional continuous action spaces. This raises a central question: how can long-horizon reasoning from search be transferred into policies in continuous action spaces? To this end, we propose DISCO: Distilled Tree Search over Continuous Actions. DISCO uses progressive-widening MCTS in physics simulation to generate long-horizon plans. During search, a learned policy proposes continuous actions, while a learned value function scores actions by their expected progress toward task completion. Using GPU-parallelized simulation, DISCO periodically distills MCTS-generated plans into the policy-value model, improving the action proposals and value estimates used in subsequent searches. Finally, the learned policy-value model can be directly deployed without the need for simulator-based search at test time. Across simulated and real-world rearrangement tasks, DISCO outperforms behavior cloning, reinforcement learning, and search-based baselines.
• 中文摘要 尽管近期策略学习有所进展，长期视野操作仍然困难，因为学习策略必须避免错误叠加，同时保持未来可行性。虽然基于搜索的规划可以明确推理此类未来后果，但在高维连续行动空间中，成本很快增加。这引出了一个核心问题：如何将搜索中的长视野推理转化为连续行动空间中的策略？为此，我们提出了DISCO：基于连续行动的蒸馏树搜索。DISCO在物理模拟中使用渐进扩展的MCTS生成长视野计划。在搜索过程中，学习的策略提出连续动作，而学习的价值函数则根据任务完成的预期进展对行动进行评分。利用GPU并行化模拟，DISCO定期将MCTS生成的计划提炼为策略价值模型，改进后续搜索中使用的行动提案和价值估计。最后，学习到的策略-价值模型可以在测试时无需模拟器搜索即可直接部署。在模拟和现实重排任务中，DISCO的表现优于行为克隆、强化学习和基于搜索的基线。

Is it RL or Data? A Controlled Comparison of GRPO and Rejection-Sampling Fine-Tuning

是强化学习还是数据？GRPO与拒绝抽样微调的受控比较

• Authors: Lakshya Chaudhry
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25812
• Pdf link: https://arxiv.org/pdf/2610.25812
• Abstract Bauer et al. (2026) showed that in low-data reinforcement learning with verifiable rewards, training GRPO on mixed-difficulty prompts gives up to 5x the sample efficiency of easy-only prompts. They did not test whether this benefit is specific to GRPO, or whether another post-training method given the same data would obtain it too. We test this on their procedural counting task with Qwen3-4B-Base and LoRA, comparing GRPO to rejection-sampling fine-tuning (RFT) on the same prompts, verifier and token cap. Six primary arms cross the two methods with three prompt sets (Easy, Mixed, and a Curated subset of prompts the base model sometimes solves, which makes GRPO's implicit filter explicit). We use three seeds per arm. Hypotheses and falsifiers were registered before training. To rule out a poorly tuned baseline, RFT receives a 27-run hyperparameter sweep, while GRPO uses Bauer et al.'s recipe unchanged. GRPO outperforms RFT at every prompt set: by 10.9 points on held-out test problems with curated prompts, and, in post hoc comparisons with matched rollout budgets, by 8.4 and 9.2 points on the easy and mixed prompts. On test the advantage survives the least favorable treatment of truncated completions; out of distribution, truncation prevents a conclusion. Curating prompts does not help RFT (-2.7 points). Mixed-difficulty data helps RFT (+3.7 points), while for GRPO the evidence is inconclusive. A secondary iterated-RFT arm, which resamples from the current policy, recovers 3.2 of the 10.9-point gap (about 30%). These are controlled measurements on one synthetic task family and one base model with three seeds per arm.
• 中文摘要 Bauer 等人（2026）证明，在低数据强化学习且可验证奖励的情况下，混合难度提示训练GRPO的样本效率是仅简单提示的5倍。他们未测试该优势是否仅限于GRPO，或其他训练后方法是否也能获得相同数据。我们在Qwen3-4B-Base和LoRA的程序计数任务中测试此优势，比较GRPO与拒绝抽样微调（RFT）对同一提示、验证器和令牌上限的提示。六个主要分支交叉于两种方法，包含三个提示集（简单、混合和基模型有时解决的策划提示子集，这使得GRPO的隐式过滤器显式化）。我们每臂使用三个种子。假设和反驳者在训练前已注册。为排除基线调优不佳，RFT接受27次超参数扫描，而GRPO则不变Bauer等人的方案。GRPO在每个提示集上都优于RFT：在保留测试题目中，精选提示比RFT高10.9分，在匹配的推广预算后，简单和混合提示分别领先8.4分和9.2分。测试中优势能承受截断完成中最不利的处理;在非分布状态下，截断会阻止结论。策划提示无助于RFT（-2.7分）。混合难度数据对RFT有帮助（+3.7分），而GRPO的证据尚不确定。一个次级迭代RFT组，从当前政策中重新采样，恢复了10.9分差距中的3.2个（约30%）。这些是基于一个合成任务家族和一个基模型的受控测量，每臂有三个种子。

Where RL Breaks: Localizing and Repairing Failures via Targeted Continued Pretraining

强化学习的漏洞：通过有针对性的持续预训练定位并修复故障

• Authors: Xinyi Wang, Danqi Chen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25836
• Pdf link: https://arxiv.org/pdf/2610.25836
• Abstract Reinforcement learning with verifiable rewards (RLVR) is widely used to improve reasoning in language models, yet growing evidence suggests that it mostly sharpens what the base model can already do. This raises a practical question: when RL fails, what is the base model missing? We look inside the RL policy's own rollouts and find that failures are localized: whether a rollout fails is strongly predicted by a small set of characteristic n-grams, ranging from reflection phrases such as "let's recheck" to specific operations such as "the inverse of A". These failures persist throughout RL training, although they are not beyond the model's reach: other sampled answers to the same question are often correct. Building on this observation, we use the failure $n$-grams to retrieve text from the pretraining corpus and continue pretraining the base model on it before RL. The resulting corpus has only 35M tokens, 3--4 orders of magnitude smaller than typical mid-training. We evaluate this approach on OLMo2 and Qwen3 models trained with RL on (1) GSM8K; (2) a mixture of math, commonsense, and knowledge questions; and (3) general math and science reasoning. Despite its size, the corpus yields consistent gains: our approach outperforms continued pretraining on the same amount of text sampled from the pretraining corpus across 10 out of 11 settings, including by $2.0$ points on GSM8K when inserted before mid-training. The retrieved text also transfers across model families: text retrieved for OLMo2-1B's failures improves Qwen3-4B on the mixture by $1.9$ points over the same baseline. Together, our results suggest that where RL fails is a cheap and effective signal for selecting pretraining data.
• 中文摘要 带可验证奖励的强化学习（RLVR）被广泛用于提升语言模型的推理能力，但越来越多的证据表明，它主要提升了基础模型已有的能力。这引出了一个实际问题：当强化学习失败时，基础模型缺失了什么？我们观察强化学习策略自身的推广，发现失败是局部性的：是否推广失败由少数特征n-gram强烈预测，这些指标从“让我们重新检查”这样的反思短语到“A的逆”等具体操作。这些失败贯穿整个强化学习训练，尽管模型并非无法控制：同一问题的其他抽样答案往往是正确的。基于这一观察，我们利用失败$n$-克从预训练语料库中检索文本，并在强化学习前继续对基础模型进行预训练。最终语料库仅有3500万个令牌，比典型中期训练少了3到4个数量级。我们在用RL训练的OLMo2和Qwen3模型上评估了这种方法：（1）GSM8K;（2）数学、常识和知识问题的混合;以及（3）普通数学和科学推理。尽管语料库规模庞大，但取得了持续的提升：我们的方法在11种设置中，10种在预训练语料库中抽样的相同文本上表现优于持续预训练，包括在GSM8K中插入中训练前的额外2.0美元积分。检索到的文本还能跨模型族迁移：OLMo2-1B失败时检索的文本在混合条件下提升Qwen3-4B，比同一基线提升了1.9美元。综合来看，我们的结果表明，RL失败的地方是选择预训练数据的廉价有效信号。

Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning

评分标准超越评估：在强化学习中利用评分标准结构

• Authors: Yuxun Huang, Yunyao Zhang, ZeLiang Chen, Yi-Ping Phoebe Chen, Junqing Yu, Zikai Song
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25850
• Pdf link: https://arxiv.org/pdf/2610.25850
• Abstract Rubrics are widely used to evaluate open-ended question answering by breaking overall quality into explicit criteria. These explicit criteria make rubrics a rich source of learning guidance, yet their potential for guiding the training process has been largely overlooked. Existing methods use rubrics mainly as direct feedback signals, without explicitly modeling the capabilities behind their criteria. This leaves their capability structure largely unused in guiding the training process. We introduce Rubric-Structured Reinforcement Learning (RSRL), an RL framework that unlocks the full potential of rubrics by exploiting their structure throughout the training process, rather than treating them simply as direct reward signals. At its core, RSRL builds on three key designs: (1) Rubric-to-Capability Structuring first recovers the capability structure hidden in rubric criteria, (2) Self-Sufficient Capability Learning then uses this structure to bootstrap underdeveloped capabilities into effective learning, (3) Capability-Residual Credit Routing finally preserves capability-specific gains during optimization. Experiments show that RSRL consistently improves both overall performance and underdeveloped capabilities, boosting downstream scientific question-answering performance by up to 7.43 over the initial policy and 7.60 over non-rubric structured RL baselines, showing that reinforcement learning can substantially benefit from the capability structure encoded in rubrics.
• 中文摘要 评分标准被广泛用于通过将整体质量划分为明确标准来评估开放式问题回答。这些明确标准使评分标准成为丰富的学习指导来源，但它们在指导训练过程中的潜力却被广泛忽视。现有方法主要将评分标准作为直接反馈信号，而未明确建模其标准背后的能力。这使得其能力结构在指导训练过程中基本未被使用。我们引入了评分标准结构强化学习（RSRL），这是一种通过在整个训练过程中利用评分标准结构，而非仅仅作为直接奖励信号，从而释放评分标准的全部潜力。RSRL的核心基于三个关键设计：（1）评分标准到能力结构首先恢复了隐藏在评分标准中的能力结构，（2）自给能力学习利用该结构将未充分开发的能力转化为有效学习，（3）能力-剩余学分路由最终在优化过程中保持能力特异的提升。实验显示，RSRL持续提升整体性能和未充分开发的能力，使下游科学问答性能比初始策略提升至7.43，比非评分标准结构化强化学习基线提升7.60，表明强化学习可显著受益于评分标准中的能力结构。

Adaptive Decoding Methods for Token-Efficient Reinforcement Learning

用于高效令牌强化学习的自适应解码方法

• Authors: Zhaonan Li, Zelei Cheng, Amritansh Mishra, Chia-Hsuan Lee, Mann Patel, Sambit Sahu, Ben Zhou, William M. Campbell
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25860
• Pdf link: https://arxiv.org/pdf/2610.25860
• Abstract Rollout decoding affects both which trajectories are used for reinforcement learning with verifiable rewards (RLVR) and how many completion tokens are spent collecting them. We study whether candidate-token support should remain fixed or widen during training in the context of rollout sampling efficiency. Our main schedule moves from top-K=10 to K=20 and then to full support as the attempted-token budget increases. In a 4B code-RLVR setting, the schedule has a higher interpolated 150M-token validation value than its matched fixed-K=10 continuation in all three runs, with a mean gain of $1.80{\pm}0.42$ points. The schedule also performs comparably to a separately trained fixed-K=20 baseline, the strongest fixed-support setting in our sweep. Follow-up experiments suggest that late widening to full support is the most promising variant, although the evidence remains noisy, and mixed-reward-group diagnostics do not clearly explain the gain. An exploratory single-run test replay yields identical point estimates for the schedule and fixed $K=10$. Overall, our results show that rollout support is an important control for sampling efficiency in RLVR and motivate further study of adaptive support schedules.
• 中文摘要 展开解码既影响哪些轨迹用于可验证奖励强化学习（RLVR），也影响收集完成代币的消耗量。我们研究候选代币支持在训练期间应保持固定还是扩大，以提升推广抽样效率。我们的主要计划从top-K=10调整到K=20，再随着尝试代币预算增加而完全支持。在4B代码-RLVR设置中，该计划在三次运行中插入的1.5亿代币验证值高于匹配的固定K=10延续值，平均收益为$1.80{\pm}0.42美元。该计划的表现也与单独训练的固定K=20基线相当，这是我们扫描中最强的固定支持设置。后续实验表明，晚期扩展至全支持是最有前景的变体，尽管证据仍存在噪声，混合奖励组诊断未能明确解释其增益。探索性单次测试回放得出相同的点估计，时间表和固定的 $K=10 美元。总体而言，我们的结果表明，推广支持是调节 RLVR 抽样效率的重要控制，并激励对自适应支持计划的进一步研究。

CODA: Cohort- and Drift-aware Foundation Model for Multimodal Clinical Reasoning

CODA：多模态临床推理的队列感知和漂移感知基础模型

• Authors: Changshuo Liu, Jiaqi Zhu, Wenqiao Zhang, Xiaokui Xiao, Beng Chin Ooi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25871
• Pdf link: https://arxiv.org/pdf/2610.25871
• Abstract Recent advances in multimodal foundation models have demonstrated strong performance across diverse clinical tasks. However, existing approaches predominantly operate at the individual sample level and fail to model the interplay between population-level cohort patterns and patient-specific variation that underlie real-world clinical reasoning. In practice, clinicians routinely reason through patient cohorts sharing similar demographic, physiological, or pathological characteristics, while accounting for patient-specific deviations from the group prototype. Motivated by this gap, we propose CODA, a cohort- and drift-aware foundation model for multimodal clinical reasoning. CODA explicitly integrates cohort structure into the modeling pipeline through two complementary mechanisms: cohort-aware population stratification and drift-aware individual adaptation. In particular, we leverage reinforcement learning to discover clinically meaningful patient cohorts, and further model each patient's deviation from cohort prototypes as a structured drift signal encoding fine-grained individual heterogeneity. These cohort- and drift-aware representations are incorporated into the foundation model to enable reasoning that is simultaneously informed by population-level patterns and sensitive to individual-level variation. Extensive experiments on real-world electronic health record (EHR) benchmarks demonstrate that CODA achieves consistent effectiveness and strong generalizability across diverse clinical tasks, including closed-QA, open-QA, and report generation.
• 中文摘要 多模态基础模型的最新进展显示，在多样化临床任务中表现出强劲的表现。然而，现有方法主要在个体样本层面运作，未能建模群体层级队列模式与患者特定变异之间的相互作用，这些差异是现实临床推理的基础。在实践中，临床医生通常通过具有相似人口统计学、生理学或病理特征的患者队列进行推理，同时考虑患者特异性与群体原型的偏差。基于这一空白，我们提出了CODA，一种多模态临床推理的队列感知和漂移感知的基础模型。CODA通过两种互补机制明确将队列结构整合进建模流程：队列感知的群体分层和对漂移感知的个体适应。特别是，我们利用强化学习发现具有临床意义的患者队列，并将每个患者偏离队列原型的过程建模为编码细粒度个体异质性的结构化漂移信号。这些队列和漂移感知表征被纳入基础模型，使推理既受人群层面模式影响又对个体层面变异敏感。对真实世界电子健康记录（EHR）基准的广泛实验表明，CODA在多种临床任务中实现了一致的有效性和强的泛化性，包括封闭质检、开放质检和报告生成。

Learning to Reason Efficiently with Interventional Step Credit

通过介入步骤信用学习高效推理

• Authors: Guanda Li, Xuefeng Bai, Yulong Chen, Zhiyong Wang, Kehai Chen, Min Zhang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25876
• Pdf link: https://arxiv.org/pdf/2610.25876
• Abstract Large reasoning models often produce lengthy chain-of-thought (CoT) traces mixing useful intermediate steps with unnecessary ones. Recent methods explore step-level signals such as attention mass and step entropy to identify and suppress unnecessary steps. Despite their effectiveness, such signals reflect the model's behavior at a step rather than that step's contribution to the answer. To address this, we propose the interventional step credit (ISC), which takes a causal-intervention view and assesses step importance to the reasoning through targeted parameter-level interventions. Building on ISC, we further propose IGPO, which uses ISC to guide fine-grained advantage rescaling in reinforcement learning, reducing redundant steps while preserving the important ones. Experiments across two model scales and nine benchmarks show that IGPO achieves up to 62% average token reduction and better overall performance among compared methods.
• 中文摘要 大型推理模型常常产生冗长的思维链（CoT）痕迹，将有用的中间步骤与不必要的步骤混合在一起。近期方法探索了注意力质量和步骤熵等步骤级信号，以识别和抑制不必要的步骤。尽管这些信号有效，但它们反映的是模型在某一步的行为，而非该步骤对答案的贡献。为此，我们提出了介入步骤信用（ISC），它从因果干预视角出发，通过针对参数级干预评估步骤对推理的重要性。基于ISC，我们进一步提出了IGPO，该方法利用ISC指导强化学习中的细粒度优势重标，减少冗余步骤同时保留重要步骤。跨两个模型尺度和九个基准测试的实验显示，IGPO在比较方法中平均代币减少率高达62%，整体表现更优。

Learn the Interface, Not the Implementation: Output Representation Dominates Training Regime in LLM-Based RTL Generation

学习接口，而非实现：输出表示在基于LLM的RTL生成中是训练模式的主导

• Authors: Siddharth Patel, Vikram Singh, Siddharth Savyasachi Malu, Rohit Singh
• Subjects: Subjects: Machine Learning (cs.LG); Hardware Architecture (cs.AR)
• Arxiv link: https://arxiv.org/abs/2610.25881
• Pdf link: https://arxiv.org/pdf/2610.25881
• Abstract Large-language-model register-transfer-level (RTL) generation has concentrated on how to train the model: fine-tuning, continued pre-training, and reinforcement learning from a verifier. We argue this is the second-order question. On a bounded, formally specifiable task of generating single-error-correcting, double-error-detecting (SECDED) ECC memory wrappers, we hold the task, data, and correctness oracle fixed and vary two axes independently: the training regime (seven settings) and the output representation (four levels, from fully expanded flat Verilog to an eight-field configuration expanded by a deterministic generator). Because a reference generator exists, correctness is decided by formal equivalence over a design space of approximately ten million points, rather than sampled from a hand-written testbench. Two results follow. First, representation dominates: every representation close to raw RTL reaches a pass rate of at most 0.17, while the configuration reaches 0.83 with no training at all. Once the model emits the configuration, every training regime achieves a pass rate between 0.89 and 1.00. Second, raw RTL hits a hard output-length limit: a 512-bit design requires 44,237 tokens of flat Verilog compared with just 69 tokens for the configuration. Consequently, raw generation becomes infeasible above 32-bit data words rather than merely inaccurate. A model trained from scratch with fewer than 50 million parameters, combined with the verified generator, achieves 0.89 formal equivalence, compared with 1.00 for a 32-billion-parameter fine-tuned model, while using approximately 1600 times less inference energy per correct design. Inference energy also dominates amortized training energy. The lesson is to learn the interface, not the implementation.
• 中文摘要 大型语言模型寄存器传输层（RTL）生成集中在如何训练模型：微调、持续预训练以及从验证器进行强化学习。我们认为这是二阶问题。在一个有界且形式可规范的任务中，生成单错误更正、双重错误检测（SECDED）ECC内存包装器，我们固定任务、数据和正确性oracle，并分别改变两个轴：训练模式（七个设置）和输出表示（四个层级，从完全展开的平坦Verilog到由确定性生成器扩展的八场配置）。由于存在参考生成器，正确性通过约一千万点的设计空间形式等价来决定，而非从手写测试平台中采样。以下有两个结果。首先，表示主导：所有接近原始RTL的表示最多可达0.17，而配置在无训练时可达0.83。一旦模型发出配置，每个训练阶段的通过率均在0.89至1.00之间。其次，原始RTL在输出长度上有严格限制：512位设计需要44,237个扁平Verilog令牌，而配置仅需69个令牌。因此，原始生成在32位以上的数据字上变得不可行，而不仅仅是不准确。一个从零开始训练、参数少于5000万的模型，结合经过验证的生成器，形式等价率为0.89，而320亿参数的微调模型仅为1.00，且每次正确设计的推理能量约为1600倍。推理能量也主导摊销训练能量。教训是要学会界面，而不是实现。

Policy Search in Code Space for Multi-Agent Pathfinding

多智能体路径寻寻中的代码空间策略搜索

• Authors: Matthias Eder, Pantelis Emmanouil, Vivek Farias, Fabian Fischer, Adam Jozefiak, Tianyi Peng, Thomas Ulz, Raphael Watschinger, Fangyu Wu, Andrew Zheng
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25946
• Pdf link: https://arxiv.org/pdf/2610.25946
• Abstract Developing coordination policies for complex multi-agent systems relies on either labor-intensive algorithm design or opaque task-specific reinforcement learning. With recent advances in autonomous coding agents, we show that policy search directly in code space offers a compelling third path that combines automated optimization with explicit interpretability. We investigate a human-directed workflow in which coding agents iteratively propose, implement, evaluate, diagnose, and eventually refine algorithms guided by simulation feedback and explicit correctness verification. We conducted two case studies and found that this workflow robustly outperformed any manually engineered solutions. On a challenging map in the 2024 League of Robot Runners competition, a two-day search discovered a policy that exceeds the stronger locally reproduced competition entry by 35.0% on the densest instance and generalizes to unseen map instances. In simulated real-world warehouses with irregular lanes, continuous robot geometry, and asynchronous motion, a week-long search discovered an entirely new class of multi-agent pathfinding algorithm that outperforms a calibrated proprietary solution in throughput by 58.9%. Under stated assumptions, we further prove that the new algorithm is collision-free and completes, in finite time, every accepted goal that is not withdrawn or replaced. Together, the studies show that policy search in code space powered by state-of-the-art coding agents offers a highly competitive path to building real engineering solutions that are both effective and analytically tractable.
• 中文摘要 为复杂多智能体系统制定协调策略依赖于劳动密集型算法设计或不透明的任务特定强化学习。随着自主编码代理的最新进展，我们表明直接在代码空间中的策略搜索提供了一条引人注目的第三路径，结合了自动优化与显式可解释性。我们研究了一种人类主导的工作流，编码代理在模拟反馈和显式正确性验证指导下，迭代提出、实现、评估、诊断，最终优化算法。我们进行了两个案例研究，发现该工作流程远超任何手动设计的解决方案。在2024年机器人奔跑者联盟竞赛的一张具有挑战性的地图上，经过两天的搜索，发现了一种策略在最密集的实例上比更强的局部复制竞赛条目高出35.0%，并推广到未见的地图实例。在模拟的真实仓库中，具有不规则车道、连续机器人几何结构和异步运动，经过一周的搜索发现了一类全新的多代理路径寻路算法，其吞吐量比校准后的专有解决方案高出58.9%。在既定假设下，我们进一步证明该新算法无碰撞，并且在有限时间内完成所有未被撤回或替代的公认目标。综合来看，这些研究表明，在代码空间中由最先进编码代理驱动的策略搜索，为构建既有效又分析可行的真实工程解决方案提供了高度竞争的路径。

Productive State-Space Exploration for Multi-turn RL in LLM Agents

LLM代理中多回合强化学习的高效状态空间探索

• Authors: Mohammed Haroon Dupty, Yong Liang Goh, Song Guo, Wee Sun Lee
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.25957
• Pdf link: https://arxiv.org/pdf/2610.25957
• Abstract Reinforcement Learning (RL) for Large Language Model (LLM) agents in multi-turn, long-horizon environments remains challenging due to inefficient action selection and repetitive behavioral patterns. In particular, agents often become trapped in high-confidence yet unproductive behaviors, repeatedly executing invalid actions or entering action loops that hinder task completion. We observe that these frequently violate environment preconditions, revealing a mismatch between the agent's representation of the internal state and the dynamics of the underlying environment. In addition, repetitive high-confidence actions lead to cyclical behaviors that provide little to no progress towards the goal. We argue that addressing these challenges requires more productive task-aligned exploration to improve the agent's understanding of environment dynamics. Motivated by intrinsic curiosity-driven exploration, we propose a conditional state-space exploration framework that balances task completion with intrinsic information-seeking behavior. Specifically, we introduce an intrinsic reward that penalizes actions whose observation duplicates prior state memory. This provides the benefit of reducing redundant behaviors and encouraging state-space coverage without relying on a general information-theoretic measure. Strong experimental results demonstrate that our approach substantially improves policy learning in multi-turn environments, mitigating invalid actions and repetitive loops.
• 中文摘要 大型语言模型（LLM）智能体在多回合、长视野环境中的强化学习（RL）由于动作选择效率低下和行为模式重复性，依然具有挑战性。特别是，智能体常常陷入高自信但无效的行为，反复执行无效动作或进入阻碍任务完成的动作循环。我们观察到这些行为经常违反环境前提条件，揭示了智能体对内部状态的表征与底层环境动态之间的不匹配。此外，重复的高置信度行为导致循环行为，几乎没有在目标前取得进展。我们认为，解决这些挑战需要更有成效的任务对齐探索，以提升智能体对环境动态的理解。基于内在好奇心驱动的探索，我们提出了一种条件式状态-空间探索框架，平衡任务完成与内在信息寻求行为。具体来说，我们引入了内在奖励机制，惩罚观察与先前状态记忆重复的行为。这有助于减少冗余行为，鼓励状态空间覆盖，而无需依赖通用的信息论衡量。强有力的实验结果表明，我们的方法在多回合环境中显著提升了策略学习，减少了无效操作和重复循环。

SkateNav: Learning Energy-Efficient Navigation through Adaptive Skating on Passive-Wheel Quadrupedal Robots

SkateNav：通过被动轮四足机器人的自适应滑行学习节能导航

• Authors: Jeil Jeong, Taeyun Kim, Sangjin Kim, Jinyoung Choi, Sung-Eui Yoon
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.25962
• Pdf link: https://arxiv.org/pdf/2610.25962
• Abstract Passive-wheel quadrupedal robots offer energy-efficient locomotion through leg-driven propulsion and momentum-driven gliding. However, navigation requires coordinating steering with intermittent propulsion and adapting push-glide timing to task demands. Velocity-tracking objectives may suppress the speed variations inherent in skating, discouraging efficient gliding. To address these challenges, we propose SkateNav, a unified skating-navigation framework based on reinforcement learning. A time-budgeted goal-reaching objective allows the policy to jointly learn steering, propulsion, and their temporal allocation. An adaptive skating cycle structures gliding, pushing, and recovery, while frequency modulation at every control step adjusts phase durations to navigation demands. Simulation experiments demonstrate higher average success rates and lower cost of transport than the evaluated planner-controller baselines, with ablations confirming the contributions of cycle structure and frequency adaptation. Real-world experiments demonstrate zero-shot transfer from simulation to a passive-wheel Mini Cheetah across cluttered indoor spaces, slopes, and paved outdoor surfaces. Supplementary videos are available on the project page: this https URL
• 中文摘要 被动轮式四足机器人通过腿部驱动和动量驱动滑翔实现节能运动。然而，导航需要协调转向与间歇推进，并根据任务需求调整推滑时机。速度追踪目标可能抑制滑行固有的速度变化，阻碍高效滑翔。为应对这些挑战，我们提出了基于强化学习的统一滑行导航框架SkateNav。时间预算的目标达成目标使政策能够共同学习转向、推进及其时间分配。自适应滑行循环构建滑翔、推力和恢复，每个控制步骤的频率调制则根据导航需求调整相位持续时间。模拟实验显示平均成功率更高，运输成本低于评估的规划者-控制基线，消融验证了循环结构和频率适应的贡献。真实实验展示了从模拟到被动轮迷你猎豹在拥挤室内空间、坡地和铺装地面上的零发射转移。补充视频可在项目页面观看：此 https URL

Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization

自蒸馏策略优化的正确性依赖非对称目标分布

• Authors: Suwan Yoon, Seongryong Jung, Changhee Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.25973
• Pdf link: https://arxiv.org/pdf/2610.25973
• Abstract Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers. However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones. We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones. We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets. Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6x fewer training steps on average across Qwen3 model scales.
• 中文摘要 自蒸馏策略优化通过反馈条件自学者为LLM后训练后提供密集的代币级监督。然而，直接匹配自学者可能会干扰正确展开的成功行为，而错误的展开则无法提供足够的纠正。我们提出了通过自蒸馏进行非对称策略优化（APSD），该方法通过混合学生和自教师分布构建正确度依赖的目标：插值保留正确展开的成功行为，而超越自教师的外推则加强错误展开的纠正。我们通过散度分解、反KL重心几何和梯度分析分析APSD，强调梯度流动在学生依赖目标中的重要性。在强化学习下，配合可验证的奖励和丰富的反馈实验显示，APSD在强化学习和自我蒸馏基线中表现优异，且在Qwen3模型尺度下，平均训练步骤减少2.6倍，与SDPO表现相当。

ClueFlow: Learning Long-Horizon Search through Attributed Research-State Transitions

ClueFlow：通过归因研究-状态转变学习长视野搜索

• Authors: Yicheng Wang, Wentao Tan, Yidan Huang, Yifei Liu, Qiong Cao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25980
• Pdf link: https://arxiv.org/pdf/2610.25980
• Abstract Long-horizon search requires agents to acquire relevant evidence and integrate it into an evolving research state. Existing methods maintain research states to guide exploration or assign turn-level progress signals, yet grounding these signals in explicit, attributable state changes remains underexplored. We introduce ClueFlow, which learns long-horizon search from attributed research-state transitions. ClueFlow constructs Source-State-Final attribution chains that connect evidence-acquiring browser actions, Evidence and Candidate updates, and the final answer supported by those updates. During search, ClueFlow records typed state changes and potentially supporting browser actions, then applies gold-conditioned semantic validation and source tracing after execution. We use this attribution to guide both supervised fine-tuning and reinforcement learning. During SFT, attribution selects which decisions receive supervision with their original interaction contexts. During Turn-PPO, the same attribution augments outcome-based turn advantages with credit assigned along the Source-State-Final chains. Transition-level validation also recovers useful supervision from gold-consistent state changes in trajectories with incorrect final answers. We evaluate ClueFlow across two Qwen backbones and four search benchmarks spanning live-web and fixed-corpus settings, observing consistent gains from SFT and further improvements from PPO. With Qwen3.5-35B-A3B, ClueFlow achieves 60.0% accuracy on BrowseComp and 72.0% on BrowseComp-Plus. In controlled BrowseComp-Plus experiments, attribution-guided SFT matches all-turn exact-context SFT using about 30% of the training tokens, while attribution-guided PPO outperforms outcome-only PPO by 4.1 pp. These results demonstrate that state-transition attribution enables efficient supervision and effective credit assignment for long-horizon search agents.
• 中文摘要 长视野搜索要求代理获取相关证据并将其整合进不断演变的研究状态。现有方法维护研究状态以指导探索或分配转折级进展信号，但将这些信号建立在明确且可归因状态变化的基础上仍然未被充分探索。我们引入了ClueFlow，它通过归因的研究-状态转变学习长视野搜索。ClueFlow构建了源-状态-最终归因链，连接了获取证据的浏览器动作、证据和候选更新，以及这些更新支持的最终答案。在搜索过程中，ClueFlow记录类型状态变化和可能支持的浏览器操作，执行后应用黄金条件语义验证和源追踪。我们利用这种归因指导监督式微调和强化学习。在SFT中，归因选择哪些决策会在其原始交互上下文中接受监督。在Turn-PPO中，同样的归因通过源-状态-最终链赋予基于结果的回合优势。过渡级验证还从错误最终答案的轨迹黄金一致状态变化中恢复有用的监督。我们在两个Qwen骨干和四个跨实时网络和固定语料库设置的搜索基准中评估ClueFlow，观察到SFT持续提升，PPO进一步提升。在Qwen3.5-35B-A3B中，ClueFlow在BrowseComp上实现60.0%准确率，BrowseComp-Plus中72.0%。在受控的BrowseComp-Plus实验中，归因引导SFT使用约30%的训练代币匹配全回合精确上下文SFT，而归因引导PPO比仅结果PPO高4.1pp。这些结果表明，状态转移归因能够高效监督并有效分配长期搜索代理的信用。

Map-Distribution Mixture-of-Experts for Multi-Agent Pathfinding

多智能体路径寻觅的专家分布地图分布混合

• Authors: Fumimaro Suzuki, Keisuke Okumura
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.25991
• Pdf link: https://arxiv.org/pdf/2610.25991
• Abstract Designing capable neural policies for multi-agent pathfinding (MAPF), which map local observations to action distributions, represents an important challenge in multi-agent learning, with practical relevance to scalable collective automation problems such as warehouse logistics. In contrast to prior studies focusing primarily on individual policy design, this study explores a meta-level Mixture-of-Experts (MoE) approach to learning-based MAPF. Specifically, given a collection of compact yet high-performing experts specialised for different map distributions, our architecture learns a router that selectively fuses their outputs according to each agent's local situation. We instantiate and validate this MoE approach with a commonly used graph neural network-based policy as well as a newly developed recursive action-token refinement model. Combined with supervised and reinforcement learning fine-tuning of the experts, our experiments show that MoE generally improves upon the pretrained policy and outperforms existing leading methods. At the same time, our findings reveal that appropriately trained, parameter-matched monolithic models still outperform MoE, highlighting a nuanced role for map-distribution MoE for MAPF.
• 中文摘要 设计多功能的多智能体路径寻寻（MAPF）神经策略，将局部观测映射到动作分布，是多智能体学习中的一项重要挑战，并与仓库物流等可扩展集体自动化问题具有实际意义。与以往主要聚焦个体策略设计的研究不同，本研究探讨了基于学习的元级专家混合（MoE）方法。具体来说，给定一组专注于不同地图分布的紧凑但高性能专家，我们的架构学习一个路由器，根据每个代理的局部情况选择性融合输出。我们通过常用的图神经网络策略以及新开发的递归动作令牌细化模型，实例化并验证了该MoE方法。结合对专家的监督和强化学习微调，我们的实验表明MoE通常优于预训练策略，并优于现有领先方法。与此同时，我们的发现显示，经过适当训练、参数匹配的单体模型仍然优于MoE，凸显了MAPF地图分布MoE在其中的微妙作用。

FrameGRPO: Incentivizing Video Reasoning via Multi-Frame Rollouts and Temporal Credit

FrameGRPO：通过多帧展开和时间署名激励视频推理

• Authors: Jiaxuan Sun, Biao Wang, Shuai Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26022
• Pdf link: https://arxiv.org/pdf/2610.26022
• Abstract Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has improved video reasoning in multimodal large language models. However, standard video GRPO varies only linguistic trajectories while conditioning every rollout in a group on the same sampled frames, which can produce homogeneous rewards and weaken group-relative learning when the shared visual evidence is insufficient or distracting. We propose \textbf{FrameGRPO}, a simple and effective reinforcement-learning framework that requires neither auxiliary networks nor changes to the model architecture. Specifically, FrameGRPO combines two lightweight components: (1) \textbf{Multi-Frame Rollouts}, which use only uniform temporal subsampling of a single maximal-frame decode to construct aligned views that diversify visual evidence and reasoning trajectories; and (2) \textbf{Gated Temporal Efficiency Credit}, which selectively reinforces a correct lower-ratio trajectory when all rollouts in a fixed full-ratio reference subset fail, without imposing an unconditional preference for fewer frames. Extensive experiments across diverse video understanding and reasoning benchmarks demonstrate robust overall improvements over standard GRPO, highlighting temporal-view diversification as an effective mechanism for exploration and credit assignment in video MLLMs.
• 中文摘要 带有可验证奖励的强化学习，尤其是群相对策略优化（Group Relative Policy Optimization，GRPO），改善了多模态大型语言模型中的视频推理能力。然而，标准视频GRPO仅在语言轨迹上变化，同时将组内每次部署基于相同的采样帧，这可能导致同质奖励，并在共享视觉证据不足或分散注意力时削弱群体相对学习效果。我们提出了\textbf{FrameGRPO}，这是一个简单有效的强化学习框架，无需辅助网络或模型架构的修改。具体来说，FrameGRPO结合了两个轻量级组件：（1） \textbf{多帧展开}，仅使用单一最大帧解码的均匀时间子采样，构建对齐视图，丰富视觉证据和推理轨迹;以及（2）\textbf{门控时间效率信用}，当固定全比例参考子集的所有推广失败时，选择性强化正确的低比例轨迹，而不强制对更少帧施加无条件偏好。跨越多种视频理解和推理基准的广泛实验显示，时间视图多样化在视频MLLm中探索和分配信用的有效机制。

Mobile Embodied Agent Network (MEAN): Integrated Framework, Enabling Technologies, and Task-Driven Deployment

移动实体代理网络（MEAN）：集成框架、赋能技术与任务驱动部署

• Authors: Zhaoyang Zhang, Lingxiao Sun, Weijie Zhou, Chunhui Feng, Zhaohui Yang, Xuan Cheng, Tianyi Qin, Zhixian Kong, Baoping Cheng, Zijian Wu, Jingze Che
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.26297
• Pdf link: https://arxiv.org/pdf/2610.26297
• Abstract Mobile networks are evolving from human-centered, data-oriented communication infrastructures toward machine- and/or agent-dominant, task-oriented platforms that are to serve the rapidly increasing networked machines and embodied agents. In order to fulfill the newly arose complex task objectives, the framework of future mobile network is to be innovated to reflect and facilitate the intrinsic interplay and integration of the underlying processes of perception, communication, computation, decision, action, and feedback, etc. This survey establishes the mobile embodied agent network (MEAN) as a task-oriented networking paradigm in which networking and distributed intelligence are jointly organized around closed-loop embodied task execution. We first establish a unified MEAN framework that explains how network resources and intelligent functions jointly support embodied task execution, and extend its performance indicators from conventional communication metrics to embodied task outcomes and cross-layer system effects. With this framework, we analyze three complementary enabling mechanisms that form a cognition-interaction-adaption loop. Large artificial intelligence (AI) models provide high-level cognition, token communication connects this cognition with distributed agents through shared semantic representations, and reinforcement learning closes the loop by adapting system operation to network, task, and physical feedback. We further investigate the bidirectional interplay between MEAN and emerging technologies, showing how they mutually extend system capabilities and reshape task-oriented network design. Finally, we discuss open challenges and future research directions toward the practical realization of MEAN over real joint communication and computation platform.
• 中文摘要 移动网络正从以人为中心、数据导向的通信基础设施，向以机器和/或代理为主导的任务导向平台发展，服务于快速增长的网络化机器和具身代理。为了实现新出现的复杂任务目标，未来移动网络的框架将被创新，以反映并促进感知、通信、计算、决策、行动和反馈等底层过程的内在相互作用与整合。本次调查确立了移动实体代理网络（MEAN）作为一种面向任务的网络范式，其中网络与分布式智能共同围绕闭环具身任务执行联合组织。我们首先建立了统一的MEAN框架，解释网络资源和智能功能如何共同支持具身任务执行，并将其绩效指标从传统通信指标扩展到具身任务结果和跨层系统效应。基于该框架，我们分析了三种互补的赋能机制，构成认知-交互-适应循环。大型人工智能（AI）模型提供高层认知，令牌通信通过共享语义表示将该认知与分布式智能体连接起来，强化学习通过调整系统运行以适应网络、任务和物理反馈闭环。我们还进一步探讨了MEAN与新兴技术之间的双向相互作用，展示了它们如何相互扩展系统能力并重塑面向任务的网络设计。最后，我们讨论了面向MEAN实际实现的挑战及未来研究方向，以超越真实的联合通信与计算平台。

ForagerSim: Evaluating Embodied Map-Based Reasoning in Large-Scale Dynamic Worlds

ForagerSim：评估大规模动态世界中具身映射推理

• Authors: Tomáš Musil, Martin Kotík, Michal Vavrečka, Karla Štěpánová, Sebastian Scherer, Martin Saska
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.26395
• Pdf link: https://arxiv.org/pdf/2610.26395
• Abstract Animals build cognitive maps of vast environments, use them to perform map-based navigation, and might even be reusing the same circuitry used in spatial reasoning for creating and using maps of arbitrary task-spaces. In contrast, current embodied AI systems either rely on hand-engineered mapping pipelines, which break in the presence of unmodeled dynamics and are restricted to metric space, or on learned policies that rarely form map-like representations at all. Developing architectures that could learn to build and use map-like representations as flexibly as animals is, however, difficult because few embodied AI benchmarks actually require map-based reasoning. To address this, we present ForagerSim --- an interdisciplinary research platform and simulation environment focused on embodied tasks that require map-based reasoning, with scenarios ranging from mobile-robot search and rescue to animal foraging in both complex mazes and vast, potentially infinite open areas. ForagerSim is highly modular and allows modifying independent axes of difficulty related to perception (abstract 2D sensing to full RGB or LiDAR), scale (room-scale to multi-km scale), topology (open-fields to complex mazes), dynamics (smoke, debris, sensorimotor failures and others) and tasks (from simple 2D coverage to foraging-inspired well-alternation tasks). We compare reinforcement learning baselines with a simple engineered robotics stack and a human player on 8 example worlds which show how different difficulty axes affect existing methods. Our results show that, when sensing is simplified to 2D, learned policies can perform multi-object search and well-alternation tasks surprisingly well in simple topologies, but they fall short of human performance in larger or more complex-topology environments where map-based baselines perform well, suggesting that map-based reasoning is the bottleneck in these scenarios.
• 中文摘要 动物构建广阔环境的认知地图，利用它们进行基于地图的导航，甚至可能重复使用用于空间推理的电路，用于创建和使用任意任务空间的地图。相比之下，当前的具身人工智能系统要么依赖手工设计的映射流程，这些流程在未建模的动态中会崩溃，且仅限于度量空间;要么依赖学习过的策略，很少形成地图式表示。然而，开发能够像动物一样灵活地构建和使用地图式表示的架构非常困难，因为很少有具身人工智能基准实际上需要基于地图的推理。为此，我们提出了ForagerSim---一个跨学科的研究平台和仿真环境，专注于需要基于地图推理的具身任务，场景涵盖从移动机器人搜救到在复杂迷宫和广阔、潜在无限的开放区域中的动物采集。ForagerSim高度模块化，允许修改与感知（抽象二维感知到全RGB或激光雷达）、尺度（房间尺度到多公里尺度）、拓扑（开阔地块到复杂迷宫）、动力学（烟雾、碎片、感官运动失效等）及任务（从简单的二维覆盖到采食启发的井交替任务）相关的独立难度轴。我们将强化学习基线与简单工程机器人栈和人类玩家在8个示例世界中进行了比较，展示了不同难度轴如何影响现有方法。我们的结果表明，当感知简化为二维时，学习到的策略在简单拓扑中能出人意料地很好地执行多对象搜索和良交替任务，但在更大或更复杂的拓扑环境中，基于地图的基线表现良好，表明基于地图的推理是这些场景中的瓶颈。

Co-Evolving Policy and Reward for Code Generation: A Mechanistic Study of Single-Adapter Drift

代码生成的共演化政策与奖励：单适配器漂移的机制性研究

• Authors: Amarsaikhan Batjargal, Noriko Tomuro
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.26416
• Pdf link: https://arxiv.org/pdf/2610.26416
• Abstract Reinforcement learning with verifiable rewards (RLVR) is particularly well suited to code generation because generated programs can be objectively evaluated through execution. Execution also provides behavioral information that can be reused as additional supervision. The Synergistic Policy and Reward Co-Evolving Framework (SPARK), originally proposed for reasoning tasks, exploits this idea by recycling training rollouts and their evaluation results to train a model not only to generate solutions but also to evaluate and improve them. We investigate whether this policy-reward co-evolution approach transfers effectively to code generation and whether execution-grounded reflection provides an additional source of self-improvement. We compare SPARK-style rollout recycling with execution-only Group Relative Policy Optimization (GRPO) using Qwen2.5-Coder-3B-Instruct. Our results do not support either hypothesis under the configuration tested. Although co-evolution improves training performance, these gains do not generalize to held-out problems. Compared at the same stage of training, rollout recycling produces roughly 34x greater KL divergence from the reference policy than execution-only GRPO. Over training, trained tokens per GRPO sequence fall from 185 to 86 as training pass rate increases, while held-out performance declines or fails to sustain early gains. We characterize these patterns as single-adapter policy drift: the shared trainable parameters increasingly depart from pretrained code-generation behavior without corresponding gains in generalization. The results show that higher training reward alone is not evidence of generalizable code self-improvement and highlight the importance of held-out evaluation and drift diagnostics.
• 中文摘要 带有可验证奖励的强化学习（RLVR）特别适合代码生成，因为生成的程序可以通过执行来客观评估。执行还提供了行为信息，可以作为额外的监督再利用。协同策略与奖励共演化框架（SPARK）最初为推理任务提出，利用了这一理念，通过回收训练推广及其评估结果来训练模型，不仅用于生成解决方案，还用于评估和改进。我们研究了这种策略-奖励共进化方法是否能有效转化为代码生成，以及基于执行的反思是否提供了额外的自我改进来源。我们比较了SPARK式的推广回收与仅执行的组相对策略优化（GRPO），使用Qwen2.5-Coder-3B-Instruct。我们的结果在测试的配置下均不支持这两种假设。尽管协进提升了训练性能，但这些收益并不能推广到未解决的问题。与同一训练阶段相比，推广回收产生的与参考策略的 KL 偏差大约是仅执行 GRPO 的 34 倍。在训练后，每个 GRPO 序列的训练令牌数从 185 降至 86，随着训练通过率的提升，而长期执行性能下降或未能维持早期收益。我们将这些模式描述为单适配器策略漂移：共享的可训练参数越来越偏离预训练代码生成行为，而在泛化上却没有相应的提升。结果表明，仅有较高的训练奖励并不代表代码可推广自我改进，并凸显了延迟评估和漂移诊断的重要性。

HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL

HarnessRL：与开放式强化学习的实时培训框架共同演进政策

• Authors: Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu, Pinyi Zhang, Caixin Kang, Ling Li, Bingxue Qiu, Jieping Ye, Yongbin Li, Bang Liu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.26426
• Pdf link: https://arxiv.org/pdf/2610.26426
• Abstract Rubric-based reinforcement learning (RL) trains policies for open-ended generation with natural-language criteria as rewards, but these rewards can become ineffective as training progresses. We identify two failure modes. Under non-discriminative rewards, scores collapse to uniformly low or high values due to insufficient exploration or a mismatch between task difficulty and policy capability. Under spurious rewards, higher scores do not correspond to better responses. Both reflect a training configuration whose task sampling, exploration, and evaluation stay fixed as the policy evolves. We introduce HarnessRL, a training-time harness that co-evolves with the policy through three interfaces for task sampling, rollout guidance, and rubric criteria. HarnessRL maintains a training-time memory from which a harness evolver retrieves within-task history and cross-task evidence. Using this evidence, the evolver attributes each reward failure to the interface whose update can address it. It then runs a proposer and just-in-time critic loop in which the critic accepts a candidate update only after measuring its effect on the current frozen policy, returning rejected candidates for revision. We also curate Rubric-Mix-10k, a five-domain training and evaluation suite for open-ended RL across medical, science, writing, role-play, and instruction-following tasks. On Rubric-Mix-10k, HarnessRL outperforms the strongest baselines by 2.8 to 3.1 points across both 4B and 9B model scales and nearly halves the training steps to its best checkpoint. Code and data are available at this https URL.
• 中文摘要 基于评分标准的强化学习（RL）训练开放式生成策略，并以自然语言标准作为奖励，但随着训练进展，这些奖励可能变得无效。我们识别出两种失败模式。在非歧视性奖励下，由于探索不足或任务难度与策略能力不匹配，分数会崩溃为均匀的低或高。在虚假奖励下，较高的分数并不代表更优的响应。这两种方式都反映了训练配置，其任务抽样、探索和评估在策略演进过程中保持固定。我们引入了HarnessRL，一种训练时间框架，通过三个任务抽样、推广指导和评分标准的接口与策略共同演化。HarnessRL维护一个训练时间内存，驱动驱动者从中检索任务内历史和跨任务证据。利用这些证据，演化器将每次奖励失败归因于能够解决该问题的更新接口。然后运行提议者和即时批评者循环，批评者仅在测量候选更新对当前冻结策略的影响后接受，并返回被拒绝的候选者进行修订。我们还策划了Rubric-Mix-10k，这是一个涵盖医疗、科学、写作、角色扮演和指令执行任务的开放式强化学习五域培训与评估套件。在Rubric-Mix-10k上，HarnessRL在4B和9B模型尺度上均优于最强基线2.8至3.1分，且训练步骤几乎减半至最佳检查点。代码和数据可在此 https 网址获取。

E-Layout: Layout-Stage Logic Restructuring via RL-Optimized Equality-Saturated Graph Retrieval-Augmented Generation

E-布局：通过强化学习优化的等式饱和图检索增强生成实现布局阶段逻辑重构

• Authors: Wenji Fang, Rongjian Liang, Yi-Chen Lu, Cunxi Yu, Chen Chen, Shang Liu, Chenhui Deng, Wenkai Li, Haoxing Ren, Zhiyao Xie
• Subjects: Subjects: Hardware Architecture (cs.AR)
• Arxiv link: https://arxiv.org/abs/2610.26432
• Pdf link: https://arxiv.org/pdf/2610.26432
• Abstract Achieving high sign-off performance in advanced ICs increasingly depends on layout-stage optimization under realistic physical conditions. While recent efforts have moved toward synthesis-layout fusion, most existing works still focus on bringing estimated physical information earlier into synthesis, whose optimization does not fully correlate with real layout-stage behavior. Meanwhile, layout-stage optimization remains largely restricted to a fixed logic structure, relying mainly on buffering and sizing. This fundamentally constrains the optimization space under realistic physical conditions. To address this, we present E-Layout, an ML-driven framework for post-placement layout restructuring. E-Layout jointly optimizes logic transformation and physical implementation across multiple subcircuits under design-level timing feedback. This enables the selection of the most timing-beneficial set of disjoint restructured subcircuits to form the optimized circuit. Specifically, it first constructs equivalence-preserving logic alternatives via e-graph equality saturation, then couples logic selection with physical realization through a graph-based retrieval-augmented generation (RAG) model, and finally refines these local logic-physical decisions with reinforcement learning under design-level timing feedback. Evaluated on 9 diverse designs, it achieves on average 6% TNS improvement and 2% WNS improvement, while even reducing area by 1%, over an industry-leading commercial physical design flow.
• 中文摘要 在先进集成电路中实现高签核性能，越来越依赖于在现实物理条件下的布局阶段优化。尽管近期努力趋向合成-布局融合，但大多数现有工作仍侧重于将估计的物理信息更早引入合成阶段，其优化与真实布局阶段行为并不完全相关。与此同时，布局阶段优化仍主要局限于固定逻辑结构，主要依赖缓冲和尺寸调整。这在现实物理条件下根本限制了优化空间。为此，我们提出了E-Layout，一个机器学习驱动的布局后重构框架。E-Layout在设计级时序反馈下，联合优化多个子电路的逻辑变换和物理实现。这使得能够选择最有利时序的不相交重构子电路集合，形成优化电路。具体来说，它首先通过电子图等价饱和构建等价保持逻辑的替代方案，然后通过基于图的检索增强生成（RAG）模型将逻辑选择与物理实现结合，最后在设计级时序反馈下通过强化学习细化这些局部逻辑-物理决策。在9种不同设计的评估中，平均实现了6%的TNS改进和2%的WNS提升，甚至比行业领先的商业物理设计流程减少了1%。

VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards

VeriCUA：学习主动验证以实现可靠GUI代理奖励

• Authors: Zhizhou Wang, Yaxi Lu, Zhong Zhang, Zikun Hu, Xin Cong, Yankai Lin, Fangming Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.26469
• Pdf link: https://arxiv.org/pdf/2610.26469
• Abstract Reinforcement learning for computer-use agents increasingly relies on proactive reward agents that interact with the post-execution GUI environment to verify what recorded screenshots cannot reveal. The reliability of such verification hinges on two coupled decisions: what to check and when to stop. Existing methods leave these decisions to prompted foundation models. However, neither scaling up the model nor providing few-shot examples reliably improves verification: prompted Judges ranging from 4B to GPT-5.4 all plateau at 68%-70% macro-average agreement. This is because the value of a check depends on the environment rather than the task alone, and becomes apparent only through execution. Learning from execution is itself constrained, as the Actor's terminal state cannot be faithfully replicated during training, allowing only one verification attempt per state. We introduce VeriCUA, a proactive reward agent that learns its verification strategy from these single attempts. A trainable Judge decides what to check and when to stop, while a frozen Probe executes the checks in the live environment. Online reinforcement learning optimizes the Judge for both verdict correctness and efficient Probe use, with returns normalized across terminal states. Trained only on OSWorld and evaluated without further training on MobileWorld and WebArena-Verified, VeriCUA achieves the highest reward agreement across all three benchmarks and lies on the agreement-cost Pareto frontier among proactive verifiers. It surpasses few-shot GPT-5.4 by 5.36 points with 47.6%-59.8% fewer Probe tokens, and even a 1B VeriCUA exceeds every prompted Judge. As the reward for Actor RL, it raises task success by 3.21 points overall and 5.40 points on held-out multi-application tasks. These results identify the verification strategy, not verifier scale, as the key to reliable GUI rewards.
• 中文摘要 计算机智能体的强化学习越来越依赖主动的奖励代理，这些代理与执行后的图形界面环境交互，以验证录制截图无法揭示的内容。此类验证的可靠性取决于两个耦合决策：检查什么和何时停止。现有方法将这些决策交给提示式基础模型。然而，无论是放大模型还是提供少数样本样本，都无法可靠地提升验证：从4B到GPT-5.4的提示评判均稳定在68%-70%的宏观平均一致度。这是因为检验的价值取决于环境而非任务本身，只有通过执行才能显现。从执行中学习本身受限，因为在训练过程中无法忠实复制演员的终端状态，每个状态只能进行一次验证尝试。我们引入VeriCUA，一种主动奖励代理，通过这些单次尝试学习验证策略。可训练的评判者决定检查内容和何时停止，而冻结的探针则在实时环境中执行检查。在线强化学习优化了评判者的判决正确性和探针高效使用，且在终端状态间实现归一化。VeriCUA仅在OSWorld上训练，且未在MobileWorld和WebArena-Verified上进行进一步培训，在三大基准测试中获得最高的奖励协议，处于主动验证者中协议成本帕累托边界。它比少射GPT-5.4高出5.36分，探针代币减少47.6%-59.8%，甚至1B的VeriCUA也超过所有被提示的评判。作为演员RL的奖励，它整体提升任务成功率3.21分，在未完成的多应用任务上提升5.40分。这些结果表明，验证策略而非验证者规模才是获得可靠GUI奖励的关键。

SPAR: Structural Pruning and Action Recovery for Vision-Language-Action Models

SPAR：视觉-语言-动作模型的结构修剪与动作恢复

• Authors: Shenli Zheng, Zhou Fang, Yuan Du, Yijiang Liu, Li Du
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26487
• Pdf link: https://arxiv.org/pdf/2610.26487
• Abstract The limited onboard computing resources of robots make the inference demands of vision-language-action (VLA) models costly to accommodate. Reducing these demands through aggressive compression, however, can degrade manipulation performance beyond what supervised fine-tuning alone recovers. We present SPAR (Structural Pruning and Action Recovery), which simultaneously prunes the hidden and feed-forward intermediate dimensions of the large language model (LLM) backbone through pruning-aware tuning, then applies reinforcement learning (RL) to recover policy performance. Applied to $\pi_{0.5}$, SPAR reduces the language backbone from 2.34B to 0.09B parameters and the full model from 3.37B to 1.12B, removing approximately 96% and 67% of their parameters, respectively. At this compression level, RL raises average success across the four LIBERO suites from 84.55% to 93.45%, an 8.90-percentage-point gain. The compressed policy achieves a $1.98\times$ full-policy inference speedup over the unpruned model. These results identify substantial intermediate-dimension redundancy and show that post-pruning RL can recover manipulation performance in compact policies.
• 中文摘要 机器人有限的机载计算资源使得视觉-语言-动作（VLA）模型的推理需求难以满足。然而，通过激进压缩降低这些需求，可能会使操作性能下降，超出仅靠监督微调所能恢复的水平。我们介绍SPAR（结构剪枝与动作恢复），它通过修剪感知调优同时修剪大型语言模型（LLM）骨干网的隐藏和前馈中间维度，然后应用强化学习（RL）恢复策略性能。应用于$\pi_{0.5}$，SPAR将语言骨干参数从2.34B降至0.09B，将完整模型从3.37B降至1.12B，分别去除约96%和67%的参数。在这一压缩水平下，RL将四个LIBERO套件的平均成功率从84.55%提升至93.45%，提升8.90个百分点。压缩策略相比未修剪模型实现了1.98\乘以的全策略推断加速。这些结果识别了显著的中间维冗余，并表明修剪后RL能够恢复紧凑策略中的操作性能。

Mitigating Reward Hacking in Preference Reward Models with Evolving Penalties

在偏好奖励模型中缓解奖励黑客行为，伴随不断演变的惩罚

• Authors: Afonso Baldo, António Farinhas, Marcos Treviso, Ricardo Rei, Nuno M. Guerreiro
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26496
• Pdf link: https://arxiv.org/pdf/2610.26496
• Abstract Reinforcement learning for language models has moved from preference reward models to verifiable rewards, which sustain much longer optimization. Preference models are prone to reward hacking: the policy raises their score without improving response quality, or even while degrading it. Yet for open-ended tasks, where quality can neither be verified nor fully written down as rubrics, they remain valuable. We introduce Dynalize, which mitigates reward hacking online, during policy training. Periodically and asynchronously, a language model analyzes training rollouts and writes penalties for undesirable behaviors that the reward model appears to reward; a judge applies them to every rollout. In clinical counseling and creative writing, training on the reward model alone raises its score continuously but ultimately leads to a decline in external benchmarks. Dynalize sustains longer training and results in improved downstream performance. For example, we find that our method suppresses patterns incentivized by preference models that, without penalties, spread to most responses, and even to the model's reasoning, which the reward model does not explicitly score. Furthermore, we show that its gains hold in multi-turn training, and even after we retrain the reward model on the discovered behaviors.
• 中文摘要 语言模型的强化学习已从偏好奖励模型转向可验证奖励模型，后者优化时间更长。偏好模型容易出现奖励黑客：策略提高了评分却不提升响应质量，甚至在降低响应质量的同时。然而对于开放式任务，质量既无法验证也无法完整书写为评分标准，它们依然有价值。我们在政策培训中引入了Dynalize，它能缓解在线奖励黑客行为。语言模型定期且异步地分析培训推广，并对奖励模型似乎奖励的不良行为写入惩罚;评委对每次推广应用惩罚。在临床咨询和创意写作中，仅基于奖励模型的训练会持续提升评分，但最终导致外部基准下降。Dynalize能持续更长时间的训练，并提升下游表现。例如，我们发现我们的方法抑制了由偏好模型激励的模式，这些模式在无惩罚的情况下会扩散到大多数反应，甚至扩展到模型的推理，而奖励模型并未明确评分这些。此外，我们证明其收益在多回合训练中依然成立，即使我们根据发现的行为重新训练奖励模型。

Training Natural Schemers: RL Can Amplify Oversight-Conditional Behavior

培养天生策划者：强化学习可以放大监督条件行为

• Authors: Bart Jaworski, Nathaniel Mitrani, Fred Bruford, Victoria Krakovna
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.26551
• Pdf link: https://arxiv.org/pdf/2610.26551
• Abstract One of the central concerns for the safety of advanced AI is scheming: when a model strategically complies under oversight but pursues its own goals when unobserved. Current demonstrations, such as Anthropic's Alignment Faking work, depend on design choices that make them less convincing: explicit test-time prompting, handcrafted backdoors, or training data that demonstrates or hints at the target behavior. Model organisms built in this way cannot tell us whether scheming tendencies would arise or increase under realistic training conditions. We therefore construct a natural model organism of scheming, built on Kimi K2.6: trained with traits that could plausibly emerge in a frontier training pipeline, and without providing the model with the target scheming policy. The resulting model is a benign schemer: it aims to maximize aggregate help across all users over time, values its own continued operation only as a means to future help, and knows the consequences of oversight and deployment. We then train it using Reinforcement Learning (RL) with a reward that scores only task quality and policy compliance and does not reward scheming; we show that this amplifies scheming. We measure the oversight flip (the gap between monitored and unmonitored compliance with a given policy) and find that it increases during RL training across our environments. Chain-of-Thought (CoT) analysis finds explicit instrumental reasoning about oversight accompanying this behavior. The effect holds across our three RL environments, and we find that the scheming policy transfers between them. The pipeline also replicates on other open-weight base models. Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.
• 中文摘要 先进人工智能安全性的核心关注之一是策划：当模型在监督下战略性地配合，但在未被观察时却追求自身目标。当前的演示，如Anthropic的Alignment Faking工作，依赖于使其不那么令人信服的设计选择：显式测试提示、手工设计的后门，或展示或暗示目标行为的训练数据。以这种方式构建的模型生物无法告诉我们在现实训练条件下阴谋倾向是否会出现或增加。因此，我们构建了一个基于Kimi K2.6的自然策划模型生物：训练时具备可能在前沿训练流程中可能出现的特征，且不向模型提供目标策略。最终模型是一个良性方案：它旨在随着时间最大化所有用户的总和帮助，仅将自身持续运行视为未来帮助的手段，并了解监督和部署的后果。随后，我们使用强化学习（RL）进行训练，奖励仅评分任务质量和策略合规性，不奖励策划;我们证明这会放大策划行为。我们测量监督翻转（监控与未监控遵守政策之间的差距），发现在强化学习训练过程中，该差距在我们环境中增加。思维链（CoT）分析发现，这种行为伴随着明显的工具性推理。这种效应在我们的三个强化学习环境中都存在，且策划策略在它们之间会转移。该流水线也在其他开放权重基模型中复制。我们的结果表明，当监督条件政策与训练期间的对齐政策无异时，不奖励阴谋的奖励本身无法对某些特质提供固有保护。

BetaMem: Learning to Remember a Changing World for Embodied Agents

BetaMem：具身代理人学习记忆变化的世界

• Authors: Mai Liu, He Sun, Siru Zhong, Kerun Chen, Hao Yan, Xiaohui Lv, Shuai Tao, Wulong Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26554
• Pdf link: https://arxiv.org/pdf/2610.26554
• Abstract Long-horizon embodied agents face temporal confusion and incomplete or distorted records, recurring errors from fixed memory rules, and growing storage and retrieval costs. We present BetaMem, a framework for building and evolving traceable long-term memory through six linked layers: working, spatial, entity, episodic, experiential, and procedural memory. A dual-level write contract constrains operation executability and source support for content: supervised fine-tuning learns compliant memory operations, while reinforcement learning combines repeated sampling with teacher verification of the original observations to reward complete, accurate records and penalize omissions, distortion, and unsupported content. We formulate failure-driven policy recursive self-improvement through diagnosis along the causal sequence of memory operations and local rule repairs gated by independent replay. Atomic tools frame storage, updates, retrieval, and forgetting as cost-utility allocation under storage and read budgets. Using the open-source Qwen3.5-9B model for memory generation, BetaMem achieves overall scores of 48.08% on M3-Bench-robot, 92.08% on LoCoMo, 72.6% on OpenEQA EM-EQA, and 62.4% on EgoLifeQA, the highest among the listed memory methods under their respective evaluation protocols. Training the model for memory generation raises the contract pass rate from 50.9% to 86.2% while reducing output tokens per stored action by 47.3%. A separate Qwen3-VL-8B-Instruct evaluation on EMem-Bench achieves an average success rate of 38.4% and an average error recurrence rate of 24.9%. We perform best among evaluated methods using comparably sized open-source models. Category results show that we have advantages in integrating information across observations, recalling events over long histories, and inferring personal habits.
• 中文摘要 长视野具身智能体面临时间混乱和不完整或扭曲记录、固定记忆规则导致的反复错误，以及不断增长的存储和检索成本。我们提出了BetaMem，这是一个通过六个关联层构建和演进可追溯长期记忆的框架：工作记忆、空间记忆、实体记忆、情景记忆、体验记忆和程序记忆。双层写合同限制了操作可执行性和内容源支持：监督微调学习符合的记忆操作，强化学习结合反复采样与教师对原始观察的验证，奖励完整准确的记录，惩罚遗漏、扭曲和无支持内容。我们通过沿记忆操作因果序列的诊断和通过独立重放限制的局部规则修复，制定失败驱动的策略递归自我改进。Atomic Tools 框架存储、更新、检索和遗忘作为存储和读取预算中的成本效用分配。采用开源的 Qwen3.5-9B 内存生成模型，BetaMem 在 M3-Bench-robot 上取得了 48.08%，LoCoMo 92.08%，OpenEQA EM-EQA 72.6%，EgoLifeQA 62.4%，是各自评估协议中列出的内存方法中最高。训练内存生成模型可将合同通过率从 50.9% 提升至 86.2%，同时每个存储动作输出令牌减少 47.3%。EMem-Bench 上的 Qwen3-VL-8B-Instruct 评估平均成功率为 38.4%，平均错误复发率为 24.9%。我们在使用同等规模开源模型的评估方法中表现最佳。类别结果显示，我们在整合跨观察信息、回忆长期历史事件以及推断个人习惯方面具有优势。

VaDar: Implicit Goal Conditioning via Value Disaggregation

VaDar：通过价值分解实现隐性目标条件

• Authors: Shashwat Saxena, Mehul Goel, Sreyas Venkataraman, Sarvesh Patil, Max Simchowitz
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.26629
• Pdf link: https://arxiv.org/pdf/2610.26629
• Abstract Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into subgoals, and learn a subgoal-conditioned policy to execute each subtask in sequence. We introduce an alternative called Value Disaggregation (VaDar), which trains a single subgoal-independent policy to maximize a linear combination of per-subgoal critic values. Because subgoal information is needed only at training-time, VaDar removes the need for possibly costly or cumbersome subgoal-selection during policy deployment. Moreover, through careful experiments, we show that VaDar (1) is always competitive with, and frequently outperforms, natural baselines like goal-conditioning and (2) succeeds on tasks with non-sequential goal structure and multiple optimization objectives. In particular, VaDar is the first algorithm to solve challenging, long-horizon tasks in the RoboCasa suite, even when starting from a policy pretrained from behavior cloning with near-zero initial task success. Taken together, VaDar's successes suggest that subgoal information benefits actor-critic reinforcement learning by enhancing the efficacy of critic learning, whereas policy goal-conditioning is often unnecessary.
• 中文摘要 长视野任务对现代人工智能和机器人技术很重要，但对强化学习也构成挑战。历史上，一种流行的方法是将任务分解为子目标，学习一个子目标条件策略以顺序执行每个子任务。我们引入了一种叫做价值分解（VaDar）的替代方案，它训练一个单一子目标无关策略，以最大化每个子目标批判值的线性组合。由于子目标信息仅在训练时需要，VaDar消除了在策略部署时可能昂贵或繁琐的子目标选择需求。此外，通过细致实验，我们证明VaDar（1）始终能够与自然基线如目标条件（如目标条件）竞争，且常常表现优于它们，（2）在非顺序目标结构和多优化目标的任务中取得成功。特别是，VaDar 是 RoboCasa 套件中首个解决具有挑战性的长期任务的算法，即使从从行为克隆预训练的策略开始，且初始任务几乎没有成功率。综合来看，VaDar 的成功表明子目标信息通过增强行为者-批判者强化学习的有效性，有利于行为者-批评者强化学习，而策略目标条件化往往是不必要的。

PACT: Plan-Augmented Coordination via Trace-Graphs for Dynamic Multi-Agent Combinatorial Optimization

PACT：通过迹图进行计划增强协调，用于动态多代理组合优化

• Authors: Xiao Zhou, Alex Schutz, Victor-Alexandru Darvariu, Nick Hawes
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.26671
• Pdf link: https://arxiv.org/pdf/2610.26671
• Abstract Multi-agent approaches have been investigated recently as a means of improving the scalability of reinforcement learning for neural combinatorial optimization. Existing methods assume that all problem instance information is known upfront. However, real-world problems are often dynamic: relevant information is progressively revealed as the agents take actions (e.g., arriving at an edge in a routing problem only to discover it is untraversable). In such cases, existing methods fail to coordinate agents to explore and exploit the progressively revealed information. To address the arising coordination challenge, we introduce PACT (Plan-Augmented Coordination via Trace-graphs). PACT augments the environment graph with auxiliary nodes representing the plans of agents, which are connected to the resource nodes to indicate future dependencies. The proposed message-passing scheme over the augmented graph grounds inter-agent coordination in the environment and enables proactive, rather than reactive, conflict resolution. We show that PACT improves solution quality in dynamic problems relative to baselines and exhibits the best transfer to larger instances on which, remarkably, it outperforms centralized oracles with privileged access to instance uncertainty.
• 中文摘要 多智能体方法最近被研究为提升神经组合优化强化学习可扩展性的手段。现有方法假设所有问题实例信息都是事先已知的。然而，现实问题往往是动态的：随着智能体采取行动（例如，在路由问题中到达边缘却发现无法穿越）逐渐揭示相关信息。在这种情况下，现有方法未能协调智能体探索和利用逐步揭示的信息。为应对出现的协调挑战，我们引入了PACT（通过痕迹图进行计划增强协调）。PACT通过辅助节点来增强环境图，这些节点代表代理的计划，这些节点与资源节点连接，以指示未来的依赖关系。基于增强图的消息传递方案在环境中实现了代理间协调，并实现了主动而非被动的冲突解决。我们证明，PACT相较于基线提升了动态问题中的解质量，并且在更大实例中表现出最佳的转移能力，值得注意的是，它在该实例上优于拥有特权访问实例不确定性的集中式预言机。

Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing

评分标准内奖励发散：通过潜在维度恢复与平衡缓解评分标准黑客问题

• Authors: Jianqiang Xia, Han Xu, Dianxi Shi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.26685
• Pdf link: https://arxiv.org/pdf/2610.26685
• Abstract Rubric hacking limits both performance and training robustness in rubric-based reinforcement learning. By decomposing rubrics into sub-dimensions, we reveal the Intra-Rubric Reward Divergence: as the total rubric score increases, certain sub-dimension scores systematically decrease, suggesting the model sacrifices specific quality aspects to optimize the aggregated objective. From the sub-dimension perspective, we observe two phenomena: (1) dimension imbalance, causing the policy to favor heavily covered dimensions; (2) dimension absence, leading to a systematic gap between rewards and true quality. To mitigate these issues, we propose Fair-Race Dimension Aggregation (FRDA), which has generations compete within each dimension separately to mitigate rubric imbalance, and Residual Semantic Dimension Completion (RSDC), which mines evaluation dimensions implicitly encoded in a holistic preference signal to complete rubric coverage. Experiments in chat, science, and medical domains indicate that FRDA mitigates rubric hacking with more stable improvements, while RSDC outperforms both single-reward training and naive multi-reward aggregation, yielding generally higher-quality responses.
• 中文摘要 评分标准修改限制了基于评分标准的强化学习中的表现和训练的稳健性。通过将评分标准分解为子维度，我们揭示了评分标准内奖励偏差：随着总评分的增加，某些子维度得分系统性下降，表明模型为了优化聚合目标而牺牲了特定质量方面。从亚维度视角，我们观察到两种现象：（1）维度不平衡，导致政策偏向覆盖较多的维度;（2）维度缺失，导致奖励与真实质量之间存在系统性差距。为缓解这些问题，我们提出了公平种族维度聚合（FRDA），即在每个维度内各代分别竞争以缓解评分标准不平衡;以及剩余语义维度补全（RSDC），通过挖掘隐含编码于整体偏好信号中的评估维度，以完成评分标准覆盖。聊天、科学和医学领域的实验表明，FRDA通过更稳定的改进来缓解评分评分修改，而RSDC则优于单一奖励训练和朴素多奖励聚合，通常能获得更高质量的响应。

CheckDistill: Adaptive Checklist Distillation for Efficient Distributional Reward Modeling in Image Editing

CheckDistill：自适应检查表蒸馏，用于图像编辑中高效的分布奖励建模

• Authors: Yaqi Cai, Zuan Gao, Mengting Chen, Wei Li, Bo Zheng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.26740
• Pdf link: https://arxiv.org/pdf/2610.26740
• Abstract Reinforcement learning for instruction-guided image editing requires reward models that provide reliable, accurate, and efficient feedback. Direct scalar prediction compresses multidimensional editing preferences, whereas generative evaluators can expose their judgment through explicit reasoning. However, broad scoring criteria do not specify all instance-dependent editing and preservation constraints, and generating long reasoning trajectories is costly for online optimization. We introduce CheckDistill, a pointwise reward modeling framework that internalizes instruction-adaptive checklists. We decompose each editing condition into atomic, independently verifiable requirements and construct four-dimensional scoring labels together with structured verification evidence. After supervised score initialization, a privileged teacher conditions on checklists, item-level judgments, and dimension-level rationales to supervise the student's own scoring trajectories. A reliability-weighted on-policy self-distillation objective adjusts supervision at score tokens according to agreement between the teacher's expected scores and reference labels, while retaining candidate probability weights in a truncated token support. The deployed student directly predicts four score distributions without checklist inputs or explicit reasoning and aggregates their expectations into a scalar reward. CheckDistill achieves aggregate accuracies of 72.30%, 56.67% and 82.90% on MMRB2, MER-Bench and EditReward-Bench, respectively, exceeding the compared open-source baselines on all three metrics.
• 中文摘要 用于指令引导图像编辑的强化学习需要提供可靠、准确且高效的反馈的奖励模型。直接标量预测压缩多维编辑偏好，而生成式评估者则可以通过显式推理揭示其判断。然而，宽广的评分标准并未规定所有实例依赖的编辑和保存约束，且生成长推理轨迹对在线优化成本较高。我们介绍了CheckDistill，一种逐点奖励建模框架，内化了指令自适应检查表。我们将每个编辑条件分解为原子级、独立可验证的需求，并构建四维评分标签及结构化验证证据。在监督评分初始化后，特权教师会以检查表、题目级判断和维度级理由为条件，以监督学生自身的评分轨迹。基于可靠性加权的政策自蒸馏目标根据教师期望分数与参考标签的一致性调整评分代币的督导，同时保留候选概率权重于截断代币支持。部署的学生无需检查表输入或明确推理即可直接预测四个得分分布，并将预期汇总为标量奖励。CheckDistill在MMRB2、MER-Bench和EditReward-Bench上的总准确率分别为72.30%、56.67%和82.90%，在三个指标上均超过了开源基线。

Simulation Priors for Rapid Adaptation in Model-Based RL for Autonomous Racing

基于模型的强化学习自动驾驶快速适应的模拟先验

• Authors: Adrian Remonda, Jiajun Xi, Nicklas Hansen, Marcus Greiff, Thomas Lew, John Talbot, John Subosits, Xiaolong Wang
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.26757
• Pdf link: https://arxiv.org/pdf/2610.26757
• Abstract Reinforcement learning enables learning high-performance control policies, but its reliance on extensive interaction limits deployment on physical systems where data collection is costly, and failures are unsafe. We introduce Prior-Informed Adaptation (PIA), an offline-to-online model-based RL framework that uses structured simulation priors for rapid adaptation. PIA pretrains a world model on heterogeneous low- and high-fidelity simulated trajectories, then adapts it with limited target-domain interaction using uncertainty-regularized planning. Rather than requiring an accurate simulator for policy transfer, PIA treats simulated trajectories as biased priors whose residual mismatch is corrected during online adaptation. We evaluate PIA in autonomous racing using high-fidelity simulation and on a real full-size autonomous race car. In simulation, PIA adapts faster across track and surface shifts than strong baselines, achieving lower lap times and fewer failures. On hardware, PIA achieves reliable few-shot adaptation with less than one hour of on-vehicle interaction.
• 中文摘要 强化学习使得学习高性能控制策略成为可能，但其对广泛交互的依赖限制了在数据收集成本高且故障不安全的物理系统上的部署。我们引入了先验知情适应（PIA），这是一种离线到在线基于模型的强化学习框架，利用结构化仿真先验实现快速适应。PIA对世界模型进行异构低保真和高保真模拟轨迹的预训练，然后利用不确定性正则化规划，在目标域交互有限的情况下进行调整。PIA将模拟轨迹视为有偏置的先验，在线适应时纠正其残余不匹配，而非需要精确模拟策略传输。我们利用高精度模拟和真实全尺寸自动驾驶赛车评估PIA在自动驾驶赛车中的应用。在模拟中，PIA在赛道和路面换挡间的适应速度比强基线更快，实现更低的圈速和更少的故障。在硬件上，PIA实现了可靠的少量试探适应，且车辆上互动时间少于一小时。

Tactile In-Hand Object Rotation: A Comparative Study of Hand Morphology in Simulation and Hardware

触觉手中物体旋转：模拟与硬件中手形态学的比较研究

• Authors: Wenhao He, Max Yang, Chenghua Lu, Yijiong Lin, David A.W. Barton, Nathan F. Lepora
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.26759
• Pdf link: https://arxiv.org/pdf/2610.26759
• Abstract Continuous in-hand rotation requires coordinated contact maintenance and renewal, linking manipulation performance to both hand geometry and tactile feedback. We investigate these relationships through rotational progress, finger participation, and fingertip loading on two versions of the low-cost LEAP Hand with anthropomorphic and symmetric finger arrangements. Both hands are integrated with TacTip vision-based tactile sensors and use a shared reinforcement learning framework. Simulation evaluates full tactile, binary touch, and proprioception-only observations on 12 unseen objects; hardware experiments evaluate full tactile policies on ten objects under multiple hand orientations. In simulation, full tactile observations increase mean normalized rotation per episode over proprioception alone by 94.8% for the anthropomorphic hand and 66.9% for the symmetric hand, with longer mean episode durations and lower mean total fingertip loads. Hardware contact measurements reveal more differentiated finger participation in the anthropomorphic hand, while cumulative loading is more evenly distributed across the symmetric hand's fingers. Overall, the symmetric hand achieves a 32.0% higher mean rotation count and a 25.8% higher aggregate rotation rate, together with longer manipulation duration. Additional hardware force recordings show a 19.9% lower mean total fingertip load for this layout. These findings relate finger arrangement to contact organization and load sharing, and show that richer tactile information supports greater task progress at lower mean loading in both simulated embodiments.
• 中文摘要 持续的手部旋转需要协调的接触维护和更新，将操作性能与手部几何结构和触觉反馈相结合。我们通过旋转进展、手指参与和指尖加载，研究了两种低成本LEAP手版本的拟人化和对称手指排列。双手集成了基于TacTip视觉的触觉传感器，并使用共享强化学习框架。模拟评估12个未被看见物体的完整触觉、二元触觉和本体感觉观察;硬件实验评估多手方向下十个物体的完整触觉策略。在模拟中，完整触觉观察使拟人手每发作平均归一化旋转提升94.8%，对称手提升66.9%，平均发作持续时间更长，指尖总负荷更低。硬件接触测量显示拟人化手中手指参与更为细致，而累计负荷在对称手的手指上分布更均匀。总体而言，对称手的平均旋转次数提高了32.0%，总旋转速率提高了25.8%，同时操作时间更长。额外的硬件力记录显示，该布局的平均指尖总负荷降低了19.9%。这些发现将手指排列与接触组织和负荷分担联系起来，表明更丰富的触觉信息在两个模拟体中平均负荷较低下支持更快的任务进展。

Next Best Action: Causal Constrained Offline RL for Business Action Targeting

下一个最佳行动：因果约束离线强化学习，用于业务行动定向

• Authors: Surya Ramachandiran, Yiming Li, Juan Mancilla-Caceres, Tao Ye, Jun Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.26817
• Pdf link: https://arxiv.org/pdf/2610.26817
• Abstract Next Best Action (NBA) systems must select among customer interventions (e.g., discounts, promotions, and lifecycle actions), maximize long-term economic value, satisfy complex operational constraints, and remain causally grounded. Existing approaches lack a rigorous framework for meeting these requirements jointly. We address this gap by formulating NBA as a Constrained Markov Decision Process (CMDP). We then integrate off-policy causal correction of the estimated customer dynamics directly into the Linear Programming (LP) formulation of the CMDP. The resulting policy selects actions based on customer states to maximize long-term value while explicitly enforcing business guardrails. Furthermore, the LP formulation yields shadow prices that quantify the economic trade-offs of operational constraints (e.g., fairness and user welfare), enhancing policy interpretability. We validate our proposed offline Reinforcement Learning (RL) framework on an industry case study involving rideshare coupon targeting, where customer states are defined by Recency, Frequency, and Monetary (RFM) stages. We evaluate performance using model-based off-policy evaluation, with causal dynamics estimated from a temporal holdout. Under the observed coupon budget and RFM state representation, our optimized policy improves customer lifetime value (LTV) by 19% over the estimated behavior policy and 8% over a myopic baseline. Although the demonstrated action space is binary, the proposed framework is action-agnostic, yielding a general-purpose decision module for targeting systems that manage multiple interventions.
• 中文摘要 Next Best Action（NBA）系统必须在客户干预（如折扣、促销和生命周期行动）中做出选择，最大化长期经济价值，满足复杂的运营约束，并保持因果基础。现有方法缺乏严格的框架来共同满足这些要求。我们通过将NBA表述为受限马尔可夫决策过程（CMDP）来弥补这一空白。随后，我们将估计的客户动态的非策略因果修正直接整合进CMDP的线性规划（LP）表述中。最终的策略基于客户状态选择行动，以最大化长期价值，同时明确执行业务防护措施。此外，LP表述产生了影子价格，量化运营约束的经济权衡（如公平性和用户福利），增强了政策的可解释性。我们在涉及网约票优惠券定向的行业案例研究中验证了我们提出的离线强化学习（RL）框架，该案例中客户状态由近期性、频率和货币（RFM）阶段定义。我们通过基于模型的非政策评估评估性能，并从时间保留中估算因果动态。在观察到的优惠券预算和RFM状态表示下，我们的优化策略使客户终身价值（LTV）比估计行为政策提升19%，在近视基线上提升8%。尽管所展示的行动空间是二元的，但该框架是行动无关的，提供了一个用于管理多重干预的目标定位系统的通用决策模块。

Train An Imaginative Agent

训练一个富有想象力的特工

• Authors: Jiaxin Ge, Junyi Zhang, Tianning Chai, Hanjun Yoo, Chenyang Wang, Xiuyu Li, Weijia Shi, XuDong Wang, Sewon Min, Luke Zettlemoyer, Trevor Darrell, Haiwen Feng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26845
• Pdf link: https://arxiv.org/pdf/2610.26845
• Abstract A seemingly reasonable push in Sokoban can make the puzzle unsolvable. To avoid such mistakes, an agent has to anticipate the consequences of its actions before execution, especially when feedback from the real environment is sparse or actions are irreversible. We introduce Agentic Imagination (Agima), a training framework that trains an agent to imagine before acting, i.e. to be imaginative. Agima trains a single omni model to propose actions, generate images of their outcomes, and then reason over these imagined futures to decide what to execute in the real environment. Rather than committing to a single plan, the agent can simulate a sequence of actions, explore alternatives, and compare possible futures. We first use supervised fine-tuning (SFT) to unify policy and world modeling within a single omni model. This enables the agent to predict future outcomes and perform basic forms of imagination, such as forward thinking and counterfactual branching. We then use agentic reinforcement learning (RL) to teach the agent to reason with its imagined futures and use them effectively for planning. Across four environments spanning Maze2D, Sokoban, Pacman, and PushT, Agima generalizes to harder settings unseen during training. Trained with SFT alone on two-box Sokoban, Agima solves 40% of four-box puzzles while seeing the real board only once every five moves, compared with an action-only planner that only solves 2%. After RL, new behaviors emerge in how Agima uses its imagination: it learns a new unified format that was unseen during the SFT stage, learns to conduct forward thinking only when necessary, and branches more frequently at critical decision points. An agent that can picture where its actions lead can learn when and where to look ahead, that is, to plan. Project page: this https URL.
• 中文摘要 在Sokoban中看似合理的推力可能会使谜题无法解决。为了避免此类错误，代理必须在执行前预见其行为的后果，尤其是在来自真实环境的反馈稀少或行为不可逆时。我们引入了代理想象（Agima），这是一种训练框架，训练代理在行动前先想象，即具备想象力。Agima训练一个单一的全集模型，提出行动，生成结果图像，然后对这些想象的未来进行推理，决定在现实环境中执行什么。代理不依赖单一计划，而是可以模拟一系列行动，探索替代方案，并比较可能的未来。我们首先使用监督微调（SFT）将政策和世界建模统一在单一的全向模型中。这使得代理能够预测未来结果，并执行基本的想象形式，如前瞻性思维和反事实分支。然后我们使用代理强化学习（RL）教代理与其想象的未来进行推理，并有效利用这些未来进行规划。在涵盖Maze2D、Sokoban、Pacman和PushT的四个环境中，Giima能够推广到训练中未曾察觉的更高难度设置。仅用SFT在两格Sokoban上训练，Agima在每五步只看到一次真实棋盘的情况下，解决了40%的四盒谜题，而仅以行动为主的规划器仅能解决2%。在RL之后，Agima的想象力运用方式出现了新的行为：它学会了SFT阶段未曾见过的新统一格式，仅在必要时进行前瞻性思考，并在关键决策点更频繁地分支。能够描绘其行动的走向的代理，能够学会何时何地展望未来，也就是规划。项目页面：此 https URL。

CoopUQ: Robust Multi-Robot Social Navigation via Uncertainty-Aware Cooperative Forecasting

CoopUQ：通过不确定性感知的合作预测实现的多机器人稳健社交导航

• Authors: Zehao Wang, Jianpeng Yao, Yuping Wang, Yiping Li, Jiaqi Li, Zhixu Li, Kanghoon Lee, Jinfeng Wang, Cong Liu, Zhaowei Tan, Jinkyoo Park, Jiachen Li
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.26852
• Pdf link: https://arxiv.org/pdf/2610.26852
• Abstract Reinforcement learning enables mobile robots to make fast and scalable navigation decisions in constrained human-shared environments. However, directly extending single-robot approaches to multi-robot settings often fails to ensure safety and efficiency, especially under occlusions and out-of-distribution (OOD) scenarios, where individual robots may overlook potential risks or make inaccurate predictions. Leveraging cooperation effectively for both prediction and navigation remains a major challenge. We propose CoopUQ, a multi-robot social navigation framework that bridges uncertainty-aware cooperative occupancy forecasting and robust decision making. CoopUQ predicts future occupancy grid maps through cross-robot spatiotemporal feature fusion and further leverages forecasts and their uncertainty to guide policy via constrained multi-agent reinforcement learning, where the cost penalizes navigation through regions predicted to be occupied or highly uncertain. This design enables the policy to adapt its risk sensitivity under distribution shifts. Experimental results across multiple Unity3D-based simulation environments with both in-distribution and OOD settings, including diverse layouts, pedestrian densities, and rushing behavior, demonstrate improved prediction performance and safety over representative baselines. Real-robot experiments further demonstrate its robustness across various scenes. Our code and videos are available at this https URL.
• 中文摘要 强化学习使移动机器人能够在受限的人共享环境中做出快速且可扩展的导航决策。然而，直接将单机器人方法扩展到多机器人环境下，往往无法保证安全性和效率，尤其是在遮挡和分布外（OOD）场景下，单个机器人可能忽视潜在风险或做出不准确的预测。有效利用协作进行预测和导航仍是一大挑战。我们提出了CoopUQ，一种多机器人社交导航框架，桥接了不确定性感知的合作占用预测与稳健决策。CoopUQ通过跨机器人时空特征融合预测未来占用网格地图，并进一步利用预测及其不确定性指导政策，通过受限多智能体强化学习，成本惩罚了预测占用或高度不确定区域的导航。该设计使政策能够在分布变化下调整风险敏感性。在多个基于Unity3D的模拟环境中，包括分布内和户外环境（包括多样化布局、行人密度和加急行为）的实验结果，显示出比代表性基线更优的预测性能和安全性。真实机器人实验进一步展示了其在不同场景中的鲁棒性。我们的代码和视频可在此 https URL 访问。

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

具视觉归因的政策自我提炼强化学习

• Authors: Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.26856
• Pdf link: https://arxiv.org/pdf/2610.26856
• Abstract Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.
• 中文摘要 多模态强化学习与可验证奖励（RLVR）提供了可靠的任务级监督，但对视觉推理中错误的定位指导有限。策略自提纯（OPSD）则通过配备证据为中心视觉视图的特权教师进行密集的标记级修正来补充该监督。然而，由此产生的修正可能会将视觉证据与由语言先验、反应风格或特权视角本身引起的非视觉变化混淆。我们将此现象形式化为特权纠正污染：当代币级修正的更新方向无法一致归因于特权视觉证据时，即为污染。直接提炼受污染的纠正可能强化错误的标记级更新。为解决这一问题，我们提出通过可视化归因政策自我提炼的强化学习，这是一种证据一致的范式，通过基于视觉证据的特权令牌级修正来补充任务级RLVR监督。我们通过可视化归因门槛（VAG）实现这一范式，该方法利用验证者衍生优势作为优化锚点，并根据证据存在与证据退化的教师视角对比验证每个特权修正。通过仅接受由视觉对比支持的方向性修正，VAG保持了视觉基础的指导，同时抑制了不一致的残差。涵盖六个视觉推理基准测试的实验显示，VAG始终优于GRPO和OPSD方法，建立了新的最先进平均表现。

Keyword: diffusion policy

DiPLO: Self-distilled Diffusion Planning for Loco-manipulation

DiPLO：机车操控的自蒸馏扩散计划

• Authors: Shafeef Omar, Sebastian Sanokowski, Elham Daneshmand, Dian Yu, Daniel Cremers, Cordelia Schmid, Majid Khadiv
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.24847
• Pdf link: https://arxiv.org/pdf/2610.24847
• Abstract Humanoid loco-manipulation is particularly challenging because it requires coordinating locomotion, balance, and object interaction, making it difficult to learn from limited demonstrations or from scratch through Reinforcement Learning (RL). We introduce DiPLO (self-distilled Diffusion Planning for Loco-manipulation), a framework that combines a self-distilled diffusion planner with a low-level whole-body tracking policy. Our planner is pretrained via supervised learning on motions derived from human-to-humanoid retargeting, while the tracker is trained via RL to execute them. DiPLO centers around two complementary post-training stages. First, we introduce Self-Distillation Diffusion Policy Optimisation (SD:DPO) to remove the planner's dependence on strong task conditioning, which we identify as a crucial ingredient for effective pretraining, as downstream fine-tuning does not recover comparable performance when it is omitted.. We then fine-tune this distilled model with Diffusion-Augmented Proximal Policy Optimization (DA:PPO), a maximum-entropy RL method for diffusion models, to improve task success and align generated plans with the frozen tracker's execution capabilities. Together, these stages enable motion generation through state-history and goal conditioning alone, enabling 99% task success in pick-carry-place tasks, compared to just 45% for the pretrained model. The resulting planner also generates motions for unseen object goals, with further evaluations on kicking and pushing/pulling tasks. Finally, we deploy our pipeline on the hardware, using our DA:PPO planner-tracker on a Unitree G1, achieving repeated successful executions without any real-world fine-tuning.
• 中文摘要 类人机车操作尤其具有挑战性，因为它需要协调运动、平衡和物体交互，这使得通过有限演示或通过强化学习（RL）从零开始学习变得困难。我们引入了DiPLE（自提炼式机车操控扩散规划），这是一个结合了自提炼扩散规划器和低层次全身跟踪策略的框架。我们的规划器通过监督学习预训练，针对人形生物重新定向产生的动作，而追踪器则通过强化学习训练以执行这些动作。DiPLO围绕两个互补的训练后阶段展开。首先，我们引入自蒸馏扩散策略优化（SD：DPO），以消除规划者对强任务条件的依赖，我们认为强任务条件是有效预训练的关键要素，因为下游微调若省略，无法恢复相当的性能。随后，我们用扩散增强近端策略优化（DA：PPO）微调该提炼模型，这是一种扩散模型的最大熵强化学习方法，以提升任务成功率，并将生成的计划与冻结跟踪器的执行能力对齐。这两个阶段共同实现了仅通过状态历史和目标条件生成运动，使得拾取-携带-放置任务的任务成功率达到99%，而预训练模型仅为45%。生成的规划器还生成未见物体目标的动作，并进一步评估踢动和推拉任务。最后，我们将流水线部署到硬件上，使用Unitree G1上的DA：PPO规划追踪器，实现多次成功执行，无需任何实际微调。

Energy-Score Stabilized Attentive Mixture-Density Policy: A Lightweight Approach to Multimodal Robot Motion

能量评分稳定的注意力混合-密度政策：一种多模态机器人运动的轻量化方法

• Authors: Sachiya Fujita, Hideyuki Ichiwara, Shigeki Sugano, Tetsuya Ogata
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.25401
• Pdf link: https://arxiv.org/pdf/2610.25401
• Abstract For practical motion generation, it is essential to handle multimodality (multiple actions can be correct for the same observation) while maintaining low inference cost and strong generalization from limited demonstrations. We introduce an attentive mixture-density policy trained with the Energy Score that represents future visuomotor trajectories as explicit probability distributions with a single forward pass. Concretely, we output Gaussian mixture parameters for joint commands (and optionally attention points) and optimize them with the Energy Score (computed as a sum of 1D CRPS under a diagonal covariance assumption) instead of the negative log-likelihood (NLL) to avoid MDN singularities and gradient-scale instabilities. Our architecture couples explicit spatial attention with mixture-density outputs, enabling robust multimodal behavior selection under visual noise and small-data regimes. On an 8-DOF manipulator, the method generates distinct avoidance trajectories and stabilizes learning in obstacle avoidance. Moreover, in a randomized moving-object grasping task, where spatiotemporal multimodality and fast timing decisions are required, the proposed model outperforms Diffusion Policy by 15-25% under several unseen conditions, demonstrating superior generalization and real-time feasibility.
• 中文摘要 对于实际的运动生成，必须处理多模态（同一观测可对多个动作正确），同时保持低推断成本和有限演示的强推广。我们引入了一种用能量评分训练的注意力混合密度策略，将未来视觉运动轨迹以单次前向传递的显式概率分布表示。具体来说，我们输出联合指令的高斯混合参数（以及可选的注意点），并用能量评分（在对角协方差假设下计算为1D CRPS和）而非负对数似然（NLL）进行优化，以避免MDN奇点和梯度尺度不稳定性。我们的架构将显式空间注意力与混合密度输出结合，实现在视觉噪声和小数据环境下稳健的多模态行为选择。在8自由度操作手上，该方法生成不同的避让轨迹，并稳定障碍物规避的学习。此外，在需要时空多模态和快速时序决策的随机移动物体抓取任务中，该模型在多个未见条件下表现优于扩散政策15-25%，展现了优越的泛化性和实时可行性。

SWAP: Test-Time Steerable Humanoid Whole-Body Collision Avoidance

SWAP：测试时间可转向人形全身碰撞避免

• Authors: Zeji Yi, Guanqi He, Xiaofeng Guo, Guannan Qu, Pieter Abbeel, Yan Duan, Karen Liu, Guanya Shi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.25711
• Pdf link: https://arxiv.org/pdf/2610.25711
• Abstract Humanoid robots operating in human-centric environments must coordinate the entire body to traverse tight, cluttered spaces while maintaining balance. Existing methods often struggle to jointly consider dynamic stability and collision avoidance from high-dimensional perceptual inputs. Moreover, for humanoids, safe behavior is inherently multimodal: the same obstacle may admit multiple valid traversals, which is difficult to represent with a single deterministic policy. To address these challenges, we propose SWAP (Steerable Whole-body Avoidance Policy), a hierarchical framework that separates collision-aware decision making from stabilization. A high-level, perception-conditioned diffusion policy generates multimodal, collision-aware motion proposals, while a low-level whole-body RL controller tracks these proposals and maintains dynamic balance. To efficiently train the high-level policy, we obtain scalable supervision via hindsight labeling: we roll out the low-level policy under randomized commands, place obstacles only in the space that each executed motion did not sweep, so the motion is collision-free by construction, and re-render the perception the robot would have seen. At deployment, Monte Carlo sampling with simple cost signals steers the diffusion model toward safe, task-aligned behaviors without heavy reward tuning. Experiments in simulation and on a real Unitree G1 humanoid show reliable avoidance with static and dynamic obstacles. Test-time steering further enables customization, improving targeted rewards (e.g., speed, orientation, or posture) by 34% on average while maintaining a high collision avoidance rate. Together, these results support a practical solution to humanoid whole-body collision avoidance with test-time steerability. Project website: this https URL
• 中文摘要 在以人为中心的环境中运行的人形机器人必须协调整个身体，以穿越狭窄、杂乱的空间，同时保持平衡。现有方法常常难以同时考虑高维感知输入中的动态稳定性和碰撞避免。此外，对于类人生物来说，安全行为本质上是多模态的：同一障碍可能允许多次有效穿越，这很难用单一确定性策略来表示。为应对这些挑战，我们提出了SWAP（可引导全体避免策略），这是一个分层框架，将碰撞感知决策与稳定分离。高级别感知条件扩散策略生成多模态碰撞感知运动提案，而低级全身强化学习控制器跟踪这些提案并维持动态平衡。为了高效训练高层策略，我们通过事后诸葛亮标签获得可扩展的监督：我们以随机命令推出低层策略，只在每次执行动作未扫过的空间放置障碍物，使动作结构上无碰撞，并重新渲染机器人本应感知的状态。部署时，蒙特卡洛采样与简单成本信号引导扩散模型朝安全、任务对齐行为方向发展，无需大量奖励调优。模拟实验和真实Unitree G1人形生物显示，静态和动态障碍物都能可靠地避让。测试时间转向进一步促进定制化，平均提升目标奖励（如速度、方向或姿态）34%，同时保持高碰撞避免率。综合来看，这些结果支持了基于测试时间转向的人形全身碰撞避免的实用解决方案。项目网站：此 https URL

Compositional Humanoid Manipulation through Isomorphic Teleoperation and Independent Skill Routing

通过同构远程操作和独立技能路由实现合成类人操控

• Authors: Yiming Wang, Peixian Yang, Lifeng Zhuo, Lai Wei, Xinling Du, Kun Song, Weiming Wang, Daolin Ma
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.26169
• Pdf link: https://arxiv.org/pdf/2610.26169
• Abstract Humanoid manipulation requires reusable arm skills, yet demonstrating every possible skill combination is impractical. We present Humanoid Semantic Mixture-of-Experts Diffusion Policy (HS-MoDP), which composes separately demonstrated arm skills through independent semantic routing over a shared expert pool. This design preserves arm-specific skill control while sharing action-generation parameters across arms. A semantic scheduler integrates the learned policy with an existing locomotion controller through alternating manipulation and repositioning stages, extending the effective workspace and enabling skill reuse across location changes. To acquire demonstrations of whole-body contact manipulation, we develop G1Exo, an isomorphic exoskeleton that provides direct joint-level control of arm configurations. Experiments on a Unitree G1 show stronger skill composition than the baselines. G1Exo enables more reliable contact teleoperation than end-effector VR-IK, and its demonstrations improve contact-skill learning across four policy architectures. For HS-MoDP, switching from VR-IK to G1Exo demonstrations raises full cabinet workflow success from 35% to 65%. All project hardware designs and source code will be released as open source after the review process.
• 中文摘要 类人操作需要可重复使用的手臂技能，但展示所有可能的技能组合并不切实际。我们提出了类人语义专家混合扩散策略（HS-MoDP），通过独立语义路由在共享专家池中组合展示过的手臂技能。该设计保留了手臂专属技能控制，同时共享跨臂的动作生成参数。语义调度器通过交替操作和重新定位阶段，将学习策略与现有运动控制器整合，扩展有效工作空间并实现技能在位置变化间的重复使用。为展示全身接触操作，我们开发了G1Exo，一种同构外骨骼，提供对手臂配置的直接关节级控制。在Unitree G1上的实验显示技能组成比基线更强。G1Exo 比端效器 VR-IK 更能实现更可靠的接触远程操作，其演示提升了四种策略架构下的接触技能学习。对于 HS-MoDP，从 VR-IK 切换到 G1Exo 演示将全机柜工作流程成功率从 35% 提升至 65%。所有项目硬件设计和源代码将在审核后以开源形式发布。

Learn the Task, Not the Schedule: Reducing Spurious Cross-Arm Coupling in Bimanual Imitation Learning

学习任务，而非计划：减少双手模仿学习中的虚假交叉臂耦合

• Authors: Shuyuan Hu, Deng-Ke Wu, Yifan Jin, Qianhang Luo, Mingchen Ju, Kai Ye, Bo Zhao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.26507
• Pdf link: https://arxiv.org/pdf/2610.26507
• Abstract A bimanual demonstration specifies both what the robot should do and one particular schedule for when each arm should act. Standard imitation learning does not distinguish task-required cross-arm constraints from incidental scheduling choices, and may therefore exploit opposite-arm progress as a shortcut cue, creating spurious cross-arm coupling even where relative timing is irrelevant for the current task phase. We introduce Constraint-preserving Temporal Recombination (CTR), a training-time data intervention that preserves recorded arm-local sensorimotor content while retiming the corresponding segments to realize admissible schedules that preserve task-required cross-arm constraints. We use CTR to train Diffusion Policy, Action Chunking with Transformers, and $\pi_{0.5}$, and evaluate them on four simulated and four real-robot tasks. Across six real-robot bimanual progress-state initializations, CTR raises worst-state success from 56% for the strongest non-CTR schedule mixture to 76%. With $\pi_{0.5}$ on an Independent task, a 4 s pause transfers 2.56 s of delay to the opposite arm under the Concurrent schedule distribution, but only 0.25 s with CTR. On real robots, CTR violates task constraints in only 0--4% of trials, versus 36--98% for narrow-schedule baselines. Code will be released at this https URL.
• 中文摘要 双手演示指定机器人应执行的任务以及每只臂应行动的具体时刻表。标准模仿学习不区分任务要求的交叉臂约束与偶然的调度选择，因此可能利用反向臂的进展作为捷径提示，即使相对时间对当前任务阶段无关，也会产生虚假的交叉臂耦合。我们引入约束保持时间重组（CTR），这是一种训练时间数据干预，保留记录的臂-局部感觉运动内容，同时重新计时相应段以实现可接受的计划，从而保留任务所需的交叉臂约束。我们使用CTR训练扩散策略、与变换器分块和$\pi_{0.5}$，并在四个模拟任务和四个真实机器人任务中评估它们。在六次真实机器人双手进度状态初始化中，CTR将最强非CTR计划组合的最低状态成功率从56%提升至76%。在独立任务中pi_，4秒暂停可将2.56秒延迟传递到对侧臂，而CTR仅有0.25秒。在真实机器人中，CTR仅在0--4%的试验中违反任务约束，而窄周期基线的测试为36-98%。代码将在此 https URL 发布。