生成时间: 2026-10-23 21:11:29 (UTC+8); Arxiv 发布时间: 2026-10-23 20:00 EDT (2026-10-24 08:00 UTC+8)

今天共有 44 篇相关文章

Keyword: reinforcement learning

Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning

编辑，别模仿：多模态推理的策略提纯最小编辑

• Authors: Wen Ye, Ailin Deng, Zirui Wang, Shixing Yu, Han Zhang, Tian Yun, Nikhil Mehta, Tarun Kalluri, Xuefei Cao, Xuezhe Ma, Hengduo Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30231
• Pdf link: https://arxiv.org/pdf/2610.30231
• Abstract Visual mathematical reasoning can fail before reasoning begins: when a vision-language model (VLM) misreads a diagram, subsequent reasoning proceeds from false premises. This perceptual bottleneck is especially pronounced in smaller models, while accurate textual descriptions can substantially narrow the gap to larger models. Because obtaining such descriptions at inference requires additional annotation or model calls, we propose minimal-edit corrective on-policy distillation. Using reference figure descriptions and verified reference solutions as training-only information, a teacher scores incorrect student rollouts under a minimal-edit objective, and only teacher-disfavored tokens contribute to the corrective loss. Correct rollouts receive task-reward reinforcement learning. This preserves on-policy supervision while avoiding full-response imitation. Across model scales, the method requires no descriptions at test time and meets or exceeds the corresponding base model supplied with a reference figure description at inference. It also improves perception-oriented performance, particularly low-level geometric perception, and transfers to external mathematics and perception benchmarks. Because the minimal-edit teacher evaluates corrections rather than generating its own solution, the method remains effective with a frozen same-size teacher, although its gains are smaller than with a larger teacher on most metrics.
• 中文摘要 视觉数学推理可能在推理开始前就失败：当视觉语言模型（VLM）误读图示时，后续推理基于错误前提。这种感知瓶颈在较小模型中尤为明显，而准确的文本描述则能显著缩小与大型模型之间的差距。由于在推理时获得此类描述需要额外的注释或模型调用，我们提出最小编辑纠正策略提炼。使用参考图描述和经过验证的参考解作为仅训练信息，教师在最小编辑目标下对错误的学生展开进行评分，只有教师不偏好的标记才会贡献纠正损失。正确的展开会获得任务奖励强化学习。这既保持了政策上的监督，又避免了全响应的模仿。在不同模型尺度上，该方法在测试时无需描述，且在推断时满足或超过对应的参考图模型。它还提升了以感知为导向的表现，特别是低层次几何感知，并可迁移到外部数学和感知基准测试。由于最小编辑教师评估纠正而非自行生成解，该方法在固定的同规模教师下依然有效，尽管在大多数指标上其提升幅度小于大型教师。

TerraTransfer++: Demonstration-Free End-to-End Planner

TerraTransfer++：无需演示的端到端规划器

• Authors: Zikang Xiong, Weixin Li, Zhouchonghao Wu, Akshay Rangesh, Saarth Bonde, Grantland Hall, Chen Tang, Yihan Hu, Wei Zhan
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30319
• Pdf link: https://arxiv.org/pdf/2610.30319
• Abstract End-to-end autonomous driving has achieved state-of-the-art performance on benchmarks and real-world deployments. Its standard training recipe, however, is expensive across all existing paradigms: imitation learning requires fleet-scale driving logs whose behavior distribution only thinly covers the safety-critical tail; closed-loop reinforcement learning in a photorealistic simulator, while offering diverse behaviors, is bottlenecked by sample-efficiency, per-step cost of rendering, and a forward pass through a large vision backbone. Self-play in vectorized simulators changes the economics: at millions of rollout steps per second, one can obtain a state distribution with natural diversity in collisions, near-misses, and recoveries that no driving log contains, and a highly reactive policy emerges from a basic driving-quality reward. Our approach exploits this asymmetry. We pretrain a single policy by self-play, distill its rollouts into a flow-matching trajectory teacher that also plans, and train lightweight adapters on a frozen pretrained vision backbone to reproduce the latent space of either teacher through the action KL divergence, a batch-relational low-rank structural loss, and, for the planner, a trajectory-matching loss. Every target traces back to the self-play policy, so alignment never supervises against a logged trajectory: a paired dataset of (image, scene-state) frames suffices, with no need for the curated expert demonstrations that imitation pretraining is built on. From this one self-play teacher we obtain two end-to-end students: a closed-loop policy that outperforms every published imitation-based end-to-end method on the photorealistic 3D Gaussian splatting closed-loop benchmark HUGSim, and a planner that reaches 59.4 EPDMS on NAVSIM navhard, above every published method including DriveZero-Scale (57.1), and 95.5 PDMS on navtest, without human demonstrations.
• 中文摘要 端到端自动驾驶在基准测试和实际部署中已达到最先进的性能。然而，其标准训练方案在所有现有范式中成本高昂：模仿学习需要车队规模的驾驶日志，其行为分布仅覆盖安全关键的尾部;在写实模拟器中进行闭环强化学习，虽然提供多样化行为，但受样本效率、每步渲染成本和通过大型视觉骨干的前向传递所限制。矢量模拟器的自我游戏改变了经济学：以每秒数百万次展开步，可以获得碰撞、擦撞和恢复中自然多样性的状态分布，且无驾驶日志包含这些，且从基本驾驶质量奖励中产生高度反应的策略。我们的方法利用了这种不对称性。我们通过自我对单一策略进行预训练，将其推出的推举提炼为一个同时规划的流程匹配轨迹教师，并在冻结的预训练视觉骨干上训练轻量级适配器，通过动作 KL 发散、批次关系的低秩结构性损失，以及规划器的轨迹匹配损失，再现任一教师的潜在空间。每个目标都追溯到自玩策略，因此对齐不会监督已记录的轨迹：配对的（图像、场景状态）帧数据集就足够了，无需模仿预训练所依赖的精心策划的专家演示。从这位自学教师那里，我们得到了两个端到端的学生：一个闭环策略，在逼真的3D高斯喷溅闭环基准测试HUGSim上表现优于所有已发表的基于模仿的端到端方法;一个在NAVSIM navhard上达到59.4 EPDMS，超过所有已发表方法，包括DriveZero-Scale（57.1）和95.5 PDMS，且无需人工演示。

MPBRS: Multi-Potential-Based Reward Shaping for Unified Reward Configuration in Humanoid Locomotion

MPBRS：基于多势位的奖励塑造，用于类人机动中统一奖励配置

• Authors: Van Chung Nguyen, Hung Manh La
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30334
• Pdf link: https://arxiv.org/pdf/2610.30334
• Abstract Designing and tuning reward functions remains one of the main challenges in building effective reinforcement learning (RL) pipelines for humanoid locomotion. While shaping rewards can accelerate learning, naively formulated terms often conflict with the desired behavior, overfit to a specific robot, or become unstable under small weight changes. Potential-Based Reward Shaping (PBRS) offers an alternative solution: shaping the reward via differences of a potential function to guide exploration without changing the optimal policy. In this paper, we study Multi-PBRS for humanoid locomotion and propose a unified reward configuration. We then evaluate the same reward structure and weights across nine humanoid robots, requiring only robot-specific metadata (joint/link mappings and default poses) rather than per-robot reward redesign. Our experiments show that, compared to standard shaping, MPBRS provides more consistent scaling behavior and reduced tuning sensitivity across robots, making it easier to tune without altering the optimal policy.
• 中文摘要 设计和调优奖励函数仍是构建有效强化学习（RL）人形运动流水线的主要挑战之一。虽然塑造奖励可以加速学习，但简单表达的术语常常与期望行为冲突，导致对特定机器人过拟合，或在微小的权重变化下变得不稳定。基于势能的奖励塑形（PBRS）提供了另一种解决方案：通过潜在函数的差异塑造奖励，指导探索而不改变最优策略。本文研究了类人机动的多重PBRS，并提出了统一的奖励配置。随后，我们评估了九台类人机器人的相同奖励结构和权重，只需机器人特定的元数据（关节/链接映射和默认姿势），而无需每个机器人重新设计奖励。我们的实验表明，与标准整形相比，MPBRS在机器人间提供了更一致的尺度行为和降低调谐灵敏度，使得调优更为便捷，而无需改变最优策略。

Consistency Is Not Validity: Self-Rewarded Moral Argumentation Makes Language Models Decisive but Pliable

一致性不等于有效性：自我奖励的道德论证使语言模型既果断又具可塑性

• Authors: Marius De Kuthy Meurers, Aaron Parisi, Bernd Bohnet
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30349
• Pdf link: https://arxiv.org/pdf/2610.30349
• Abstract LLMs are increasingly consulted for moral judgment. We aim to improve performance on such tasks by rewarding good arguments, on the premise that arguing well increases the likelihood of judging well. We train a reinforcement learning policy initialized from a state-of-the-art language model in a self-judged tournament with masked thinking: the generated arguments are judged pairwise on argument quality by the policy itself. As an empirical basis, we extract authentic dilemmas from classic novels, using a logic solver to verify that each constitutes a conflict. We show that our judge rates argument quality like human annotators, and the policy then adopts the language register that it rewards. Testing on the external benchmarks MoralChoice, ELEPHANT, and MORABLES, we find that the policy becomes more decisive and invariant to the order in which options are presented, but sycophancy and reframing remain a challenge to validity. Two ablation studies with synthetic dilemmas and unmasked thinking match the reordering gains, but show a weaker register shift and higher sycophancy. We conclude that consistency is not validity: consistency under reordering can be optimized without improving the moral judgment it is meant to signal.
• 中文摘要 LLMs越来越多地被用于道德判断。我们旨在通过奖励优秀论证来提升此类任务的表现，前提是论证得当能提高判断能力。我们在自我评判的比赛中训练一套基于最先进语言模型的强化学习策略，并结合掩蔽思维进行自我评判：生成的论证由策略本身根据论证质量成对评判。作为实证基础，我们从经典小说中提取真实的两难，使用逻辑求解器验证每个困境构成冲突。我们证明我们的评判者像人类注释者一样评定论证质量，策略随后采用其奖励的语言寄存器。在外部基准测试MoralChoice、ELEPHANT和MORABLES时，我们发现该策略变得更具决定性，且对选项呈现顺序保持不变，但谄媚和重构仍然是有效性的挑战。两项具有合成困境和无掩饰思维的消融研究与重排序的收益相符，但显示出更弱的音域移位和更高的谄媚。我们得出结论：一致性不等于有效性：重排下的一致性可以在不提升其传递的道德判断的情况下进行优化。

Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models

自我评估强化学习用于检索增强大型语言模型中搜索状态意识

• Authors: Taewhoo Lee, Minju Song, Jungwoo Park, Chanwoong Yoon, Junha Jung, Hyeon Hwang, Sanghoon Lee, Minbyul Jeong, Jaewoo Kang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30435
• Pdf link: https://arxiv.org/pdf/2610.30435
• Abstract Recent advances in search agents have enabled large language models to tackle complex tasks by retrieving and synthesizing external information. However, existing search agents often lack search-state awareness, i.e., the ability to recognize what information has been obtained and what remains missing, leading to hallucinated reasoning and prematurely terminated search. To address this limitation, we propose Self-Assessment Reinforcement Learning (SARL), a reinforcement learning framework in which models learn to recognize their own search state during training and improve their search behavior accordingly, without requiring additional assessment at test time. SARL achieves this through a dual reward design that captures the corrective effect of each assessment while penalizing excessive reliance on it, jointly encouraging effective and efficient search trajectories. Under this setup, SARL achieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% on multi-hop reasoning and 27.9% on deep research benchmarks against the strongest per-benchmark baselines. SARL further surpasses methods built on auxiliary supervision or multi-agent coordination, despite operating as a standalone RL framework with no inference-time scaffolding. Beyond final performance, SARL maintains training stability under extended search horizons, a regime where existing methods consistently degrade, enabling sustained gains as task complexity grows.
• 中文摘要 搜索代理的最新进展使大型语言模型能够通过检索和综合外部信息来应对复杂任务。然而，现有的搜索代理通常缺乏搜索状态意识，即识别已获得信息和缺失信息的能力，导致产生幻觉推理和过早终止搜索。为解决这一限制，我们提出了自我评估强化学习（SARL），这是一种强化学习框架，模型在训练时学习识别自身搜索状态并相应改进搜索行为，而无需在测试时额外评估。SARL通过双重奖励设计实现这一目标，既捕捉每次评估的纠正效果，又惩罚过度依赖评估，共同鼓励有效且高效的搜索轨迹。在这种配置下，SARL相较于最强基线实现了24.3%的相对提升，在多跳推理方面平均提升35.9%，在深度研究基准中对比最强的每个基准基线提升27.9%。尽管作为独立的强化学习框架运行，SARL仍超越了基于辅助监督或多代理协调的方法，且没有推断时间支架。除了最终性能外，SARL还能在延长搜索时间下保持训练稳定性，在这一模式下现有方法持续退化，使得随着任务复杂度的增加，能够持续获得进步。

Tactile-Based Robotic Manipulation: From Mechanics-Driven to Data-Driven Perspectives

基于触觉的机器人操作：从机械驱动到数据驱动视角

• Authors: Jingyi Xiang, Yixuan Wang, Wenzhen Yuan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.30438
• Pdf link: https://arxiv.org/pdf/2610.30438
• Abstract Tactile sensing provides robots with direct access to physical interactions at the contact interface, making it a critical modality for robust and dexterous manipulation. Over the past several decades, tactile-based manipulation research has evolved from mechanics-driven approaches, which explicitly interpret tactile signals through contact mechanics and task-specific reasoning, to data-driven approaches that learn tactile representations and manipulation policies directly from data. This paper reviews the development of tactile-based manipulation from these two complementary perspectives. We first introduce the fundamentals of tactile sensing, and then review, from a mechanics-driven perspective, how those signals form manipulation-relevant state representations and subsequently influence the robot action. We then examine the recent advances in data-driven tactile manipulation, covering imitation learning, reinforcement learning, vision-language-action models, and predictive models. Finally, we discuss open challenges and future opportunities toward scalable, generalizable, and robust tactile-enabled robotic systems.
• 中文摘要 触觉感知为机器人提供了直接接触接触界面物理交互的通道，使其成为实现稳健且灵活操作的关键方式。在过去几十年里，基于触觉的操作研究已从机械驱动的方法（通过接触力学和任务特定推理明确解释触觉信号）发展到直接从数据中学习触觉表征和操作策略的数据驱动方法。本文从这两个互补视角回顾了基于触觉的操作的发展。我们首先介绍了触觉感知的基础知识，然后从机械驱动的角度回顾这些信号如何形成与操作相关的状态表征，进而影响机器人动作。接着，我们考察了数据驱动触觉操作的最新进展，涵盖模仿学习、强化学习、视觉-语言-动作模型和预测模型。最后，我们讨论了面向可扩展、通用且稳健的触觉机器人系统的开放挑战及未来机遇。

SpecMAGIC: Specification-Guided Multi-Task Generalization in Multi-Agent Reinforcement Learning

SpecMAGIC：多智能体强化学习中的规范引导多任务泛化

• Authors: Zijian Guo, İlker Işık, H. M. Sabbir Ahmad, Wenchao Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30443
• Pdf link: https://arxiv.org/pdf/2610.30443
• Abstract Multi-task generalization is an important yet underexplored problem in multi-agent reinforcement learning (MARL). Many real-world applications require multi-agent systems to generalize across complex, temporally extended tasks with diverse inter-agent coordination requirements. We introduce a specification language that combines temporal-logic task specifications with operators for expressing such requirements. We propose SpecMAGIC, a hierarchical framework that leverages the automaton structure of task specifications for subtask decomposition. A centralized high-level planner identifies and allocates compatible subtask sequences across agents, while decentralized low-level policies execute individual subtasks. This enables zero-shot generalization to new tasks by composing reusable subtask policies according to new specifications, without retraining. We establish soundness and probabilistic completeness of the high-level planner. Experiments on multi-agent navigation tasks show that SpecMAGIC substantially outperforms existing approaches on unseen task specifications and maintains strong performance as the number of agents increases.
• 中文摘要 多任务泛化是多智能体强化学习（MARL）中一个重要但尚未被充分探索的问题。许多现实应用需要多智能体系统在复杂、时间扩展且具多样代理间协调需求的任务中进行泛化。我们引入了一种结合时序逻辑任务规范与表达此类需求的操作符的规范语言。我们提出了SpecMAGIC，一种利用任务规范自动机结构进行子任务分解的层级框架。集中式高级规划器识别并分配兼容的子任务序列在各个智能体之间，而去中心化的低级策略执行单个子任务。这使得通过根据新规范组合可复用子任务策略，实现零样本泛化到新任务而无需重新训练。我们建立了高层规划器的合理性和概率完整性。多智能体导航任务的实验表明，SpecMAGIC在未见任务规范上远超现有方法，并且随着智能体数量的增加，性能依然强劲。

Short-to-Long: Detached Features for Long-Horizon Value Learning

短至长：远视野价值学习的分离特征

• Authors: Tong Che, Boris Ivanovic, Marco Pavone
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30461
• Pdf link: https://arxiv.org/pdf/2610.30461
• Abstract Learning useful representations from continually changing value targets remains a challenge in deep reinforcement learning. We study the temporal horizon of this supervision: smaller discounts make these targets less sensitive to distant bootstrap changes, but using them for behavior changes the control objective. We introduce Short-to-Long (S2L), which supplies a long-horizon behavior learner with features trained by smaller-discount source networks. Detached connections prevent the behavior value loss from updating the sources, allowing each source to follow its own value objective while the behavior discount remains unchanged. Across Atari, online continuous control, and offline continuous control, the approach yields aggregate performance gains, improving the paired terminal median on 40 of 57 Atari games. Matched Atari ablations support the roles of feature transfer, smaller source discounts, and gradient isolation. Fixed-trajectory measurements further show reduced drift in source value targets with mixed discounts.
• 中文摘要 从不断变化的价值目标中学习有用的表示仍然是深度强化学习中的挑战。我们研究了该监督的时间视角：较小的折扣使这些目标对远距离自助变化的敏感度降低，但用于行为则改变控制目标。我们引入了短至长（S2L），为长视野行为学习者提供由小折扣源网络训练的特征。分离连接防止行为值损失更新源，使每个源能够遵循其自身的价值目标，而行为折扣保持不变。在Atari、在线连续控制和离线连续控制中，该方法实现了整体性能提升，改善了57款Atari游戏中40款的配对终端中位数。匹配Atari消融支持特征转移、更小的源折扣和梯度隔离的作用。固定轨迹测量进一步显示，源值目标在混合折扣下漂移减少。

Learning to Incentivise: Using Reinforcement Learning for Sustainable Urban Mobility

学习激励：利用强化学习实现可持续城市出行

• Authors: Germán Pardo-González, Shaghayegh Vosough, Katerina Papadaki, Claudio Roncoli
• Subjects: Subjects: Multiagent Systems (cs.MA); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.30469
• Pdf link: https://arxiv.org/pdf/2610.30469
• Abstract Traffic management has traditionally focused on toll-based and road pricing solutions. However, road pricing often raises concerns about accessibility and public dissatisfaction, leading to its prohibition in some places, such as Finland. This study optimises the dynamic allocation of incentives to drivers, encouraging them to reroute onto alternative (potentially longer) paths to achieve greater societal benefit, reduced total travel time (TTT) and total emissions in the transportation network, contributing to sustainable urban mobility. We employ a multi-agent reinforcement learning approach to dynamically assign incentives to drivers to reduce both TTT and emissions, with travel times estimated using traffic simulation software. We demonstrate that, with an unlimited budget and an objective of minimising travel time, the incentive scheme reduces TTT by 16%, compared to the dynamic User-Equilibrium (UE) with a budget equivalent to about 11% of the UE total time. When the goal is to minimise emissions, a 9% reduction in CO2 emissions is observed under an unlimited budget. We demonstrate a critical trade-off: minimising TTT leads to an increase in emissions, while prioritising emission reductions raises TTT. However, with the right combination of weights in the multi-objective function, both TTT and total emissions are improved beyond the baseline.
• 中文摘要 交通管理传统上专注于基于通行费和道路收费的解决方案。然而，道路收费常常引发对可达性和公众不满的担忧，导致芬兰等地禁止。本研究优化了激励措施的动态分配，鼓励他们改道至替代（可能更长）路径，以实现更大的社会利益，减少总出行时间（TTT）和交通网络的总排放，促进可持续的城市出行。我们采用多智能体强化学习方法，动态为驾驶员分配激励措施，以减少TTT和排放，并通过交通模拟软件估算出行时间。我们证明，在无限预算和最小化出行时间的目标下，激励方案可将TTT减少16%，相比动态用户均衡（UE）约占UE总时间11%的预算。当目标是最小化排放时，在无限预算下可观察到二氧化碳排放减少9%。我们展示了一个关键权衡：最小化TTT会导致排放增加，而优先减少排放则提高TTT。然而，通过多目标函数权重的正确组合，TTT和总排放量都会超越基线水平。

Motion Retargeting with Spatiotemporal Preferences for Humanoid Loco-Manipulation

基于时空偏好的人体机车操控运动重定向

• Authors: Xiongbo Shi, Shihang Du, Zongyou Yu, Ao Zhou, Xudong Zhang, Ziyu Lin, Haodong Zhang, Junqiao Zhao, Guang Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.30540
• Pdf link: https://arxiv.org/pdf/2610.30540
• Abstract Human motion retargeting bridges the embodiment gap between humans and humanoid robots by converting human demonstrations into reference trajectories for whole-body control and loco-manipulation. For object-interaction tasks, successful execution depends on accurately preserving body--object relations at critical stages. However, uniform per-frame iteration budgets do not account for these localized accuracy requirements, limiting the computational efficiency of retargeting. To address this limitation, we present Spatiotemporal Retargeting (STR), which uses keyframes and key body parts to allocate optimization effort. Specifically, a vision-language model selects relevant motion signals and body parts, and keyframes are identified from changes in the selected signals. Based on these selections, STR assigns additional solver iterations to keyframes and higher interaction weights to the selected parts. To further improve contact fidelity, we introduce Mechanics-Aware Contact Anchoring (MCA) to refine object-surface contact positions using an approximate object-wrench model while retaining the selected surface directions. Building on the resulting references, we develop Spatiotemporal Reinforcement Learning (STRL), which combines adaptive key-stage sampling with a contact reward based on target proximity and object contact forces for downstream control. Simulation experiments on a Unitree G1 and real-world experiments on a Unitree R1 demonstrate that STR improves geometric contact coverage and continuity at lower computational cost, while the resulting references improve downstream task completion across diverse object-interaction motions.
• 中文摘要 人体运动重定向通过将人体演示转换为全体控制和机动操作的参考轨迹，弥合了人类与类人机器人之间的身体化差距。对于物体交互任务，成功执行依赖于在关键阶段准确保持身体-物体关系。然而，统一的每帧迭代预算未能满足这些局部精度要求，限制了重定向的计算效率。为解决这一限制，我们提出了时空重定向（STR），它利用关键帧和关键身体部位分配优化工作。具体来说，视觉语言模型选择相关的运动信号和身体部位，并通过所选信号的变化识别关键帧。基于这些选择，STR为关键帧分配额外的求解器迭代，并为所选部分赋予更高的交互权重。为进一步提升接触保真度，我们引入了力学感知接触锚定（MCA），利用近似对象扳手模型优化物体-表面接触位置，同时保持所选表面方向。基于所得参考，我们开发了时空强化学习（SPATIOTEMPORAL Reinforcement Learning，STRL），结合了自适应关键阶段采样与基于目标接近度和物体接触力的接触奖励，用于下游控制。在Unitree G1上的模拟实验和在Unitree R1上的实际实验表明，STR以更低的计算成本提升几何接触覆盖率和连续性，而所得参考则提升了在多种对象交互运动中的后续任务完成率。

When News Gets Old: Portfolio Allocation with Reinforcement Learning and Large Language Models

当新闻变得陈旧：强化学习与大型语言模型的投资组合分配

• Authors: Bolin Song (Cornell University), Surui Xiang (United Nations)
• Subjects: Subjects: Machine Learning (cs.LG); Portfolio Management (q-fin.PM)
• Arxiv link: https://arxiv.org/abs/2610.30564
• Pdf link: https://arxiv.org/pdf/2610.30564
• Abstract Financial news can influence investment decisions long after publication. Timely news may reach an investor before prices fully adjust, whereas older news may already be reflected in prices. We examine two ways to handle aging signals: retain them and give a portfolio policy timing information, or filter them out before portfolio selection. A fixed large language model (LLM) extracts signals from public SEC disclosures. We compare these treatments in reinforcement learning (RL) policies and LLMs that choose portfolios directly. We evaluate returns after trading costs in historical tests over 2023-2025. RL choices respond to timing information in a controlled test, but historical returns show no consistent benefit from adding it. For the LLM, filtering old news produced the highest average net return, reaching 18.43% per year when news delivery was delayed by four weeks. This exceeded the returns of all RL strategies that used news.
• 中文摘要 财经新闻在发布后很长时间仍会影响投资决策。及时新闻可能在价格完全调整前传达投资者，而较旧的新闻可能已反映在价格中。我们探讨处理老旧信号的两种方法：保留它们并提供投资组合政策时机信息，或在选择投资组合前过滤掉。固定大型语言模型（LLM）从公开的SEC披露中提取信号。我们将强化学习（RL）政策与直接选择投资组合的LLM进行比较。我们在2023-2025年历史测试中评估交易成本后回报。在受控测试中，RL选择对时机信息有响应，但历史回报并未显示添加时机带来持续收益。对于LLM而言，过滤旧新闻产生了最高的平均净回报，在新闻发布延迟四周后达到每年18.43%。这一收益超过了所有使用新闻的RL策略的回报。

Visual Grounding First, Multimodal In-context Learning Follows

先是视觉基础，随后是多模态语境内学习

• Authors: Minhyuk Seo, Minjae Lee, Chaeeun Lee, Wei Lin, Muhammad Jehanzeb Mirza, Tinne Tuytelaars, Jonghyun Choi
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30592
• Pdf link: https://arxiv.org/pdf/2610.30592
• Abstract LLMs have shown strong in-context learning (ICL) capability, but extending it to Multimodal LLMs (MLLMs) remains challenging. Prior multimodal ICL methods often rely on ICL-specific datasets for additional training or task vector extraction, which can improve performance on the same ICL benchmark used for adaptation. However, they do not generalize well to other benchmarks and often induce forgetting in previously well-solved tasks. For this reason, we identify 'visual grounding' as the key bottleneck in multimodal ICL; MLLMs often fail to attend to task-relevant visual evidence in demonstrations, instead over-relying on textual cues and producing hallucinated outputs. To address this, we propose ProCoRe to enhance visual grounding through contrastive reinforcement learning on multimodal contrastive data for improving multimodal ICL. ProCoRe generates captions for contrastive image pairs, decomposes them into propositions, and optimizes contrastive rewards that encourage alignment with corresponding images while discouraging mismatched ones. Despite never seeing ICL-formatted training examples, ProCoRe improves average multimodal ICL classification accuracy by 5.6% and captioning ROUGE-L by 13.8% over the off-the-shelf Qwen-VL-3-8B model, outperforming ICL-data-dependent state of the arts across benchmarks. Finally, we introduce PICL, a personalized multimodal ICL captioning benchmark that tests whether MLLMs genuinely understand visual demonstrations rather than relying on pre-trained semantic priors.
• 中文摘要 LLMs展现出强大的上下文学习（ICL）能力，但将其推广到多模态LLMs（MLLMs）仍具挑战性。以往的多模态ICL方法通常依赖ICL专用数据集进行额外训练或任务向量提取，这可以提升同一ICL基准测试的性能。然而，这些方法未能很好地推广到其他基准测试，且常常导致先前已解决的任务中遗忘。因此，我们将“视觉基础化”视为多模态ICL的关键瓶颈;MLLM在演示中常常忽视任务相关的视觉证据，反而过度依赖文本线索，产生幻觉输出。为解决这一问题，我们提出ProCoRe通过对多模态对比数据的对比强化学习来增强视觉基础化，以提升多模态ICL。ProCoRe为对比图像对生成说明，将其分解为命题，并优化对比奖励，鼓励与对应图像对齐，同时防止不匹配图像。尽管从未见过ICL格式的训练示例，ProCoRe在多模态ICL分类的平均准确率提升了5.6%，ROUGE-L的字幕比现成的Qwen-VL-3-8B模型提升了13.8%，在基准测试中优于依赖ICL数据的先进水平。最后，我们介绍PICL，这是一款个性化的多模态ICL字幕基准，测试MLLM是否真正理解视觉演示，而非依赖预训练语义先验。

PATH: Preference-Aware Traversability Estimation for Robotic Teleoperation in Off-Road Navigation

PATH：非公路导航机器人远程操作的偏好感知可穿越性估计

• Authors: Xianghao Meng, Wenshuo Wang, Cheng Shao, Lewei Feng, Qingxiao Liu, Canyu Wang, Junqiang Xi
• Subjects: Subjects: Robotics (cs.RO); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.30605
• Pdf link: https://arxiv.org/pdf/2610.30605
• Abstract Traversability estimation is critical for ground vehicle teleoperation, yet existing methods often neglect the human operator's preferences for speed and risk tolerance. To address this gap, we propose PATH-Net, a Preference-Aware Traversability estimation Hierarchical Network that adaptively generates preference-conditioned traversability cost maps and speed profiles for off-road teleoperation. Our framework integrates visual and geometric bird's-eye-view features with short-term historical vehicle states through a multi-head shared-backbone architecture. Our PATH-Net infers traversability cost maps while optimizing speed recommendations by using maximum entropy inverse reinforcement learning and sparse speed supervision. Our method incorporates operator preference via an asymmetric feature aggregation mechanism, splitting inference into a cost branch that prioritizes high-risk traversability regions and a speed branch that outputs preference-adjusted recommended speeds. Experiments with 15 operators across 225 teleoperated trajectories show alignment with user-specific navigation strategies, achieving modified Hausdorff distance of 0.17 m (aggressive) and 0.15 m (conservative). In the questionnaire assessment, participants reported lower NASA-TLX workload scores with PATH-Net recommendations than without recommendations, decreasing from 34.7 to 22.2. These results support preference-conditioned trajectory and speed recommendation for off-road teleoperation.
• 中文摘要 可穿越性估算对于地面车辆远程操作至关重要，但现有方法常常忽视人类操作员对速度和风险承受能力的偏好。为弥补这一空白，我们提出了PATH-Net，一种偏好感知可穿越性估算的分层网络，能够自适应生成偏好条件的可穿越性成本图和速度剖面，用于越野远程操作。我们的框架通过多脑共享骨干架构，将视觉和几何鸟瞰图与短期历史车辆状态整合起来。我们的PATH-Net通过最大熵逆强化学习和稀疏速度监督，推断可通行性成本图，同时优化速度推荐。我们的方法通过非对称特征聚合机制整合操作员偏好，将推断分为优先考虑高风险可穿越区域的成本分支和输出偏好调整推荐速度的速度分支。在225条远程操作轨迹中，15名操作员参与的实验显示，符合用户特定的导航策略，实现了0.17米（激进）和0.15米（保守）的修正豪斯多夫距离。在问卷评估中，参与者报告使用PATH-Net推荐后NASA-TLX工作负荷分数低于未推荐，从34.7降至22.2。这些结果支持基于偏好条件的轨迹和速度建议，用于越野远程操作。

Group Relative Policy Optimisation Improves Multilingual Speech Recognition in Low-Resource Languages

群体相对策略优化提升低资源语言中的多语言语音识别能力

• Authors: Tim Wenjie Hu, Evelyn Nafula Ouma, Benjamin Akera, John A. Quinn
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.30663
• Pdf link: https://arxiv.org/pdf/2610.30663
• Abstract Autoregressive encoder-decoder speech recognition models trained with teacher forcing can suffer from exposure bias, limiting their ability to recover from their own prediction errors at inference time. Recent work has shown that reinforcement learning with Group Relative Policy Optimisation (GRPO) improves speech recognition in major languages with LLM-based models. We extend this line of work to smaller, non-LLM encoder-decoder ASR models, using Whisper-large-v3 as a case study across 51 African languages. The benefits of this approach are most pronounced when training data is scarce or when input audio is noisy or degraded. Our resulting model, SunflowerASR, achieves the best published macro-averaged WER on the AfriVox-v2 (0.227) and SimbaBench (0.372) benchmarks, and is publicly available.
• 中文摘要 用教师强制训练的自回归编码器-解码器语音识别模型可能存在暴露偏差，限制其在推断时从自身预测误差中恢复的能力。近期研究表明，基于LLM的组相对策略优化（Group Relative Policy Optimisation，GRPO）强化学习提升了主要语言中的语音识别能力。我们将该研究方向扩展到较小的非LLM编码-解码器ASR模型，以Whisper-large-v3为案例，涵盖51种非洲语言。当训练数据稀缺或输入音频噪声大、质量下降时，这种方法的优势最为明显。我们所得的模型SunflowerASR在AfriVox-v2（0.227）和SimbaBench（0.372）基准测试中实现了最佳已发表的宏平均WER，并已公开使用。

Fast Iterative Combinatorial Auctions via Deep Reinforcement Learning

通过深度强化学习实现快速迭代组合拍卖

• Authors: Luis F. Paul Tosta, Quinn Arnold, Gianluca Brero
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30667
• Pdf link: https://arxiv.org/pdf/2610.30667
• Abstract Iterative combinatorial auctions are used in spectrum allocation, transportation, and industrial procurement, but rounds are costly. We introduce FastCombo, a deep reinforcement learning approach to sequential price discovery. Bayesian pricing has sharply reduced auction rounds even against subgradient benchmarks tuned with hindsight for each instance, but relies on a one-step objective and repeated online inference and optimization. We prove that greedily maximizing immediate clearing probability can require arbitrarily more rounds in expectation than an informative policy, even with the true valuation distribution known. FastCombo learns price policies offline, assigning credit to queries for the future decisions they enable and producing each online price update with a single neural-network forward pass. We formulate the objective of minimizing expected rounds to clearing in single-minded auctions as a partially observable Markov decision process and train a recurrent bidder-item graph policy over complete auction trajectories. In a one-item experiment, all five trained policies learn an informative opening, reducing expected rounds from 1.85 to 1.65 relative to the tested Bayesian one-step rule. Across four domains from the Combinatorial Auction Test Suite, FastCombo achieves competitive clearing while reducing measured online price-selection time per auction by a factor of roughly 300-900 relative to the standard Bayesian baseline. These results demonstrate the value of learning sequential price policies for both information acquisition and online computational efficiency.
• 中文摘要 迭代组合拍卖用于频谱分配、运输和工业采购，但轮次成本较高。我们引入FastCombo，一种深度强化学习方法，用于顺序价格发现。贝叶斯定价即使在事后调优的次梯度基准测试下，也大幅减少了拍卖轮次，但依赖一步目标和反复的在线推断和优化。我们证明，贪婪地最大化即时清算概率可能需要比信息性策略更多的预期轮次，即使真实估值分布已知。FastCombo离线学习价格政策，将其未来决策的查询归功于其实现的未来决策，并通过一次神经网络前向传递生成每次在线价格更新。我们提出将单一竞拍中预期轮次最小化为清算的目标，作为一个部分可观测的马尔可夫决策过程，并训练一个重复竞标者-物品图策略，覆盖完整的拍卖轨迹。在一项实验中，所有五个训练策略都学习了一个有信息量的开局，相较于测试的贝叶斯一步规则，期望轮次从1.85降至1.65。在组合拍卖测试套件的四个领域中，FastCombo实现了竞争清算，同时将每次拍卖的在线价格选择时间比标准贝叶斯基线缩短约300-900倍。这些结果展示了学习顺序价格策略对信息获取和在线计算效率的价值。

An Introduction to World Models for Generalist Agents

通用智能体世界模型导论

• Authors: Zhihui Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30789
• Pdf link: https://arxiv.org/pdf/2610.30789
• Abstract World models have recently emerged as a promising direction toward developing generalist agents capable of understanding, predicting, and interacting with diverse environments. However, despite growing interest, the concept of a world model remains loosely defined, encompassing approaches ranging from predictive dynamics models and learned simulators to models that jointly generate future observations and actions. This introductory paper aims to clarify the concept of world models and provide a unified perspective on their role in developing generalist agents. Starting from the partially observable Markov decision process (POMDP) framework, we connect world modeling to classical control, dynamical systems, signal processing, and reinforcement learning. We introduce different approaches to world modeling, emphasizing the knowledge they capture, their learning objectives, and how they support decision-making. We further explore the relationship between world modeling and policy learning, highlighting how these capabilities can be unified through conditional prediction without requiring separate models. Rather than offering an exhaustive survey of existing methods, this paper focuses on fundamental concepts, modeling principles, and connections, with the goal of providing an accessible and unified introduction to world models for generalist agents.
• 中文摘要 世界模型近年来被视为开发能够理解、预测和与多样环境互动的通用智能体的有前景方向。然而，尽管兴趣日益增长，世界模型的概念仍然定义较为松散，涵盖了从预测动力学模型、学习模拟器到共同生成未来观察和行动的模型等多种方法。本导论旨在澄清世界模型的概念，并统一地了解它们在开发通用智能体中的作用。从部分可观测的马尔可夫决策过程（POMDP）框架出发，我们将世界建模与经典控制、动力系统、信号处理和强化学习相结合。我们介绍了不同的世界建模方法，强调它们所捕捉的知识、学习目标以及如何支持决策。我们进一步探讨了世界建模与政策学习之间的关系，强调这些能力如何通过条件预测实现统一，而无需单独的模型。本文并非对现有方法进行详尽综述，而是侧重于基本概念、建模原则和联系，旨在为通才代理提供一个易于理解且统一的世界模型入门。

Agentic Knowledge-Model Co-Evolution via Consistency Regularization

通过一致性正则化实现的智能知识-模型共进化

• Authors: Sirui Chen, Zhining Liu, Chao Zhang, Jingrui He, Zheng Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30790
• Pdf link: https://arxiv.org/pdf/2610.30790
• Abstract LLM agents can evolve by learning from accumulated experience. Existing approaches extract reusable external knowledge, such as skills and memories, from experience to guide subsequent task solving, with some further internalizing this knowledge through model training. However, different ways of analyzing and transforming the same experience can yield knowledge with substantially different utility, even when the resulting guidance appears reasonable. Inspired by consistency regularization, we identify consistent utility gains across different ways of analyzing the same experience, rather than relying on any single way of analyzing it. We introduce SPLICE, a framework for consistency-regularized knowledge-model co-evolution. It constructs evolution views by perturbing how the same experience is analyzed, producing different knowledge candidates. Knowledge updates are then guided by common gain, which accounts for utility gains and their consistency across views. During model evolution, self-distillation internalizes the evolved knowledge alongside reinforcement learning, using knowledge associated with consistent utility degradation as a contrastive reference. The two processes interleave, with model updates driving further knowledge evolution. Experiments across diverse domains show that SPLICE outperforms knowledge evolution and model training baselines by up to 7.1 and 4.6 points in average accuracy, while its evolved knowledge reduces output tokens per correct solution by up to 44% and transfers across model sizes and families.
• 中文摘要 LLM代理可以通过积累经验来进化。现有方法从经验中提取可重复使用的外部知识，如技能和记忆，以指导后续任务解决，有些方法通过模型训练进一步内化这些知识。然而，不同分析和转化同一经验的方法，即使最终指导看似合理，也可能产生具有显著效用的知识。受一致性正则化启发，我们识别出不同分析同一经验方法的一致性效用增益，而非依赖单一分析方式。我们引入SPLICE，一个一致性正则化知识模型共进化框架。它通过扰动同一经验的分析方式构建进化视图，产生不同的知识候选。知识更新随后由共同收益指导，共同收益考虑效用收益及其在不同视图间的一致性。在模型演化过程中，自我蒸馏将进化后的知识与强化学习相结合，并以与效用退化一致相关的知识作为对比参考。这两种过程交织，模型更新推动了进一步的知识演进。跨多个领域的实验表明，SPLICE在平均准确率上比知识演化和模型训练基线高出最多7.1和4.6分，而其进化后的知识则将每个正确解的输出令牌减少多达44%，并且能跨模型规模和族进行转移。

Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training

Trace2World：从真实交互痕迹重建应用世界，实现可扩展的图形界面代理训练

• Authors: Zezhou Wang, Hanzhang Zhou, Panrong Tong, Quyu Kong, Pengxiang Li, Steven Hoi, Zhuzhong Qian, Yue Wang
• Subjects: Subjects: Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.30792
• Pdf link: https://arxiv.org/pdf/2610.30792
• Abstract Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce \textbf{Trace2World}, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4--16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it.
• 中文摘要 扩展图形界面强化学习需要保留真实应用行为的环境。然而，将行为推断与编码结合，可能会将部分追踪变成可运行的模拟，且状态转换不正确。我们引入了 \textbf{Trace2World}，它将全局行为重建与实现分离。它将跟踪整合为共享的源链应用模型，然后编译可执行环境。独立重放源工作流程本地化不匹配，实现有界实现修复。验证过的世界随后支持源支持行为中的学习者自适应任务和环境增强。从 1,570 条痕迹中，Trace2World 构建了 50 个环境。在十个应用的 200 个工作流程中，91.1% 的端到端重现，而最强匹配的任务-本地基线仅为 59.8%。基于源的验证保留了94.0%的模拟训练收益，而自验证后为54.7%。学习者指导将基准测试成功率从56.9%提升至61.1%。Trace2World训练进一步提升了基准20.2分，四个外部图形界面基准提升5.4-16.2分，100个真实设备任务提升16.3分。可靠的重建将行为源证据支持与实现是否保留行为区分开来。

GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL

GIFT：用于蒸馏增强推理的梯度知情过滤

• Authors: Shijie Li, Yilin Gao, Tieyuan Chen, Yun Luo, Weiyao Lin, Hang Yu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30796
• Pdf link: https://arxiv.org/pdf/2610.30796
• Abstract Training large language models for complex reasoning tasks relies on two complementary signals: sparse, outcome-grounded reinforcement learning (RL) for correctness, and dense, token-level distillation for guidance. However, complementarity in supervision does not guarantee compatibility in optimization; imitation-focused distillation can conflict with the outcome-driven RL objective, and existing hybrids lack a principled resolution. We propose \textbf{GIFT} (\textbf{G}radient-\textbf{I}nformed \textbf{F}il\textbf{T}ering for Distillation-Augmented Reasoning RL), a novel method that directly addresses the allocation problem of which teacher updates to apply. GIFT performs a token-level geometric analysis, decomposing the distillation gradient into a component \textbf{parallel} to the RL gradient, which measures first-order conflict and provides a \textbf{safety} filter, and a component \textbf{orthogonal} to it, which quantifies novel directional guidance and provides a measure of \textbf{value}. This decomposition yields a principled scoring rule to rank token-level guidance. We combine this with a progressive \textbf{coverage annealing} schedule that shrinks the budget of tokens receiving distillation, ensuring that the most conflicting updates are discarded first while the most complementary ones are retained at full strength. This facilitates a smooth transition from dense, geometrically-filtered guidance to a pure, outcome-grounded objective at convergence. Extensive experiments on mathematical and logical reasoning benchmarks show that GIFT consistently outperforms strong baselines across multiple model scales, achieving relative gains of up to 9.5\% on difficult mathematical reasoning tasks. Code: this https URL
• 中文摘要 训练复杂推理任务的大型语言模型依赖两种互补信号：稀疏、基于结果的强化学习（RL）以保证正确性，以及密集的代币级蒸馏作为指导。然而，监督中的互补性并不保证优化兼容性;以模仿为中心的提炼可能与结果驱动的强化学习目标相冲突，且现有混合型缺乏原则性的解决方案。我们提出了 \textbf{GIFT}（\textbf{G}radient-\textbf{I}nformed \textbf{F}textbf{T}ering for Distillation-Augmented Reasoning RL），这是一种新颖的方法，直接解决教师更新应用的分配问题。GIFT执行代币级几何分析，将蒸馏梯度分解为与强化语言梯度平行的\textbf{平行}组成部分，该梯度测量一阶冲突并提供\textbf{safety}过滤器;另一个component为\textbf{orthogonal}，量化新方向指导并度量_textbf{value}。这种分解产生了一个原则性的评分规则，用于对标记级指导进行排序。我们将此与渐进式\textbf{覆盖退火}计划结合，缩减提取代币的预算，确保最冲突的更新优先丢弃，而最互补的更新保持最大强度。这促进了从密集几何滤波指导向纯粹、以结果为基础的目标的平滑过渡。数学和逻辑推理基准测试的广泛实验表明，GIFT在多个模型尺度上始终优于强基线，在困难的数学推理任务中相对提升高达9.5%。代码：此 https URL

DRILL: Training World Models to Improve Policies, Not Predict Pixels

训练：训练世界模型以改进政策，而非预测像素

• Authors: Zhaolu Kang, Tailong Luo, Lei Wei, Guangyuan Dong, Sihan Liu, Yingjie He, Siheng Wang, Shuaibo Li, Rongchao Zhang, Zhen Tian, Richeng Xuan, Zhichao Hu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.30814
• Pdf link: https://arxiv.org/pdf/2610.30814
• Abstract World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically trained to predict pixels one step ahead, even though their role here is not merely to render visually faithful videos but to produce learning signals from imagined rollouts that improve the policy. We show that this mismatch is substantial: lower pixel MSE can produce worse policy gradients, and a world model trained for policy utility can yield better downstream policies despite 17% higher pixel MSE on the next frame. We propose the Downstream-Return Imagination Learning Loop (DRILL), a bilevel framework that updates the world model to maximize the policy's return after an inner GRPO update on imagined rollouts. DRILL has two instantiations. DRILL-VWI is a closed-form surrogate, requiring no additional simulator rollouts, that reweights prediction errors by policy visitation, the group-relative GRPO advantage, and model confidence. DRILL-IMG is a full meta-gradient method that differentiates through the inner update using truncated Hessian-vector products. We show that DRILL-VWI recovers the local one-step component of the DRILL-IMG meta-gradient under score-compatibility and small-residual assumptions. Across five manipulation simulators and two VLA backbones, DRILL-IMG raises mean success rate by 13.1 points over WMPO and 7.9 points over RLVR-World, and complements RLVR pretraining rather than competing with it. More broadly, we believe video world models for VLA reinforcement learning can be productively trained by the policy gradients they produce, not only by pixel-level prediction.
• 中文摘要 世界模型越来越多地用于减少与环境交互的过程，尤其是在强化学习的视觉-语言-行动（VLA）策略训练后。然而，它们通常被训练为提前预测像素，尽管它们的作用不仅仅是呈现视觉忠实的视频，而是从想象中的推展中产生改进策略的学习信号。我们证明这种不匹配存在显著性：较低的像素MSE可能导致更差的策略梯度，而训练为策略效用的世界模型即使下一帧像素MSE高出17%，也能产生更好的下游策略。我们提出了下游-返回想象学习循环（DRILL），这是一个双层级框架，在想象的推展后，更新世界模型以最大化策略的回报。DRILL有两个实例化。DRILL-VWI是一种封闭形式替代，无需额外的模拟器部署，通过策略访问、组相对GRPO优势和模型置信度重新加权预测误差。DRILL-IMG是一种全元梯度方法，通过内部更新通过截断黑森向量积进行差异。我们证明DRILL-VWI在分数兼容性和小残差假设下恢复了DRILL-IMG元梯度的局部一步分量。在五个操作模拟器和两个VLA骨干中，DRILL-IMG的平均成功率比WMPO提高了13.1个百分点，比RLVR-World提高了7.9个百分点，并且是与RLVR预训练的补充，而非竞争。更广泛地说，我们认为视频世界模型用于VLA强化学习，可以通过其产生的策略梯度有效训练，而不仅仅是像素级预测。

RECAP: Looking Once Is Not Enough for Vision-Language Reasoning

回顾：仅仅看一次还不够进行视觉-语言推理

• Authors: Zhaolu Kang, Tailong Luo, Chenxin Li, Yingjie He, Zhenyu Yu, Fengyu Zhou, Jiachen Qian, Lei Wei, Shuang Chen, Jiachen Li, Eric Hanchen Jiang, Rongchao Zhang, Zhengtao Yao, Hoi Leong Lee, Guansu Wang, Kaiyue Zhou
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.30834
• Pdf link: https://arxiv.org/pdf/2610.30834
• Abstract Long chain-of-thought reasoning improves vision-language models (VLMs), but it also exposes a temporal grounding failure: as generation unfolds, models may drift from image evidence while reinforcement learning still provides only a final-answer reward. We study this gap through visual revisit, a temporally localized reactivation of image evidence during reasoning. Across three VLM backbones, revisit peaks are predictive of correctness and causally linked to performance: masking them degrades accuracy more than masking non-peaks, while injecting revisit-like peaks into failed rollouts partially restores correct answers. Control analyses show that this signal is not explained by raw attention strength, uncertainty, logit margin, or hidden-state norm. We propose RECAP, a GRPO-compatible training method that turns detached revisit traces into a credit-assignment signal. RECAP uses revisit-conditioned advantage estimation, dynamic discounting, and gated value prediction to propagate final rewards through visually important reasoning steps, without extra rewards, additional rollouts, or inference-time changes. Across Qwen2.5-VL-7B, InternVL3-8B, and LLaVA-OV-7B, RECAP consistently improves over GRPO and VPPO, yielding $+6.1$--$+6.5$ HallusionBench gains over the SFT base while preserving reasoning and text-only performance. Results on larger LoRA-tuned models, Chinese zero-shot benchmarks, visual perturbations, and human evaluation further demonstrate robust improvements in image-grounded reasoning.
• 中文摘要 长链思考推理提升视觉语言模型（VLM），但也暴露出时间基础失败：随着生成展开，模型可能偏离图像证据，而强化学习仍仅提供最终答案奖励。我们通过视觉重访研究这一空白，即推理过程中图像证据的局部时间激活。在三种VLM骨干中，重访峰值预测正确性并与表现因果相关：掩盖它们比掩蔽非峰值更降低准确性，而在失败的展开中注入类似重访峰值则部分恢复正确答案。对照分析显示，该信号无法由原始注意力强度、不确定性、logit余裕或隐藏状态规范来解释。我们提出了RECAP方法，这是一种与GRPO兼容的训练方法，将脱离的重访痕迹转化为学分分配信号。RECAP利用重访条件优势估计、动态贴现和门控价值预测，通过视觉上重要的推理步骤传递最终奖励，无需额外奖励、额外推广或推理时间变化。在Qwen2.5-VL-7B、InternVL3-8B和LLaVA-OV-7B中，RECAP相较GRPO和VPPO持续提升，在保持推理和纯文本性能的同时，HallusionBench相较SFT基础提升$+6.1$-$+6.5$。在更大型LoRA调优模型、中国零样本基准测试、视觉扰动和人类评估中的结果进一步显示了图像基准推理的强劲提升。

MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback

MOER：专家前缀指导下的混合策略优化，含选择性回滚

• Authors: Xuyang Zhong, Jiuding Yang, Yiwen Guo, Chen Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30873
• Pdf link: https://arxiv.org/pdf/2610.30873
• Abstract Mix-policy optimization incorporates off-policy expert data into the reinforcement learning (RL) loop to transcend the capability boundary of on-policy RL. However, existing methods either provide static full-trace guidance that suppresses self-exploration, or adopt expert prefix guidance without correctness guarantees, leading to suboptimal performance or even training collapse. In this paper, we propose MOER, a mix-policy optimization framework that harmonizes expert guidance with self-exploration. MOER injects a decaying expert chain-of-thought prefix into the prompt and lets the model complete the remaining reasoning, with the prefix length progressively annealed to encourage self-exploration. A selective rollback mechanism replaces failed completions with the original full expert trace, guaranteeing the correctness of every off-policy trajectory. Furthermore, a harmonized mix-policy objective optimizes on-policy tokens with a GRPO loss while supervising both expert prefix and completion tokens with a trust-region SFT loss, substantially stabilizing training. Extensive experiments show that MOER achieves the highest average scores in mathematical and general domain reasoning tasks, outperforming all basic and mix-policy baselines, and generalizes consistently across diverse tasks, model scales, and base model categories. Codes are available at this https URL.
• 中文摘要 混合策略优化将非策略专家数据纳入强化学习（RL）循环，以超越策略上强化学习的能力边界。然而，现有方法要么提供静态的全追踪指导，抑制自我探索，要么采用专家前缀指导但没有正确性保证，导致性能不优甚至训练崩溃。本文提出MOER，一种混合策略优化框架，协调专家指导与自我探索。MOER在提示中注入衰减专家思维链前缀，让模型完成剩余推理，前缀长度逐步退火以鼓励自我探索。选择性回滚机制用原始完整专家追踪替代失败完成，保证每个非策略轨迹的正确性。此外，统一混合策略目标在管理带有 GRPO 损失的策略上令牌优化，同时监督专家前缀和完成令牌的信任区域 SFT 丢失，从而大幅稳定训练。大量实验表明，MOER 在数学和一般领域推理任务中获得最高平均分，优于所有基础和混合策略基线，并在多样化任务、模型尺度和基模型类别中持续推广。代码可在此 https 网址获取。

Inherited, Not Invented: Tracing the Provenance of Wait in Reasoning Models

继承而非发明：追溯推理模型中等待的起源

• Authors: Zhizhuo Kou, Sizai Hou, Long Chen, Jian Yang, Junyu Luo, Jingshu Peng, Wenkai Zhang, Sirui Han, Yike Guo
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.30882
• Pdf link: https://arxiv.org/pdf/2610.30882
• Abstract Reinforcement learning (RL) is credited with teaching reasoning models to write {wait} before reconsidering a step, yet large RL accuracy gains do not require the word. This view can confuse what RL discovers with what a model inherits from its training data. We trace the word through open models' training histories, reading it with label-free indices of frequency, position, form and usage and an injected-error test, all computed from sampled outputs. We find that zero-RL almost never coins the marker: on bases that lack the word, RL raises MATH-500 accuracy substantially while {wait} stays near absent. It arrives with training data: mid-training in OLMo-3, and fine-tuning on reasoning traces installs it in most models whose base lacks it. Where the base already uses it, both released RL-Zero recipes (OLMo-3, MiMo) multiply it, whereas KL-free GRPO from the same base does not; OLMo-3's recipe wears down its form and makes it a more sensitive but more liberal alarm; RL after supervised fine-tuning barely changes it, and our GRPO recipe erases a marker seeded by fine-tuning. Finally, we show that marker frequency alone does not indicate RL-induced reasoning and caution against LLM-as-a-judge revision metrics, as automated judges can reverse human-labelled trends.
• 中文摘要 强化学习（RL）被认为教会推理模型在重新考虑步骤前先写{wait}，但大幅提升的强化学习准确率并不需要该词。这种观点可能混淆强化学习发现的内容与模型从训练数据继承的内容。我们通过开放模型的训练历史追踪词，使用无标签的频率、位置、形式和用法索引，以及注入错误测试，所有这些都是从抽样输出计算出来的。我们发现零强化学习几乎从不创造该标记：在缺乏该词的基底上，强化学习显著提升了MATH-500的准确率，而{wait}几乎没有。它伴随着训练数据而来：在OLMo-3中训练中途，推理痕迹的微调使它安装在大多数缺乏该词的基础模型中。在基底已有使用的情况下，发布的RL-Zero配方（OLMo-3、MiMo）都会乘以该频率，而同一基底的无KLGRPO则不会;OLMo-3的配方会削弱其形式，使其更灵敏但更宽松;经过监督微调后的RL几乎不改变它，而我们的GRPO配方则擦除了通过微调植入的标记。最后，我们表明仅凭标记频率并不表明强化学习引发的推理和对LLM作为评判的修正指标的谨慎，因为自动评判可以逆转人工标记的趋势。

Building Presentation Intelligence for High-Quality Slide Generation

构建高质量幻灯片生成的演示智能

• Authors: Yuhang Xie, Cheng Li, Chenliang Li, Mingyu Xu, Mingwei Sun, Suhang Zheng, Ming Yan, Xiaoxiao Xu, Lin Qu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30888
• Pdf link: https://arxiv.org/pdf/2610.30888
• Abstract PowerPoint presentations (PPTs) are a dominant medium for communication across diverse domains, yet high-quality PPT generation still depends heavily on labor-intensive human effort. Recent advances in large language models (LLMs) for agentic tasks have created both new opportunities and challenges (Chen et al., 2026) for automatic PPT generation. In this work, we present a unified training framework for automatic PPT generation toward presentation intelligence, integrating cross-validated data synthesis, guided teacher enhancement, and verifiable reward optimization. We first construct a cross-validated pipeline for data synthesis, with explicit control over both difficulty and validity. We then develop a teacher-enhancement framework grounded in rules and prior knowledge to substantially strengthen the instructional capacity of teacher models. Building on this, we further improve supervised fine-tuning (SFT) through a mixed-teacher strategy and a lightweight outcome-driven trajectory filtering method. Finally, based on the resulting SFT models, we introduce a suite of verifiable aesthetic metrics together with a GRPO-based reinforcement learning framework to optimize slide-level visual attributes beyond text-only supervision. Experiments show that our enhanced SFT pipeline improves Qwen-3.6-27B and Qwen-3.6-35B-A3B by 7.45 and 9.35 points on PresentBench, respectively, surpassing GLM-5.2 while maintaining strong generalization. Reinforcement learning further improves performance across five aesthetic dimensions, compensating for the limitations of SFT in visual refinement. Overall, the results demonstrate the effectiveness of our framework for high-quality PPT generation.
• 中文摘要 PowerPoint演示文稿（PPTs）是跨多个领域沟通的主要媒介，但高质量的PPT生成仍高度依赖于劳动密集型人力劳动。大型语言模型（LLMs）在代理任务中的近期进展，为自动PPT生成带来了新机遇和挑战（Chen 等，2026）。本研究提出了一个统一的自动PPT生成培训框架，旨在实现演示智能，集成交叉验证数据综合、指导教师提升和可验证的奖励优化。我们首先构建了一个交叉验证的数据综合流程，明确控制难度和有效性。随后，我们基于规则和先验知识开发教师增强框架，以显著增强教师模型的教学能力。基于此，我们通过混合教师策略和轻量级成果驱动轨迹过滤方法进一步改进监督微调（SFT）。最后，基于所得SFT模型，我们引入了一套可验证的美学指标，结合基于GRPO的强化学习框架，优化幻灯片级视觉属性，超越仅文本的监督。实验显示，我们增强的SFT流水线在PresentBench上分别提升了Qwen-3.6-27B和Qwen-3.6-35B-A3B的7.45分和9.35分，超越GLM-5.2，同时保持了强有力的泛化能力。强化学习进一步提升了五个美学维度的性能，弥补了SFT在视觉细化方面的局限性。总体而言，结果证明了我们框架在高质量PPT生成方面的有效性。

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

ReflectPO：面向长期视野LLM代理的反思引导策略优化

• Authors: Rumei Li, Xunliang Cai
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.30933
• Pdf link: https://arxiv.org/pdf/2610.30933
• Abstract With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent--child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.
• 中文摘要 借助外部工具，大型语言模型（LLMs）可以通过持续的环境互动来处理复杂任务。因此，基于LLM的代理越来越多地被用于处理现实世界中复杂的长视野任务。然而，这些任务涉及顺序且相互依赖的决策和环境状态的变化。早期执行错误会影响后续的交互历史和环境状态，其影响会在决策链中积累。为应对这一挑战，我们提出了反思引导策略优化（ReflectPO），这是一种用于长期任务自主错误恢复的强化学习框架。ReflectPO包含两个核心组件。第一个是反射引导采样。我们利用分支差异和交互轨迹中的重复访问构建错误恢复的提示。第二个是树结构化的学分分配。我们将抽样轨迹组织成树状结构，并利用父子奖励差异和恢复收益分配行动层级的信用。两个长期工具使用基准测试的实验显示，ReflectPO在大多数领域中优于多个现有基线。它也实现了平均最佳的整体表现。进一步分析显示，ReflectPO有效提升了代理识别自身偏差、从错误状态中恢复以及持续推进任务完成的能力。

Verbalizing Activation Differences with Natural Language Autoencoders

用自然语言自编码器进行口语激活差异

• Authors: Andrea Blasi Núñez, Gianluca Barmina, Federico Torrielli, Thiusius Rajeeth Savarimuthu, Peter Schneider-Kamp, Lukas Galke Poech
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.30934
• Pdf link: https://arxiv.org/pdf/2610.30934
• Abstract Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.
• 中文摘要 激活语言器将大型语言模型的内部表征翻译成人类可读文本。最近提出的自然语言自编编码器（NLA）通过通过自然语言瓶颈重建激活来学习这一点。然而，这些方法在训练环境之外的推广效果尚不明确。在这里，我们将围绕四个密切相关的问题研究NLA的问题：（i）NLA的语言表达是否在输入中表现出组合结构？（ii）NLA是否会推广到其训练模型的微调版本？（iii）NLA能否恢复模型微调目标？（iv）NLA能否专门用于模型差异？在三个模型家族的四个模型中，我们的实验提供了支持这四个问题总体上积极答案的证据。具体来说，合成类比激活的口头化大多只标明目标，很少提及减去词，NLA可以在其原始模型的轻量级微调上重用而无需重新训练，仅最大模型的重建质量会明显下降。口头化Taboo模型生物与其原始模型之间的激活差异，可以恢复大多数生物体隐藏的秘密词，而对激活差异的强化学习进一步提升了对被保留生物的恢复率。我们还发现，NLA可以在微调模型不生成任何令牌的情况下恢复微调目标，因为最具信息量的令牌位置先于用户提示。NLA尚未达到最先进的模型差分方法，但表现优于黑箱基线。我们的发现表明，NLA能够有意义地在激活空间中表达差异，使NLA成为未来研究的有前景方向。

The Geometry of Meaning in Neural Manifolds

神经流形中的意义几何

• Authors: Valentino Maiorca
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30938
• Pdf link: https://arxiv.org/pdf/2610.30938
• Abstract Just as different languages express the same concept through different words, neural networks express similar functions through incompatible coordinate systems. Different architectures, random initializations, and optimization trajectories produce incompatible parameterizations in extrinsic coordinates, obscuring functional relationships between independently trained models. When models independently solve similar tasks on related data, do their learned representations share exploitable structure despite incompatible coordinates? This thesis shows that semantic correspondence is a fundamental primitive revealing shared geometric structure beneath incompatible parameterizations. By identifying which elements refer to the same semantic content, correspondence breaks the symmetry of arbitrary implementation choices, anchoring incompatible systems to a shared semantic frame and revealing the intrinsic geometry that enables zero-shot transformation between independently trained models. Building on this principle, fourteen contributions explore correspondence-based alignment across zero-shot model stitching, cross-modal translation, policy transfer, activation steering, and neural decoding. They span domains (vision, language, reinforcement learning, neuroscience), architectures (CNNs, Transformers, graph networks), and scales (from small supervised models to billion-parameter generative systems). Each exploits different geometric properties (angular, metric, spectral, sparse, or distributional), yet all instantiate the same principle. It extends beyond artificial systems: fMRI data show that different human brains processing identical stimuli exhibit alignable geometric structure. By revealing intrinsic geometric structure beneath extrinsic parameterization, semantic correspondence transforms model isolation into interoperability.
• 中文摘要 正如不同语言通过不同词汇表达同一概念，神经网络通过不兼容的坐标系统表达类似功能。不同的架构、随机初始化和优化轨迹会导致外部坐标中的参数化不兼容，模糊了独立训练模型之间的功能关系。当模型独立解决相关数据的类似任务时，其学习到的表示是否在坐标不兼容的情况下共享可利用的结构？本论点表明语义对应是一种基本的原始，揭示了不兼容参数化下共享的几何结构。通过识别哪些元素对应相同的语义内容，对应打破了任意实现选择的对称性，将不兼容系统锚定在共享语义框架上，揭示了实现独立训练模型之间零样本转换的内在几何结构。基于这一原则，十四篇论文探讨了零样模型拼接、跨模态翻译、策略转移、激活引导和神经解码等基于对应的对齐。它们涵盖了视觉、语言、强化学习、神经科学）、架构（卷积神经网络、变换器、图网络）和尺度（从小型监督模型到数十亿参数生成系统）。每种都利用不同的几何属性（角度、度量、谱、稀疏或分布），但都体现了同一原理。它超越了人工系统：fMRI数据显示，处理相同刺激的不同人脑表现出可对齐的几何结构。通过揭示外在参数化下的内在几何结构，语义对应将模型孤立转化为互操作性。

Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks

通过逆最优性对强化学习进行基于地面真实的基准测试：从平滑、完全观察的系统到非平滑或部分观察任务

• Authors: Grégoire Ouerdane, Sinan Ibrahim, Stefan Streif, Pavel Osinenko
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30954
• Pdf link: https://arxiv.org/pdf/2610.30954
• Abstract Benchmarking in reinforcement learning is usually relative: algorithms are ranked by returns on environments with unknown optima. Distance to optimality remains unobservable, while conclusions may depend on reward design, stochasticity, hyperparameters, environment selection, and simulator details. We develop a ground-truth-first framework based on converse optimality, constructing each task with a certified optimal value function and policy withheld from learners and used only for evaluation. The framework unifies two modes: sensitivity benchmarking samples certified environments from continuous-time generators to assess robustness to task variation, while oracle-referenced comparison uses native discrete-time generators to evaluate policies via optimality gap, regret, and policy error. Continuous-time models support physically interpretable sensitivity studies but introduce dependence on the simulator step ($\Delta t$) and numerical scheme. Native discrete-time models align the environment, oracle, and learner on the same transition law, eliminating external integration choices as a confounder. We reproduce quantitative results covering optimizer sensitivity and oracle-referenced comparisons on serial planar arms, dynamically extended nonholonomic vehicles, and quadrotor-inspired tasks. The methodology extends beyond smooth affine Gaussian systems: virtual-input pullback handles non-affine raw-action interfaces and finite-covariance non-Gaussian disturbances. A non-smooth converse optimality formulation enables certified benchmarks with kinked value functions and switching surfaces. For partially observed evaluation, an information-state formulation reduces benchmarking to converse optimality on a sufficient statistic and provides an information-penalty reference. Code, fixtures, and diagnostics are available on the public project page.
• 中文摘要 强化学习中的基准测试通常是相对的：算法通过对未知最优环境的回报进行排名。最优距离仍不可观察，而结论可能取决于奖励设计、随机性、超参数、环境选择和模拟器细节。我们基于逆最优性开发了一个以真实为先的框架，构建每个任务的经过认证的最优价值函数和策略，这些策略对学习者保密，仅用于评估。该框架统一了两种模式：敏感度基准测试通过连续时间生成器采样认证环境，以评估任务的鲁棒性与变异性;而oracle参考比较则使用原生离散时间生成器，通过最优差距、遗憾和策略误差来评估策略。连续时间模型支持物理可解释的敏感性研究，但引入了对模拟器步骤（$\Delta t$）和数值方案的依赖。原生离散时间模型将环境、预言机和学习者对齐在同一过渡律上，消除了作为混杂因素的外部积分选择。我们重现了涵盖序列平面臂、动态扩展非全息载具以及四旋翼启发任务的优化器灵敏度和预言机参考比较的定量结果。该方法论超越了光滑仿射高斯系统：虚拟输入拉回处理非仿射原始作用界面和有限协方差非高斯扰动。非光滑逆最优表述使得带有折曲值函数和切换曲面的认证基准测试成为可能。对于部分观测评估，信息状态表述将基准测试降低为在足够统计量上的逆最优性，并提供信息惩罚参考。代码、夹具和诊断可在公开项目页面获取。

Revisiting Distribution Matching for State-to-Vision Policy Distillation in Laparoscopic Robotic Manipulation

重新审视腹腔镜机器人操作中状态到视觉政策提炼中的分布匹配

• Authors: Mingyu Sheng, Jianan Fan, Dongnan Liu, Guoyan Zheng, Ron Kikinis, Weidong Cai
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.31010
• Pdf link: https://arxiv.org/pdf/2610.31010
• Abstract End-to-end control policies trained via Reinforcement Learning (RL) have emerged as an alternative to conventional surgical robotic control approaches, with recent studies exploring state-based policies trained on privileged state observations. However, such approaches rely on precise physical states that are difficult to obtain reliably in surgical environments. Vision-based RL provides an alternative by learning manipulation policies directly from RGB observations, avoiding explicit state estimation and manually engineered perception pipelines while enabling policies to acquire task-relevant visual representations. Nevertheless, obtaining vision-based policies via pure RL methods for surgical manipulation remains challenging. Thus, imitation learning (IL) becomes another solution, yet it requires a large number of expert demonstrations, which are difficult to obtain in minimally invasive surgery. In this work, we propose a framework for state-to-vision policy distillation in laparoscopic robotic manipulation. Specifically, a state-based policy with privileged observations is employed as the teacher, and a vision-based policy learns from RGB observations and robot proprioceptive information, which requires less expert demonstrations than IL and introduces superior performance than RL methods. We conduct extensive experiments on various surgical manipulation tasks in the simulated environment. Experimental results show that, in the context of laparoscopic robotic manipulation, training vision-based policies with general-purpose RL methods is challenging, whereas our proposed state-to-vision policy distillation leads to more consistent and stable performance across tasks.
• 中文摘要 通过强化学习（RL）训练的端到端控制策略已成为传统手术机器人控制方法的替代方案，近期研究探讨了基于特权状态观察训练的基于状态的策略。然而，此类方法依赖于精确的物理状态，而这些在手术环境中难以可靠获得。基于视觉的强化学习提供了替代方案，通过直接从RGB观察学习操作策略，避免了显式状态估计和人工工程的感知流水线，同时使策略能够获得与任务相关的视觉表征。然而，通过纯强化学习方法获得基于视觉的策略进行手术操作仍然具有挑战性。因此，模仿学习（IL）成为另一种解决方案，但需要大量专家演示，而这些演示在微创手术中较为困难。本研究提出了一个用于腹腔镜机器人操作中状态到视觉策略提炼的框架。具体来说，采用基于状态的策略和特权观察作为教师，基于视觉的策略则从RGB观察和机器人本体感觉信息中学习，这比IL需要的专家演示更少，且性能优于强化学习方法。我们在模拟环境中对各种外科操作任务进行了大量实验。实验结果表明，在腹腔镜机器人操作的背景下，用通用强化学习方法训练基于视觉的策略具有挑战性，而我们提出的状态到视觉策略提炼则能实现更一致和稳定的任务表现。

CARL: Chat Assisted Reinforcement Learning

CARL：聊天辅助强化学习

• Authors: Sanjiv Das, Harshad Khadilkar, Sukrit Mittal, Daniel Ostrov, Jacob Sisk
• Subjects: Subjects: Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.31014
• Pdf link: https://arxiv.org/pdf/2610.31014
• Abstract Failure of applied Reinforcement Learning (RL) to solve a given problem predominantly stems from unaddressed specification gaps, where the intent of human experts in the problem's domain is inaccurately or incompletely translated by AI into mathematical objectives, constraints, and reward functions. Failure can also stem from unaddressed performance gaps, where human domain experts are unable to determine the validity of the RL based solution or where parameters (i.e., hyperparameters or weights on the reward terms comprising the reward) are poorly chosen. Chat Assisted Reinforcement Learning (CARL) bridges these gaps through a human-AI co-design interface architecture that enforces interactive specification prior to code generation and then enables subsequent verification and parameter alterations via iteration. More specifically, CARL leads domain experts through a structured ten-step workflow for each iteration using step-specific skill files, domain-aware suggestions via an assistant persona (called Trillian), and compact decision summaries that maintain context across iterations without context overload. Once the domain expert confirms the Markov decision process (MDP) specified by this procedure, CARL's autonomous tool-use loop generates, tests, executes, and debugs an end-to-end RL training pipeline, followed by an interactive post-training diagnosis and revisit step that allows for verification and parameter alteration before running a new iteration if desired.
• 中文摘要 应用强化学习（RL）未能解决给定问题，主要源于未被解决的规范漏洞，即人工智能将该领域人类专家的意图不准确或不完整地转化为数学目标、约束和奖励函数。失败也可能源于未解决的性能差距，即人类领域专家无法判断基于强化学习方案的有效性，或参数（即奖励项的超参数或权重）选择不当。聊天辅助强化学习（CARL）通过人机协设计接口架构弥合这些空白，该架构在代码生成前强制执行交互式规范，然后通过迭代实现后续验证和参数修改。更具体地说，CARL通过结构化的十步工作流程引导领域专家，使用步骤特定技能文件、通过助手角色（称为Trillian）提供领域感知建议，以及紧凑的决策摘要，确保跨迭代保持上下文且不造成上下文过载。一旦领域专家确认该过程指定的马尔可夫决策过程（MDP），CARL的自主工具使用循环将生成、测试、执行和调试端到端的强化学习训练流水线，随后进行交互式训练后诊断和重访步骤，允许在运行新迭代前进行验证和参数调整。

Stability-Preserving Reinforcement Learning for Heterogeneity-Aware Composite Max-Pressure Traffic Signal Control

稳定性保持强化学习，用于异质感知复合最大压力交通信号控制

• Authors: Jichen Zhu, Chaopeng Tan
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.31037
• Pdf link: https://arxiv.org/pdf/2610.31037
• Abstract Reinforcement learning (RL) provides a flexible mechanism for adapting traffic signal control to heterogeneous traffic and sensing conditions, but directly learned signal actions do not automatically inherit the analytical properties of structured traffic-control methods. This paper proposes Com-MP, a theory-guided RL framework that learns a context-dependent composition of multiple max-pressure (MP) control preferences while retaining decentralized MP phase selection. The learned policy adapts the relative influence of constituent MP formulations according to local traffic and sensing conditions, allowing data-driven adaptation to operate within a structured traffic-control framework. We show that such intersection-specific and time-varying composition preserves strong queue stability for every demand in the strictly stabilizable region under any admissible non-anticipative composition policy, thereby separating the performance-adaptation role of learning from the stability property of the underlying controller. In a representative implementation combining detector-supported queue-based MP and connected-vehicle-supported travel-time MP, experiments on a real-world 45-intersection network show that Com-MP reduces average delay by 15.6\% and 8.6\% relative to the two constituent controllers, respectively, and by about 7.0\% relative to the best fixed composition. The learned composition responds systematically to local traffic and sensing conditions, remains effective under heterogeneous and degraded sensing, and can be transferred without retraining to an unseen network with an average delay only 2.4\% higher than that of a policy trained directly on the target network. These results show how learning-based adaptation can be introduced within a stability-preserving traffic-control structure, providing a theory-guided pathway toward trustworthy learning-enabled traffic signal control.
• 中文摘要 强化学习（RL）提供了一种灵活机制，用于适应异构交通和传感条件，但直接学习的信号动作并不自动继承结构化交通控制方法的分析特性。本文提出了Com-MP，一种理论引导的强化学习框架，能够学习多种最大压力（MP）控制偏好的上下文相关组合，同时保留分散式的MP阶段选择。该策略根据本地交通和传感条件调整组成MP表述的相对影响，使数据驱动的适应能够在结构化的交通控制框架内运行。我们证明，这种路口特定且时间变化的组合在任何允许的非预期组合策略下，能为严格稳定区域内的每个需求保持强的队列稳定性，从而将学习的性能适应角色与底层控制器的稳定性分离开来。在一种结合了探测器支持队列的多元存储器和连接车辆支持的行车时间多元的代表性实现中，现实世界45交叉点网络上的实验显示，Com-MP相对于两个组成控制器分别将平均延迟降低了15.6%和8.6%，相对于最佳固定组合减少约7.0%。学习到的组合能够系统地响应本地交通和传感条件，在异构且降级的传感下依然有效，并且可以在不重新训练的情况下转移到一个未被看见的网络，平均延迟仅比直接在目标网络上训练的策略高2.4/%。这些结果表明，基于学习的适应可以在保持稳定性的交通控制结构中引入，为实现可信的学习驱动交通信号控制提供了理论引导的路径。

Cross-Region Cooperation of UAV Swarms Towards Air-Ground ISAC via Hierarchical Multi-Agent Reinforcement Learning

跨区域无人机协作，通过分层多智能体强化学习，向空地ISAC推进

• Authors: Linghui Miao, Nhan Thanh Nguyen, Shijian Gao
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.31054
• Pdf link: https://arxiv.org/pdf/2610.31054
• Abstract The limited communication resources and sensing perspectives of a single uncrewed aerial vehicle (UAV) motivate UAV swarms for integrated sensing and communication (ISAC) in air-ground networks. However, in multi-region scenarios, regional UAV allocations may not match vehicle demand, while local observations may lack sufficient diversity for accurate sensing. To address these issues, this paper proposes a cross-region cooperation framework for UAV swarms. For communication, cross-region UAV reallocation reduces service disparities by matching resources to vehicle demand. For sensing, combining UAV observations from different regions increases observation diversity, while adaptive handshaking guided by marginal Cramér--Rao bound (CRB) utilities balances phase synchronization gains against handshaking energy consumption. Joint optimization of UAV positions and transmit power within each region then further improves communication service and sensing performance. These coupled decisions are coordinated by a hierarchical multi-agent reinforcement learning (H-MARL) approach through slow-timescale inter-region UAV reallocation and fast-timescale intra-region UAV control. Under the considered simulation settings, reallocation can improve quality of service (QoS) by over 20 percentage points compared with retaining the initial allocation under severe resource-demand mismatches. Cooperative sensing with adaptive handshaking incurs only about a 5.0% increase in mean CRB relative to perfect synchronization.
• 中文摘要 单一无人机（UAV）有限的通信资源和传感视角促使无人机群用于空地网络中的综合感测与通信（ISAC）。然而，在多区域场景中，区域无人机分配可能无法匹配车辆需求，而局部观测可能缺乏足够的多样性以实现准确感测。为解决这些问题，本文提出了一个跨区域无人机群合作框架。在通信方面，跨区域无人机重新分配通过匹配资源与车辆需求来减少服务差异。在传感方面，结合不同区域的无人机观测数据可增加观测多样性，而由边际Cramér-Rao束缚（CRB）公用设施引导的自适应握手则平衡相位同步收益与握手能耗。在各区域内联合优化无人机位置和发射功率，进一步提升通信服务和传感性能。这些耦合决策通过分层多智能体强化学习（H-MARL）方法协调，通过慢时间尺度的区域间无人机重新分配和快速时间尺度的区域内无人机控制。在考虑的模拟设置下，重新分配相比在严重资源需求不匹配下保留初始分配，可提升服务质量（QoS）超过20个百分点。配合自适应握手的协同感测相比完美同步仅使平均CRB增加约5.0%。

HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

HiSTEVE 1.0：利用人机界面VLA代理进行开放世界游戏

• Authors: Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.31162
• Pdf link: https://arxiv.org/pdf/2610.31162
• Abstract Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.
• 中文摘要 构建像人类玩家一样在开放游戏中运作的虚拟具身代理，是一个长期目标，需要既具备长视野自主性，又需要可靠的原生接口控制。作为一个程序生成的开放世界，Minecraft为研究这些能力整合提供了自然的试验平台。现有代理通常要么提供原生视觉运动控制，无需长期自主性，要么通过环境API或预定义技能执行高层规划。为弥合这一差距，我们提出了HiSTEVE，一种具有能力感知代理的分层VLA代理。在编排层面，我们为框架配备了结构化内存以跟踪进度，并配备了VLM规划器，用于选择VLA能力范围内的子任务。为可靠地确定子任务完成度，我们进一步引入了策略耦合的任务完成状态检测器，根据VLA策略上下文进行条件，以指示候选任务完成。在控制层，单个VLA以仅视觉感知和原生键鼠操作执行所有子任务。为减少异构GUI与非GUI交互之间的干扰，我们引入了特定状态的RMSNorm分支和独立的鼠标移动令牌嵌入。我们还通过世界知识注入、模仿学习和环境反馈强化学习训练VLA，以提高可靠性和恢复能力。HiSTEVE实现了最先进的原子任务性能，在MineStudio Bench上得分85.0%，OpenHA平均成功率为37.4%，同时还能完成如从零制作铁镐等长远任务，就像人类玩家一样。

Reward-Weighted Preference Optimization for Selective Combinatorial Optimization Problems

选择性组合优化问题的奖励加权偏好优化

• Authors: Haohui Zhang, Xinyu Hu, Wouter van Heeswijk, Neil Yorke-Smith, Martijn Mes
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31239
• Pdf link: https://arxiv.org/pdf/2610.31239
• Abstract Reinforcement learning has emerged as an effective approach for combinatorial optimization, but most existing methods focus on problems such as the Travelling Salesperson Problem (TSP) in which all elements must be served. Selective combinatorial optimization problems (SCOPs), including the Orienteering Problem (OP), Prize-Collecting TSP (PCTSP), Stochastic PCTSP and Knapsack Problem (KP), additionally require selecting which elements to include. This selection creates sampled solutions with substantially different objective values, yet standard preference optimization reduces such comparisons to binary rankings and discards the magnitude of these differences. We introduce Reward-Weighted Preference Optimization (RWPO), a pairwise training objective that incorporates reward-gap information into preference learning. For candidate trajectories sampled from the same instance, RWPO weights each preference loss according to the normalized difference in their rewards, assigning greater training emphasis to comparisons that represent larger improvements in solution quality. We further combine RWPO with scale-specific fine-tuning and reward- and likelihood-weighted tilted sampling to adapt trained policies to larger and shifted instance distributions without relying on local-search post-processing. Experiments on the OP, PCTSP, Stochastic PCTSP, and KP show that RWPO consistently improves over unweighted preference optimization and policy-gradient baselines across a wide range of instance sizes. On the OP, the resulting solver exceeds the mean reward of the Compass metaheuristic at scales from 100 to 2,000 nodes. The results demonstrate that preserving reward-gap information can substantially improve preference-based learning for selective combinatorial optimization and support generalization to large-scale instances.
• 中文摘要 强化学习已成为组合优化的有效方法，但大多数现有方法仍关注诸如必须满足所有元素的旅行销售员问题（TSP）等问题。选择性组合优化问题（SCOP），包括定向障碍问题（OP）、奖品收集TSP（PCTSP）、随机PCTSP和背包问题（KP），还需要选择包含哪些元素。这种选择产生了具有显著不同目标值的抽样解，但标准偏好优化将此类比较简化为二元排名，并摒弃了差异的大小。我们引入奖励加权偏好优化（RWPO），这是一种两对训练目标，将奖励差距信息纳入偏好学习。对于从同一实例抽样的候选轨迹，RWPO根据奖励的归一化差异加权每个偏好损失，给予更大训练重视代表更大解质量提升的比较。我们还进一步将RWPO与尺度特定微调以及奖励加权及似然加权倾斜抽样结合，将训练策略适配到更大且偏移的实例分布，无需依赖局部搜索后处理。在OP、PCTSP、随机PCTSP和KP上的实验显示，RWPO在无加权偏好优化和策略梯度基线下，在广泛的实例规模下持续提升。在OP中，所得求解器在100至2000节点尺度下超过Compass元启发式的平均奖励。结果表明，保留奖励缺口信息可以显著提升基于偏好的学习，实现选择性组合优化，并支持向大规模实例的推广。

Reinforcing intra-operative tumour localisation with weak supervision from pre-operative appearances

在术前表现不足的情况下强化术中肿瘤定位

• Authors: Sidhart Sami, Ali Sharjeel, Sri Atluri, Miao Zhao, Caroline Roney, Eirini Marouli, Shaheer U. Saeed
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.31253
• Pdf link: https://arxiv.org/pdf/2610.31253
• Abstract Limited tumour visibility on intra-operative images complicates surgical navigation, including liver hepatocellular carcinoma (HCC) targeting on computed tomography (CT) imaging. This makes labelled datasets difficult to curate for deep learning, and transferring pre-operative delineations through registration remains challenging due to intra-operative deformation and lack of paired pre- and intra-operative data. We propose to transfer tumour appearances learned from pre-operative magnetic resonance (MR) images to guide localisation on intra-operative CT through weak supervision. A classifier is trained to classify tumour presence on pre-operative images. This is applied to windows on intra-operative images, and despite the domain gap, it assigns higher tumour-presence probabilities on average to tumour-containing windows. These weak cross-domain signals guide a reinforcement learning agent to localise tumours on intra-operative images. We compare direct transfer, few-shot adaptation and synthetic transfer (trained using unpaired MR and CT), which translates CT into synthetic MR before localisation. On an external HCC cohort, synthetic transfer outperformed other tested methods and CT-only baselines. Our framework enables cross-modality tumour localisation without paired MR-CT training images.
• 中文摘要 术中图像上的肿瘤可见性有限，使手术导航复杂，包括计算机断层扫描（CT）中肝细胞癌（HCC）靶向。这使得深度学习中标记数据集难以整理，且由于术中变形及缺乏配对的术前和术中数据，术前界定通过注册传递仍具挑战性。我们计划通过从术前磁共振（MR）图像中获得的肿瘤表现，通过薄弱监督引导术中CT定位。分类器被训练用于在术前图像中分类肿瘤存在。该方法应用于术中图像中的窗口，尽管存在领域空白，但它平均赋予含肿瘤窗口更高的肿瘤存在概率。这些弱的跨域信号引导强化学习剂定位术中图像上的肿瘤位置。我们比较直接转移、少量样本适应和合成转移（使用未配对的MR和CT训练），后者在定位前将CT转化为合成MR。在外部HCC队列中，合成移植优于其他测试方法和仅CT基线。我们的框架实现了无需配对MR-CT训练图像的跨模态肿瘤定位。

RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding

RIPPLE：通过过程级政策学习和效应感知认可实现的恢复，实现长期代理编码

• Authors: Jingxuan Xu, Zihang Xu, Yuechen Luo, Ziqi Miao, Wenqiang Zhu, Weihao Li, Junhao Fan, Xinshun Feng, Zhiyi Lai, Yanan Wu, Haoyang Huang, Ken Deng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31276
• Pdf link: https://arxiv.org/pdf/2610.31276
• Abstract Long-horizon coding agents can observe a failure and generate a plausible critique yet continue along the same incorrect trajectory. We call this the Critique-to-Action Handoff Gap. We introduce RIPPLE, a recovery-oriented training framework that internalizes process-level guidance into a single self-recovering policy. A prefix-only teacher provides evidence-grounded recovery supervision without access to future actions or terminal outcomes. After lightweight distillation, reinforcement learning assigns stage-specific credit to reflection triggering, diagnosis, and corrective action. Local validity signals gate trajectory advantages so that intermediate recovery decisions do not indiscriminately inherit terminal outcomes. Across four repository-level software-engineering benchmarks, RIPPLE improves the base policy by an arithmetic mean of 4.8 score points, with no auxiliary critic at deployment. Controlled ablations isolate the benefits of local, stage-aligned credit, and matched-prefix interventions show that critique semantics affect subsequent correction. Recovery gains concentrate where the prefix contains sufficient diagnostic evidence and the repository state remains reversible within the remaining budget.
• 中文摘要 长视野编码代理可以观察到失败并生成合理的批评，但仍然沿同一错误路径发展。我们称之为批评到行动的交接缺口。我们引入了RIPPLE，一种以恢复为导向的培训框架，将流程级指导内化到单一的自我恢复策略中。仅带前缀的教师提供基于证据的恢复监督，无需访问未来行动或终端结果。经过轻量级提炼后，强化学习将阶段特定的功劳归功于反思触发、诊断和纠正行动。局部效度信号传递门道轨迹优势，使中间恢复决策不会无差别地继承终端结果。在四个仓库级软件工程基准测试中，RIPPLE以4.8分的算术平均改进基础策略，部署时无辅助批评者。受控消融分离出局部、阶段对齐信用和匹配前缀干预的益处，显示批判语义会影响后续纠正。回收收益集中在前缀包含足够诊断证据且存储状态在剩余预算内保持可逆性。

Training Small Language Models for Transferable Tool Selection

训练小型语言模型以实现可迁移工具选择

• Authors: Zhaoyi Zhou, Akshayaa Magesh, Jeongyeol Kwon, Artem Kuriksha, Daniel Jiang, Andrea Zanette, Ayush Jain
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31483
• Pdf link: https://arxiv.org/pdf/2610.31483
• Abstract As LLM agents operate over corpora of more than ten thousand tools, selecting the right tool for a complex multi-step user instruction becomes a central bottleneck. Tool descriptions cannot all fit in context, and relevance-based retrievers only rank tools without knowing which are actually needed. Recent work instead leverages proprietary commercial LLMs like Claude or ChatGPT to drive a retriever, decomposing the instruction and selecting over retrieved candidates. We show that a small, open model can be trained via reinforcement learning to perform the same functionality. On ToolBench, a Qwen3-8B model post-trained with our recipe outperforms Llama-4-Maverick (402B). In addition, the trained model transfers directly to different retrievers and generalizes to new tool corpora in different domains.
• 中文摘要 由于LLM代理运行于超过一万个工具的语料库中，选择合适的工具来应对复杂的多步用户指令成为一个核心瓶颈。工具描述无法全部符合上下文，基于相关性的检索器只是对工具进行排名，却不知道哪些工具真正需要。近期工作则利用专有商业LLM如Claude或ChatGPT驱动检索器，分解指令并选择检索对象。我们展示了通过强化学习训练一个小型开放模型来执行同样的功能。在ToolBench上，Qwen3-8B模型用我们的方案后训练后表现优于Llama-4-Maverick（402B）。此外，训练好的模型可以直接转移到不同的检索器，并推广到不同领域的新工具语料库。

Towards User Simulators That Stay On Task And Still Sound Natural

对于那些能保持任务且听起来自然的用户模拟器

• Authors: Mustafa Omer Gul, Philippe Laban, Tanya Goyal, Jennifer Neville
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.31512
• Pdf link: https://arxiv.org/pdf/2610.31512
• Abstract Users increasingly rely on AI systems for extended, multi-turn tasks, yet evidence indicates that LLM-based system performance degrades over prolonged interaction. To evaluate and improve AI assistants in such settings, we need user simulators that faithfully follow complex user intents while exhibiting realistic conversational behavior. Existing approaches manifest a trade-off: prompted assistants can follow complex user intents but lack behavioral fidelity, while specialized user simulators better approximate conversational behavior but struggle to remain faithful to complex intents. In this paper, we propose a reinforcement learning (RL) recipe that jointly rewards task intent adherence and behavioral naturalness. A crucial contribution of the method is operationalizing naturalness with a lightweight discriminator trained in an alternating, adversarial loop with the user simulator. We release the resulting model, UserLM-2.0, which produces valid simulated conversations (i.e., meeting both task-faithfulness and naturalness criteria) 46.2% of the time, over three times the rate of the strongest prior user simulator (14.1%).
• 中文摘要 用户越来越依赖AI系统完成长时间、多回合的任务，但证据表明基于LLM的系统性能随着长时间交互而下降。为了评估和改进AI助手在此类环境中，我们需要用户模拟器能够忠实跟踪复杂用户意图，同时展现真实的对话行为。现有方法存在权衡：提示式助手可以跟踪复杂的用户意图，但行为忠实度不足;而专门的用户模拟器更能近似对话行为，但难以忠实于复杂意图。本文提出一种强化学习（RL）配方，共同奖励任务意图的遵循和行为自然性。该方法的一个关键贡献是通过轻量级判别器与用户模拟器交替对抗循环训练，实现自然性。我们发布了该模型UserLM-2.0，其有效模拟对话的有效率为46.2%，是最强先前用户模拟器（14.1%）的三倍以上。

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

同样的运气，不同的选择：能动强化学习的双世界学分作业

• Authors: Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.31544
• Pdf link: https://arxiv.org/pdf/2610.31544
• Abstract Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by $1-\rho$, where $\rho$ is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size $\Delta$ falls from $\Theta(\Delta^{-2})$ to $\Theta(\Delta^{-1})$. A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.
• 中文摘要 代理强化学习奖励一个完整的事件，但策略需要为其中的每个决策获得认可。反事实分支通过在备选决策后重跑该事件来估计该功劳，但该分支抽取采样噪声独立于事实事件，因此结果差异将决策效果与后续所有事件的运气混合。我们观察到，在代理搜索中，检索器、语料库和评分器是确定性的，因此一个事件是策略决策及其采样噪声的确定性函数。因此，双胞胎世界是可实现的：相同的噪声，不同的决策。我们引入TWICE（TWIn-world Credit Estimation），它通过轮流和线键化每个代币的Gumbel噪声，推出继承事实事件下游噪声的反事实孪生，并将它们的配对结果差异转化为策略梯度更新的回合级优势;仅部署行为者。孪生信用对任何耦合质量无偏，其方差为独立反事实，按$1-\rho$缩放，其中$\rho$为双胞胎的结果相关性;在耦合下，解决大小优势所需的滚动次数从$\Theta（\Delta^{-2}）$降至$\Theta（\Delta^{-1}）$。平方根规则用于计算双生预算，预测结果按单位成本不一致。在七个问答基准测试中，TWICE在Qwen2.5-3B和7B上平均精确匹配提升了5.3点和4.5分，优于树搜索和反事实分支基线，辅助标记使用量比VinePPO少68%。

Humanoid Mobility via Attention-based Contacts

通过注意力型接触实现类人机动

• Authors: Junzhe He, Kehan Wen, Chong Zhang, Clemens Schwarke, Zewei Zhang, Kaixian Qu, Siyuan Hu, Matthias Müller, Vladlen Koltun, Marco Hutter
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.31608
• Pdf link: https://arxiv.org/pdf/2610.31608
• Abstract Humanoid mobility over complex terrain requires accurate foot placement and coordinated whole-body motion capable of robust adaptation. Reinforcement learning (RL) policies trained without motion references generalize well across terrains, yet eliciting effective whole-body coordination through reward design and exploration alone remains challenging. In contrast, reference-based methods yield natural motions but struggle to adapt beyond the predefined trajectories in their training data. We introduce a learning framework that bridges these two approaches by using motion data strictly as a training prior. An explicit, self-supervised, attention-based contact planner predicts future body-environment contacts, while a contact-conditioned tracker translates these plans into coordinated whole-body motion. Together, they establish contact as an interface between sensory observations and control actions. This abstraction allows the policy to internalize the contact-aware structure of locomotion rather than directly replicating kinematics. Consequently, a single policy can exploit data-driven motion priors for natural whole-body coordination while retaining the flexibility to discover environment-dependent behaviors through reinforcement learning. Building on the learned contact planner and tracker, a subsequent fine-tuning stage bootstraps their acquired capabilities to explore environment variations beyond those covered by the original motion data. Operating entirely from onboard observations without any reference inputs at deployment, this framework enables versatile humanoid mobility in challenging environments.
• 中文摘要 在复杂地形上实现类人机动需要准确的脚步位置和协调的全身运动，具备稳健的适应能力。在无运动参考的情况下训练的强化学习（RL）策略在地形上具有良好泛效性，但仅通过奖励设计和探索引发有效的全身协调仍然具有挑战性。相比之下，基于参考的方法产生自然动作，但在适应训练数据预设轨迹之外时遇到困难。我们引入了一个学习框架，通过严格将运动数据作为训练先验，连接这两种方法。显性、自我监督、基于注意力的接触规划器预测未来的身体-环境接触，而接触条件追踪器则将这些计划转化为协调的全身运动。它们共同建立了感官观察与控制行动之间的接触接口。这种抽象化使策略能够内化接触感知的运动结构，而非直接复制运动学。因此，单一策略可以利用数据驱动的运动先验实现自然的全身协调，同时保持通过强化学习发现环境依赖行为的灵活性。基于已学习到的接触规划器和追踪器，后续的微调阶段使其获得的能力启动，探索原始运动数据覆盖之外的环境变异。该框架完全基于机载观察运行，无需部署时任何参考输入，使得在复杂环境中实现多样化的人形机动。

DiffDreamer: Coupling World Models with Differentiable Simulation for Agile Quadrotor Control

DiffDreamer：结合世界模型与可微分仿真以实现敏捷四旋翼控制

• Authors: Fanxing Li, Yuxiang Huang, Shengyang Wang, Fangyu Sun, Shuyu Wu, Tianbao Zhang, Dexin Zuo, Yan Yufei, Danping Zou, Wenxian Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.31617
• Pdf link: https://arxiv.org/pdf/2610.31617
• Abstract Tasks such as quadrotor dynamic-gate racing require agents to anticipate external motion while maintaining precise high-frequency control. World-model-based reinforcement learning provides a natural solution by learning latent dynamics for future imagination, yet it often entangles environmental evolution and the robot's own dynamics in a single learned rollout, where model bias can affect policy optimization efficiency. We propose DiffDreamer, a differentiable reinforcement learning framework that tightly couples a learned world model with differentiable simulation for actor optimization. In one optimization step, the learned world model propagates latent states that capture external dynamics and history-dependent features, while the differentiable simulator propagates the robot's explicit physical state. We further decompose the actor objective into a critic-driven pathway for non-differentiable task rewards and a simulator-driven pathway for differentiable motion rewards. Experiments on aggressive tracking and dynamic-gate racing show that DiffDreamer improves training convergence over the model-free, model-based, and differentiable-RL baselines, and final performance over these baselines and classical planning and control methods. Additional experiments assess reward decomposition and the odometry-conditioned relative-state representation through ablations. We also provide proof-of-concept real-world flight experiments for validation.
• 中文摘要 诸如四旋翼动态门竞速等任务要求智能体在保持精确高频控制的同时预判外部运动。基于世界模型的强化学习通过学习潜在动力学为未来想象提供了自然解决方案，但它常常将环境演化与机器人自身动力学纠缠在单一学习的展开中，模型偏差可能影响策略优化效率。我们提出了DiffDreamer，一种可微强化学习框架，紧密结合已学世界模型与可微模拟以优化演员。在一个优化步骤中，学习世界模型传播捕捉外部动态和历史相关特征的潜在状态，而可微模拟器传播机器人的显式物理状态。我们进一步将演员目标分解为非微分任务奖励的批评者驱动路径和可微运动奖励的模拟器驱动路径。激进跟踪和动态门竞速的实验显示，DiffDreamer在无模型、基于模型和可微化的强化学习基线上提升了训练收敛性，最终性能优于这些基线和经典规划与控制方法。其他实验通过消融评估奖励分解和里程条件下的相对状态表示。我们还提供概念验证的真实飞行实验。

Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3

质疑RLVR中的工具性趋同：OLMo-3的检查点证据

• Authors: Barış Alparslan, Sandy Tanwisuth, Caleb Petterson
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31624
• Pdf link: https://arxiv.org/pdf/2610.31624
• Abstract Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline. The result is specific to this model family and benchmark, and judge agreement is moderate, so absolute rates should be read cautiously.
• 中文摘要 工具性趋同常被用作一般理由，期望有能力的智能体获得或表达对社会构成重要威胁的权力追求子目标：自我保护、资源获取、欺骗或对监督的抵抗。对于大型语言模型（LLM）而言，强化学习是训练后直接针对目标优化行为的第一阶段;因此，基于IC的收敛工具行为最有可能出现。带可验证奖励的强化学习（RLVR）是否使模型更可能表现出工具性收敛行为，还是这取决于训练体系、评估框架和训练后阶段？我们将此问题作为一个检查点条件的行为轨迹来研究，而非单一的安全前后评分。利用InstrumentalEval，我们评估了OLMo-3-7B-Think在SFT、DPO、13个中间RLVR检查点及最终RLVR模型中的表现。该轨迹为非单调：工具率从SFT的25.0%上升至RLVR第200步附近的46.1%，最终检查点降至11.8%。配对题级测试和分步回归支持最终SFT到RLVR的减少。提示框架无法解释这一模式：有害提示指标结合其他框架效应平均仅解释检查点内分数方差的15%，而迭代零空间投影（INLP）则去除可检测的框架方向而不改变SFT到最终RLVR的差距。结合32B的初步终点结果（35.53% SFT至22.37% RLVR最终），这些发现表明RLVR在此环境中不会单调地增加工具行为：速率在早期上升，随后降至SFT基线以下。结果针对该模型家族和基准，评判意见中等，因此绝对速率应谨慎解读。

METTA: Memory-Enhanced Test-Time Adaptation for Meta-Reinforcement Learning

METTA：记忆增强测试时间适应技术，用于元强化学习

• Authors: Ali Najar, Hosein Hasani, Amir Kooshan Fattah, Yasamin Rajabi, Mahdieh Soleymani Baghshah
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31656
• Pdf link: https://arxiv.org/pdf/2610.31656
• Abstract Meta-reinforcement learning aims to train agents that adapt quickly to unseen tasks from limited interaction, but this requires representing task evidence over long interaction histories. Classical recurrent agents compress experience into hidden states, while transformer-based agents retrieve from past trajectories at quadratic memory and computation cost. In this work, we propose Memory-Enhanced Test-Time Adaptation (METTA) for meta-reinforcement learning. It decomposes the fast adaptation phase into two temporal scales: a causal neural memory performs fine-grained within-episode adaptation with linear sequence complexity, while compact episode-level summaries support long-range cross-episode task inference without attending over the full interaction history. Moreover, equipping agents with neural memory provides sufficient expressive capacity and test-time training capability, enabling effective adaptation to unseen held-out tasks and task families at meta-test time. On unseen held-out Meta-World ML10/45 test environments, METTA achieves a success rate over $50\%$ while maintaining a low training budget and short wall-clock time. On long-context sparse-reward PointMaze tasks, it outperforms transformer-based baselines by exhibiting more structured exploration behavior. These results show that METTA provides an efficient path toward more adaptive and scalable long-context meta-RL.
• 中文摘要 元强化学习旨在训练能够快速适应有限交互中未见任务的智能体，但这需要在较长的交互历史中表示任务证据。经典循环智能体将经验压缩为隐藏状态，而基于变换器的智能体则以二次级记忆和计算成本从过去轨迹中检索数据。本研究提出内存增强测试时间适应（METTA）用于元强化学习。它将快速适应阶段分解为两个时间尺度：因果神经记忆执行线性序列复杂度的细粒度剧内适应，而紧凑的剧集级摘要支持长距离跨剧集任务推断，而无需关注完整的交互历史。此外，装备神经记忆为代理提供了足够的表达能力和测试时间训练能力，使得在元测试时间有效适应未被发现的任务和任务族。在未被发现的Meta-World ML10/45测试环境中，METTA在保持低训练预算和短时钟时间的同时，成功率超过50%。在长上下文稀疏奖励的PointMaze任务中，METTA通过更具结构化的探索行为优于基于变换器的基线。这些结果表明，METTA为更适应性和可扩展的长上下文元强化学习提供了一条高效的路径。

Keyword: diffusion policy

VecTile: A Low-Cost Flexible Tactile Array for 3-DoF Force Sensing

VecTile：一种低成本的柔性触觉阵列，用于3-dF力感测

• Authors: Jian Hou, Xin Zhou, Adam J. Spiers
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.31472
• Pdf link: https://arxiv.org/pdf/2610.31472
• Abstract Tactile sensing provides robots with contact information difficult to obtain from vision alone. Shear force is particularly important for understanding tangential contact, guiding corrective motions, and maintaining grasp stability, yet flexible and low-cost tactile arrays often measure only normal force. We introduce VecTile, a low-cost, 6 mm thin, flexible tactile array capable of three-axis force sensing. VecTile combines a piezoresistive layer with passive silicone nubs that convert normal and shear forces into distinguishable pressure patterns. The sensor can be fabricated in-house without specialized equipment. The complete system costs less than 35 USD for a 4x4 array. We validate the sensing mechanism through finite element analysis and hardware experiments. In controlled single-taxel tests over 1-5 N normal force and up to 1.5 N shear force, VecTile achieves root-mean-square errors of 0.39 N and 0.16 N for normal and shear forces. A VecTile calibrated while flat retains accuracy when curved, while a five-minute manual calibration is sufficient for practical use. Two applications demonstrate its utility. First, shear readings are sensitive enough to stop a sharp scalpel from puncturing a balloon after contact. Second, the sensor substantially improves Diffusion Policy-guided power-plug insertion. All designs and code will be released upon paper acceptance. Project website: this https URL.
• 中文摘要 触觉感应为机器人提供了难以仅凭视觉获得的接触信息。剪切力对于理解切线接触、引导修正运动和保持抓握稳定性尤为重要，但柔性且低成本的触觉阵列通常仅测量法向力。我们介绍了VecTile，一种低成本、6毫米薄且柔性触觉阵列，具备三轴力感应能力。VecTile结合了压阻层和被动硅胶凸起，将法向力和剪切力转换为可区分的压力模式。该传感器可在内部自行制造，无需专用设备。整个系统4x4阵列成本低于35美元。我们通过有限元分析和硬件实验验证了感测机制。在1-5牛的法向力和最高1.5牛顿剪切力的受控单量衡测试中，VecTile在法向力和剪切力下均方根误差分别为0.39牛顿和0.16牛顿。在平面校准的Vectile在弯曲时保持准确性，而五分钟的手动校准即可实际使用。有两个应用展示了其实用性。首先，剪切读数足够灵敏，能阻止锋利手术刀在接触气球后刺破气球。其次，传感器显著提升了扩散政策指导电源插头插入的效率。所有设计和代码将在纸质接受后发布。项目网站：此 https 网址。