生成时间: 2026-10-14 21:32:00 (UTC+8); Arxiv 发布时间: 2026-10-14 20:00 EDT (2026-10-15 08:00 UTC+8)

今天共有 130 篇相关文章

Keyword: reinforcement learning

GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games

GameSmith：用合成游戏扩展零和自玩

• Authors: Weixian Waylon Li, Jiaxin Zhang, Han Zhou, Chengzu Li, Ke Wang, Fengxiang He, Yiwen Guo
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.13995
• Pdf link: https://arxiv.org/pdf/2610.13995
• Abstract Post-training large language models (LLMs) on two-player zero-sum games provides a verifiable reward, and the gains have been shown to transfer to general reasoning. Existing pipelines, however, rely on a handful of hand-written games and on advantage estimators tied to them, so both the supply of games and the advantage estimate on a large corpus limit scaling. We present GameSmith, which scales both. An LLM synthesises games inside templates whose reward, observation, and hidden-information code is fixed. Each game is verified by seed-paired replay and by an independent re-implementation of its specification, then expanded into encoding, rule, and size variants, giving 26 new games in 29 families and 336 verified variants from the three SPIRAL templates. The model is distilled from the self-play of a larger teacher (Qwen3-32B) on these games and then trained by self-play reinforcement learning with CRAB, an advantage estimator that credits each decision against a shrinkage estimate of the expected return of its position and is defined on every variant. On Qwen3-4B-Base, distillation raises the 13-benchmark mean pass@1 from 45.5 to 62.5, above the three SPIRAL games at the same budget, and self-play with CRAB reaches 64.9. The ordering holds across models, and transfer grows with the number of game families.
• 中文摘要 在双人零和游戏上进行大型语言模型（LLM）后训练，提供了可验证的奖励，且其收益已被证明能转化为通用推理。然而，现有的流程依赖少数手写游戏及其关联的优势估计器，因此游戏的供应和优势估计均基于大语料库极限缩放。我们介绍GameSmith，它对两者进行扩展。LLM在固定奖励、观察和隐藏信息代码的模板中综合游戏。每个游戏通过种子配对重玩和其规范的独立重构进行验证，随后扩展为编码、规则和大小变体，形成29个家族中的26个新游戏，以及来自三个SPIRAL模板的336个经过验证的变体。该模型由较大教师（Qwen3-32B）在这些游戏中的自玩中提炼出来，然后通过CRAB的自玩强化学习进行训练，CRAB是一个优势估计器，将每个决策与其位置预期回报的缩小估计相匹配，且定义于每个变体。在Qwen3-4B-Base上，蒸馏将13基准的平均pass@1从45.5提升到62.5，超过三款同预算的SPIRAL游戏，CRAB自玩达到64.9。该排序在各模型中成立，转移随游戏家族数量增加而增长。

Splicing Hints into Reasoning Traces: Learning from Demonstrator Failures

将提示拼接进推理痕迹：从演示器失败中学习

• Authors: Hamed Mahdavi (Pennsylvania State University), Pouria Mahdavinia (Pennsylvania State University), Majid Farhadi (Google), Vasant Honavar (Pennsylvania State University)
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13996
• Pdf link: https://arxiv.org/pdf/2610.13996
• Abstract Reinforcement learning with verifiable rewards improves mathematical reasoning when a final answer can be checked, but a correct answer does not certify the derivation behind it, and proof problems offer no answer to check. Replacing the check with a reasoning model as an in-loop verifier faces a fundamental obstacle: the policy must already produce occasional correct attempts for any reward to exist, which it may never do at olympiad level. We instead move verification and repair out of the training loop and into offline data generation with SPLICE, a workflow in which a second model, given only a reference solution, edits a reasoning model's traces as they are generated: it verifies progress, truncates a failed trace at its last verified step, and splices in an adaptively strengthened hint until progress resumes. The resulting dataset includes repaired examples whose sampled unaided trajectory failed, with failed continuations removed and progress repeatedly checked against the reference solution. A model finetuned on SPLICE traces outperforms the same model distilled from the reasoning model's unedited traces on AIME and, under a grading protocol validated against human grades, on olympiad proofs. The trained model's own traces show a corresponding behavioral signature: at difficult points it announces a branch in the style of the spliced hints and takes riskier steps, with predictive entropy rising there and just after, a pattern absent from the model trained on unedited traces.
• 中文摘要 带有可验证奖励的强化学习提升了数学推理能力，当最终答案可以被检查，但正确答案并不能证明其推导，证明题也没有答案需要验证。用推理模型取代检查作为循环验证器面临一个根本障碍：策略必须先产生偶尔的正确尝试，才能获得任何奖励，而在奥林匹克级别下可能永远不会做到。我们取而代之的是，将验证和修复从训练循环转向离线数据生成，采用SPLICE技术。该工作流程中，第二个模型仅给出参考解，在生成时编辑推理模型的痕迹：验证进展，在最后验证步骤截断失败的痕迹，并插入自适应强化的提示，直到进展恢复。最终数据集包含那些样本中无辅助轨迹失败的已修复样本，失败的延续被移除，进度反复与参考解进行核对。在SPLICE痕迹上微调的模型，表现优于AIME上从推理模型未编辑轨迹中提炼出来的同一模型，并在基于人类评分验证的评分协议下，在奥林匹克证明上表现优异。训练模型自身的轨迹显示出相应的行为特征：在困难点，它会以拼接提示风格的分支宣布，并采取更大风险的步骤，预测熵在该处及之后上升，而模型在未编辑轨迹上训练时则缺少这种模式。

Enhanced Trading Systems with Intelligent Market Analysis

增强的交易系统与智能市场分析

• Authors: Vedanth Padmaraman, Shreen Anand, Vamshi Vishruth J, Vineet V Pai, Arvind Srinath K.T, Subramanyam Natarajan, Shylaja S S
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13998
• Pdf link: https://arxiv.org/pdf/2610.13998
• Abstract This paper proposes a hybrid reinforcement learning trading framework integrating intraday market data with transformer-based financial sentiment signals. The system extracts sentiment dynamics and semantic event clusters from financial news and incorporates them into a PPO-based trading agent. Experiments on intraday NSE data demonstrate improved risk-adjusted returns compared to price-only baselines.
• 中文摘要 本文提出了一种混合强化学习交易框架，将日内市场数据与基于变压器的金融情绪信号整合。该系统从金融新闻中提取情绪动态和语义事件簇，并将其整合进基于PPO的交易代理中。对日内NSE数据的实验显示，风险调整后收益优于仅价格基线。

VH-GPO: Verified Handoff for LLM Agent Reinforcement Learning

VH-GPO：LLM 代理强化学习的验证交接

• Authors: Haoze Li, Peng Zhou, Peiyuan Wu, Weijie Zhao, Yinan Xin
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14000
• Pdf link: https://arxiv.org/pdf/2610.14000
• Abstract Group-relative reinforcement learning methods such as GRPO train large language model (LLM) agents by comparing rollouts for the same task. When every trajectory in a group fails with the same reward, relative advantages collapse toward zero, leaving no useful learning signal. A frozen teacher can recover from a student-visited state, but letting it complete the task makes the recovered suffix entirely teacher-generated. We ask: after repairing a student failure, when should control return to the student? We propose outcome-verified teacher-student handoff, instantiated in Verified Handoff for Group Policy Optimization (VH-GPO). The teacher provides enough support to reach a state from which the student can succeed, then returns control so that successful experience is again student-generated. Given an environment-verified teacher recovery, VH-GPO selects the earliest independently confirmed handoff point found within a bounded search budget. With Qwen2.5-1.5B, VH-GPO increases success rates over GRPO from 59.38% to 94.79% on ALFWorld and from 71.62% to 77.34% on WebShop. Improvements also hold with the 7B student, for which additional teacher and student output tokens amount to 1.77% and 2.61% of regular student rollout output tokens on ALFWorld and WebShop, respectively. Ablation studies demonstrate the effectiveness of the key components. These results support using verified teacher assistance to repair failures while reinforcing the student's ability to complete tasks independently.
• 中文摘要 群体相对强化学习方法如GRPO通过比较同一任务的展开来训练大型语言模型（LLM）代理。当一个小组中每个轨迹都失败且奖励相同时，相对优势会趋近于零，没有有用的学习信号。冻结的教师可以从学生访问状态恢复，但让其完成任务后，恢复的后缀完全由教师生成。我们提出：修复学生失败后，何时应将控制权返回学生？我们提出结果验证师生交接，具体体现在“验证交接用于小组策略优化”（VH-GPO）中。教师提供足够的支持以达到学生能够成功的状态，然后返回控制，使成功体验再次由学生生成。在环境验证的教师恢复条件下，VH-GPO选择在有界搜索预算内找到的最早独立确认的交接点。通过Qwen2.5-1.5B，VH-GPO在ALFWorld上将GRPO的成功率从59.38%提升至94.79%，WebShop上从71.62%提升至77.34%。7B学生的改进同样显著，教师和学生输出代币分别占ALFWorld和WebShop常规学生推广代币的1.77%和2.61%。消融研究证明了关键组成部分的有效性。这些结果支持利用经过验证的教师协助修复失败，同时增强学生独立完成任务的能力。

PreSO: Privileged Representation Isolation for On-Policy Self-Distillation

PreSO：特权代表隔离，用于政策自我提炼

• Authors: Yuhang Yang, Mingxuan Xia, Chao Ye, Shuai Zhu, Shouqing Yang, Haobo Wang, Shenzhi Yang, Zhengqing Zang, Chenglong Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14001
• Pdf link: https://arxiv.org/pdf/2610.14001
• Abstract On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learning with verifiable rewards (RLVR) into dense token-level supervision. Yet the teacher's advantage may depend on information unavailable to the student at inference, raising a central question: which privileged changes are worth distilling? We use cross-problem recurrence as an empirical cue for potentially reusable guidance. Comparing the representation shifts induced by full solutions and compact hints, we find that hint-induced shifts are more concentrated and directionally consistent across problems, and yield more sustained downstream gains despite revealing less information. Motivated by this association, we propose Privileged Representation Isolation for On-Policy Self-Distillation (PreSO). At the source, PreSO uses compact hints to limit the solution-specific detail entering the privileged shift. At the target, it estimates a leading low-rank subspace from shifts pooled across related problems, projects the privilege-induced correction onto this subspace, and adds it to the student's unprivileged representation to form the representation-level distillation target. The selected subspace serves as an operational proxy for recurring structure, rather than a guaranteed separation of transferable and instance-specific information. On five competition-level mathematics benchmarks, PreSO consistently outperforms strong baselines across Qwen3-1.7B, 4B, and 8B, improving average accuracy over the base model by 6.8, 9.0, and 10.0 points. Beyond peak accuracy, PreSO attains the highest final-checkpoint average at every model scale, with improvements that persist through the end of training.
• 中文摘要 策略自提纯（OPSD）利用以特权信息为条件的自教师，将带有可验证奖励的强化学习（RLVR）稀疏结果信号转化为密集的代币级监督。然而，教师的优势可能依赖于学生在推理时无法获得的信息，这引发了一个核心问题：哪些特权变化值得提炼？我们以跨问题重复性作为潜在可重用指导的经验线索。比较全解与紧凑提示引发的表征转移，我们发现提示诱导的转变在问题间更集中且方向一致，尽管揭示的信息较少，但下游收益更持久。基于这一关联，我们提出了策略上自我蒸馏的特权表示隔离（PreSO）。在源头，PreSO使用紧凑提示限制进入特权转移的解决方案特定细节。在目标空间中，它从相关问题间的移位中估算一个领先的低秩子空间，将特权引起的修正投射到该子空间上，并将其与学生的非特权表示相加，形成表征层级的提炼目标。所选子空间作为重复结构的操作代理，而非可转移信息和实例特定信息的保证分离。在五项竞赛级数学基准测试中，PreSO在Qwen3-1.7B、4B和8B中持续优于强基线，基础模型的平均准确率提升了6.8、9.0和10.0分。除了峰值准确性外，PreSO在所有模型尺度上都达到了最高的最终检查点平均值，且提升持续到训练结束。

Showing versus Doing: Teaching by Demonstration

展示与实践：以示范为教学

• Authors: Mark K. Ho, Michael L. Littman, James MacGlashan, Fiery Cushman, Joseph L. Austerweil
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.14004
• Pdf link: https://arxiv.org/pdf/2610.14004
• Abstract People often learn from others' demonstrations, and inverse reinforcement learning (IRL) techniques have realized this capacity in machines. In contrast, teaching by demonstration has been less well studied computationally. Here, we develop a Bayesian model for teaching by demonstration. Stark differences arise when demonstrators are intentionally teaching (i.e. showing) a task versus simply performing (i.e. doing) a task. In two experiments, we show that human participants modify their teaching behavior consistent with the predictions of our model. Further, we show that even standard IRL algorithms benefit when learning from showing versus doing.
• 中文摘要 人们常常从他人的演示中学习，逆向强化学习（IRL）技术已在机器中实现了这一能力。相比之下，演示式教学在计算上研究较少。在这里，我们开发了一个贝叶斯演示教学模型。当示范者有意地教学（即展示）任务与仅仅执行（即执行）任务时，差异显著。在两个实验中，我们表明人类参与者会根据模型的预测调整教学行为。此外，我们表明即使是标准的现实学习算法，在通过展示与执行学习中也能受益。

Residual Compliant Policy: Learning Compliant Policies from Few Demonstrations

剩余合规政策：从少数演示中学习合规政策

• Authors: Chunyu Jiang, Yichen Cai, Shuai Li, Wenbin Sun, João Carvalho, Kostas Daniilidis, Oleg Arenz, Cristiana de Farias, Jan Peters
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.14035
• Pdf link: https://arxiv.org/pdf/2610.14035
• Abstract Contact-rich manipulation requires a robot to be precise in free space to establish the intended contacts, and compliant once in contact to avoid excessive forces. Imitation learning has proven effective for simpler manipulation tasks, but the compliance underlying a demonstration is not directly observable and is therefore difficult to infer from demonstration data alone. We present a framework that makes compliance explicit at every stage. During data collection, the operator teleoperates the real robot while a synchronized digital twin visualizes the error between the commanded reference and the achieved pose, making the current compliance visible, and selects between two discrete compliance modes. To generalize across object poses, we combine Laplacian editing, which retargets trajectories efficiently but is not physics-consistent during contact, with simulation-based trajectory optimization, which enforces physical plausibility but offers no manual guidance. Finally, a residual reinforcement learning policy refines the interaction forces by adjusting the reference pose of the underlying admittance controller. We evaluate on three flipping and pushing tasks on a Franka FR3 based on a total of 7 demonstrations across tasks. Our method is successful on all 40 out-of-distribution trials, compared to 33/40 and 24/40 for two baselines, with our base policy alone reaching 35/40.
• 中文摘要 丰富接触操作要求机器人在自由空间内精确建立预期接触，并在接触后顺应以避免过度受力。模仿学习已被证明对较简单的操作任务有效，但演示背后的顺应性无法直接观察，因此仅凭演示数据难以推断。我们提出了一个框架，使每个阶段都明确了顺应性。在数据收集过程中，操作员远程操作真实机器人，同步数字孪生可视化指令参考与实现姿态之间的误差，使当前顺应性可见，并在两种离散顺应模式中选择。为了跨物体姿态推广，我们结合了拉普拉斯编辑（能高效重新定位轨迹但接触时物理不一致）与基于仿真的轨迹优化（强化物理可行性但不提供手动指导）。最后，残差强化学习策略通过调整底层导纳控制器的参考姿态来细化交互力。我们基于7个任务演示，在Franka FR3上评估了三个翻转和推送任务。我们的方法在所有40个非分布试验中均取得成功，而两个基线的试验分别为33/40和24/40，仅基础策略就达到35/40。

A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learning

反向强化学习中奖励函数更新的几何视角

• Authors: Anish Diwan, Jan Peters, Oleg Arenz
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.14041
• Pdf link: https://arxiv.org/pdf/2610.14041
• Abstract Inverse Reinforcement Learning (IRL) is classically solved using Lagrangian optimisation, by jointly optimising a primal-dual problem with gradient descent to obtain the optimal reward function parameters and corresponding optimal policy. While this algorithmic view of IRL has been highly influential, gradient-based reward updates only paint a partial picture, largely ignoring the geometry of the problem in occupancy space. In this paper, we present an occupancy-space view of reward updates in IRL. We leverage the insight that both the optimal solution and its geometry in occupancy space are known, and show that the geometry along the e-geodesic (straight line in occupancy space) toward the optimal occupancy directly affects the gradient of the IRL dual problem in reward parameter space. This perspective naturally suggests preconditioning the dual gradient with the average Fisher information along the occupancy trajectory. We show that this preconditioned gradient is just a convex interpolation of the old reward and a new target reward, yielding efficient natural-gradient-style updates with little additional computational overhead. We then analyse these occupancy-space reward updates from a Mirror Descent (MD) perspective, and show that policy search over the resulting rewards induces approximate entropic MD on the policy. Finally, we conclude with convergence results of such geometry-aware IRL algorithms.
• 中文摘要 逆强化学习（IRL）经典地通过拉格朗日优化来求解，通过联合优化带有梯度下降的原始对偶问题，以获得最优奖励函数参数和相应的最优策略。虽然这种基于算法的IRL观点影响深远，但基于梯度的奖励更新仅描绘出部分图景，基本忽略了占用空间中问题的几何结构。本文提出了IRL中奖励更新的占用空间视角。我们利用已知的最优解及其占用空间中的几何，并证明沿e-测地线（占用空间中的直线）指向最优占用率的几何结构直接影响IRL对偶问题在奖励参数空间中的梯度。这一观点自然暗示对双梯度进行预条件，并沿占用轨迹的平均费舍尔信息。我们证明该预条件梯度只是旧奖励和新目标奖励的凸插值，从而实现高效的自然梯度式更新，且计算开销较小。随后，我们从镜像下降（MD）视角分析这些占用空间奖励更新，证明对所得奖励进行策略搜索会诱导策略上的近似熵MD。最后，我们以此类几何感知的真实现实学习算法的收敛结果作结论。

Reinforcement Learning from Real-Robot Interaction without High-Quality State Estimates

基于真实机器人交互的强化学习，无需高质量状态估计

• Authors: Ryan Gao, Ashvin Anilkumar, Nicholas Corrado, William Cong, Josiah Hanna
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14042
• Pdf link: https://arxiv.org/pdf/2610.14042
• Abstract A longstanding goal for robot reinforcement learning (RL) is that robots learn from their own physical experience, as humans and animals do. A key obstacle to learning directly from physical experience, especially outside instrumented lab environments, is that the state of the world is usually unobservable to the learner and must be estimated from on-board sensing. Errors in that estimate corrupt both the policy input and potentially the reward signal. We study this challenge using the contact-rich task of a bipedal robot learning to score soccer goals. We estimate all key state variables on-board with an unscented Kalman filter (UKF) and no external instrumentation (e.g., motion capture or overhead cameras). Because the UKF may incorrectly estimate state when the ball is unobserved, we learn a new state representation with a recurrent state-space model (RSSM) that, critically, only corrects the RSSM's latent state when the ball is observed. Building on RLPD, we introduce RSSM-RLPD and apply it to the task of a NAO robot learning to dribble a soccer ball into a goal. We show that the robot learns to score from multiple starting positions after 30 minutes of real-world interaction, reaching 100% success against 0% for a feed-forward policy under the same training budget. We complement our physical robot experiments with a simulation study and show that RSSM-RLPD learns policies that achieve a higher success rate than behavior cloning, a feed-forward, and GRU baseline. Taken together, our results show that the addition of a learned state representation to existing on-robot learning methods enables RL from real-robot data even with a high degree of partial observability.
• 中文摘要 机器人强化学习（RL）长期以来的一个目标是让机器人像人类和动物一样，从自身的物理体验中学习。直接从物理经验中学习，尤其是在仪器实验室环境之外，一个关键障碍是学习者通常无法观察到的世界状态，必须通过机载感测来估算。这种估计中的误差会破坏策略输入，甚至可能破坏奖励信号。我们通过双足机器人学习足球得分的接触丰富任务来研究这一挑战。我们用无香味卡尔曼滤波器（UKF）估算所有关键状态变量，且不使用外部仪器（如动作捕捉或顶置摄像头）。由于UKF可能错误估计未被观察球的状态，我们学习一种新的状态表示，采用循环状态空间模型（RSSM），关键是只有在观察球时才修正RSSM的潜伏状态。基于RLPD，我们引入了RSSM-RLPD，并将其应用于NAO机器人学习运球入网的任务。我们展示了机器人在30分钟的真实世界互动后，从多个起始位置学习得分，在同一训练预算下，前馈策略成功率为100%，对0%。我们通过模拟研究补充了物理机器人实验，显示RSSM-RLPD学习的策略成功率高于行为克隆、前馈和GRU基线。综合来看，我们的结果表明，在现有机器人学习方法中加入学习状态表示，使得即使具有高度部分可观测性，也能从真实机器人数据实现强化学习。

Disentangling Task-Relevant Latent Dynamics for Reward Modeling

解开任务相关的潜在动态以进行奖励建模

• Authors: Dayou Li, Jionghao Wang, Peihao Wang, Jiuzhou Lei, Hao Wang, Jichen Chai, Hezhen Hu, Minghui Zheng, Zhangyang Wang, Xin Li, Wenping Wang, Zhiwen Fan
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.14159
• Pdf link: https://arxiv.org/pdf/2610.14159
• Abstract Reward design remains a major bottleneck for reinforcement learning in robotic manipulation. Handcrafted dense rewards can compensate for the limited guidance provided by sparse rewards, but require substantial task-specific reward engineering. Learning rewards from videos provides a scalable, data-driven alternative to manual reward design. However, existing video-based rewards are often computed in entangled visual spaces where task-relevant state changes are confounded with extraneous visual variation. As a result, the learned reward may respond to visual distractors instead of actual task execution. To address this limitation, we propose DiTaR (Disentangling Task-Relevant Latent Dynamics for Reward Modeling), a framework for learning dense rewards from heterogeneous videos. DiTaR learns a continuous task-relevant representation from multi-frame video latents through an asymmetric two-stage procedure that suppresses task-irrelevant visual variation. In the task-relevant latent space, we train a history- and language-conditioned rectified-flow model to capture task dynamics and derive a directional dense reward from temporal changes in dynamics compatibility. Together, these components enable reward modeling directly over task-relevant dynamics rather than generic visual variation. DiTaR substantially improves policy learning on MetaWorld-v3 and ManiSkill over video-based reward baselines, and transfers to real-world offline RL through reward relabeling. Reward analyses further show that DiTaR provides more reliable rewards.
• 中文摘要 奖励设计仍然是机器人操作强化学习的主要瓶颈。手工制作的密集奖励可以弥补稀疏奖励所提供的有限指导，但需要大量针对任务的奖励工程。从视频中学习奖励提供了一种可扩展、数据驱动的替代选择，替代手动奖励设计。然而，现有基于视频的奖励常常在纠缠的视觉空间中计算，任务相关状态变化与外来视觉变化混淆。因此，学习到的奖励可能对视觉干扰物产生反应，而非实际任务执行。为解决这一限制，我们提出了DiTaR（解开任务相关潜在动态以实现奖励建模）框架，用于从异构视频中学习密集奖励。DiTaR通过非对称的两阶段过程，从多帧视频潜在状态中学习连续的任务相关表示，抑制任务无关的视觉变化。在任务相关潜在空间中，我们训练了一个历史和语言条件的整流流模型，以捕捉任务动态，并从动态兼容性的时间变化中推导出方向性密集奖励。这些组件共同使得基于任务相关动态而非泛泛的视觉变化进行奖励建模。DiTaR在MetaWorld-v3和ManiSkill上的策略学习相较于基于视频的奖励基线显著提升，并通过奖励重新标记转移到现实世界的离线强化学习。奖励分析进一步表明，DiTaR提供了更可靠的奖励。

AutoEnvScaling: Automating the Data Flywheel with Terminal Agents

自动环境扩展：使用终端代理自动化数据飞轮

• Authors: Simon Yu, Baolin Peng, Bo Liu, Zhengyan Shi, Zhaoyang Wang, Hao Cheng, Qianhui Wu, Wenlin Yao, Natasha Jaques, Weiyan Shi, Jianfeng Gao
• Subjects: Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14163
• Pdf link: https://arxiv.org/pdf/2610.14163
• Abstract Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.
• 中文摘要 终端代理通过在终端中执行命令来解决软件工程和科学研究中的任务。通过强化学习（RL）进行训练需要多样且可验证的环境，但这些环境大多仍由人类研究人员设计，因此其供给增长速度取决于研究人员的工作速度。我们引入了AutoEnvScaling，将环境设计视为终端任务，利用代理自动化数据飞轮。提案代理从被训练求解器的部署中构建新的强化学习环境，求解器的新部署引导下一轮。通过将环境生成转变为终端任务，AutoEnvScaling使有效环境速率提升多达25倍，且每个有效环境的成本降低高达75%。我们展示了，在 GPT-5.6-sol 的 AutoEnvScaling 环境中训练 Qwen3.5-35B-A3B 时，其 Terminal-Bench 2.1 分数比匹配的 Tmax 基线提升了 4.3 分。AutoEnvScaling 的更广泛目标是实现递归自我改进（RSI）：通过同时实现环境生成和终端任务，使同一模型在两个角色中共同进化。我们显示，Qwen3.6-35B-A3B 在 Terminal-Bench 2.1 上提升了 12.2 分，并将 Terminal-Bench 4.0 得分从 1.6 提升至 3.2。该流水线还跨越了代理领域，生成软件工程、计算机使用、专业工作和 GPU 内核编程的环境。总体来看，这些结果显示出了智能体构建自身训练环境、减少人类依赖、迈出 RSI 核心一步的路径。

Visualizing the World Model in an Agent's Value Function

在智能体价值函数中可视化世界模型

• Authors: Shawheen Naderi, Fatemeh Akherati, Yubei Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14186
• Pdf link: https://arxiv.org/pdf/2610.14186
• Abstract Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.
• 中文摘要 无模型强化学习算法实际上会学习环境模型吗？在本研究中，我们应用了反事实干预，使状态的所有内容都固定，除了目标变化外，以探讨近端策略优化代理的价值函数是否构成了制造者环境的世界模型。尽管仅被训练以优化奖励，我们发现价值函数学习了诸如钻石可收藏规则、水和食物对健康的影响、库存启发式等概念。最后，我们讨论了可视化的安全性影响以及当前强化学习代理可能阻碍我们构建真正先进机器智能的一些不足。

Learning to Sync at the Edge: Client-Side Adaptive Synchronization for Federated Learning

学习边缘同步：联邦学习的客户端自适应同步

• Authors: Ebasa Temesgen, Ammar Ahmed, Nathan Lechamo, Ali Anwar, Maria Gini
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.14268
• Pdf link: https://arxiv.org/pdf/2610.14268
• Abstract The scalability and performance of edge federated learning are fundamentally constrained by synchronization bottlenecks. Clients operate under heterogeneous compute, time-varying communication conditions, and, in some settings, privacy constraints, yet synchronization is often still handled with fixed schedules. However, synchronization decisions are often made outside the client, even though the client directly observes local progress, resource state, and communication conditions. We present a client-side reinforcement learning synchronization framework in which each client decides during training whether to upload, download, synchronize bidirectionally, or defer communication. The framework places a lightweight scheduler alongside the local model and formulates synchronization as a resource-aware control problem driven by local learning and system state. We study this framework on two representative edge workloads, spatiotemporal traffic forecasting and image classification. Across both settings, adaptive client-side synchronization improves the accuracy-cost trade-off, reducing system cost in traffic forecasting while maintaining competitive cost and improving final accuracy in image classification. We also consider privacy-constrained settings, where synchronization must account for not only latency and energy but also privacy expenditure. Overall, our results show that synchronization in edge federated learning is better treated as an adaptive client-side control problem than only as a server-side coordination mechanism.
• 中文摘要 边缘联邦学习的可扩展性和性能从根本上受到同步瓶颈的限制。客户端在异构计算、时变通信条件下运行，且在某些环境中存在隐私限制，但同步通常仍以固定的计划处理。然而，同步决策往往在客户端外部做出，尽管客户端直接观察本地进展、资源状态和通信条件。我们提出了一个客户端强化学习同步框架，每个客户端在训练过程中决定是上传、下载、双向同步还是延迟通信。该框架将一个轻量级调度器置于本地模型旁边，并将同步表述为由本地学习和系统状态驱动的资源感知控制问题。我们研究该框架基于两种代表性的边缘工作负载：时空交通预测和图像分类。在这两种环境下，自适应客户端同步改善了准确性与成本的权衡，降低了流量预测中的系统成本，同时保持了竞争成本并提升了图像分类的最终准确性。我们还考虑了隐私受限的环境，其中同步不仅要考虑延迟和能源，还要考虑隐私消耗。总体来看，我们的结果表明，边缘联邦学习中的同步更应被视为一个自适应的客户端控制问题，而不仅仅是服务器端的协调机制。

The Thinking Leakage in Reinforcement Learning for LLMs

大型语言模型强化学习中的思维泄漏

• Authors: Wenxuan Zhu, Shuming Liu, Bang An, Aznaur Aliev, Bing Li, Yibo Yang, Bernard Ghanem
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.14270
• Pdf link: https://arxiv.org/pdf/2610.14270
• Abstract Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens. A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance. We provide evidence that much of this gain can be explained by thinking leakage: the gain comes largely from partially reactivating the reasoning abilities of the original model's THINK mode instead of learning enhanced reasoning behavior. We train Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK mode. As accuracy improves, responses become longer and more reflective, are increasingly judged by the original model to resemble its own thinking, and the model's internal activations shift toward its THINK mode. We then investigate at the level of weights, activations, and attention heads: scaling the RL update further elicits THINK-mode behavior and approaches THINK-mode accuracy but does not exceed it; in Qwen3-8B, replacing a low-dimensional THINK-related activation subspace of the RL-trained model with its original-model value removes most of the RL gain, and the reverse exchange transfers a considerable fraction of the gain to the original model; and the RL-induced shift passes through the attention heads that read the empty think block. We further identify a possible mechanism in the GRPO training signal: responses favored by the advantage-weighted update are more THINK-like, and a gradient step computed from this signal moves the model toward its THINK state. Thus, gains in NO-THINK mode can largely reflect reactivation of reasoning abilities already present in the original model.
• 中文摘要 混合推理模型如Qwen3系列通过几个聊天模板标记在THINK模式和NO-THINK模式之间切换。一种流行的做法是用强化学习训练这些模型，并在NO-THINK模式下报告推理表现有所提升。我们提供了证据表明，这种提升很大程度上可以用思维泄漏来解释：这种提升主要来自于部分重新激活原模型THINK模式的推理能力，而不是学习增强推理行为。我们用GRPO在NO-THINK模式下训练Qwen3-1.7B、4B和8B。随着准确率的提升，反应变得更长且更具反思性，原始模型越来越多地判断其思维方式，模型的内部激活也逐渐向其THINK模式转变。随后我们在权重、激活和注意力集中力层面进行研究：放大强化学习更新进一步引发THINK模式行为，接近THINK模式的准确性，但未超过该水平;在Qwen3-8B中，用原始模型值替换RL训练模型中低维THINK相关的激活子空间，去除大部分RL增益，反向交换将相当一部分增益转移给原模型;而RL引起的偏移则通过读取空白思考块的注意力头。我们进一步识别了GRPO训练信号中的一种可能机制：优势加权更新偏好的反应更接近THINK，而从该信号计算的梯度步进使模型向THINK状态移动。因此，NO-THINK模式下的收益很大程度上反映了原始模型中已具备的推理能力的重新激活。

ASSET: Adaptive Speech Segmentation via Reinforcement Learning

优势：通过强化学习实现自适应语音分割

• Authors: Cihan Xiao, Yuan Tseng, Matthew Wiesner, Desh Raj, Leibny Paola Garcia Perera, Sanjeev Khudanpur, David Harwath
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.14309
• Pdf link: https://arxiv.org/pdf/2610.14309
• Abstract Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder. However, LLMs scale poorly with input sequence length, while acoustic encoders tend to produce substantially longer sequences than typical textual inputs. This motivates learning to compress encoder outputs while preserving task-oriented information. To this end, we introduce ASSET, a reinforcement learning (RL)-based framework that learns to adaptively compress encoder outputs by first partitioning them into variable-duration segments and then pooling each segment into a continuous audio token. We use group-relative policy optimization (GRPO) to train a segmenter with task performance feedback and a token-rate penalty. In automatic speech recognition (ASR) experiments, ASSET achieves performance comparable to a transcript-assisted character-boundary oracle with approximately 25% fewer audio tokens on English LibriSpeech, while improving on a phone-boundary oracle at a similar token rate on Mandarin AISHELL. Exploratory experiments on non-ASR tasks further illustrate how learned segmentation changes when the framework is adapted to tasks beyond transcription.
• 中文摘要 语音大型语言模型（LLMs）通常使用预训练声学编码器作为LLM解码器的音频接口构建。然而，LLMs随着输入序列长度的扩展性较差，而声学编码器往往产生比典型文本输入更长的序列。这促使我们学习压缩编码器输出，同时保留任务导向信息。为此，我们引入了ASSET，一个基于强化学习（RL）的框架，通过先将编码器输出划分为可变时长段，然后将每个段池化为连续的音频令牌，学习自适应压缩编码器输出。我们使用组相对策略优化（GRPO）训练分段器，并带有任务性能反馈和令牌率惩罚。在自动语音识别（ASR）实验中，ASSET在英语LibriSpeech上以约25%的音频标记数量，实现了与转录辅助字符边界预言机相当的性能，同时在普通话AISHELL上以类似的令码率提升了电话边界预言机。非ASR任务上的探索性实验进一步展示了当框架适应转录以外任务时，所学分割的变化。

OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

OATH：灵巧手部的物体无关追踪，实现基于物理的快速重定向

• Authors: Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon, Joel Burdick, Dingqi Zhang, Jiuguang Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.14379
• Pdf link: https://arxiv.org/pdf/2610.14379
• Abstract Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.
• 中文摘要 将人类演示重新定位为动态可行的灵巧手部轨迹需要解决复杂的接触物理，但当前方法要么省略物理约束，要么依赖计算量高的基于采样的模拟优化。强化学习此前曾被用于通过接触动力学实现规划和控制;然而，标准策略需要针对对象进行特定训练，且无法在任意几何上泛化。这项工作将基于物理的重定向重新定位为残差学习问题，我们证明了零点泛化依赖于结构化状态表示。具体来说，将观察空间设计为包含引导每个机器人链接到目标坐标的局部接触误差信号，从而实现单一统一的残差策略，成功把握新颖且未见的物体几何形态。在运动学参考轨迹模拟中推广该策略，得到基于物理的重定向器（OATH）。OATH以每秒参考计算0.4秒运行，实现了比实时快2.4倍的性能，计算成本比基于采样的方法降低了60-350倍。该系统在保持DexYCB轨迹的58%和未可见物体的DexYCB轨迹中恢复了58%，同时保持了不到2厘米的跟踪误差。

Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations

稀疏-TD-JEPA：利用稀疏潜在任务表示改进零样本强化学习

• Authors: Duaa Nakshbandi, Zhiyu An, Wan Du
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14400
• Pdf link: https://arxiv.org/pdf/2610.14400
• Abstract Successor-feature-based zero-shot reinforcement learning methods pre-train a task encoder and successor-feature predictor on offline, reward-free data, then recover a policy at test time by solving a closed-form linear regression over a small labeled dataset. We identify an underexplored limitation of this approach: when the task encoder produces dense representations, the inferred reward signal becomes diffuse across the state space, reducing the contrast between goal-relevant states and the background and providing a weaker signal for the policy to follow. We address this limitation by introducing an l1 sparsity penalty that concentrates reward-predictive structure into a smaller effective subspace of the task encoder's latent space. The penalty is applied only during training and requires no architectural change or modification to the closed-form inference procedure. We instantiate this approach as Sparse-TD-JEPA, a single-line of code addition based on Temporal-Difference Joint-Embedding Predictive Architecture (TD-JEPA) that significantly improved its performance in zero-shot reinforcement learning. We evaluate Sparse-TD-JEPA on 65 tasks across the DeepMind Control (DMC) Suite and OGBench, covering locomotion, navigation, trajectory stitching, puzzle solving, and robotic manipulation. Sparse-TD-JEPA improves over TD-JEPA in all 13 domain-level comparisons. On DMC, it increases the average return from 661 +- 6 to 714 +- 3. On OGBench, it increases the average success rate from 38.0 +- 0.8 to 48.4 +- 2.3, with improvements of 13.4-18.0 percentage points on the evaluated antmaze navigation and stitching domains. Overall, Sparse-TD-JEPA achieves the highest mean performance among established zero-shot baselines in nine of the 13 domains. Analysis further shows that sparsity produces a sharper and more task-informative inferred reward landscape.
• 中文摘要 基于后继特征的零样本强化学习方法在离线无奖励数据上预训练任务编码器和后继特征预测器，然后在测试时通过对一个小标记数据集的闭式线性回归求解恢复策略。我们发现了该方法的一个未被充分探讨的局限性：当任务编码器产生密集表示时，推断出的奖励信号会在状态空间中分散，降低目标相关状态与背景之间的对比，并为策略提供较弱的信号。我们通过引入L1稀疏性惩罚来解决这一限制，将奖励预测结构集中到任务编码器潜在空间的较小有效子空间中。该惩罚仅在训练期间应用，且不需对封闭形式推理过程进行架构调整或修改。我们将该方法实例化为Sparse-TD-JEPA，基于时间差联合嵌入预测架构（TD-JEPA）的单行代码添加，显著提升了其在零样本强化学习中的表现。我们在DeepMind Control（DMC）套件和OGBench中评估了Sparse-TD-JEPA的65项任务，涵盖运动、导航、轨迹拼接、解谜和机器人操作。在所有13个领域级比较中，Sparse-TD-JEPA均优于TD-JEPA。在DMC中，平均回报从661+-6提升至714+-3。在OGBench上，平均成功率从38.0 +-0.8提升至48.4+-2.3，评估的迷宫导航和缝合领域提升了13.4-18.0个百分点。总体而言，稀疏-TD-JEPA在13个领域中有9个领域实现了既有零射击基线的平均表现最高。分析进一步表明，稀疏性产生了更锐利且更具任务信息量的推断奖励景观。

SAT-RL: Training Language Models for Logical Reasoning with Symbolic Solvers

SAT-RL：用符号求解器训练逻辑推理的语言模型

• Authors: Jialiang Sun, Kuldeep Meel
• Subjects: Subjects: Artificial Intelligence (cs.AI); Symbolic Computation (cs.SC)
• Arxiv link: https://arxiv.org/abs/2610.14402
• Pdf link: https://arxiv.org/pdf/2610.14402
• Abstract Large language models (LLMs) have achieved strong performance in mathematical and logical reasoning, yet they still suffer from the curse of complexity and struggle as problem difficulty increases with the number of constraint interactions and the size of the search space. In contrast, satisfiability solvers such as Z3 provide scalable and verifiable reasoning through constraint solving under formal theories, but lack the flexibility to interpret natural-language problem descriptions. In this work, we propose SAT-RL, a reinforcement-learning (RL) framework that trains LLMs to interact with symbolic solvers through multi-turn tool execution. SAT-RL leverages synthetic data generation with verifiable solutions, supervised warm-up on solver-augmented traces, and solver-grounded reinforcement learning with tool-aware rewards. Experiments show that SAT-RL substantially improves symbolic reasoning with tools on both in-domain and out-of-domain logic tasks. Our code is available on this https URL.
• 中文摘要 大型语言模型（LLMs）在数学和逻辑推理方面表现出色，但随着约束交互次数和搜索空间规模增加，问题难度增加，它们仍面临复杂性和困难。相比之下，像Z3这样的可满足性求解器通过形式理论下的约束求解提供可扩展且可验证的推理，但缺乏解释自然语言问题描述的灵活性。本研究提出SAT-RL，一种强化学习（RL）框架，通过多回合工具执行训练LLM与符号求解器交互。SAT-RL利用可验证解的合成数据生成、求解器增强轨迹的监督热身，以及基于求解器的基于工具的强化学习。实验表明，SAT-RL通过域内外逻辑任务的工具显著提升了符号推理能力。我们的代码可以在这个 https URL 上获取。

SPROUT: Scaling Agentic Reinforcement Learning for Long-Horizon Software Engineering

SPROUT：面向远视野软件工程的智能体强化学习扩展

• Authors: Mengyang Liu, Weiming Wang, Haixin Nan, Yu Wu, Peize Li, Shihong Huang, Xuning He, Xue Jiang, Yihong Dong
• Subjects: Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14405
• Pdf link: https://arxiv.org/pdf/2610.14405
• Abstract Reinforcement learning (RL) has driven substantial progress in coding agents by relying on verified successful trajectories as learning signals. However, such trajectories are scarce in long-horizon software engineering tasks, where a minor error can invalidate the entire rollout. Repeatedly restarting from the initial state is commonly used yet inefficient, and resetting only the context with working directory misses state changes such as installed dependencies. Therefore, branching from intermediate steps to preserve earlier progress requires restoring both the agent context and execution environments. In this paper, we present SPROUT (SnaPshot-based RollOUT), an agentic RL rollout framework that scales long-horizon coding tasks by resuming and branching from intermediate states. Specifically, SPROUT was built on top of a lightweight infrastructure that checkpoints both the filesystem and the agent context after each assistant turn. A verifier-guided backtracking policy then selects where to resume a failed attempt and how to proceed. Extensive experiments on long-horizon coding tasks show that SPROUT achieves up to 23.6% improvement over pass@k across various LLMs, while using $32$--$52%$ of their tokens. For RL training of Qwen3.8-27B, SPROUT stably outperforms standard GRPO by $13.9$ points on three long-horizon coding benchmarks while cutting the time per training step by $39.5%$. SPROUT thus makes RL on long-horizon coding tasks both more effective and cheaper to scale.
• 中文摘要 强化学习（RL）通过依赖已验证的成功轨迹作为学习信号，推动了编码代理的显著进展。然而，在长期的软件工程任务中，这种轨迹较为稀缺，因为一个小错误就可能使整个推广失效。从初始状态反复重启很常见但效率低下，仅用工作目录未命中重置上下文会导致状态变化，比如安装依赖。因此，从中间步骤分支以保留早期进展需要恢复代理上下文和执行环境。本文介绍了SPROUT（基于SnaPshot的RollOUT），这是一种代理式强化学习（RL）展开框架，通过从中间状态恢复和分支来扩展长期编码任务。具体来说，SPROUT建立在一个轻量级基础设施之上，该基础设施在每次助手回合后检查文件系统和代理上下文。验证者引导的回溯策略随后选择在哪里继续失败尝试，以及如何继续。对长期视野编码任务的广泛实验表明，SPROUT在使用32至52%的代币时，比pass@k提升高达23.6%。对于Qwen3.8亿至27亿的强化学习训练，SPROUT在三个长视野编码基准测试中稳定地比标准GRPO高出13.9美元，同时每个训练步骤缩短了39.5美元。因此，SPROUT使RL在长视野编码任务上更有效且更易扩展。

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

KV4-RL：解锁RLVR中带有INT4 KV缓存量化的更快异步展开

• Authors: Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14448
• Pdf link: https://arxiv.org/pdf/2610.14448
• Abstract Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail generation. Sustained concurrency also concentrates context processing into fewer decode steps, increasing the KV workload per step. We characterize this \emph{KV concentration} and find that KV cache quantization delivers substantially greater acceleration than weight-only and weight-activation quantization in our matched-workload comparisons. However, INT4 KV rollout can degrade learning even with token-level truncated importance sampling (TIS). We propose \emph{KV4-RL}, which combines calibration-free INT4 KV quantization with selective INT8 protection of attention sinks and recent context. Protecting a bounded number of positions reduces sampler mismatch while retaining most of the speedup from INT4. Across 400 iterations of Qwen3-8B-Base GRPO training, KV4-RL achieves $1.80\times$ cumulative rollout speedup and $1.27\times$ cumulative full-iteration speedup over BF16 rollout, reaching $1.97\times$ rollout speedup over the final 50 iterations. Under a common BF16 evaluator, its final six-benchmark mean accuracy is within 0.5 percentage points of BF16-rollout training for both Qwen3-4B-Base and Qwen3-8B-Base.
• 中文摘要 异步展开通过减少长尾生成过程中的不足利用率，提高了可验证奖励强化学习（RLVR）的效率。持续并发还使上下文处理集中于更少的译码步骤，增加了每步的KV工作负载。我们对这种\emph{KV集中}进行了表征，发现在匹配工作负载比较中，KV缓存量化比仅权重和权重激活量化带来显著更高的加速。然而，即使采用代币级截断重要性采样（TIS），INT4 KV的推广也可能降低学习质量。我们提出了\emph{KV4-RL}，结合了无校准的INT4 KV量化与选择性INT8对注意力汇和近期上下文的保护。保护有界数量的位置减少采样器错配，同时保留了大部分INT4带来的加速。在400次Qwen3-8B基础GRPO训练中，KV4-RL实现了比BF16部署的累计加速$1.80\x$和$$1.27\倍$的全迭代加速，最终50次迭代中达到$1.97\乘时$的推广加速。在共同的BF16评估器下，其最终六个基准的平均准确率与Qwen3-4B-Base和Qwen3-8B-Base的BF16推广训练均相差不到0.5个百分点。

Co-adaptive Rule Routing and Constraining for Structured Knowledge Retrieval

结构化知识检索的共适应规则路由与约束

• Authors: Zhisheng Qi, Haoyu Han, Jiaxi Li, Bing He, Huiyuan Chen, Franck Dernoncourt, Ryan A. Rossi, Mahantesh M. Halappanavar, Jin Lu, Yu Zhang, Tianming Liu, Yu Wang
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14465
• Pdf link: https://arxiv.org/pdf/2610.14465
• Abstract Structured knowledge retrieval acquires task-relevant evidence from structured knowledge sources for downstream tasks. Existing methods rely on unconstrained retrieval or predefined query-rule supervision, such as author-paper rules for scientific discovery and anatomy-drug rules for precision medicine. Such supervision is rarely available in practice. We propose Co-Adaptive Rule Routing and Constraining Retrieval (CoRE), which jointly learns which reasoning rule to assign to each query and how to retrieve evidence under that rule, without predefined query-rule pairs. A routing agent selects a rule, and its corresponding retriever traverses the knowledge base and ranks evidence that satisfies the rule's constraints. Because routing and retrieval are interdependent, CoRE alternates between reinforcement learning for the routing policy and contrastive learning for the rule-constrained retriever. The selected rules provide supervision for retriever training, while retrieval performance rewards the routing policy. This closed-loop process enables both components to co-adapt without explicit query-rule supervision. Experiments on STaRK-QA datasets across scientific discovery, biomedical reasoning, and e-commerce knowledge management demonstrate CoRE's effectiveness. Ablations show the benefit of co-adapting the router and retriever. Generation-based routing transfers across datasets and benefits from multi-domain pretraining followed by target-specific fine-tuning. Analysis of rule usage across queries also informs automatic rule refinement and discovery.
• 中文摘要 结构化知识检索通过结构化知识源获取与任务相关的证据，用于下游任务。现有方法依赖于不受约束的检索或预定义的查询规则监督，例如科学发现的作者-论文规则和精准医学的解剖-药物规则。但实际上很少有此类监督。我们提出了共适应规则路由与约束检索（CoRE），它共同学习每个查询应分配哪条推理规则以及如何在该规则下检索证据，无需预定义的查询-规则对。路由代理选择一条规则，其对应的检索器会穿越知识库并对满足规则约束的证据进行排序。由于路由和检索相互依赖，CoRE在路由策略的强化学习和规则约束检索器的对比学习之间交替进行。所选规则为检索器训练提供监督，而检索性能则奖励路由策略。这一闭环过程使两个组件能够在无需明确查询规则监督的情况下协同适应。在科学发现、生物医学推理和电子商务知识管理等STaRK-QA数据集上的实验展示了CoRE的有效性。消融分析展示了共适路由器和检索器的优势。基于生成的数据集间路由传输，并受益于多域预训练及针对特定目标的微调。对跨查询的规则使用分析还为自动规则细化和发现提供依据。

ScienceBuddy-PV: A Verifier-Governed Recursive Improvement Protocol for Panvascular Intervention Research

ScienceBuddy-PV：一种验证者控制的递归改进方案，用于泛血管干预研究

• Authors: Lingsen You, Yujun Guo, Xinyu Zhong, Zijie Feng, Wentong Wang, Li Shen, Junbo Ge
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.14523
• Pdf link: https://arxiv.org/pdf/2610.14523
• Abstract ScienceBuddy-PV adapts the ScienceBuddy recursive-in-recursive design to panvascular intervention research through a structured evidence ledger, device-vessel ontology, abstention contract and retained failure bank. Recursive harness optimization and outer-loop model learning, including reinforcement learning, are specified but not executed. We audit a fixed component, rather than the complete agent system, using separate ARCADE syntax and stenosis tasks (1,000 training, 200 validation and 300 test images per task). A shared task-masked convolutional model was trained once per representation setting; its parameters and the heuristic support gate were then held fixed, with the stenosis threshold selected on validation data. On the locked test sets, syntax Dice was 0.475 (95% CI 0.459-0.490) and stenosis Dice was 0.231 (95% CI 0.214-0.249). Stenosis pixel AUROC was 0.957 and AUPRC was 0.180 at prevalence 0.00975. The gate accepted 76.7% of stenosis-frame outputs versus 100% for a score-only comparator. In a constructed mask audit, it accepted 44.3% of reference candidates and rejected 55.7%, exposing the cost of abstention. Acceptance does not establish correctness or clinical safety. This public-data component audit makes support and abstention measurable and provides a reproducible protocol for future collaborative agent studies. It does not demonstrate recursive improvement, patient-level validity or clinical device recommendations.
• 中文摘要 ScienceBuddy-PV通过结构化证据账本、设备-容器本体、弃权合同和保留失效库，将ScienceBuddy递归中递归设计适应泛血管干预研究。递归束带优化和外环模型学习（包括强化学习）被指定但未执行。我们审计固定组件，而非完整代理系统，使用分别的ARCADE语法和狭窄任务（每个任务1000训练、200验证和300测试图像）。每个表示设置训练一次共享任务掩蔽卷积模型;其参数和启发式支持门保持固定，狭窄阈值由验证数据选择。在锁定测试集中，语法Dice为0.475（95% CI 0.459-0.490），狭窄Dice为0.231（95% CI 0.214-0.249）。狭窄像素AUROC为0.957，AUPRC为0.180，患病率为0.00975。该门接受了76.7%的狭窄帧输出，而仅用分数对照时为100%。在结构性掩膜审核中，接受了44.3%的参考候选人，拒绝了55.7%，暴露了弃权的成本。接受不证明正确性或临床安全性。该公开数据成分审计使支持和保留可测量，并为未来合作代理研究提供了可重复的方案。它不显示递归性改善、患者层面的有效性或临床器械推荐。

From Video to Vision: Learning Generalizable Humanoid Carrying from Human Motion

从视频到视觉：从人体运动学习可推广的人形携带

• Authors: Ziluo Ding
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.14561
• Pdf link: https://arxiv.org/pdf/2610.14561
• Abstract Most existing approaches rely on video or motion-capture data for reinforcement learning (RL) training; however, all of these methods require an upper-level module to generate reference trajectories in real time. In object carrying tasks, the humanoid can only lift objects from designated locations along the reference trajectory, resulting in poor spatial generalization. Moreover, existing policies can only be evaluated inside a motion-capture studio. To address these limitations, we propose \textbf{MOVER}, a multi-stage, Teacher-Assistant-Student, RL training framework for humanoid object carrying. We first collect data from videos and employ GENMO and GMR to retarget the video data, combining this with data augmentation techniques to generalize the data across objects of varying shapes and sizes. In the \textbf{first stage}, building upon the DeepMimic \cite{peng2018deepmimic} motion-training framework, we train the humanoid to transport objects from specified locations along designated trajectories. In the \textbf{second stage}, we relax the tracking of the reference motion and adjust the reward so that the humanoid learns to transport objects from different positions. In the \textbf{third stage}, we apply a Dagger-PPO distillation method to distill away the proprioceptive position observations and reference joint position of the humanoid. This allows the deployed policy to operate without relying on global position information from a motion-capture system; instead, the humanoid can directly perceive the object's pose from its onboard RGB-D camera to complete the carrying task. We design a series of simulation experiments to compare the success rates across stages (Stages 1 vs. 2) and before versus after distillation. We further transfer the policy to a real humanoid, successfully transporting objects at varying positions in a motion-capture-free setting.
• 中文摘要 大多数现有方法依赖视频或动作捕捉数据进行强化学习（RL）训练;然而，所有这些方法都需要一个高级模块，以实时生成参考轨迹。在携带物体的任务中，类人生物只能从参考轨迹上的指定位置举起物体，导致空间泛化较差。此外，现有策略只能在动作捕捉工作室内评估。为解决这些限制，我们提出了 \textbf{MOVER}，一个多阶段的教师-助教-学生强化学习（RL）人形物体携带培训框架。我们首先从视频中收集数据，并利用 GENMO 和 GMR 重新定位视频数据，结合数据增强技术，将数据泛化到不同形状和大小的物体上。在 \textbf{第一阶段}，基于 DeepMimic \cite{peng2018deepmimic} 运动训练框架，我们训练人形生物沿指定轨迹从指定位置运输物体。在 \textbf{第二阶段}，我们放松对参考运动的跟踪，并调整奖励，使类人生物学会从不同位置运输物体。在 \textbf{第三阶段}，我们应用 Dagger-PPO 蒸馏方法，提取人形的本体感觉位置观察和参考关节位置。这使得部署策略无需依赖运动捕捉系统的全局位置信息;相反，类人生物可以直接从其内置 RGB-D 摄像头感知物体姿态，完成携带任务。我们设计了一系列模拟实验，比较各阶段（第1与第2阶段）及蒸馏前后对比的成功率。我们进一步将该政策转移到真实的人形上，成功地在无动作捕捉的环境中以不同位置运输物体。

Chunk-wise Model-based Policy Optimization

按块计算的基于模型的策略优化

• Authors: Weixi Song, Donglin Wang, Yong-Lu Li
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14564
• Pdf link: https://arxiv.org/pdf/2610.14564
• Abstract Model-based reinforcement learning has shown superior sample efficiency by leveraging the learned latent dynamics either in the form of derivative-free planning/imagined rollout training or dynamics gradient-based policy updates. While these approaches achieve comparatively fast learning, they all suffer, to varying degrees, from issues of recurrent compounding errors, high inference-time latency, and instability in backward propagation through time (BPTT). In this work, we propose a model-based RL approach that uses the chunk world model as a differentiable training-time evaluator to optimize the chunk actor without recurrent model invocation. Through the collaborative chunk-wise prediction of the world model and policy, we can generate H-step hypothetical rollouts in a single forward pass and directly optimize value expansion, avoiding the costly and unstable BPTT. We evaluate our method on 19 challenging high-dimensional humanoid tasks from HumanoidBench and deploy our method to the real-world for quadruped bipedal walking. Project Website: this https URL
• 中文摘要 基于模型的强化学习通过利用学习到的潜在动态，无论是无导数规划/想象式展开训练，还是基于动态梯度的策略更新，都展现出了更优的样本效率。虽然这些方法实现了相对快速的学习，但它们都存在不同程度的重复复利误差、高推理时间延迟以及逆向传播（BPTT）不稳定性的问题。本研究提出一种基于模型的强化学习方法，利用区块世界模型作为可微的训练时间评估器，在不重复调用模型的情况下优化区块演员。通过对世界模型和策略的逐区块协作预测，我们可以在一次前向传递中生成H步假设性展开，并直接优化价值扩展，避免成本高昂且不稳定的BPTT。我们对HumanoidBench的19项具有挑战性的高维类人生物任务进行了评估，并将该方法应用于四足双足行走的现实世界。项目网站：此 https URL

Reward-Aligned Distillation

奖励对齐蒸馏

• Authors: Yilan Chen, Zian Su, Manish Prajapat, Dingmin Wang, Rein Houthooft, Rui Song, Vijay Lingam
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.14572
• Pdf link: https://arxiv.org/pdf/2610.14572
• Abstract Reinforcement learning with verifiable rewards improves the reasoning capabilities of large language models, but remains computationally expensive and sample inefficient. On-policy distillation (OPD) provides dense token-level supervision from a stronger teacher, yet standard OPD distills every token regardless of whether the teacher's preference actually improves task reward. Existing token-selection methods rely on proxies such as uncertainty or teacher--student divergence, which need not identify reward-improving supervision. We introduce Reward-Aligned Distillation (RAD), a bilevel framework that weights distillation signals based on their predicted effect on the student's post-update reward. RAD differentiates through a temporary optimizer step to score each token's supervision, then updates the token weights by projected ascent from the initial token weights. The resulting weights can strengthen, suppress, or reverse individual distillation-gradient contributions. At a matched training-response budget, RAD improves mean accuracy on six competition-math benchmarks over the strongest baselines by 6.5 percentage points for Qwen3-1.7B (non-thinking) and 2.5 points for Qwen3-4B-Instruct-2507. RAD-Simple, a simplified variant, achieves comparable math accuracy at the same training-response budget. On LiveCodeBench-v6, RAD reaches 54.69\% and 58.67\% pass@1 with Qwen3-1.7B (non-thinking) and OLMo3-7B-Instruct, respectively, exceeding the strongest baselines by 8.7 and 6.4 points. These results support reward-guided weighting of teacher supervision across model families and tasks.
• 中文摘要 带有可验证奖励的强化学习提升了大型语言模型的推理能力，但仍计算成本高且样本效率低下。策略提纯（OPD）提供了更强教师的密集代币级监督，而标准OPD则无论教师偏好是否实际提升任务奖励，都会提炼每个代币。现有代币选择方法依赖于不确定性或师生分歧等代理指标，这些因素无需识别奖励提升监督。我们引入了奖励对齐蒸馏（RAD）框架，这是一种双层框架，基于蒸馏信号对学生更新后奖励的预测影响加权。RAD通过临时优化器步骤对每个代币的监督进行区分，然后通过预测的初始代币权重上升来更新代币权重。所得权重可以强化、抑制或逆转单个蒸馏梯度贡献。在匹配的训练-响应预算下，RAD在最强基线上，Qwen3-1.7B（非思考）在六个竞赛数学基准中平均准确率提升6.5个百分点，Qwen3-4B-Instruct-2507提升2.5个百分点。RAD-Simple作为简化版，在相同的训练-响应预算下实现了相当的数学准确率。在LiveCodeBench-v6中，RAD分别达到54.69%和58.67%pass@1 Qwen3-1.7B（非思考）和OLMo3-7B-Instruct，分别比最强基线高出8.7分和6.4分。这些结果支持在模型家族和任务中对教师监督进行奖励引导加权。

Load-Aware Quadruped Navigation with Online Contact-based Geometry Estimation

负载感知四足导航，结合在线接触式几何估计

• Authors: Qi Yin, XiaoLi Zhang, Yangming Shi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.14596
• Pdf link: https://arxiv.org/pdf/2610.14596
• Abstract Navigating cluttered environments while carrying objects remains a challenging problem for quadruped robots. Compared with works that focused on load-aware locomotion adaptation to maintain gait stability, existing autonomous navigation systems generally ignore the influence of carried objects' geometries and use fixed footprint inflation, leading to overly conservative navigation. To fill this knowledge gap, we propose a two-phase load-aware navigation learning framework. In Phase 1, we train LOADNav, a load-aware navigation policy via reinforcement learning, enabling the robot to adapt its trajectory and clearance behavior according to the geometric size of the carried object. In Phase 2, we train CoGE, an online geometry predictor via imitation learning, which estimates unknown payload geometry from occasional contact events. We evaluate the framework in cluttered environments with loads of varying sizes and shapes. The results show that our navigation policy exhibits clear load-aware behaviors, and the online geometry predictor estimates object geometry from occasional contact events, refining its estimate and adapting the navigation behavior accordingly. Without requiring any prior knowledge of the payload, it narrows the gap to geometry-aware policies.
• 中文摘要 在携带物体时穿越杂乱环境仍是四足机器人面临的挑战。与专注于负载感知运动适应以维持步态稳定性的研究相比，现有自主导航系统通常忽视携带物体几何体的影响，采用固定足迹膨胀，导致导航过于保守。为弥补这一知识空白，我们提出了一个两阶段的负载感知导航学习框架。第一阶段，我们通过强化学习训练LOADNav，这是一种负载感知导航策略，使机器人能够根据携带物体的几何尺寸调整轨迹和间隙行为。第二阶段，我们训练CoGE，这是一种通过模仿学习的在线几何预测器，通过偶尔的接触事件估算未知有效载荷几何。我们在负载大小和形状各异的杂乱环境中评估该框架。结果显示，我们的导航策略表现出清晰的负载感知行为，在线几何预测器通过偶尔的接触事件估计物体几何体，细化估计并相应调整导航行为。无需事先了解有效载荷，它将差距缩小到几何感知策略。

PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents

PEARL：多回合代理的过程体验感知强化学习

• Authors: Jie Yang, Mianqiu Huang, Yiwei Zhao, Bo Wang, Yang Liu, Yuxin Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14636
• Pdf link: https://arxiv.org/pdf/2610.14636
• Abstract Terminal rewards make multi-turn agents trainable without step-level annotations, but group-relative advantages provide little guidance about which intermediate decisions to adjust. Failed trajectories also contain reasonable actions, while unfavorable decision patterns can recur as training proceeds. We introduce PEARL (Process-Experience Aware Reinforcement Learning), which improves local credit assignment without an additional critic. Given two continuations from the same history with different outcomes, PEARL locates their first divergence and builds a feature representation of each branch's local decision. It clusters these decision features in recent failure and success memories to identify recurring local patterns. Using this pattern memory, PEARL selectively strengthens negative advantages at lower-reward decisions, while leaving positive advantages and the base policy optimizer unchanged. On AppWorld, PEARL improves over ARPO on Normal and Challenge tasks at both 4B and 8B scales. On ALFWorld, it achieves 97.9% mean success at 4B, 2.7 points above the strongest baseline, while remaining competitive at 8B. Ablations of shaping location, success-bank filtering, and memory window support the value of selective, recurring-pattern-based updates.
• 中文摘要 终端奖励使多回合代理可训练而无需步级注释，但群体相对优势对调整哪些中间决策几乎无指导。失败轨迹也包含合理行动，而不利决策模式在训练过程中可能反复出现。我们引入PEARL（过程-体验感知强化学习），可在无额外批评者的情况下改善局部学分分配。鉴于同一历史的两个延续且结果不同，PEARL定位其首次分歧点，构建每个分支局部决策的特征表示。它将这些决策特征聚类于近期失败和成功记忆中，以识别反复出现的局部模式。利用该模式记忆，PEARL选择性强化低奖励决策中的负面优势，同时保持正向优势和基础策略优化器不变。在AppWorld上，PEARL在4B和8B尺度上均优于ARPO在正常和挑战任务上表现优于ARPO。在ALFWorld上，4B的平均成功率为97.9%，高于最强基线2.7分，同时在8B上保持竞争力。塑形位置、成功银行过滤和内存窗口的消融支持选择性、周期性模式更新的价值。

Lingdan-V2: Large language models for clinically grounded reasoning in traditional Chinese medicine

灵丹-V2：传统中医临床基础推理的大型语言模型

• Authors: Rui Hua, Yu Wei, Ziri Jin, Zhuo Sun, Kai Chang, Jianan Xia, Zixin Shu, Xiaodong Li, Dongmei Jia, Fei Dong, Runshun Zhang, Jian Yu, Hao Xu, Haibin Yu, Jiansheng Li, Baoyan Liu, Wenjia Wang, Xuezhong Zhou
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14705
• Pdf link: https://arxiv.org/pdf/2610.14705
• Abstract Large language models have advanced medical reasoning, but clinically grounded reasoning in traditional Chinese medicine (TCM) remains challenging because prescription decisions require integrating domain knowledge, syndrome differentiation, treatment principles, herbal prescription composition, and dosage determination. We develop a TCM language model family comprising models with 4 billion, 8 billion, and 14 billion parameters. The artificial intelligence contribution is a staged clinical alignment framework integrating large scale continued pretraining on TCM and clinical records, complementary reasoning supervision, and prescription-centered reinforcement learning (RL). We further formulate herbal prescription generation as a structured RL problem with prescription-level feedback, enabling direct optimization of clinical decision outputs beyond knowledge reproduction and generic instruction following. The engineering application is TCM clinical decision support, particularly herbal prescription recommendation. We evaluate the models on a public benchmark containing 25,620 expert-verified items across 13 TCM knowledge and clinical reasoning tasks and on a private retrospective dataset of 4,348 spleen and stomach disease prescription cases. The 14 billion parameter reinforcement aligned model achieves a prescription generation F1 of 37.0 on the public benchmark, compared with 32.3 for DeepSeek-R1 with 671 billion total parameters, and a herb-level F1 of 52.8 on the clinical dataset with improved dosage agreement. Reinforcement post-training consistently improves prescription recommendation across all scales, while physician-source analysis shows that expert heterogeneity affects reward stability. These findings demonstrate a scalable pathway from domain grounding and reasoning supervision to clinically oriented prescription decision support.
• 中文摘要 大型语言模型在医学推理方面已进步，但中医（TCM）中临床基础推理仍具挑战性，因为处方决策需要整合领域知识、综合征区分、治疗原则、草药处方成分和剂量确定。我们开发了一个包含40亿、80亿和140亿参数模型的中医语言模型家族。人工智能贡献是一个分阶段的临床对齐框架，集成了对中医和临床记录的大规模持续预训练、补充推理监督以及以处方为中心的强化学习（RL）。我们还进一步将草药处方生成表述为一个结构化的强化学习问题，支持处方级反馈，从而实现临床决策输出的直接优化，超越知识再现和通用指令跟随。工程应用是中医临床决策支持，特别是草药处方推荐。我们在一个包含25,620个专家验证项目、涵盖13项中医知识和临床推理任务的公开基准测试，以及一个包含4,348个脾胃病处方病例的私人回顾数据集上评估模型。140亿参数强化对齐模型在公开基准上实现了37.0的处方生成F1，而DeepSeek-R1在总参数6710亿的32.3%中，在药物层面F1为52.8，且剂量一致性有所改善。培训后强化持续提升所有量表的处方推荐，而医生-来源分析显示专家异质性影响奖励稳定性。这些发现展示了从领域基础和推理监督到临床导向处方决策支持的可扩展路径。

Vela: User Behavior as Supervision Sparks LLM Intelligence

Vela：用户行为作为监督激发LLM智能

• Authors: Yang Xiao, Yusong Sun, Haoyi Wu, Wenyang Hui, Weiqiang Hang, Luqing Wang, Zejian Wang, Zirui Tu, Jingsheng Mao, Shaobo Hu, Yang Luo, Jiale Guo, Jin Zhang, Yudan Liu, Zhaokai Luo, Ren Huan, Yao Hu, Chengyue Jiang, Wenjie Li, Chuan Mu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14709
• Pdf link: https://arxiv.org/pdf/2610.14709
• Abstract Large language models are increasingly deployed in applications that act on users' long-term behavior histories, from personal assistants to recommendation and advertising systems. These applications demand the user dimension of LLM intelligence: inferring a person's preferences, intent, and likely future actions from weeks to years of behavior data. Yet current models are trained on static text or synthetic data, neither of which captures how user behavior evolves over time. Real-world behavior records, already available at scale on content platforms and in personal-assistant agents, offer a natural supervisory signal. We propose user behavior as supervision and introduce Vela, a framework that converts real user behavior data into trainable tasks and trajectories. From the anonymized behavior records of 100,000 users, Vela generates 100,000 tasks across 91 real-world task templates; difficulty-based filtering and trajectory acceptance criteria retain approximately 75,000 tasks with selected training trajectories. Fine-tuning Qwen3.6-35B-A3B on these trajectories improves performance over the base model by up to 11.5 points on user-related tasks and by an average of 4 points on general benchmarks. Training on more user behavior data further improves user-related performance, and reinforcement learning after fine-tuning outperforms reinforcement learning from the base model by up to 7.7 points. These gains suggest that real-world user behavior is a scalable source of supervision for both user-centered capabilities and the broader intelligence of LLMs.
• 中文摘要 大型语言模型越来越多地应用于基于用户长期行为历史的应用中，从个人助理到推荐和广告系统。这些应用需要大型语言模型智能的用户维度：从数周到数年的行为数据推断个人的偏好、意图以及可能的未来行为。然而，当前模型训练于静态文本或合成数据，这两者都未能捕捉用户行为随时间演变的过程。现实世界的行为记录已在内容平台和个人助理代理中大规模提供，提供了自然的监督信号。我们提出以用户行为作为监督，并引入了Vela框架，将真实用户行为数据转换为可训练的任务和轨迹。Vela从10万用户的匿名行为记录中生成了10万个任务，覆盖91个真实任务模板;基于难度的过滤和轨迹接受标准保留约75,000个任务，且有选定的训练轨迹。对Qwen3.6-35B-A3B在这些轨迹上进行微调，在用户相关任务上可使基础模型性能提升最多11.5分，在一般基准测试中平均提升4分。对更多用户行为数据进行训练进一步提升用户相关性能，微调后的强化学习表现可高出基础模型的强化学习多达7.7分。这些提升表明，真实世界的用户行为是对以用户为中心能力和大型语言模型更广泛智能的可扩展监督来源。

Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient

将基于AIRL的奖励估计整合进使用模糊推断和策略梯度的控制方法中

• Authors: Seiji Ishihara
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14723
• Pdf link: https://arxiv.org/pdf/2610.14723
• Abstract A method integrating policy gradient with fuzzy inference has been proposed. This method, hereafter referred to as the base model, has high flexibility with respect to the configuration of fuzzy inference rules and fuzzy representation. It can also automatically adjust the rule weights and the membership functions in the antecedent and consequent parts using the same framework for learning. In methods that integrate reinforcement learning with fuzzy inference, however, the reward function must be designed in advance. It is therefore useful to add learning-based reward estimation to the base model. This paper proposes a fuzzy control method that integrates a reward estimation mechanism based on the adversarial inverse reinforcement learning (AIRL) framework into the base model. The generative adversarial nets (GAN) framework adopted in AIRL, however, tends to suffer from unstable learning. To address this, the proposed method treats the policy of the base model as the generator and also uses the set of membership functions in the policy as the cost function of the discriminator. This configuration allows the discriminator update for reward estimation and the generator update for policy estimation to be performed simultaneously on the parameters shared by both, without estimating a value function. In simulations of an illustrative vehicle speed control task, the proposed method acquired policies with high generalization performance through stable and efficient learning.
• 中文摘要 已有一种将策略梯度与模糊推理整合的方法被提出。该方法（以下简称基础模型）在模糊推理规则和模糊表示的配置上具有高度灵活性。它还能自动调整前件和后件部分的规则权重和成员函数，使用相同的学习框架。然而，在将强化学习与模糊推理整合的方法中，奖励函数必须提前设计。因此，在基础模型中加入基于学习的奖励估计是有用的。本文提出了一种模糊控制方法，将基于对抗逆强化学习（AIRL）框架的奖励估计机制集成到基础模型中。然而，AIRL采用的生成对抗网络（GAN）框架往往存在不稳定学习的问题。为此，所提方法将基模型的策略视为生成元，同时将策略中的成员函数集合作为判别器的成本函数。该配置允许奖励估计的判别器更新和策略估计的生成器更新同时对两者共享的参数进行，而无需估计价值函数。在示例车辆速度控制任务的模拟中，所提方法通过稳定高效的学习获得了具有高泛化性能的策略。

Equal-information evaluation of credit assignment in modular financial reinforcement learning

模块化金融强化学习中学分分配的等信息评估

• Authors: Gennadiy Chistiakov, Yury Yanovich
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14750
• Pdf link: https://arxiv.org/pdf/2610.14750
• Abstract Credit assignment in modular financial reinforcement learning is difficult to evaluate because an attributed signal may improve a component loss without adding decision-relevant information or improving economic outcomes. We propose an evaluation framework combining equal-information reconstruction controls, fixed-utility comparisons and prespecified downstream guardrails. The framework is instantiated in an attribution-guided modular trading architecture and tested in three controlled studies covering update allocation, supervision-target representation and portfolio risk control. We find that residual-based allocation is exactly reproduced by an ordinary loss-budget control under the disclosed interface; direct shortfall supervision improves component prediction but fails decoded-outcome preservation; and component-wise critic supervision worsens risk-condition certainty equivalent and drawdown relative to scalar supervision under fixed utility. These results identify three distinct evaluation failures: redundant attribution signals, representation-induced prediction trade-offs, and misalignment between component supervision and downstream risk outcomes. A preserved historical engineering case further shows why module-level learning claims require run-level provenance. The contribution is a reusable methodology for evaluating credit assignment in expert systems operating under partial observability, delayed feedback and economic risk constraints.
• 中文摘要 模块化金融强化学习中的信用分配难以评估，因为归因信号可能在不增加决策相关信息或改善经济结果的情况下改善组件损失。我们提出了一个结合等信息重构控制、固定效用比较和预设下游保护栏的评估框架。该框架在归因引导的模块化交易架构中实现，并在三项受控研究中进行了测试，涵盖更新配置、监督目标表示和投资组合风险控制。我们发现基于剩余的配置在披露接口下被普通损失-预算控制精确复制;直接的亏损监管提升了成分预测，但未能实现解码结果的保存;而组件层次的批评监督则使风险条件的确定性和回撤相对于固定效用下的标量监管更差。这些结果识别出三种不同的评估失败：冗余归因信号、表示诱导的预测权衡，以及组件监督与下游风险结果之间的错位。一个保存下来的历史工程案例进一步说明了为何模块级学习声明需要运行级别的来源。该贡献是一种可复用的方法论，用于在部分可观测性、延迟反馈和经济风险约束下评估专家系统中的信用分配。

Getting Space Right: Agentic Rewards for Image Generation

正确把握空间：图像生成的代理奖励

• Authors: Xiaofeng Tan, Jiaxin Lin, Haobo Yuan, Tiancheng Shen, Zihan Su, Rui Chen, Hongsong Wang, Ming-Hsuan Yang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14852
• Pdf link: https://arxiv.org/pdf/2610.14852
• Abstract Reinforcement learning (RL) is promising for spatial text-to-image generation, but existing work largely focuses on policy optimization, assuming accurate rewards and effective online rollouts. We observe in-domain gains with out-of-domain degradation, exposing two limitations: unreliable spatial rewards in open-world settings and rollouts lacking basic-to-complex progression. We propose a curriculum-guided RL framework. First, we introduce SpatialCritic, an agentic reward model equipped with a spatial perception harness that combines reusable verification guidance with specialized vision tools. We then construct spatial preference datasets through targeted counterfactual perturbations and perform agent-harness alignment to improve tool selection and evidence integration efficiently. Second, curriculum learning organizes shared foundational capabilities and their compositional dependencies in a capability graph for mastery-guided rollouts, periodic review, and targeted remediation. On SpaRW-Eval, SpatialCritic achieves 95.1% verification accuracy, outperforming 320B GLM-5.3-Flash with a 40x smaller 8B backbone and raising average baseline scores from 78.4 to 83.9 on SD3.5-Medium and 83.2 to 86.5 on BAGEL. Project page: this https URL
• 中文摘要 强化学习（RL）在空间文本到图像生成方面前景看好，但现有工作主要聚焦于策略优化，假设奖励准确且在线推广有效。我们观察到域内收益与域外退化，暴露出两个局限：开放世界环境中空间奖励不可靠，以及缺乏基础到复杂进阶的推广。我们提出了一个课程引导的强化学习框架。首先，我们介绍SpatialCritic，这是一种配备空间感知束带的代理奖励模型，结合了可复用的验证指导和专业视觉工具。随后，我们通过针对性的反事实扰动构建空间偏好数据集，并进行代理-约束对齐，以高效提升工具选择和证据整合。其次，课程学习将共享的基础能力及其组合依赖性组织成能力图，用于掌握引导的推广、定期审查和有针对性的补救。在SpaRW-Eval中，SpatialCritic的验证准确率达到了95.1%，优于320B GLM-5.3-Flash（8B骨干网小40倍），并将SD3.5-Medium的平均基线分数从78.4提升到83.9，BAGEL从83.2提升到86.5。项目页面：此链接

Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds

Terrarium：通过应用世界的可扩展综合来培养个人助理

• Authors: Mengzhou Wu, Yuzhe Guo, Jiawen Tao, Xiaokun Yuan, Yuan Cao, Gang Cao, Yuetang Deng, Zhi Gong, Dezhi Ran, Linyi Li, Wei Yang, Tao Xie
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14860
• Pdf link: https://arxiv.org/pdf/2610.14860
• Abstract Advances in large language models (LLMs) have brought the vision of a general-purpose personal assistant closer to reality. Such assistants could provide a unified natural-language interface to a broad ecosystem of applications. However, real-world applications do not readily support training at scale because access is often restricted and their state is difficult to reset. To address this challenge, we introduce Terrarium, a framework that synthesizes executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics from these worlds, and iteratively validates and refines both through agent rollouts. For App World synthesis, Terrarium organizes public Web information and in-application observations into structured app specifications that guide the construction of self-contained runtimes with callable operations and mutable, resettable state. For task synthesis, Terrarium constructs tasks grounded in App World state together with evaluator-side outcome rubrics that assess success without prescribing reference trajectories. We construct a training suite of 275 App Worlds and 13,344 tasks for outcome-based reinforcement learning across three Qwen3 model scales (4B, 8B, and 14B parameters). Across these scales, models trained with Terrarium outperform the corresponding Qwen3 models by an average of 18.34 percentage points on the $\tau^2$-bench and 10.94 percentage points on the AppWorld bench. These results suggest that app-grounded world and task synthesis offers a scalable path toward general-purpose assistants capable of operating across broad application ecosystems.
• 中文摘要 大型语言模型（LLM）的进步使通用个人助理的愿景更接近现实。此类助手可以为广泛的应用生态系统提供统一的自然语言接口。然而，现实应用不易大规模支持训练，因为访问通常受限且状态难以重置。为应对这一挑战，我们引入了Terrarium框架，该框架从现实应用信息合成可执行应用世界，生成带有评估者端评分标准的训练任务包，并通过代理部署迭代验证和优化。在应用世界综合中，Terrarium将公共网络信息和应用内观察数据组织成结构化应用规范，指导构建具备可调用操作和可变、可重置状态的自包含运行时。在任务综合方面，Terrarium构建了基于App World状态的任务，并结合评估者侧的结果评分标准，评估成功率而不规定参考轨迹。我们构建了一个包含275个App World和13,344个基于结果的强化学习任务的训练套件，涵盖三个Qwen3模型尺度（4B、8B和14B参数）。在这些尺度上，使用Terrarium训练的模型在$\tau^2$-bench上平均优于相应的Qwen3模型18.34个百分点，在AppWorld平台上高出10.94个百分点。这些结果表明，基于应用的世界和任务综合为实现通用助理提供了可扩展的路径，能够在广泛的应用生态系统中运行。

MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL

MARL-BattleGrounds：一个针对异构且竞争性多智能体强化学习的创新JAX原生基准

• Authors: Ulixes Tariq Hawili, Subramanian Ramamoorthy, Yoann Altmann, Ignacio Carlucho
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.14883
• Pdf link: https://arxiv.org/pdf/2610.14883
• Abstract We present MARL-BattleGrounds (MARL-BGs), an easy-to-use JAX-native benchmark for heterogeneous and competitive multi-agent reinforcement learning (MARL), designed to be sample-efficient, computationally efficient, and researcher-centric. MARL benchmarks are typically simple and efficient, like SMAX, where winning reduces to unit micromanagement, or rich and costly, like Dota 2. MARL-BGs fills this missing middle. Its first mode, Team Deathmatch, is a simultaneous turn-based game between two teams of five heterogeneous agents. The five classes have distinct basic, passive, and ultimate abilities designed to synergize with allies and counter enemies. Our 52 hand-designed maps with diverse obstacles include a 12-map curriculum and ten held-out validation and test maps. MARL-BGs provides two canonical evaluation modes. First, eight endgame scenarios form a hard generalization challenge: each asks a team to win an unseen, nearly lost position against a strong reactive controller within 5-10 ticks. Second, a monthly round-robin tournament retrains each submission on equal compute, plays it against every member of the current "Big Nine" field, and publishes the Elo results on the official leaderboard. All entrants are publicly released as downloadable opponents. MARL-BGs records 11,192 metrics per game in 28 topics. We also include three strong scripted opponents, an analytical replay viewer, and off-the-shelf tools for self-play, cross-play, curriculum learning, and zero-shot coordination evaluation. We even provide an API for LLM-based agents and use it to evaluate Qwen3.5-9B against our strongest policy. In our BattleClient, humans can play alongside or against any mix of policies, enabling human-AI evaluation. We benchmark MARL-BGs with six additional deep MARL baselines, which we release. On one RTX 5090 with 1,024 parallel games, the simulator runs 50,000 5v5 game steps per second.
• 中文摘要 我们介绍MARL-BattleGrounds（MARL-BGs），一个易于使用的JAX原生基准测试，用于异构和竞争性多代理强化学习（MARL），旨在实现样本高效、计算高效且以研究者为中心。MARL基准测试通常简单高效，比如SMAX，胜利简化为单位微观管理，或是丰富且昂贵的，如Dota 2。MARL-BGs填补了这一缺失的中间。其首个模式Team Deathmatch是两支由五名异质代理组成的同时回合制游戏。五个职业拥有独特的基础、被动和终极能力，旨在协同盟友并克制敌人。我们手工设计的52张地图，障碍多样，包括12张地图课程和10张已发布的验证和测试地图。MARL-BGs提供两种标准评估模式。首先，八个残局场景构成一个艰难的概括挑战：每个场景要求一支队伍在5-10秒内赢得一个看不见、几乎失控的强力控制者。其次，每月循环赛将每个提交重新训练为平等计算，与当前“大九”阵容的所有成员对战，并将Elo结果发布在官方排行榜上。所有参赛者均公开作为可下载的对手。MARL-BGs每局记录28个主题的11,192项指标。我们还包括三个强力脚本对手、分析回放查看器，以及现成的自玩、跨平台、课程学习和零机会协调评估工具。我们甚至为基于LLM的代理提供API，用于根据我们最强的策略评估Qwen3.5-9B。在我们的BattleClient中，人类可以与任意策略组合并玩或对抗，实现人AI评估。我们用六个额外的深度MARL基线对MARL-BG进行了基准测试，并发布这些基准。在一台RTX 5090上，支持1024个并行游戏，模拟器每秒运行50,000个5v5游戏步。

Verify, Act, Distill: Turning Test-Time Compute into Test-Time Adaptation for Vision-Language-Action Models

验证、行动、提取：将测试时间计算转化为视觉-语言-动作模型的测试时间适应

• Authors: Kaneyoshi Hiratsuka, Satoshi Yamamori, Jun Morimoto
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.14885
• Pdf link: https://arxiv.org/pdf/2610.14885
• Abstract Test-time compute (TTC), which scores multiple candidate actions at inference time, is a powerful way to improve the execution performance of vision-language-action (VLA) models, but existing methods keep the policy frozen, so the gain must be recomputed at every execution. We propose Verify, Act, Distill, a framework that consolidates the gains of TTC into the VLA itself. A fixed verifier scores candidates generated by a flow-matching VLA, the weighted aggregate is executed, and the observations and aggregated chunks from successful adaptation episodes are distilled directly into LoRA parameters with the standard flow-matching loss. This reuses supervision that TTC has already produced during execution, without additional reinforcement learning or rollouts dedicated to teacher generation. On the 40 LIBERO tasks with a SmolVLA 5k checkpoint, distilling from only 10 episodes per task raises the mean success rate of verifier-free single-candidate generation from 54.1% to 70.6%, retaining about 74% of the improvement obtained by aggregating eight candidates. Applying the same aggregation to the distilled policy further raises the success rate from 76.4% to 82.3%, with a particularly large gain on LIBERO-Long (44.0% to 57.8%), and the single-candidate policy also improves on LIBERO-plus, which contains distribution shifts. These results show that using verification as a learning signal, rather than only as an execution-time selector, both retains the improvements of TTC and re-amplifies the effect of subsequent TTC.
• 中文摘要 测试时计算（TTC）在推断时对多个候选动作进行评分，是提升视觉语言动作（VLA）模型执行性能的强大方法，但现有方法使策略保持冻结，因此每次执行都必须重新计算收益。我们提出了“验证、行动、蒸馏”框架，将TTC的收益整合到VLA本身。固定验证器对流量匹配VLA生成的候选对象进行评分，加权聚合被执行，成功适应事件的观察和聚合块直接提炼为LoRA参数，带着标准的流匹配损失。这重用TTC在执行过程中已产生的监督，无需额外的强化学习或专门针对教师生成的推广。在40个带有SmolVLA 5k检查点的LIBERO任务中，从每个任务仅10集提炼出来，使无验证者单候选生成的平均成功率从54.1%提升至70.6%，保留约74%的8个候选人聚合后提升。将相同聚合应用于提炼策略，成功率从76.4%提升至82.3%，其中LIBERO-Long的提升尤为显著（44.0%至57.8%），单候选策略也优于包含分布偏移的LIBERO-plus。这些结果表明，将验证作为学习信号而非仅作为执行时间选择器，既保留了TTC的改进，又重新放大后续TTC的效果。

Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling

通过上下文建模弥合模型预测任务抽样中的风险近似差距

• Authors: Jiarong Wen, Qi Tao, Kaiyu Zhang, Yun Qu, Lizhou Cai, Heming Zou, Yuhang Jiang, Qi Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14913
• Pdf link: https://arxiv.org/pdf/2610.14913
• Abstract Reinforcement learning (RL) is essential for adaptive decision-making, e.g., robust Meta-RL and domain randomization (DR), and efficient post-training of large language models (LLMs), where tasks are commonly structured as Markov decision processes or prompts. A central bottleneck in all these settings is the cost of agent-environment interactions required for task selection and policy optimization. Model predictive task sampling (MPTS) has emerged as a promising approach to improve sample efficiency by querying informative tasks via a risk predictive model (RPM) trained on optimization history. However, existing RPMs suffer from two failure modes: (i) sparse optimization histories, which limit RPMs' generalization across the task space, and (ii) a one-step lag in risk approximation, which causes biased difficulty estimation as the policy evolves. This work formalizes these failure modes, derives lagged-risk decompositions that expose the resulting temporal bias, and recasts risk prediction as an in-context sequence modeling problem. The developed RPM integrates a one-step look-ahead mechanism with a temporal-weighted sliding window to mitigate data non-stationarity and scarcity, without requiring additional rollouts. Empirically, our approach yields more reliable difficulty estimates and consistent performance gains across robust Meta-RL, DR, and prompt curriculum RL post-training of LLMs.
• 中文摘要 强化学习（RL）对于自适应决策至关重要，例如稳健的元强化学习和领域随机化（DR），以及大型语言模型（LLMs）的高效后期训练，这些模型中任务通常被结构化为马尔可夫决策过程或提示。所有这些环境中的核心瓶颈是任务选择和策略优化所需的代理-环境交互成本。模型预测任务抽样（MPTS）已成为一种有前景的方法，通过基于优化历史训练的风险预测模型（RPM）查询信息性任务，提高样本效率。然而，现有RPM存在两种失败模式：（i）优化历史稀疏，限制了RPM在任务空间中的泛化;（ii）风险近似的一步滞后，随着策略演进导致难度估计有偏差。本研究形式化了这些失败模式，推导出了暴露时间偏倚的滞后风险分解，并将风险预测重新定义为上下文序列建模问题。开发的RPM集成了一步前瞻机制与时间加权滑动窗口，以减轻数据的非平稳性和稀缺性，无需额外展开。从经验上看，我们的方法在强大的元强化学习、复原和即时课程强化学习中，提供了更可靠的难度估计和持续的性能提升。

CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

CFG-政策：通过无分类器指导实现的统一扩散政策学习框架

• Authors: Shutong Ding, Zhaoyang Liu, Yimiao Zhou, Jingya Wang, Jiayuan Gu, Ye Shi
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14936
• Pdf link: https://arxiv.org/pdf/2610.14936
• Abstract Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at this https URL.
• 中文摘要 扩散策略越来越多地被机器人控制与操作、视觉-语言-动作模型以及基于世界模型的智能体所采用。它们表示多模态动作分布的能力为强化学习（RL）中的探索提供了丰富机会。然而，利用强化学习优化扩散策略可能会减少行为多样性，进一步限制其探索能力。在强化学习后期，直接微调预训练策略也可能干扰其获得的能力，而动作空间残差可能缺乏多模态探索的表达力。这些问题共同揭示了在线及离线到在线扩散学习中的共同挑战：在保持持续适应所需的行为多样性和能力的同时，提升任务性能。我们介绍了CFG-Policy，一个统一的即插即用框架，将探索和利用分离成两种生成策略，并在采样过程中通过无分类器指导结合它们的评分或速度场。对于在线强化学习，一个冻结的探索分支维持行为多样性，而利用分支则从任务奖励中学习;对于离线到在线的强化学习，预训练的利用分支保持冻结状态，而可训练分支则引导组合后的行为。CFG-Policy 保留了底层的强化学习目标，并通过 QSM 和 QVPO 提升了 5 个 MuJoCo 任务的性能，以及 Flow-SDE+PPO 在 4 个 RoboTwin 操作任务中表现。项目页面位于此 https URL。

VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning

VC-OPD：基于政策的视觉反事实提炼，以实现扎实的视觉语言推理

• Authors: Jiahe Feng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14939
• Pdf link: https://arxiv.org/pdf/2610.14939
• Abstract GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
• 中文摘要 GRPO式强化学习具备高级视觉语言推理能力，但其序列级奖励对单个推理代币的学分分配有限。策略上提炼（OPD）通过对学生生成轨迹提供密集监督来解决这一问题，而标准OPD则对所有教师偏好的代币（包括模板短语和视觉上无关的延续）赋予了类似的重要性。我们引入了视觉反事实策略提炼（VC-OPD），这是一种基于基础感知的提炼方法，优先选择依赖图像特定证据的正确性代币。在教师监督的热启动中稳定响应格式并减少噪音推广后，VC-OPD对每个学生轨迹进行两次教师评估，进行策略上的提炼：一次基于原始图像，另一次基于实例特定证据的反事实图像。原始图像教师分布确定了提纯目标，而原始——反事实教师差距估计了词级视觉依赖并对损失进行了软性重权。为避免反应水平更新尺度不稳定，VC-OPD进一步对词权重应用了保持质量的规范化。该设计保留了OPD的密集监督，同时将学习能力转向视觉基础的推理标记。多模态推理基准测试的实验表明，VC-OPD提升了视觉推理表现，并产生更高效的词级蒸馏。

Compositional Average Velocity Modeling for Efficient Diffusion Offline Reinforcement Learning

用于高效扩散离线强化学习的组合平均速度建模

• Authors: Thanh Nguyen, Abu Hanif Muhammad Syarubany, Hwanhee Kim, Chang D. Yoo
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.14941
• Pdf link: https://arxiv.org/pdf/2610.14941
• Abstract Diffusion-based policies have recently emerged as a powerful paradigm for offline reinforcement learning, but their reliance on multi-step denoising introduces substantial computational overhead and requires backpropagation through time during training, limiting scalability and practical deployment. In this work, we propose \textbf{Instant Flow Q-Learning (InsQL)}, a single-step generative policy learning framework that retains the expressive capacity of diffusion models while eliminating iterative sampling and BPTT. InsQL builds on the flow-matching perspective of diffusion policies, but replaces marginal velocity parameterization with \emph{average-velocity} modeling and introduces a principled compositional mechanism, yielding a theoretically grounded objective that learns average velocity consistently across scales. This formulation provides a direct noise-to-action mapping and enables single-step action generation. InsQL further unifies compositional average-velocity modeling with behavior-regularized actor--critic optimization in a single framework, requiring only a single policy network and avoiding distillation, multi-stage procedures, and Jacobian computations during training. At inference time, actions are generated in a single forward pass and can be refined via value-guided candidate selection to improve performance. Extensive experiments on standard offline RL benchmarks and real-world robotic manipulation tasks show that InsQL matches or surpasses state-of-the-art diffusion-based methods while substantially reducing both training and inference time. Overall, these results establish compositional average-velocity modeling as an efficient and scalable alternative to multi-step generative policies for offline RL.
• 中文摘要 基于扩散的策略最近成为离线强化学习的强大范式，但其对多步去噪的依赖带来了巨大的计算开销，并且在训练过程中需要反向传播，限制了可扩展性和实际部署。在本研究中，我们提出了\textbf{即时流Q学习（InsQL）}，一种单步生成策略学习框架，在消除迭代采样和BPTT的同时，保留了扩散模型的表达能力。InsQL基于扩散策略的流匹配视角，但用平均速度建模取代了边际速度参数化，并引入了有原则的合成机制，从而实现了理论基础，能够在不同尺度上一致学习平均速度。该表述提供了直接的噪声到动作映射，并支持单步动作生成。InsQL 进一步统一了组合平均速度建模与行为正则化的演员-批判优化，在单一框架内实现，只需单一策略网络，且避免了训练中的蒸馏、多阶段过程和雅可比计算。在推理时，动作通过一次前向传递生成，并通过价值引导候选选择进行优化以提升性能。对标准离线强化学习基准测试和现实机器人操作任务的广泛实验表明，InsQL 能够匹配甚至超越最先进的基于扩散的方法，同时显著缩短训练和推理时间。总体而言，这些结果确立了组合平均速度建模作为离线强化学习多步生成策略的高效且可扩展的替代方案。

Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning

离线对比性目标条件强化学习的反事实推广

• Authors: Tingrui Huang, Ai Bo, Junzhe Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14953
• Pdf link: https://arxiv.org/pdf/2610.14953
• Abstract Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
• 中文摘要 对比强化学习（CRL）通过将状态-动作对与后续达到的状态关联来学习目标条件值，假设没有未观察变量共同影响记录的动作及其结果。当离线演示来自对学习者隐藏信息的演示者时，这一假设失效，批评者吸收了虚假的动作——结果关联。我们用混淆的马尔可夫决策过程形式化该设定，并引入反事实转移，将数据识别的部分与未识别的区分开来;最小化所有兼容的转移，得到最坏情况下占用率，从而降低真实的下界。因果对比强化学习（CauCRL）学习从正负的离线演示中进行悲观的反事实转换，并培训批评者反事实推广，显著提升了在混淆目标达成任务上的成功率，优于标准CRL。

MCPDisCo: Seesaw-Mitigated Tuning for MCP-Native Agents

MCPDisCo：MCP-Native 代理的跷跷板缓解调谐

• Authors: Kounianhua Du, Jianxing Liu, Shuai Shao, Ke Li, Xing Sun, Weinan Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14956
• Pdf link: https://arxiv.org/pdf/2610.14956
• Abstract The Model Context Protocol provides a shared interaction substrate for the agentic web, allowing heterogeneous services to expose capabilities through a standardized interface. Learning to use this ecosystem entails a hierarchy of capabilities: tool-format skills, server-level transfer, and cross-server composition. We present MCPDisCo, an ecosystem-scale reinforcement learning framework for MCP-native agents. Using a Thought-Markov Decision Process (Thought-MDP), the framework explicitly represents internal reasoning alongside routing, execution, and termination and trains on real-server, long-horizon interactions spanning 70 MCP servers and 527 tools across six domains. We study the reasoning--tool seesaw, a joint-learning failure mode in which improvement in one objective can accompany deterioration in the other. To mitigate potential interference, MCPDisCo combines token-masked reward attribution with bidirectional gradient surgery: the former separates direct supervision for reasoning and action spans, while the latter removes conflicting components from both objective gradients in shared parameters. MCPDisCo achieves higher success with fewer chain-of-thought tokens. On the out-of-distribution MCP-Atlas and the GAIA benchmark, MCPDisCo maintains monotone gains, confirming that the learned behavior generalizes beyond the training server distribution and broader agentic tasks. Model weight and implementation are available.
• 中文摘要 模型上下文协议为代理网络提供了共享交互基质，使异构服务通过标准化接口暴露能力。学习使用该生态系统涉及能力层级结构：工具格式技能、服务器级传输和跨服务器组合。我们介绍MCPDisCo，一个面向MCP原生代理的生态系统尺度强化学习框架。该框架采用思维-马尔可夫决策过程（Thought-MDP），明确表示内部推理与路由、执行和终止并列，并训练真实服务器的长期互动，涵盖覆盖6个域的70台MCP服务器和527个工具。我们研究推理-工具跷跷板，这是一种联合学习的失败模式，其中一个目标的进步可能伴随另一个目标的衰退。为减少潜在干扰，MCPDisCo 结合了标记掩蔽奖励归因与双向梯度手术：前者分离推理和动作跨度的直接监督，后者则移除共享参数中两个目标梯度的冲突组件。MCPDisCo 在减少思维链代币数量下取得更高成功率。在非分布的 MCP-Atlas 和 GAIA 基准测试中，MCPDisCo 保持单调增益，证实学习行为可推广至训练服务器分布和更广泛的代理任务之外。模型权重和实现均可获得。

Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning

基于LVLM的图像字幕中减少物体幻觉的度量驱动强化学习

• Authors: Thanh Nguyen, Hee Suk Yoon, Tung Minh Luu, Eunseop Yoon, Chang D. Yoo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.14976
• Pdf link: https://arxiv.org/pdf/2610.14976
• Abstract Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning. However, LVLMs remain prone to object hallucination, describing non-existent entities in the given image. Previous remedies, including dataset-enhanced supervised training and reinforcement learning from human feedback, show promise but rely heavily on costly human involvement, limiting scalability. To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics. While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training. Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability. Extensive experiments on the InstructBLIP baseline demonstrate the effectiveness and generalizability of our framework, achieving a substantial hallucination reduction of up to 41\% on the COCO dataset and 29\% on the Visual Genome. Furthermore, the framework can enhance caption quality by incorporating complementary evaluation metrics such as BERTScore and METEOR. These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.
• 中文摘要 大型视觉语言模型（LVLM）在视觉与语言联合理解方面表现出色，尤其是在图像字幕方面。然而，LVLM仍易出现物体幻觉，描述图像中不存在的实体。以往的解决方案，包括数据集增强的监督训练和基于人类反馈的强化学习，显示出潜力，但高度依赖昂贵的人类参与，限制了可扩展性。为此，我们提出了一个强化学习框架，完全依靠自然语言处理（NLP）指标驱动，以减轻基于LBLM的图像字幕中的物体幻觉。虽然仅基于NLP指标优化强化学习具有可扩展性，但也带来了陷阱：奖励黑客、反馈稀疏和昂贵的训练。我们的框架通过引入对象一致性评分（OCS）来规避这些问题，该评分同时捕捉对象的正确性和覆盖性，结合KL正则化以实现更密集的标记级指导，并设计了一种轻量级PPO变体，在冻结的LVLM骨干上微调适配器，实现效率和稳定性。在InstructBLIP基线上的大量实验展示了我们框架的有效性和普遍性，在COCO数据集上实现了高达41%的幻觉减少，在视觉基因组上减少了29%。此外，该框架还可以通过整合BERTScore和METEOR等互补评估指标来提升标题质量。这些结果揭示了多模态学习的更广泛见解：LVLM能够通过度量反馈有效自我纠正幻觉，激励人们深入探索可扩展的强化框架，显著减少人类劳动力。

Mixing Regularization for Average-Reward $Q$-Learning and $Q$-Planning

将正则化混合用于平均奖励$Q$学习与$Q$规划

• Authors: Julien Roger, Sam Bodmer, Jian Qian, Benjamin Riviere
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.14988
• Pdf link: https://arxiv.org/pdf/2610.14988
• Abstract The average-reward setting for reinforcement learning is challenging because its Bellman operator is known to contract in span only when the underlying Markov chain is mixing. Whereas existing work typically assumes a contraction property, e.g., through environment mixing, we consider the setting where we are given a reward-agnostic $m$-step mixing policy, and we control the contraction and exploration properties of the learned policy by regularizing toward the mixing policy. We provide finite-time analysis of $Q$-learning algorithms that can be instantiated with multiple choices of regularization, with guarantees measured with respect to the regularized fixed point. Our framework exposes novel regularizer-dependent objects that control contraction, exploration, and operator bias. In the one-step setting, our rates match the best known rates for $Q$-iteration and synchronous $Q$-learning, and for $m$-step mixing we derive new rates. We also study $Q$-planning variants, and show that planning acts as time-localized learning, where the planning horizon $H$ trades off the bias from the boundary value against the sample complexity. Finally, we present numerical studies validating the analysis: (i) knowledge of a strong mixing policy improves convergence and exploration, and its advantage increases with bottleneck severity and, (ii) our theory predicts the convergence and bias rankings of regularizers from the $\alpha$-divergence family.
• 中文摘要 强化学习的平均奖励设置具有挑战性，因为其Bellman算子仅在底层马尔可夫链混合时才会在间间内收缩。现有工作通常假设收缩性质，例如通过环境混合，我们考虑了奖励无关的$m$步混合策略，并通过正则化来控制所学策略的收缩和探索性质。我们提供了有限时间的分析，分析$Q$学习算法，这些算法可用多选择正则化实现，保证基于正则化不动点进行测量。我们的框架揭示了控制收缩、探索和算子偏置的新颖正则化对象。在一步设置下，我们的速率与$Q$迭代和同步$Q$学习的最佳已知速率相匹配，对于$m$步混合，我们推导出新的速率。我们还研究了$Q$规划变体，并证明规划具有时间局部化的学习作用，其中规划视野$H$在边界值偏置与样本复杂度之间进行权衡。最后，我们提出了验证分析的数值研究：（i）了解强混合策略有助于提升收敛和探索，且其优势随着瓶颈严重性增加而增强;（ii）我们的理论预测了$\alpha$发散族正则化子的收敛和偏置排名。

Self-Evolving Visual Agent via Margin Reward

通过保证金奖励实现自我演化的可视化代理

• Authors: Wenxi Yang, Yuzhong Zhao, Baochang Zhang, Yunfan Liu, Qixiang Ye
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15084
• Pdf link: https://arxiv.org/pdf/2610.15084
• Abstract Visual agents that interleave reasoning with tool invocation have posed a promising direction for multimodal tasks. However, learning effective tool invocation usually relies on curated question-answer pairs or expert demonstrations at scale, which are expensive to acquire. In this study, we propose Marvo, a simple-yet-effective self-evolution approach that enables learning tool invocation from unlabeled images. Marvo consists of two agents: (1) a challenger that generates question-answer pairs from unlabeled images, and (2) a solver that learns tool invocation by solving questions through reinforcement learning. To encourage effective tool invocation, a good training question should be difficult enough to warrant tool use, yet remain within the solver's capabilities. To this end, we introduce a margin reward to the challenger, defined as solver's performance difference with and without access to tools. With the margin reward, we create an adaptive curriculum that encourages the solver to learn effective tool invocation. Across benchmarks covering agentic reasoning, visual search, mathematical, and general real-world reasoning, Marvo improves average accuracy by up to 7.1\% --- demonstrating margin reward as a preferred way to scale question complexity without human supervision.
• 中文摘要 将推理与工具调用交织的视觉智能体为多模态任务带来了有前景的方向。然而，学习有效的工具调用通常依赖于精心策划的问答对或大规模专家演示，而这些方法获取成本高昂。本研究提出了Marvo，一种简单但有效的自我进化方法，能够从未标记图像中学习工具调用。Marvo由两个智能体组成：（1）挑战者，从未标记图像生成问答对，（2）通过强化学习解决问题来学习工具调用。为鼓励有效的工具调用，一个好的训练问题应足够难度以支持工具使用，同时又在解题器能力范围内。为此，我们引入了对挑战者的保证金奖励，定义为求解器在使用工具时与否的表现差异。通过边际奖励，我们创建了一套自适应课程，鼓励求题者学习有效的工具调用。在涵盖代理推理、视觉搜索、数学及一般现实推理的基准测试中，Marvo 将平均准确率提升了高达 7.1% ---证明边际奖励是无人工监督下扩展问题复杂度的首选方式。

Anchors Enhance Controllability and Reasoning of Diffusion Language Models

锚点增强扩散语言模型的可控性和推理能力

• Authors: Feiyu Qu, Zheng Huang, Andrea Zanette
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.15097
• Pdf link: https://arxiv.org/pdf/2610.15097
• Abstract Diffusion language models (DLMs) offer new opportunities for controllable text generation. However, their controllability, defined as the ability to satisfy task constraints while preserving their original capabilities, remains underexplored. DLMs' bidirectional attention allows control tokens to guide generation from arbitrary positions in the output sequence, but effective guidance depends on the compatibility between their content and placement. To this end, we propose AnchorDLM, a framework in which an anchor generator produces an anchor consisting of a short output fragment and its position for each query. The anchor is placed as control tokens at the specified position to initialize the output sequence of a frozen DLM and guide subsequent denoising. We train the anchor generator with reinforcement learning (RL) to jointly optimize anchor content and placement, using rewards computed from the DLM's generated outputs based on answer correctness and constraint satisfaction. Moreover, through the lens of discrete flow matching (DFM), we interpret anchor placement as helping to reshape the initial velocity field that guides generation. Experiments on code generation and mathematical reasoning across five DLM backbones show that AnchorDLM improves controllability and further yields gains in reasoning performance across diverse benchmarks.
• 中文摘要 扩散语言模型（DLM）为可控文本生成提供了新机遇。然而，其可控性，即满足任务约束且保留原始能力的能力，仍未被充分探索。DLM的双向注意力允许控制令牌从输出序列中的任意位置引导生成，但有效引导依赖于其内容与位置的兼容性。为此，我们提出了AnchorDLM，一种框架，锚生成器生成由短输出片段及其每个查询位置组成的锚点。锚点作为控制令牌放置在指定位置，以初始化冻结DLM的输出序列并指导后续去噪。我们用强化学习（RL）训练锚生成器，共同优化锚点内容和位置，利用DLM生成的输出基于答案正确性和约束满足度计算的奖励。此外，通过离散流匹配（DFM）的视角，我们认为锚点的布置有助于重塑指导生成的初始速度场。五个DLM骨干的代码生成和数学推理实验表明，AnchorDLM提升了可控性，并在多样基准测试中进一步提升了推理性能。

Structural Chain-of-Thought: Reasoning with Executable Computational Graphs

结构性思维链：可执行计算图的推理

• Authors: Petr Senatorov, Karim Vafin, Elena Tutubalina, Oleg Somov
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15107
• Pdf link: https://arxiv.org/pdf/2610.15107
• Abstract Chain-of-thought prompting enables reasoning in large language models but produces explanations that are difficult to verify, expensive to generate, and not always faithful. We introduce a Structural Chain-of-Thought framework where models generate executable computational graphs instead of free-form reasoning. The final answer is obtained by deterministically executing the graph. The model is trained using graph-based reinforcement learning that rewards structural validity, answer correctness, and node overlap with reference graphs. Experiments across five benchmarks and four model families show that our method consistently improves accuracy over supervised graph training and can reduce generation length by up to 5-6x compared with standard GRPO and chain-of-thought supervised training. Our faithfulness experiments reveal that models trained with our approach tend to rely on deriving their answers from the generated graphs rather than producing post-hoc rationalizations as usually in chain-of-thought reasoning.
• 中文摘要 思维链提示使大型语言模型中的推理成为可能，但产生的解释难以验证、生成成本高且不总是忠实。我们引入了结构性思维链框架，模型生成可执行的计算图，而非自由形式的推理。最终答案通过确定性地执行该图得到。模型采用基于图的强化学习训练，奖励结构效度、答案正确性和节点与参考图的重叠。跨五个基准测试和四个模型家族的实验表明，我们的方法在监督图训练中持续提升准确性，且与标准GRPO和思维链监督训练相比，可将生成时间缩短多达5-6倍。我们的忠实性实验显示，采用我们方法训练的模型倾向于从生成的图中推导答案，而非像通常的链式思维推理那样事后合理化。

Single-step Retrosynthesis via Reinforcement Learning from Forward Feedback

通过前向反馈强化学习实现单步逆向合成

• Authors: Zixun Lan, Zhenfu Liu, Yi Li, Xinheng Wang, Mingmin Chi, Qibiao Wang, Jie Zhu, Xiaohui Tian, Fei Ma
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15141
• Pdf link: https://arxiv.org/pdf/2610.15141
• Abstract Single-step retrosynthesis is a one-to-many inverse problem, yet most models are trained and evaluated against a single recorded reactant set. Reference-based training gives no explicit positive signal for unrecorded alternatives, and exact-match evaluation rejects non-matching predictions regardless of chemical plausibility. We propose Retrosynthesis via Reinforcement Learning from Forward Feedback (RetroRLFF), which optimizes reactant generation using both reference agreement and forward consistency with the target product. Starting from a supervised decoder-only Transformer, RetroRLFF generates complete reactant candidates and uses a frozen forward reaction model to predict products. A sequence-level reward combines reference agreement and forward consistency while penalizing invalid outputs. Direct Advantage-Based Policy Optimization updates the policy without human preference annotations or a preference-trained reward model. On USPTO-50K, RetroRLFF outperforms all 14 baselines in candidate-level round-trip accuracy at k = 1, 3, 5, and 10. With approximately 50 million parameters, it achieves 90.1% top-1 round-trip accuracy, exceeding R-SMILES, the strongest competing baseline on this metric, by 8.5 percentage points. Compared with its supervised policy, round-trip accuracy increases from 83.1% to 90.1%, while exact-match accuracy is maintained at 57.9% versus 57.5%. In an anonymized expert assessment of 131 randomly selected test cases, organic synthesis experts prefer RetroRLFF over EditRetro in 67.9% of comparisons. These results support forward feedback as a training signal for improving forward consistency while retaining top-1 reference recovery. Expert assessment provides complementary evidence of synthetic feasibility on the selected cases, although neither forward consistency nor expert preference establishes experimental feasibility.
• 中文摘要 单步逆合成是一个一对多的逆问题，但大多数模型是针对单个记录反应物集进行训练和评估的。基于参考的训练不会对未记录的替代物提供明确的正信号，而精确匹配评估无论化学可信度如何都会拒绝不匹配的预测。我们提出了通过前向反馈强化学习（RetroRLFF）的逆向合成，通过参考一致性和目标产物的前向一致性来优化反应物生成。RetroRLFF从仅监督的解码器变换器出发，生成完整的反应物候选物，并使用冻结的前向反应模型预测产物。序列级奖励结合了参考一致性和前向一致性，同时惩罚无效输出。直接优势策略优化在不使用人类偏好注释或偏好训练奖励模型的情况下更新策略。在USPTO-50K上，RetroRLFF在候选者级往返准确率（k = 1、3、5和10）中均优于所有14个基线。拥有约5000万参数，其前1往返准确率达到90.1%，比该指标上最强的竞争基线R-SMILES高出8.5个百分点。与其监督政策相比，往返准确率从83.1%提升至90.1%，而精确匹配准确率保持在57.9%对57.5%。在对131个随机选定测试案例的匿名专家评估中，有机合成专家在67.9%的比较中更倾向于RetroRLFF而非EditRetro。这些结果支持前向反馈作为提升前向一致性的训练信号，同时保留顶级引用恢复。专家评估为所选案例提供合成可行性的补充证据，尽管前瞻性一致性或专家偏好均不能确立实验可行性。

Game-Theoretic Reinforcement Learning based Topology-Agnostic Defense for Virtual Power Plants

基于博弈论强化学习的拓扑-无关虚拟电厂防御

• Authors: Pooja Algikar, Stefano Fenu, Kumarsinh Jhala, Siby Plathottam
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.15183
• Pdf link: https://arxiv.org/pdf/2610.15183
• Abstract The rapid proliferation of distributed energy resources (DERs) aggregated by virtual power plants (VPPs) elevates cybersecurity risk by expanding the attack surface. A coordinated cyberattack has the potential to disrupt distribution-system, bulk-system, and market operations. Existing cyberphysical defense strategies largely assume access to grid network models and operating conditions, which may not be available to third-party aggregators. This paper proposes a novel hybrid, perturbation-based cyber-physical defense for VPP-operated DERs that blends game-theoretic planning with model-free reinforcement learning (RL) under a Bayesian belief-update framework. Topology agnosticism is defined from the VPP information and implementation perspective: the VPP does not require access to the feeder topology, line parameters, switching configuration, or full system state. This work advances the state of the art through three contributions: (1) a multi-output Gaussian process with Hamiltonian Monte Carlo for discrete attacker-type inference, probability of detection, and confidence of detection for modeling the belief distribution; (2) moment-matched Gaussian propagation of posterior beliefs across episodes; and (3) an RL policy that adaptively integrates Stackelberg leader (VPP) actions with exploration. Furthermore, we establish equilibrium existence and derive an episode-level closed-loop utility bound connecting attacker-belief uncertainty with GT-RL action alignment. Evaluations on IEEE 13- and 123-Bus feeders with DER portfolios demonstrate effective detection under attacker-model uncertainty and evaluated feeder re-configuration conditions, while requiring no explicit feeder-topology knowledge. Overall, the results indicate that combining bi-level game-theoretic planning, nonparametric Bayesian inference, and adaptive RL is a promising direction for VPP cybersecurity.
• 中文摘要 虚拟电厂（VPP）聚合的分布式能源资源（DER）迅速扩散，通过扩大攻击面，提高了网络安全风险。协调的网络攻击有可能扰乱配电系统、大宗系统和市场运营。现有的网络物理防御策略主要假设对电网模型和运行条件的访问，而这些条件可能不对第三方聚合商开放。本文提出了一种新型混合、基于微扰的网络物理防御方案，适用于VPP运营的DER，结合博弈论规划与无模型强化学习（RL），采用贝叶斯信念更新框架。拓扑无关性从VPP信息和实现角度定义：VPP无需访问馈线拓扑、线路参数、切换配置或完整系统状态。本研究通过三项贡献推动了技术进步：（1）采用哈密顿蒙特卡洛的多输出高斯过程，用于建模信念分布的离散攻击者类型推断、检测概率和检测信心;（2）后验信念跨事件的矩匹配高斯传播;（3）一种将Stackelberg领导（VPP）动作与探索自适应整合的强化学习策略。此外，我们建立了均衡存在性，并推导出了将攻击者信念不确定性与GT-RL动作对齐连接起来的事件级闭环效用界限。对带有DER组合的IEEE 13和123-总线馈源器的评估显示了在攻击者模型不确定性和反馈器重配置条件下的有效检测，且无需显式馈源拓扑知识。总体来看，结果表明，结合双层博弈论规划、非参数贝叶斯推断和自适应强化学习，是VPP网络安全的一个有前景方向。

Reinforcement Learning Based Heterogeneous Task Allocation Under Spatio-Temporal Environmental Uncertainty for Multi-Robot Teams

基于强化学习的多机器人团队在时空环境不确定性下进行异质任务分配

• Authors: Xi Lin, Yaxuan Li, Paul Szenher, Brendan Englot
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.15237
• Pdf link: https://arxiv.org/pdf/2610.15237
• Abstract Autonomous Surface Vehicles (ASVs) have been increasingly deployed to perform different kinds of maritime tasks in recent years, which motivates the need for a reliable task allocation policy that can adapt to heterogeneous tasks and maritime disturbances. In this work, we formulate a multi-robot task allocation problem in which a team of ASVs with heterogeneous capabilities needs to complete tasks of different types and appearance times under unknown stochastic spatio-temporal uncertainties from maritime disturbances. We propose a decentralized Reinforcement Learning (RL) based policy network that is trained to be robust towards uncertainty-induced observation deviations, and is able to perform individual decision making that leads to efficient execution behaviors toward heterogeneous task requirements. We evaluate the proposed policy and four baseline methods in simulated experiments with varied proportions of different tasks types, numbers of tasks, robot species and robots per species. Experiment results show that our proposed policy achieves the lowest makespan and can be deployed to address scenarios with 100 robots and 400 tasks.
• 中文摘要 近年来，自主水面载具（ASV）越来越多地被用于执行不同类型的海上任务，这促使人们需要一种能够适应异构任务和海上扰动的可靠任务分配策略。本研究提出了一个多机器人任务分配问题，其中一支具备异构能力的ASV团队需要在未知的随机时空不确定性下完成不同类型和出现时间的任务。我们提出了一个基于去中心化强化学习（RL）的策略网络，经过训练以应对不确定性引起的观察偏差，并能够执行个体决策，从而实现针对异构任务需求的高效执行行为。我们在不同任务类型比例、任务数量、机器人种类及每物种机器人数量的模拟实验中评估了拟议的策略和四种基线方法。实验结果显示，我们提出的策略实现了最低的完成时间，并可用于应对拥有100台机器人和400个任务的场景。

ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation

观察专题：加速能动强化学习与观察推测

• Authors: Kyle Montgomery, Wang Zhang, Chenguang Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.15246
• Pdf link: https://arxiv.org/pdf/2610.15246
• Abstract Tool-using agents alternate between model generation and tool execution, waiting for each tool call to complete before starting the next generation. This sequential execution is costly during agentic reinforcement learning (RL), which requires large batches of multi-turn rollouts. We introduce ObsSpec, an observation speculation pipeline for agentic RL that uses a small world model to predict tool outputs while the environment executes, allowing the policy to begin its next generation early. ObsSpec retains speculative work only when the predicted and actual observations match exactly, making rollout collection lossless, while co-training keeps the world model up to date as the policy learns. On terminal-use tasks, predictions from a 0.6B world model exactly match actual observations in 66.4% of attempts and arrive roughly 16 times faster. Using these predictions, ObsSpec reduces agent latency by 9.3% and 14.2% for 8B and 32B policies. Across controlled experiments, we characterize the regimes in which ObsSpec reduces agent latency under shared policy resources. Finally, because ObsSpec overlaps policy generation with tool execution, it is architecturally compatible with existing techniques for accelerating RL. Code is available at this https URL.
• 中文摘要 工具使用代理在模型生成和工具执行之间交替，等待每次工具调用完成后才开始下一代。这种顺序执行在代理强化学习（RL）中代价较高，因为该学习需要大量多回合展开。我们介绍了ObsSpec，一种针对代理强化学习的观察推测流水线，利用小世界模型在环境执行时预测工具输出，使策略能够提前启动下一代。ObsSpec仅在预测与实际观测完全匹配时保留推测工作，使得推广收集无损;而协同训练则在策略学习时保持世界模型的更新。在终端使用任务中，来自0.6亿世界模型的预测在66.4%的尝试中与实际观测完全匹配，且到达速度约快16倍。利用这些预测，ObsSpec 在 8B 和 32B 策略中分别将代理延迟降低了 9.3%，降低了 14.2%。通过受控实验，我们描述了 ObsSpec 在共享策略资源下减少代理延迟的模式。最后，由于 ObsSpec 与策略生成与工具执行重叠，其架构上与现有加速 RL 技术兼容。代码可在此 https URL 获取。

Do Rollups Self-Organize? Multi-Agent Reinforcement Learning in the EIP-4844 Blob Fee Market

汇总会自组织吗？EIP-4844 Blob Fee 市场中的多智能体强化学习

• Authors: Alessio Brini
• Subjects: Subjects: Multiagent Systems (cs.MA); Computer Science and Game Theory (cs.GT); Trading and Market Microstructure (q-fin.TR)
• Arxiv link: https://arxiv.org/abs/2610.15256
• Pdf link: https://arxiv.org/pdf/2610.15256
• Abstract EIP-4844 gave Ethereum rollups a separate fee dimension for data, governed by a multiplicative update rule that targets three ``blobs'' per block. Whether this target is reached depends on how many independent, self-interested rollups choose to post data, a question that closed-form models address only for a single agent under exogenous prices or in a static equilibrium. We propose a calibrated multi-agent reinforcement learning environment for the blob fee market in which the blob base fee is endogenous, set by the EIP-4844 rule on the aggregated decisions of heterogeneous rollups fit to post-Dencun Ethereum data. After confirming that the environment reproduces the known single-agent analytical optimum, we show that independent learners, with no convergence guarantee, still settle near the target without coordination ($3.32\pm0.01$ blobs per block over five seeds). Over a fiftyfold demand range ($10\times$ to $500\times$), supply stays at the target from $10\times$ to $200\times$ and rises above it at higher demand, where the fee can no longer hold it. Posting frequency increases with rollup size, an emergent analog of the size threshold predicted by prior theory. A costless layer-1 calldata outside option pushes the fee down by roughly an order of magnitude while aggregate supply stays at the target, so the fee rule acts as a negative-feedback controller on decentralized supply.
• 中文摘要 EIP-4844为以太坊汇总提供了一个独立的数据费用维度，由乘法更新规则控制，目标每个区块有三个“blob”。该目标是否达成取决于有多少独立且利己的汇总选择发布数据，而封闭模型仅在外生价格或静态均衡下解决单个代理的问题。我们提出了一个校准的多代理强化学习环境，适用于blob费用市场，其中blob基础费用为内生，由EIP-4844关于异构汇总决策的规则设定，适用于后Dencun以太坊数据。在确认环境重现已知单代理分析最优后，我们表明，没有收敛保证的独立学习者仍然在目标附近稳定，且没有协调（五个种子中每块块3.32\pm0.01$的块状物）稳定。在50倍的需求区间（$10\乘时$到500/倍$）内，供给保持在目标值上，从$10\times$到$200\times$，并在需求更高时超过目标，此时费用已无法维持。发布频率随着汇总规模增加而增加，这是先前理论预测规模阈值的一种新兴对应。无成本的层1调用数据外部选项将费用降低大约一个数量级，而总供应保持目标，因此费用规则作为去中心化供应的负反馈控制器。

ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs

前注：预判，然后执行——从验证信念中学习编码智能体的内部世界模型

• Authors: Yi Ge, Kangrui Ruan, Jifan Chen, Manish Prajapat, Christian Bock, Xiaoyang Wang, Rein Houthooft, Aritra Sengupta, Varun Kumar
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.15368
• Pdf link: https://arxiv.org/pdf/2610.15368
• Abstract Rapid advances in large language models have made software engineering agents capable of tackling complex coding tasks. Within a containerized environment, a coding agent receives tool responses or execution results, and determines its later actions by those earlier results. However, most existing coding agents never look ahead: their policies do not anticipate the consequences of actions, and the terminal reward barely attributes a failure to certain earlier steps, so an ill-judged edit can corrupt the repository state irreversibly before any signal arrives. Recent code world models aim to address these challenges, but they are either trained and then frozen before reinforcement learning (RL), or reduced to an auxiliary next-token loss on the environment's outputs. We present ANTE (Anticipate, Then Execute), an agentic RL framework that verifies the policy's pre-action predictions by real execution and turns them into per-step credit for its actions and trust-or-abstain decisions. With an internal belief stated before each action, Belief-Anchored Policy Optimization (BAPO) turns the settled predictions into per-step advantages by anchoring credit on the policy's own forecasts. To turn the trained forecasts into better actions at inference, thereby facilitating effective test-time scaling, ANTE-TTO (test-time optimization) selects among candidate actions without an external scorer. ANTE raises Qwen3-32B from 25.0% to 57.2% pass@1 on SWE-bench Verified, 6.2% above a matched GRPO baseline trained without the belief state; its world model's accuracy climbs from 44% to 69% during RL, and the learned foresight capability transfers across harnesses and out-of-distribution domains. In summary, we aim to build a code world model that knows what has happened, what holds now, and what the next action will achieve.
• 中文摘要 大型语言模型的快速进步使软件工程代理能够处理复杂的编码任务。在容器化环境中，编码代理接收工具响应或执行结果，并根据这些先前结果决定其后续行动。然而，大多数现有编码代理从未展望未来：其策略不预见动作的后果，终端奖励几乎不将失败归因于某些早期步骤，因此一次判断不当的编辑可能在任何信号到达前就不可逆地破坏仓库状态。近期的代码世界模型旨在解决这些挑战，但它们要么被训练后在强化学习（RL）前冻结，要么被简化为环境输出上的辅助下一令牌丢失。我们介绍ANTE（Anticipate， Then Execute），这是一个代理强化学习框架，通过实际执行验证策略的动作前预测，并将其转化为每步行动和信任或弃权决策的积分。在每次操作前陈述内部信念后，信念锚定策略优化（BAPO）通过锚定策略自身预测，将既定预测转化为每步优势。为了将训练好的预测转化为更优的推断动作，从而促进有效的测试时间缩放，ANTE-TTO（测试时间优化）在候选动作中选择无外部评分器。ANTE 将 Qwen3-32B 在 SWE-bench Verified pass@1 提升 25.0% 至 57.2%，比未训练信念状态的匹配 GRPO 基线高出 6.2%;其世界模型在强化过程中准确率从 44% 提升至 69%，且学习到的前瞻性能力可跨线束和非分发领域转移。总之，我们旨在构建一个代码世界模型，了解发生了什么、当前情况以及下一步行动的成果。

A Sovereign, Indic-First, Reasoning Model

一个主权的、以印度为先的推理模型

• Authors: Sutanshu Saksena Raj, Prasanjith Pasunuti, Parijat Chatterjee, Daksh Maan, Purusharth Malik, Nayanjit Sarkar, Avinash Benki
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15371
• Pdf link: https://arxiv.org/pdf/2610.15371
• Abstract Gnani-Evon-v3.3-30B-A3B is a long-context hybrid Mamba-Transformer mixture-of-experts model for reasoning, instruction following, and tool use in English and eight Indian languages. Starting from NVIDIA's Nemotron-3-Nano-30B-A3B checkpoint, it retains the 52-layer backbone and changes the vocabulary, training data, and post-training objectives. An in-place expansion from 131,072 to 163,072 tokens reduces mean Indian-language fertility by 38.6%. The new embedding rows are initialized from their upstream decompositions and warmed before the backbone is updated. Continual pretraining then adds broad English-Indic data, high-quality annealing, and direct training at a 131,072-token sequence length. The central result appears after post-training. Reinforcement learning across four verifiable task environments improves mathematics, graduate-level science, and instruction following, but degrades Indic extractive question answering because the rewards do not require an answer in the requested language. The share of Indic extraction items answered in English rises from 8.7% to 91.7%. Accuracy conditional on answering in the target script roughly doubles, which is consistent with preserved reading ability but is measured on a self-selected subset. Two unreleased studies test repairs. A scoped Indic reward restores the language convention. A 26B-A4B generative judge outperforms a 235B-parameter model on RM-Bench, and distillation restores 81% of the capabilities displaced by optimizing against it. Relative to the architecture-identical starting checkpoint, the released model gains 5.2 points on an Indic aggregate while giving up 1.7 points of English retention. The unreleased Indic-aware branch trades further, gaining 14.6 Indic points at a cost of 5.4 in English. A planned FP8 release reduces the artifact to 32GB with near-zero mean change across 32 paired benchmarks.
• 中文摘要 Gnani-Evon-v3.3-30B-A3B 是一个长上下文混合 Mamba-Transformer 专家混合模型，支持推理、指令跟随和工具使用，支持英语和八种印度语言。从 NVIDIA 的 Nemotron-3-Nano-30B-A3B 检查点出发，保留了 52 层骨干，并改变词汇、训练数据和训练后目标。原地扩展 131,072 个令牌可降低 38.6% 的平均印度语言生育率。新嵌入行从上游分解初始化并预热，然后再更新骨干。持续预训练随后增加了广泛的英印数据、高质量退火和直接训练，长度为 131,072 个令符。核心结果在训练后显现。跨四个可验证任务环境的强化学习提升数学、研究生级科学和教学跟随能力，但因奖励不要求用所请求语言回答，降低了印度语抽取性问题回答。印度语提取题目中英语回答的比例从8.7%上升至91.7%。以目标文字回答为条件的准确率大约翻倍，这与保持阅读能力一致，但测量基于自选子集。两项未发布的研究测试修复。范围化的印度语奖励恢复了语言惯例。一个26B-A4B生成评判在RM-Bench上优于235B参数模型，蒸馏恢复了因优化而被取代的能力81%。相较于架构相同的起始检查点，发布模型在印度语汇总上提升5.2分，同时牺牲1.7个英语保留点。未发布的印度认知分支继续交易，以5.4的英语成本获得14.6印度积分。计划的第8次FP发布将使该文物降至32GB，且在32个配对基准测试中平均变化几乎为零。

A Reinforcement Learning Framework for Safety-Aware Haptic Shared Control in Surgical Robotics

外科机器人中安全意识触觉共享控制的强化学习框架

• Authors: Zihao Zhang, Karl Handwerker, Balint Varga
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.15422
• Pdf link: https://arxiv.org/pdf/2610.15422
• Abstract Haptic shared control in surgical robotics combines human-side and robot-side forces at a shared end effector while preserving operator involvement. However, the effects of geometric safety choices on task execution and force coordination remain unclear. We model the interaction as a two-agent cooperative Markov game under centralized training and decentralized execution, comparing learned human control with a task-tuned fixed-impedance baseline. This model-free multi-agent reinforcement learning framework supports joint policy adaptation without requiring a complete analytical interaction model. It compares MADDPG and rMAPPO with an epigraph-constrained approach under common geometric task definitions. In simulation, rMAPPO generally achieved the highest return and tracking performance, whereas the epigraph approach produced more conservative motions and lower geometric exposure, including no observed collision or dangerous-region exposure after geometry transfer. With real operator input, fixed-impedance-trained policies were faster and successful in all recorded trials. Policies trained with a learned partner exhibited greater command opposition, while the fixed epigraph condition produced the most aligned commands. This work provides a benchmark for studying human-control representation and constraint-handling trade-offs in haptic shared control.
• 中文摘要 手术机器人中的触觉共享控制结合了人机双方的力，同时保持操作员的参与度。然而，几何安全选择对任务执行和力协调的影响仍不明确。我们将该交互建模为一个双代理协作马尔可夫博弈，在集中训练和分散执行下进行比较，比较学习到的人类控制与任务调优的固定阻抗基线。该无模型多智能体强化学习框架支持联合策略适应，无需完整的分析交互模型。它比较了MADDPG和rMAPPO，采用题距约束方法，基于通用几何任务定义。在仿真中，rMAPPO通常实现了最高的返回和跟踪性能，而铭文方法则产生更保守的运动和较低的几何暴露，包括几何转移后无碰撞或危险区域暴露。在真实操作员输入下，固定阻抗训练的策略在所有有记录的试验中更快且成功。与学习伙伴共同训练的策略表现出更强的指令阻力，而固定引字条件则产生最一致的指令。这项工作为研究触觉共享控制中人类控制表示和约束处理权衡提供了基准。

Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning

悲观潜在任务感知优化，实现稳健的离线元强化学习

• Authors: Chang-Hoon Jeong, Kisung Shin, Hanwool Sul, Byoung-Tak Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15428
• Pdf link: https://arxiv.org/pdf/2610.15428
• Abstract Context-based offline meta-reinforcement learning (COMRL) has advanced almost entirely through better task representations, which separate task identity from artifacts of the behavior policy so that the latent reflects the task alone. The implicit expectation has been that a clean representation will transfer to out-of-support (parameter-extrapolation) tasks. On such a task, however, the encoder produces a latent outside the training support, where the policy has never been trained. We address this gap with Pessimistic Latent Task-aware Optimization (PLATO), which exposes the policy to off-support latents during training. PLATO assumes a mild manifold hypothesis: task latents lie on a low-dimensional manifold, and out-of-support tasks lie farther from the training centroid. It exploits this geometry by perturbing inferred latents outward from the centroid to synthesize counterfactual tasks. A learned decoder ensemble then rolls out short trajectories at each perturbed latent, and the policy is updated on the imagined transitions under a pessimism weight that down-weights steps on which the ensemble disagrees. We prove that the perturbation leaves the training support by a controllable margin and that the out-of-support generalization gap is bounded by the same perturbation reach, and we give explicit conditions, verified empirically, under which the epistemic disagreement at the perturbed latent stays bounded away from zero. On eight MuJoCo continuous-control benchmarks, PLATO consistently improves out-of-support returns over strong COMRL baselines while remaining competitive on in-distribution tasks. Our code is available at this https URL.
• 中文摘要 基于上下文的离线元强化学习（COMRL）几乎完全通过更好的任务表示方式发展，这些表示法将任务身份与行为策略的产物分离，使潜在变量仅反映任务本身。隐含的预期是，干净的表示会转移到支持之外（参数外推）任务。然而，在此类任务中，编码器产生了一个训练支持之外的潜在变量，而该策略从未被训练过。我们用悲观潜在任务感知优化（PLATO）来弥补这一空白，该方法在训练过程中将策略暴露给非支持的潜在变量。PLATO假设一个温和流形假设：任务潜在变量位于低维流形上，支持外的任务则远离训练中心。它利用该几何结构，扰动从重心向外推断的潜在变量，以合成反事实任务。学习到的解码器集合随后在每个扰动潜在变量处推出短轨迹，并在悲观权重下更新策略，降低群组不同意的步骤权重。我们证明扰动以可控幅度离开训练支持，且支持外泛化差距受扰动距离限制，并给出经验验证的显式条件，使得扰动潜在变量的认识论不一致保持远离零。在八个MuJoCo连续控制基准测试中，PLATO在支持外收益上持续优于强COMRL基线，同时在分布内任务中保持竞争力。我们的代码可在此 https URL 获取。

Learning Safety-Constrained Whole-Body Loco-Manipulation for Lunar Surface Operations

学习安全受限的全身机车操作以实现月球表面操作

• Authors: Alvaro Belmonte-Baeza, Miguel Cazorla, Gabriel J. García, Carlos J. Pérez-Del-Pulgar, Jorge Pomares
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.15486
• Pdf link: https://arxiv.org/pdf/2610.15486
• Abstract Robotic systems for lunar surface operations must act autonomously and preserve their own integrity in an environment where failures are unrecoverable and continuous supervision is impossible. This paper presents a Constrained Reinforcement Learning framework for quadrupedal mobile manipulators that learns a single whole-body policy for tracking a 6D end-effector pose specified in task space, with operational limits enforced during learning through a constrained learning mechanism. We introduce a reward formulation in which the transition between locomotion and manipulation emerges from the reward geometry rather than from an explicit gating mechanism, a stability term derived from the Gravito-Inertial Inclination Margin and motivated by the reduced-gravity dynamics of the lunar surface, and a design rule for reward regularization that follows from the multiplicative structure of the termination-based objective. The framework is evaluated in a lunar-analog simulation spanning three terrain morphologies at five difficulty levels under lunar gravity, with regolith contact properties calibrated to the JSC-1a simulant and varied from loose to dense. The learned policy attains mean position and orientation errors of 3.22 cm and 6.83 deg. under nominal conditions, retains this precision across regolith densities and over the operationally representative range of terrain difficulty, and reduces soft-constraint violation time by up to an order of magnitude relative to an unconstrained baseline while incurring no hard-constraint violations.
• 中文摘要 用于月球表面操作的机器人系统必须自主行动，并在故障无法恢复且无法持续监督的环境中保持自身完整性。本文提出了一个针对四足移动机械臂的受限强化学习框架，该框架学习任务空间中指定的6维末端执行器姿态的单一全身策略，并在学习过程中通过受限学习机制强制执行操作限制。我们引入了一种奖励表述，其中运动与操作之间的转换源自奖励几何，而非显式门控机制，稳定性项源自重力-惯性倾角裕，并由月球表面的减重动力学驱动，以及基于终止目标的乘法结构的奖励正则化设计规则。该框架在月球模拟中评估，涵盖三种地形形态，在月球重力下以五个难度级别进行，风化层接触性质校准至松散至密集。该策略在名义条件下实现平均位置和方向误差为3.22厘米和6.83度，保持风化层密度及地形操作代表范围的精度，且相较无约束基线，软约束违约时间减少最多一个数量级，同时不发生硬约束违规。

Phase-Gated Constrained Reinforcement Learning for End-to-End Capture and Detumbling of Non-Cooperative Targets

相门约束强化学习用于端到端捕获和非合作目标的分离

• Authors: Álvaro Belmonte-Baeza, Celia Redondo-Verdú, José L. Ramón, Miguel Cazorla, Leonard Felicetti, Jorge Pomares
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.15491
• Pdf link: https://arxiv.org/pdf/2610.15491
• Abstract Active debris removal requires a servicer spacecraft to approach a non-cooperative tumbling object, capture it, and arrest its rotation. This sequence is conventionally divided among phase-specific controllers, each depending on knowledge of the target's inertia, a dedicated grappling fixture, or a hand-designed switch at the contact-rich seam where such pipelines are most fragile. We show that a single constrained reinforcement learning policy can execute the entire mission, driving a dual-arm free-flying servicer from free-space closing through friction-based fixture-free capture to full detumbling of a target whose mass and inertia are unknown and whose state is observed only through noisy estimates. The enabling mechanism is a smooth proximity gate applied at three loci of a constrained Markov decision process. In the reward it induces the phase structure rather than imposing it; in the action space it forecloses a degenerate grab-and-pull exploit as a hard guarantee; and in the constraint set it resolves a normalization failure of constraint-based learning, in which a constraint spanning phases that load it unequally is rendered nearly inert in the phase where precision matters most. Both the failure mode and the remedy apply to any multi-phase problem of this kind. Over 1000 randomized trials the policy succeeds in 99.5% while holding its contact-force, actuator, base-reaction and collision limits throughout, transfers zero-shot to unseen geometries and centre-of-mass offsets, and extrapolates across a tenfold range of target mass.
• 中文摘要 主动碎片移除需要服务者航天器接近一个不合作的翻滚物体，捕获并阻止其旋转。该序列通常在相位特定控制器之间分配，分别取决于对目标惯性的了解、专用抓取装置，或在此类管道最脆弱接触富裕缝隙处的手工设计开关。我们展示了单一约束强化学习策略可执行整个任务，驱动双臂自由飞行服务器从自由空间闭合经基于摩擦的夹具捕获，到目标的质量和惯性未知且仅通过噪声估计观察到的完全翻滚。使得机制是在受限马尔可夫决策过程的三个位点施加一个平滑邻近门。在奖励过程中，它诱导相位结构而非强加;在动作空间中，它排除了退化抓取拉取利用作为硬性保证;在约束集中，它解决了基于约束学习的规范化失败，即跨越多个阶段、负载不均的约束在精度最关键阶段几乎无效。失败模式和补救方法均适用于此类多阶段问题。超过1000次随机试验，该策略在保持接触力、执行器、基底反作用和碰撞限制的同时，成功率达99.5%，将零射程转移到未见几何和质心偏移，并外推目标质量范围的十倍。

Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

Apertus 1.5：从开放文本模型到开放多模推理器

• Authors: Project Apertus: Yixuan Xu (1), Yassine Turki (2), Vinko Sabolčec (2), Valentina Pyatkin (1), Tommy Chu (2), Sven Najem-Meyer (2), Sean F. Carroll (2), Robert Smith (1), Raphael Kreft (1), Oscar Alessandro Tazza (1), Oleg Lavrovsky (2), Nicola Irmiger (1), Matteo Santelmo (2), Marian Schneider (1), Juan García Giraldo (2), Hanna Yukhymenko (1), Eduard Ďurech (1), Davit Melikidze (1), Dani Tamayo (2), Bettina Messmer (2), Aryan Ahadinia (2), Anunay Yadav (2), Allen Huang (2), Ahmad Fraij (1), Alessandro Bombarda (2), Alex Padula (1), Alexander Hägele (2), Alexander Sternfeld (3), Anna Hedström (1), Anna Kosovskaia (1), Arthur Wuhrmann (2), Arundhati Balasubramaniam (1), Badr Al Mahouri (2), Barna Pasztor (1), Bartosz Szostakiewicz (1), Bich Ngoc Doan (2), Bryan Schmid (4), Burak Sezen (1), Camille Challier (2), Clément Charmillot (2), Dana Shayakhmetova (2), Daniel Perruchoud (4), David Brüggemann (5), David Isztl (6), Elia Palme (6), Emma Anastassova (2), Fabian Bösch (6), Faruk Zahiragić (2), Fawzi Roberto Mohamed (6), Fiona Panicali (4), Florian Kolly (2), Flynn Guerra (4), Francesco Monti (2), Frederike Lübeck (1), Gennaro Oliva (6), Haofeng Yang (1), Haoliang Shang (1), Helen El Khoury (2), Henrique Mendonça (6), Jannis Vamvas (7), Jérome Landolt (4), Jessica Lam (1), Jingchun Xiao (1), Jonathan Coles (6), Joshua Oyebanji (2), Khanh Nguyen (2), Konstantinos Pitas (1), Lasse B. Strand (1), Li Mi (1), Loïc Deslarzes (2), Lorenzo Baggi (1), Lorenzo Noci (1), Luca Baumann (1), Luca Sartori (1), Lukas Drescher (6), Marco De Negri (1), Marco Passerini (6), Mark Klein (6), Marko Stojanovic (2), Martin Majsec (2), Matin Ansaripour (2), Mattea Busato (2), Mauro Pellonara (2), Max Neuwinger (1), Michael Graber (4), Michal Tešnar (1), Michelle Wastl (7), Miguel Gila (6), Mikolaj Boronski (2), Morgane Magnin (2), Moritz Reihs (1), Mukhammadali Sayfiddinov (1), Nathan Ranchin (1 and 2), Nicolas Cuveillier (2), Oscar Wohlfahrt (2), Qianqian Zou (5), Rada Kamysheva (1), Rita Sevastjanova (1), Roman Machacek
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.15542
• Pdf link: https://arxiv.org/pdf/2610.15542
• Abstract Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.
• 中文摘要 语言模型正逐渐成为基础设施，而公共机构基于文档数据公开构建的基础设施，是依赖少数提供者的替代方案;Apertus 证明了这种方式在大规模上是可行的。Apertus 1.5 将完全开放的 Apertus 家族从文本扩展到多模态推理。Apertus 1.0 模型的持续预训练，8B 增加了 4T 标记，70B 增加了 2T 标记，增加了图像和音频理解，并将上下文窗口扩展四倍至 262,144 个标记;多模态通过分符器进入，语言模型头仍限制于文本，架构保持不变。监督微调、长上下文微调、可验证奖励的强化学习和偏好对齐完善了发布的模型。思维是聊天模板的一个标志，而非独立模型，在8B模型上将数学推理提升了最多20分。发布的模型在几乎所有基准测试上都优于前代;在同类完全开放模型中，8B在知识、指令跟踪和文化基准方面领先，其指令跟踪超过所有对比模型，包括开放权重模型。在图像和音频理解方面，这是该系列的首次，模型可与全开放多模态模型竞争，在文档理解和长格式及低资源语音识别方面最为强劲，而专业视觉和音频模型在英语及其他高资源语言的感知与转录方面仍领先。与Apertus 1.0类似，我们仅在公开数据上训练，追溯尊重爬虫退出权，排除受限性许可材料。我们以宽松许可发布模型权重、数据准备和训练配方及评估套件;开放性延伸至价值观，Apertus宪章指导了与模型一同发布的对齐阶段。

PointWalker: Mapless Quadruped Locomotion Directly From Raw LiDAR Point Clouds

PointWalker：直接从原始激光雷达点云实现无地图四足行走

• Authors: Sicen Li, Zhen Chu, Chao Li, Jun Zheng, Qiuguo Zhu, Jun Wu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.15548
• Pdf link: https://arxiv.org/pdf/2610.15548
• Abstract Quadruped robots deployed for inspection and field missions must cross heterogeneous terrain, from stairs, slopes, and gaps to high platforms, stepping stones, and balance beams, where safe support is sparse or see-through. Existing methods interpose a terrain map or a depth image between sensing and control. Maps are costly to maintain, corrupted by odometry drift and outliers, and misled by see-through gratings, while depth cameras degrade with lighting and surface material. In this paper, we present PointWalker, a mapless framework whose single reinforcement learning policy directly consumes raw point clouds from four light detection and ranging (LiDAR) sensors. Voxelized points are encoded by linear self-attention and fused with proprioception through cross-attention, so terrain geometry reaches the policy without maps or foothold estimation. One proximal policy optimization (PPO) run, with terrain-aware rewards and dual-channel simulation of see-through gratings, yields one policy for all seven terrain classes with a 90.1% mean success rate in simulation. Deployed zero-shot on a Deep Robotics X30 at 50 Hz onboard, it crosses 0.8-m gaps, climbs 0.8-m platforms and 45° steel-grating stairs, carries a 50-kg payload, and completes a 2.5-km mountain route.
• 中文摘要 用于检查和现场任务的四足机器人必须穿越异质地形，从楼梯、斜坡和间隙到高平台、踏脚石和平衡木，这些地方的安全支撑稀疏或透视。现有方法将地形图或深度图像置于感测与控制之间。地图维护成本高昂，受里程漂移和异常值影响，且被透视格栅误导，而深度相机则随着光照和表面材料的劣化而退化。本文介绍PointWalker，一个无地图框架，其单一强化学习策略直接消耗四个光线探测与测距（LiDAR）传感器的原始点云数据。形素化点通过线性自注意编码，并通过交叉注意与本体感觉融合，因此地形几何体能够到达策略而无需地图或足迹估计。一次近端策略优化（PPO）运行，配合地形感知奖励和双通道透明光栅模拟，七类地形均有一套策略，模拟成功率为90.1%。该机以零发射方式部署在Deep Robotics X30上，频率50赫兹，穿越0.8米间隙，攀爬0.8米高平台和45°钢栅楼梯，携带50公斤有效载荷，完成2.5公里山路。

Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning

Genesis：多模态思维链推理的视觉增强原语

• Authors: Fan Yang, Lifeng Chen, Yipu Wang, Han Wang, Feilong Chen, Yidong Ma, Zhiyang Chen, Chaoyang Zhao, Lingxiang Wu, Hanqi Jiang, Yibo Chen, Xu Tang, Yao Hu, Ming Tang, Xiaojie Jin, Xiaolong Zheng, Jinqiao Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.15565
• Pdf link: https://arxiv.org/pdf/2610.15565
• Abstract Recent multimodal large language models achieve promising results on visual reasoning tasks, approaching human-level performance on existing STEM benchmarks. However, these capabilities degrade under complex real-world scenes. We identify that chain-of-thought reasoning remains largely confined to the linguistic space, rarely anchoring each step in visual evidence. The main cause is the absence of such data during training. In this work, we introduce a training data augmentation pipeline, \textbf{Visual Augmentation Primitives} (VAP), which leverages state-of-the-art vision generative models to synthesize diverse photorealistic training samples and corresponding reasoning text. VAP introduces some basic operators and applies them to a seed image and gets a variant. It also constructs coherent multimodal reasoning chains throughout the generation process. Such training data force models to perceive visual content and perform logical reasoning according to them at every step. We further introduce multimodal chain-of-thought rubric-based reinforcement learning to enhance this. It rewards not only text reasoning but also visual reasoning by scoring intermediate reasoning steps for logical validity, completeness, and consistency. Moreover, we construct the UNREAL benchmark to systematically evaluate visual reasoning on hyperrealistic scenes across controllable difficulty levels. Experiments show that a 4B model built on this framework surpasses 8B baselines, with performance improving consistently as the model scales to 32B, validating the scaling law for visual reasoning.
• 中文摘要 近期多模态大型语言模型在视觉推理任务上取得了令人满意的成果，接近现有STEM基准的人类水平表现。然而，这些能力在复杂的现实场景下会有所退化。我们发现，思维链推理主要局限于语言空间，很少将每一步都作为视觉证据的基础。主要原因是训练过程中缺乏此类数据。在本研究中，我们引入了训练数据增强流水线\textbf{视觉增强原语}（VAP），利用最先进的视觉生成模型，合成多样的写实训练样本和相应的推理文本。VAP引入了一些基本运算符，并将其应用于种子图像，并获得变体。它还在整个生成过程中构建连贯的多模态推理链。这类训练数据迫使模型感知视觉内容，并在每一步都依据其进行逻辑推理。我们还进一步引入了多模态思维链基于评分标准的强化学习。它不仅奖励文本推理，也通过对逻辑有效性、完整性和一致性的中间推理步骤进行评分，奖励视觉推理。此外，我们构建了UNREAL基准，系统地评估超现实场景中可控难度水平下的视觉推理。实验显示，基于该框架构建的4B模型超过8B基线，且随着模型扩展至32B，性能持续提升，验证了视觉推理的缩放律。

SIGN-MAPPO: State-Intent-Aware Graph Multi-Agent Reinforcement Learning for Cooperative Driving under Imperfect V2X

SIGN-MAPPO：状态意图感知图：针对不完美V2X下合作驾驶的多智能体强化学习

• Authors: Zide Fan, Jiahui Yuan, Kaixin Zhang, Junhao Fan, Xiaohe Li, Yiru Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.15577
• Pdf link: https://arxiv.org/pdf/2610.15577
• Abstract Cooperative driving at mixed-traffic unsignalized intersections is challenged by imperfect vehicle-to-everything (V2X) communication, human-driven vehicle (HV) intentions that cannot be directly communicated, and the potential incompatibility of independently generated crossing decisions. This paper proposes State-Intent-Aware Graph Multi-Agent Reinforcement Learning for Cooperative Driving under Imperfect V2X (SIGN-MAPPO) to address these challenges through three coordinated mechanisms. Each decision-eligible connected and automated vehicle (CAV) first reconstructs a receiver-specific traffic graph from its ego state and the peer-state records available to it, from which a shared graph-attention policy generates a raw Stop/Go intention augmented with predicted HV-intent features. CAV intentions are then exchanged through timestamped vehicle-to-vehicle (V2V) messages, and a local conflict-aware arbitration layer converts each CAV's intention into a safe executable action despite potentially inconsistent peer information. In the principal Simulation of Urban MObility (SUMO) evaluation, SIGN-MAPPO achieves a 97.53% completion rate, a 0.10% collision rate, and a mean intersection traversal time of 35.28 s. Ablation studies further confirm the complementary contributions of HV-intent prediction and CAV-intent exchange to this performance. The trained policy also remains robust under the evaluated communication-delay and packet-loss ranges, and cross-scenario tests show complete traffic clearance across the tested settings, though safety margins narrow under compound unpredictable HV behavior.
• 中文摘要 在混合交通且无信号灯的交叉口，合作驾驶面临着不完善的车辆对所有（V2X）通信、无法直接传达的人驾驶车辆（HV）意图以及独立生成的交叉口决策可能存在的不兼容性挑战。本文提出了状态感知图谱多代理强化学习，用于在不完美V2X下（SIGN-MAPPO）下通过三种协调机制应对这些挑战。每个具备决策资格的连接自动驾驶（CAV）首先从其自我状态和可用对等状态记录重建一个接收器特定的交通图，基于该图的共享注意力策略生成原始的停/走意图，并辅以预测的高意图意图特征。随后通过带时间戳的车辆间（V2V）消息交换CAV意图，本地冲突感知仲裁层将每个CAV的意图转换为安全可执行的动作，尽管可能存在对等信息不一致。在主要的城市动力模拟（SUMO）评估中，SIGN-MAPPO实现了97.53%的完成率，0.10%的碰撞率，平均交叉穿越时间为35.28秒。消融研究进一步证实了高压意图预测和CAV意图交换对该性能的互补贡献。训练出的策略在评估的通信延迟和丢包范围内依然稳健，跨场景测试显示在测试设置下完全通行，尽管在复合不可预测的高压行为下安全余裕会收窄。

Imitation Learning via Discrete and Rhythmic Primitive Composition

通过离散和节奏原始构图的模仿学习

• Authors: Rafid Saif, Auke Ijspeert, Hsiu-Chin Lin
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.15639
• Pdf link: https://arxiv.org/pdf/2610.15639
• Abstract Long-horizon robotic manipulation is inherently composed of semantically meaningful sub-tasks. While Movement Primitives (MPs) can model these components, their composition typically relies on manual engineering or sample-inefficient reinforcement learning. Existing supervised methods are limited by the lack of MP labels in standard datasets and often fail to generalize across discrete and rhythmic movements. We propose Neural Dynamic Composition (NDC), a supervised framework to compose Dynamic Movement Primitives (DMPs) using only raw trajectories for supervision, given a single example of each primitive. Our approach exploits parallel combinations of DMPs as a differentiable relaxation of categorical primitive selection, and utilizes a warp-invariant loss formulation to identify primitives despite spatial or temporal variations. A staged training curriculum is employed to learn the selection policy and DMP parameters, unifying discrete and rhythmic movements in one framework. Evaluated on 2D handwriting and 3D robotic tasks, our method outperforms baselines on most tasks and metrics. An ablation study demonstrates how each design choice contributes to the system's performance.
• 中文摘要 长视野机器人操作本质上由语义有意义的子任务组成。虽然运动原语（MP）可以建模这些组件，但其组合通常依赖于手工工程或样本效率低下的强化学习。现有的监督方法受限于标准数据集中缺乏MP标签，且常常无法在离散和节律运动间推广。我们提出了神经动力学组合（NDC），这是一种监督框架，仅用原始轨迹作为监督，给定每个基元的单一实例，组合动态运动原语（DMP）。我们的方法利用DMP的并行组合作为类别原始选择的可微松弛，并利用曲速不变损失公式在空间或时间变化下识别原语。采用分阶段训练课程，学习选择政策和DMP参数，将离散和节律运动统一在一个框架中。通过二维手写和三维机器人任务评估，我们的方法在大多数任务和指标上优于基线。消融研究展示了每一项设计选择如何促进系统性能。

Drift, Then Spread: Learning One-Step Generative Policies for Online Reinforcement Learning

先漂移再传播：在线强化学习中的一步生成策略学习

• Authors: Chunyu Luo, Hongyu Zhao, Guangyi Ni, Yue Song, Lingkai Kong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.15662
• Pdf link: https://arxiv.org/pdf/2610.15662
• Abstract Generative policies offer expressive action distributions for online reinforcement learning (RL), but many diffusion- and flow-based policies require costly iterative sampling. We propose DriftSpread, a one-step generative policy based on Drifting for online maximum-entropy (MaxEnt) RL. To incorporate entropy without a tractable policy density, we build on an energy-based view of Drifting and derive critic-guided Drift and entropy-induced Spread from the negative MaxEnt objective. Our method learns these updates sequentially in the same actor, using action regression for value improvement and distribution matching to capture the effect of entropy. The actor updates require neither direct samples from the critic-defined Boltzmann target nor policy-density or score estimation. Under exact fitting with a fixed critic, we prove convergence to the Boltzmann target in the small-step limit followed by the long-time limit. On MuJoCo, DriftSpread achieves the highest aggregate normalized return among all evaluated methods, while outperforming the diffusion- and flow-based baselines in both training and inference efficiency.
• 中文摘要 生成策略为在线强化学习（RL）提供了表达式动作分布，但许多基于扩散和流的策略需要昂贵的迭代采样。我们提出了DriftSpread，一种基于在线最大熵（MaxEnt）RL漂移的一步生成策略。为了在没有可处理策略密度的情况下纳入熵，我们基于基于能量的漂移视角，并从负极大目标推导出批评者引导的漂移和熵诱导的扩散。我们的方法在同一演员中依次学习这些更新，利用行动回归进行价值改进，分布匹配以捕捉熵的影响。演员更新既不需要来自批评者定义的玻尔兹曼目标的直接样本，也不需要策略密度或分数估计。在与固定批评者精确拟合下，我们证明在小步长极限和长时间极限下收敛于玻尔兹曼目标。在MuJoCo上，DriftSpread在所有评估方法中实现了最高的总归一化收益，同时在训练和推断效率上均优于基于扩散和流量的基线。

Recovering Low-Bit Failures in Vision-Language-Action Policies via Quantization-Conditioned Reinforcement Learning

通过量化条件强化学习恢复视觉-语言-行动策略中的低位失败

• Authors: Seongmin Park, Jaewon Lee, Yujun Heo, Yoonsik Kim, Seheon Choi, Hyoung-rock Kim, Jungwook Choi
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.15675
• Pdf link: https://arxiv.org/pdf/2610.15675
• Abstract Quantization lowers deployment costs of vision-language-action (VLA) policies but can compromise task completion. Across five VLA models and 4,050 real-world trials, we systematically reveal that quantization-aware training (QAT) generally improves on post-training quantization (PTQ) but leaves substantial gaps to full precision (FP) on long-horizon tasks and under visual and semantic shifts, while its imitation-based objective does not explicitly use rollout success or partial completion. To exploit these outcome signals, we introduce Quantization-Conditioned Reinforcement Learning (QCRL), an offline RL method coupling progress-based advantage estimation with quantization-aware source conditioning. FP rollouts supply reference experience beyond action-level distillation, progress-scaled terminal penalty distinguishes partial completion, and quantization-aware conditioning selects advantageous reference behavior within the quantized policy. We evaluate weight-only INT4 W4A16 on $\pi_{0.5}$ and GR00T N1.5 and joint weight-activation NVFP4 W4A4 on $\pi_{0.5}$. Using only in-distribution data, QCRL reaches 99.3% of FP's mean score across three visual-shift conditions for $\pi_{0.5}$ with INT4 W4A16, alongside 42.9% model-memory savings. With NVFP4 W4A4, it reaches 100.7% of the same FP baseline, alongside a $1.73\times$ speedup using NVFP4/FP8 deployment. Hardware gains are relative to FP16 on Jetson Thor.
• 中文摘要 量化降低了视觉-语言-动作（VLA）策略的部署成本，但可能影响任务完成度。通过五个VLA模型和4050个真实世界试验，我们系统地发现量化感知训练（QAT）通常优于训练后量化（PTQ），但在长视野任务及视觉和语义转换下存在显著的完全精度差距，而其基于模仿的目标并未明确使用推广成功或部分完成。为利用这些结果信号，我们引入了量化条件强化学习（QCRL），这是一种离线强化学习方法，将基于进展的优势估计与量化感知的源条件结合。FP推广提供超越动作级蒸馏的参考体验，进度尺度的终端惩罚区分部分完成，量化感知条件反射则在量化策略中选择有利的参照行为。我们评估了仅权重的INT4 W4A16（基于$pi_{0.5}$）和GR00T N1.5及关节权重激活的NVFP4 W4A4（基于$pi_{0.5}$）。仅使用分布内数据，QCRL在三种视觉偏移条件下，基于INT4 W4A16实现了pi_FP平均得分的99.3%，并节省了42.9%的模型内存。使用NVFP4 W4A4时，其FP基准达到100.7%，并通过NVFP4/FP8部署实现了1.73美元/倍数的加速。硬件提升相对于Jetson Thor的FP16。

Rethinking Human Intent-to-CAD: Parametric CAD Model Generation via Cooperative Multi-Task Alignment and Spatial-Aware Reinforcement Learning

重新思考人类意图到CAD：通过协作式多任务对齐和空间感知强化学习生成参数化CAD模型

• Authors: Qingwang Zhang, Jiahao Li, Xiangdong Zhou
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15681
• Pdf link: https://arxiv.org/pdf/2610.15681
• Abstract Parametric Computer-Aided-Design (CAD) modeling from human intent remains challenging, particularly during the conceptual design stage, where design goals are expressed through incomplete and unstructured modalities (e.g., hand-drawn sketches and textual descriptions). In this work, we rethink the human intent-to-CAD pipeline and propose a unified method that directly maps multi-level human intents to executable codes, without assuming the prior existence of target CAD models. To support our study, we construct HiCAD, the first large-scale dataset aligning hand-drawn sketches, textual descriptions, and parametric CAD codes. Based on this, we introduce HiCAD, a two-stage framework comprising Cooperative Multi-Task Alignment to bridge the representational gap between heterogeneous inputs, and Spatial-Aware Reinforcement Learning to enforce geometric and topological consistency. Extensive experiments demonstrate that our method significantly outperforms existing baselines across multiple tasks, validating its effectiveness and robustness in transforming heterogeneous human intents into high-fidelity parametric CAD models. Our project page: this https URL.
• 中文摘要 从人类意图进行参数化计算机辅助设计（CAD）建模仍具挑战性，尤其是在概念设计阶段，设计目标通过不完整且无结构化的模态（如手绘草图和文本描述）表达。本研究重新思考人类意图到CAD的流程，提出一种统一方法，直接将多层次人类意图映射到可执行代码，而无需假设目标CAD模型的存在。为支持我们的研究，我们构建了HiCAD，这是首个将手绘草图、文本描述和参数化CAD代码对齐的大规模数据集。基于此，我们引入了HiCAD，这是一个两阶段框架，包括协作多任务对齐（COOPERATIVE Multi-Task Alignment）以弥合异构输入之间的表示鸿沟，以及空间感知强化学习（Spatial-Aware Reinforcement Learning）以强化几何和拓扑一致性。大量实验表明，我们的方法在多项任务中显著优于现有基线，验证了其在将异构人类意图转化为高精度参数化CAD模型方面的有效性和鲁棒性。我们的项目页面：此 https URL。

FARL: A Fully Automated Pipeline of RL Interface and Solver Design

FARL：一套全自动化的强化学习接口与求解器设计流程

• Authors: Tianyu Jin, Liuyu Xiang, Chaowen Yang, Xingyang Chang, Pei Pei Li, Zhaofeng He
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15691
• Pdf link: https://arxiv.org/pdf/2610.15691
• Abstract Existing reinforcement learning (RL) largely focuses on optimizing learning algorithms within a pre-defined task formulation, while applying RL to new tasks still relies heavily on human expertise to design the learning pipeline: what an agent observes, how it is rewarded, which algorithm it uses, and how training is configured. Automating these choices is challenging because they are tightly coupled. We introduce FARL (Fully Automated Reinforcement Learning), a framework that automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings. FARL uses training statistics and observed behavior to diagnose failures, propose changes, and test whether they improve learning. Its search preserves diverse candidates and the controls needed for these tests, using a staged budget to screen designs before investing in longer training. Experimental memory records each finding together with its conditions, so an unsuccessful component can be reconsidered in a different combination. Across expert-withheld and expert-available settings, FARL outperforms the evaluated automated baselines on challenging navigation, locomotion, and manipulation tasks. On Adroit Hand Relocate, FARL achieves 97.1% success at 5M training steps per policy, compared with 0% for Eureka and LiMEN and 1.0% for native Proximal Policy Optimization (PPO) under the same policy training budget.
• 中文摘要 现有的强化学习（RL）主要侧重于在预设任务体系内优化学习算法，而在新任务中应用RL仍高度依赖人类专业知识来设计学习流程：智能体观察到什么、如何获得奖励、使用哪种算法以及如何配置训练。自动化这些选择具有挑战性，因为它们紧密耦合。我们介绍了FARL（全自动强化学习），这是一个通过联合搜索观察和奖励程序、学习算法及训练设置来自动化RL设计的框架。FARL利用训练统计和观察到的行为来诊断失败、提出变更建议并测试是否能改善学习。其搜索保留了多样化的候选对象和测试所需的对照，利用分阶段预算筛选设计，然后再投入更长时间的训练。实验记忆会连同每个发现及其条件一起记录，因此不成功的部分可以以不同组合重新考虑。在专家保留和专家可用设置中，FARL在具有挑战性的导航、移动和操作任务上表现优于评估后的自动基线。在Adroit Hand Relocate中，FARL在每策略500万训练步的成功率为97.1%，而Eureka和LiMEN在相同策略培训预算下为0%，原生近端策略优化（PPO）为1.0%。

RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

RSIPO：通过技能-策略共进化实现代理强化学习的递归自我改进策略优化

• Authors: Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Xiao Luo, Yefeng Zheng, Bo Zhang, Haoxuan Li, Pheng-Ann Heng
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.15772
• Pdf link: https://arxiv.org/pdf/2610.15772
• Abstract Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.
• 中文摘要 近期对能动强化学习（RL）的关注点在于改善长期任务中的学分分配。一个有前景的方向是将强化学习与可验证奖励（RLVR）和策略内自我蒸馏（OPSD）结合起来，将轨迹层面优势与密集的代币级监督相结合。这一工作方向常常让教师模型在特权情境（如代理技能）上进行评分，以评分由学生政策生成的代币。尽管取得了进展，但它们大多忽视了代理技能的政策演进以及教师监督的可靠性。为此，我们引入递归自我改进策略优化（RSIPO），这是一种自我演进的代理强化学习框架，使智能技能和策略模型能够在整个培训过程中共同演化。具体来说，RSIPO使该策略能够从已完成的轨迹中提取技能，并通过轨迹重放明确优化技能生成能力。它还进一步运用实证贝叶斯收缩法动态校准学生对教师评分的信任度，考虑到技能增强型教师指导有时可能不那么可靠。在具身互动和基于搜索的代理任务中，使用Qwen家族模型的实验表明，RSIPO的表现优于所有比较的基线方法。以Qwen2.5-7B为基础模型，RSIPO在ALFWorld上的平均成功率为98.6%，相比第二佳基线和标准GRPO分别提升了2.5%和17.4%的绝对成功率。

LLM Value Functions for Credit Assignment in Agentic RL

代理强化学习中学分分配的LLM价值函数

• Authors: Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15840
• Pdf link: https://arxiv.org/pdf/2610.15840
• Abstract Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assigning credit to individual actions is difficult. Methods such as GRPO assign the same trajectory-level advantage to all actions, while critic-based methods require learning an additional value function that can be costly and unstable. We introduce LLM Value Function RL (LVRL), which uses the pretrained knowledge and in-context learning ability of an LLM to estimate policy-aware state values without finetuning a separate critic. LVRL conditions the value model on group context and calibrates its predictions using known values at the initial and terminal states, enabling step-level credit assignment from estimated value differences. We further extend this idea to learning from an expert policy through on-policy distillation (OPD), introducing LLM Value Function OPD (LVOPD), which focuses teacher supervision on actions associated with decreases in estimated value. We evaluate our approach across diverse agentic benchmarks: AppWorld, BabyAI, and SWE-bench Verified. Across policy scales (4B to 32B), LVRL consistently and substantially outperforms both GRPO and PPO. On AppWorld and SWE-bench Verified, LVRL yields absolute improvements of up to 13.6 and 8.6 percentage points over GRPO. Notably, on tasks requiring extended exploration like BabyAI, LVRL bridges the gap from total failure to 75.0% success. Furthermore, LVOPD yields consistent improvements over standard distillation baselines on AppWorld. Finally, Monte-Carlo analysis confirms that our frozen critic accurately identifies semantic pitfalls and recoveries, proving that LVRL extracts robust, policy-aware step-level credit from sparse terminal outcomes.
• 中文摘要 对于大型语言模型（LLM）代理的强化学习（RL）在长期任务中仍然具有挑战性，且结果奖励稀疏，且难以将功劳归属于单个行动。如GRPO等方法赋予所有行为相同的轨迹层优势，而基于批评者的方法则需要学习一个额外的价值函数，这可能成本高昂且不稳定。我们引入了LLM价值函数RL（LVRL），利用LLM的预训练知识和上下文学习能力，在不微调独立批评者的情况下估计策略感知状态值。LVRL以群体上下文为条件值，并利用初始和终端状态的已知值校准预测，从而从估计值差异中实现阶级信用分配。我们将这一理念进一步扩展到通过策略内提炼（OPD）从专家政策学习，引入了LLM价值函数OPD（LVOPD），将教师监督重点放在与估计值下降相关的动作上。我们在多种代理基准中评估我们的方法：AppWorld、BabyAI 和 SWE-bench Verified。在政策尺度（4B 到 32B）中，LVRL 持续且显著地优于 GRPO 和 PPO。在 AppWorld 和 SWE-bench Verified 上，LVRL 比 GRPO 分别实现了高达 13.6 和 8.6 个百分点的绝对提升。值得注意的是，在需要长时间探索的任务如 BabyAI 上，LVRL 将完全失败与成功率的差距拉近了 75.0%。此外，LVOPD 在 AppWorld 上相较标准蒸馏基线持续有改善。最后，蒙特卡洛分析证实了我们的冻结批评者准确识别语义陷阱和恢复，证明 LVRL 从稀疏的终端结果中提取了稳健且具政策意识的阶级功劳。

Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction

带有语义ID的个性化LLM推荐：自回归束搜索与多词汇预测

• Authors: Sejoon Oh, Fernando Amat Gil, Evan Smothers, Dan Iter, Mark Thornburg, Dawit Mureja Argaw, Sourabh Medapati, Thea Wang, Ria Verma, Qiuling Xu, Harshit Syal, Ehsan Gholami, Moumita Bhattacharya, Ashish Rastogi
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.15864
• Pdf link: https://arxiv.org/pdf/2610.15864
• Abstract Large-scale personalization is shifting from embedding-table retrieval to LLM-based recommenders that generate the Semantic ID (SID) of the item to recommend, where each SID is a short sequence of discrete hierarchical content codes. A single instruction-tuned model can then unify retrieval, content understanding, and language, and recommend brand-new titles from their content alone using SID; then, how should the model generate the SID? Traditionally, models rely on a single language modeling head that generates tokens autoregressively, typically using constrained beam search. In contrast, an emerging body of work explores multi-token prediction (MTP), where dedicated heads predict all tokens in parallel, substantially reducing the decoding latency associated with beam search. We report an industry-scale, apples-to-apples comparison of these two generation paradigms: single-head autoregressive decoding vs. MTP. We introduce a new auxiliary ranking loss, defined over the joint SID score, to train the MTP objective. We evaluate the SID training objectives on both quality and training/inference cost, inside a multi-task LLM that jointly does personalized retrieval, content question-answering (QA), and instruction following over a large streaming-media catalog. Our experiments show single-head beam search consistently beats MTP on most product-critical metrics: ranking and content QA, and trains at roughly 2.6x lower dollar cost on Qwen3-8B scale; MTP wins only on raw inference throughput. Around this finding, we detail an efficient training and inference stack and report end-to-end gains over the previous non-SID baseline equipped with an auxiliary ranking head: +77% item cold-start ranking, +22% content-QA accuracy, +2.8x faster training, with language metric regression traced to SID vocabulary expansion. We address the language regression by reinforcement learning post training.
• 中文摘要 大规模个性化正从嵌入表检索转向基于LLM的推荐器，后者生成推荐项目的语义ID（SID），每个SID是一系列离散层次内容代码。单一指令调优模型可以统一检索、内容理解和语言，仅凭内容推荐新书名，仅用SID进行;那么，模型应如何生成SID？传统上，模型依赖单一语言建模头，通过自回归方式生成代币，通常使用受限束搜索。相比之下，新兴研究探索多标记预测（MTP），专用标记并行预测所有符号，显著降低束搜索相关的解码延迟。我们报告了这两种世代范式的行业级、苹果对苹果比较：单头自回归译码与MTP。我们引入了新的辅助排名损失，定义在联合SID分数上，以训练MTP目标。我们在一个多任务LLM中评估SID训练目标的质量和训练/推理成本，该模型结合了个性化检索、内容问答（QA）和指令跟踪，覆盖大型流媒体目录。我们的实验显示，单头束搜索在大多数产品关键指标上持续优于MTP：排名和内容质量保证，且在Qwen3-8B尺度上训练成本约低2.6倍;MTP仅在原始推理吞吐量上获胜。围绕这一发现，我们详细介绍了高效的训练和推理堆栈，并报告了相较于之前非SID基线的端到端提升，并配备了辅助排名首：+77%的项目冷启动排名，+22%的内容质量准确率，+2.8倍的训练速度，语言指标回归可追溯到SID词汇量的扩展。我们通过训练后强化学习来解决语言回归问题。

Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards

方差最优状态抽样用于可验证奖励的强化学习

• Authors: Yibo Wang, Yuanxin Liu, Zhaoran Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15888
• Pdf link: https://arxiv.org/pdf/2610.15888
• Abstract Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $\mu_{\mathrm{var}}^\pi(s) \propto d^\pi(s)\,\sigma_\pi(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.
• 中文摘要 带有可验证奖励的强化学习将大量推广预算用于无学习信号的状态。我们将此重新定义为关于抽样分布本身的问题：在固定的推广预算下，哪种状态分布最小化策略梯度估计量的方差？答案允许闭式Neyman分配，$\mu_{\mathrm{var}}^\pi（s） \propto d^\pi（s）\，\sigma_\pi（s）自然连接GRPO的群体相对优势，并随着政策改进引入一个涌现的逆向课程，无需外部强制的时间表。基于该原则，我们提出VSR（方差驱动状态重抽样），这是一种使用前缀重放缓冲区和零成本方差代理的GRPO插件修改。在数学推理基准和长期代理任务（Sokoban、BabyAI）中，VSR在大多数基准测试中表现优于GRPO和GiGPO，且每次更新生成的代币数更少。

ADAPT: Audio-Depth Action Planning with Transition-Aware Rewards for Lights-Out Industrial Navigation

ADAPT：音频深度行动规划，并提供关灯工业导航的过渡感知奖励

• Authors: Wenjun Huang, Shenghao Fu, Yian Jin, Yang Ni, Yifan Li, Lichi Li, Xingjian Diao, Pengfei Zhang, Yutong Song, Hanning Chen, Jiarui Cui, Tianyu Yang, Shilong Li, Yu Kong, Mohsen Imani
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.15956
• Pdf link: https://arxiv.org/pdf/2610.15956
• Abstract Vision-language-action models for embodied navigation typically rely on RGB observations, limiting their applicability in lights-out industrial environments where visible illumination may be unavailable. We study illumination-independent navigation using spatial audio for target localization and depth for obstacle-aware geometry. To support this setting, we introduce DarkNav-Bench, a benchmark with 24 industrial scenes and four progressively challenging acoustic conditions. We further propose ADAPT, an audio-depth action-planning framework equipped with spatial-audio perception and optimized for closed-loop navigation through reinforcement learning (RL). Our key RL contribution, transition-aware reward shaping (TARS), defines a pose-aware shortest-action potential over a position-heading graph. Unlike conventional geodesic-progress rewards, TARS accounts for both feasible movements and in-place rotations, providing action-consistent feedback for policy optimization. Experiments show that ADAPT consistently outperforms training-based navigation baselines and zero-shot multimodal models on both seen and unseen scenes. On unseen environments, it achieves 90.6% success under Clean conditions and retains 60.0% success under the most challenging Localized Noise setting. These results demonstrate the potential of spatial audio and depth as complementary modalities for reliable RGB-free navigation in lights-out industrial environments.
• 中文摘要 具身导航的视觉-语言-动作模型通常依赖RGB观测，限制了其在灯光熄灭工业环境中的适用性，因为这些环境可能无法获得可见光照。我们研究光照无关导航，利用空间音频定位目标，利用深度定位障碍物感知几何。为支持这一设定，我们引入了DarkNav-Bench，这是一个包含24个工业场景和4种逐步复杂声学条件的基准测试。我们还提出了ADAPT，这是一个配备空间音频感知的音频深度动作规划框架，并优化用于通过强化学习（RL）实现闭环导航。我们的关键强化语言贡献是过渡感知奖励塑形（TARS），定义了位置方向图上的姿态感知最短动作势能。与传统的测地线进展奖励不同，TARS同时考虑可行移动和原位旋转，提供动作一致的反馈以优化策略。实验显示，ADAPT在可见和未见场景中持续优于基于训练的导航基线和零拍摄多模态模型。在未见环境下，在清洁条件下成功率为90.6%，在最具挑战性的局部噪声环境中保持60.0%。这些结果展示了空间音频和深度作为辅助模式，在灯光熄灭的工业环境中实现可靠无RGB导航的潜力。

PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordination

PopGen：生成语义多样化的人群以实现零射点协调

• Authors: Paul Teiletche, Kanghyun Ryu, Kyungjoon Park, Dhruva Tirumala, Markus Wulfmeier, Negar Mehr
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.15969
• Pdf link: https://arxiv.org/pdf/2610.15969
• Abstract In multi-agent reinforcement learning, exposing a cooperator to partners with diverse strategies during training is key to better collaboration at test time. However, existing methods typically rely on trajectory-level proxies to measure diversity, which may label two distinct trajectories of the same strategy as diverse. Instead, our key insight is to leverage LLMs' ability to describe behaviorally distinct agents in environments, similar to how humans can recognize high-level strategies. To this end, we introduce PopGen, a framework that synthesizes semantically diverse agents from scratch using LLMs. PopGen first generates a diverse set of textual descriptions of agent behaviors, then translates each behavior into a code policy that can be executed within the environment. We show that cooperators trained against PopGen populations achieve robust zero-shot cooperation with unseen partners, outperforming population-play baselines across four collaborative environments. We further show that semantic diversity scales with PopGen population size, improving the cooperator's worst-case performance, and we identify design choices that preserve population quality while maximizing diversity. Our code is available at this https URL.
• 中文摘要 在多智能体强化学习中，在训练期间让合作者接触到拥有多样策略的伙伴，是测试时更好协作的关键。然而，现有方法通常依赖轨迹级代理来衡量多样性，这可能会将同一策略的两种不同轨迹标记为多样性。相反，我们的关键见解是利用LLM描述环境中行为不同代理的能力，类似于人类识别高级策略。为此，我们引入了PopGen框架，该框架利用LLM从零合成语义多样性的代理。PopGen首先生成一套多样化的代理行为文本描述，然后将每种行为转化为可在环境中执行的代码策略。我们展示了针对PopGen群体训练的合作者能与未见伙伴实现强有力的零样本合作，优于四个协作环境中的群体游戏基线。我们还进一步证明，语义多样性会随着PopGen种群规模的增长而扩展，从而提升合作者在最坏情况下的性能，并识别出既能保持种群质量又最大化多样性的设计选择。我们的代码可在此 https URL 获取。

Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents

错误与回声：工具使用代理的责任意识信用分配

• Authors: Xingjian Diao, Tianyu Yang, Wenjun Huang, Xiangchi Yuan, Chunhui Zhang, Xingchen Zhao, Weiyi Wu, Haotian Xu, Soroush Vosoughi, Xiangliang Zhang, Jiang Gui
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16027
• Pdf link: https://arxiv.org/pdf/2610.16027
• Abstract Reinforcement learning improves tool-use agents, yet fine-grained credit assignment remains challenging even with step-level feedback. An observed error may be inherited from an earlier decision, while multiple responsible decisions may share the same downstream consequence or participate in later recovery. We introduce Error-Chain-Aware Optimization (ECHO), a responsibility-aware credit assignment framework that separates responsibility identification from credit allocation. A frozen judge combines retrospective trajectory analysis with decision-time assessment from the policy-visible prefix to distinguish newly introduced errors, inherited consequences, missed corrections, and successful corrections. ECHO then counts shared downstream consequences once across responsible decisions and constrains rewards for repairing self-induced errors through source-funded recovery, producing bounded action-level supervision alongside terminal supervision. Experiments on six diverse tool-use benchmarks at both 4B and 8B scales show strong and consistent improvements over the compared baselines, with ECHO achieving the highest score on five of six benchmarks at each scale. These results support decision responsibility as a useful basis for fine-grained credit assignment in tool-use agents. Our code is available at this https URL.
• 中文摘要 强化学习提升工具使用代理，但即使有步骤级反馈，细粒度的信用分配仍然具有挑战性。观察到的错误可能继承自早期决策，而多个负责任的决策可能共享相同的下游后果或参与后续恢复。我们引入错误链感知优化（ECHO），一种责任感知的信用分配框架，将责任识别与信用分配分离。冻结法官结合回顾性轨迹分析与策略可见前缀的决策时评估，区分新引入的错误、继承后果、漏补和成功纠正。ECHO随后在负责任决策中计数共享的下游后果，并通过源头资助的恢复限制自我造成错误修复的奖励，实现有限制的行动级监督与终端监督。在6个不同的工具使用基准（4B和8B尺度）上的实验显示，ECHO在两个基准中均获得最高分，分别在6个基准中获得最高分。这些结果支持决策责任作为工具使用代理细粒度信用分配的有用基础。我们的代码可在此 https URL 获取。

PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary

鸽子：渐进式长视频字幕，具持久身份画廊和不断演变的整体叙事摘要

• Authors: Xinlong Chen, Weihong Lin, Yuanxing Zhang, Xinyu Liu, Yue Ding, Bozhou Li, Bohan Zeng, Yang Shi, Qiang Liu, Pengfei Wan, Liang Wang, Tieniu Tan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.16066
• Pdf link: https://arxiv.org/pdf/2610.16066
• Abstract Detailed video captioning has become an essential semantic interface for multi-modal understanding and generation. While existing methods mainly target short videos, extending them to long videos remains challenging, primarily due to limited context windows of current video understanding models. A naive solution is to adopt segment-wise captioning followed by concatenation, yet this paradigm often suffers from inconsistent subject references and fragmented long-range narratives. To mitigate these issues, we introduce PIGEONS, an incremental long-video captioning model built upon a Persistent Identity Gallery and an Evolving Overarching Narrative Summary. Before caption generation, PIGEONS scans the entire video to construct an identity gallery that captures key subjects along with their visual references, thereby providing a stable referential anchor. The model then performs clip-by-clip captioning conditioned on the persistent gallery and an evolving overarching narrative summary, which compactly maintains and updates essential long-range contextual information after each captioning step. To enable these capabilities, we develop a three-stage training strategy that integrates supervised fine-tuning and reinforcement learning. Extensive experiments demonstrate that PIGEONS markedly improves long-video captioning performance, achieving superior accuracy, completeness, and subject referential consistency over existing methods, while maintaining competitive performance on short-video captioning.
• 中文摘要 详细的视频字幕已成为多模态理解和生成的重要语义接口。虽然现有方法主要针对短视频，但将其扩展到长视频仍具挑战性，主要原因是当前视频理解模型的上下文窗口有限。一个天真的解决方案是采用分段字幕再串接，但这种范式常常存在主体引用不一致和远距离叙事碎片化的问题。为缓解这些问题，我们引入了PIGEONS，一种基于持续身份画廊和不断演进的整体叙事摘要的增量长视频字幕模型。在生成字幕之前，PIGEONS 扫描整个视频，构建一个身份画廊，捕捉关键主题及其视觉参考，从而提供稳定的指涉锚点。模型随后基于持久画廊和不断演进的整体叙述摘要执行逐段字幕，紧凑地维护并更新每个字幕步骤后重要的长距离上下文信息。为实现这些能力，我们开发了一套三阶段训练策略，整合监督微调和强化学习。大量实验表明，PIGEONS 显著提升了长视频字幕的性能，实现了优于现有方法的准确性、完整性和主题引用一致性，同时保持短视频字幕的竞争性能。

Understanding and Reducing Misalignment in Persuasion Optimization

理解并减少说服优化中的错位

• Authors: Boyi Wei, Ruta Desai, Peter Henderson, Wenyue Hua
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16070
• Pdf link: https://arxiv.org/pdf/2610.16070
• Abstract Frontier models are now trained for complex multi-agent tasks, from running a company to engaging in negotiations. These tasks necessitate training models to be persuasive. But what does this optimization pressure mean for model alignment? In this work, we study how reinforcement learning (RL) training for persuasion affects models' use of misaligned strategies such as evidence fabrication and belief coercion, and what techniques can reduce such misalignment. We find that optimizing sender models solely for receiver belief shifts causes the average number of distinct misaligned strategies used per game to increase by more than 25%, and fabrications per game increase to more than 4 times relative to pre-RL models. We compare various mitigation approaches, including stronger strategy guidance, clean demonstrations, penalizing misaligned strategy usage, and improving the model's ability to detect fabrications. We find that navigating the persuasiveness-misalignment trade-off in this setting is challenging, with the best trade-offs coming from combining guidance on aligned persuasion strategies with reward penalties on misalignment. Overall, our work provides a testbed for assessing misalignment stemming from RL-based persuasion training and shows concrete paths for reducing misalignment. Importantly, it highlights that without direct intervention, persuasion-involved RL environments are likely to naturally lead to large increases in the use of misaligned strategies.
• 中文摘要 前沿模型现在被训练用于复杂的多智能体任务，从经营公司到参与谈判。这些任务要求训练模型具有说服力。但这种优化压力对模型对齐意味着什么？本研究中，我们研究强化学习（RL）训练对模型使用错位策略（如证据伪造和信念胁迫）的影响，以及哪些技术可以减少这种错位。我们发现，仅针对接收方信念转移优化发送者模型，使每局游戏中使用的不同错位策略平均数量增加超过25%，而每局制造策略相比强化学习前模型增加了4倍以上。我们比较了多种缓解方法，包括更强的策略指导、干净的演示、惩罚错位策略使用以及提升模型检测伪造的能力。我们发现，在这种环境中应对说服力与错位的权衡具有挑战性，最佳权衡来自于结合对齐说服策略的指导与对错位的奖励惩罚。总体而言，我们的工作为评估基于强化学习的说服训练引起的错位提供了试验平台，并展示了减少错位的具体路径。重要的是，它强调，如果没有直接干预，涉及说服的强化学习环境很可能会自然导致错位策略的使用大幅增加。

SelectTTRL: Representation-Guided Data Selection for Efficient Test-Time Reinforcement Learning

SelectTTRL：高效测试时强化学习的表征引导数据选择

• Authors: Yi Li, Linan Yue, Weibo Gao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16104
• Pdf link: https://arxiv.org/pdf/2610.16104
• Abstract Test-Time Reinforcement Learning (TTRL) enables large language models to adapt directly on unlabeled test problems by constructing pseudo-labels from multiple sampled responses. However, existing TTRL methods typically apply the same expensive sampling and optimization procedure to all test problems, despite substantial differences in their training utility. In this paper, we study whether such problems can be identified before costly test-time training, without access to ground-truth answers. We uncover a stable relationship between internal response representations and problem-level empirical accuracy. Specifically, we define COS as the cosine similarity between a model's response representation and a correctness-related direction constructed from an independent question-answering dataset. COS consistently correlates with problem solvability across different models, datasets, and experimental settings, and we further provide a theoretical analysis explaining this relationship. Building on this finding, we propose SelectTTRL, a representation-guided data selection method that ranks test problems using COS and applies TTRL only to a selected percentile interval. SelectTTRL leaves the standard TTRL pipeline unchanged for the selected problems, while avoiding unnecessary multi-response sampling and reinforcement learning updates on the remaining ones. Experiments across multiple reasoning benchmarks demonstrate that SelectTTRL substantially reduces test-time computation while maintaining or improving the performance of standard TTRL. Code is available at this https URL.
• 中文摘要 测试时强化学习（TTRL）使大型语言模型能够直接适应未标记的测试问题，通过从多个抽样回答构建伪标签。然而，现有TTRL方法通常对所有测试问题应用相同的昂贵抽样和优化程序，尽管其训练效用存在显著差异。本文研究了在昂贵的测试时间训练前，在无法获得真实答案的情况下，是否能识别此类问题。我们揭示了内部响应表示与问题级经验准确性之间的稳定关系。具体来说，我们定义COS为模型响应表示与由独立问答数据集构建的正确性相关方向之间的余弦相似度。COS在不同模型、数据集和实验环境中与问题可解性保持一致，我们进一步提供了理论分析来解释这一关系。基于这一发现，我们提出了SelectTTRL，一种表示引导的数据选择方法，利用COS对测试问题进行排序，并仅对选定的百分位区间应用TTRL。SelectTTRL对所选问题保持标准TTRL流程不变，同时避免对剩余问题进行不必要的多响应抽样和强化学习更新。多项推理基准测试的实验表明，SelectTTRL在保持或提升标准TTRL性能的同时，显著减少了测试时间的计算。代码可在此 https 网址获取。

Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR

超越政策推广：基于锚定噪声的嵌入空间定向探索，适用于LLM RLVR

• Authors: Daijiao Liu, Huiyi Wang, Lina Yao, Dong Gong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16105
• Pdf link: https://arxiv.org/pdf/2610.16105
• Abstract Reinforcement learning with verifiable rewards (RLVR) typically uses small rollout groups per prompt, where repeated sampling from the same policy can yield highly similar trajectories, reducing within-group reward variation and weakening group-relative advantage signals. Existing remedies such as higher sampling temperature, entropy bonuses, asymmetric clipping, and larger rollout budgets broaden sampling or modify policy updates but leave rollout conditioning unchanged. In this paper, we propose EDEN (Embedding-space Directional Exploration via anchored Noise), which intervenes at the rollout source through anchored input perturbations. EDEN explores beyond the policy's own rollouts. Within the same rollout budget, it replaces a small fraction of standard rollouts in each group with rollouts that the same policy generates from the prompt plus a single appended, perturbed embedding position, rather than from the prompt alone. Gaussian noise drives variation, while a shared anchor provides a common conditioning direction within each prompt group. Selected perturbed rollouts contribute to GRPO updates with downweighted advantages, and the model architecture, discrete decoding, and test-time inference remain unchanged. Across model scales and mathematical reasoning benchmarks, EDEN outperforms GRPO and the exploration controls evaluated in the same pipeline, with the largest gains in sampled solution coverage. Analyses show that more prompt groups provide non-zero group-relative learning signals and that noise and anchoring play complementary roles. Intervening at the rollout source is thus an effective lever for exploration in RLVR.
• 中文摘要 带可验证奖励的强化学习（RLVR）通常每个提示使用较小的展开组，重复抽样同一策略可产生高度相似的轨迹，减少组内奖励的差异并削弱群体相对优势信号。现有的补救措施如更高的采样温度、熵加成、非对称裁剪和更大的推广预算，扩大了抽样范围或修改策略更新，但保持推广条件不变。本文提出EDEN（通过锚定噪声进行嵌入空间定向探索），通过锚定输入扰动介入推广源。EDEN探索策略自身的推广。在同一推广预算内，它用同一策略由提示加上一个附加的扰动嵌入位置生成的推送替代每组中一小部分标准推广，而非仅仅由提示。高斯噪声驱动变化，而共享锚点则在每个提示组内提供统一的条件方向。选定的扰动展开对GRPO更新贡献，具有降权优势，模型架构、离散译码和测试时间推断保持不变。在模型尺度和数学推理基准中，EDEN表现优于GRPO及同一流水线中评估的探索控制，采样解覆盖率提升最大。分析显示，更多提示组提供非零组相对学习信号，噪声与锚定相互作用。因此，在扩展源介入是RLVR探索的有效杠杆。

EventLens: Event-Structure Reinforcement Learning for Video Understanding

EventLens：视频理解中的事件结构强化学习

• Authors: Guowen Zhang, Boshen Xu, Zihao Yue, Ziheng Wang, Xiaokun Liu, Xin Tao, Wenyu Qin, Pengfei Wan, Qin Jin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2610.16172
• Pdf link: https://arxiv.org/pdf/2610.16172
• Abstract Recent multimodal large language models (MLLMs) achieve strong results on video question answering, yet often fail to recover the temporal structure that makes a video coherent. They may merge neighboring events, miss single-event progression, or infer implausible relations across events. We study structured temporal understanding as the ability to recover event structure from visual evidence, including event boundaries, single-event progression, and multi-event relations. We argue that this gap arises from a mismatch between current post-training objectives and temporal reasoning, where caption-style supervision allows models to rely on language shortcuts rather than visual temporal grounding. To address this, we propose EventLens, a vision-centric reinforcement learning framework that learns video understanding through event-structure recovery. Instead of reconstructing captions, EventLens trains models to recover temporal structure under controlled perturbations, such as altered temporal granularity, reversed progression, and shuffled event order. We instantiate this framework with a three-level temporal ontology and derive three verifiable task families: event segmentation, progression discrimination, and multi-event relation reasoning. These tasks admit deterministic rewards, enabling task-specialized GRPO training without human preference labels or learned reward models. We further introduce multi-teacher on-policy distillation (MT-OPD) to consolidate specialized policies into a unified model. Experiments on Qwen3-VL backbones show consistent improvements on temporal grounding and event-sensitive reasoning benchmarks, while preserving general video-QA performance. Learning curves further demonstrate that EventLens achieves comparable downstream transfer with significantly reduced training cost compared to direct mixed RL optimization.
• 中文摘要 近期多模态大型语言模型（MLLM）在视频问答方面取得了强劲成绩，但往往未能恢复使视频连贯的时间结构。它们可能合并邻居事件，遗漏单事件进程，或推断事件间不合理的关系。我们研究结构化时间理解，即从视觉证据中恢复事件结构的能力，包括事件边界、单事件进程和多事件关系。我们认为这一差距源于当前训练后目标与时间推理之间的不匹配，后者字幕式监督使模型依赖语言捷径而非视觉时间基础。为此，我们提出了EventLens，一种以视觉为中心的强化学习框架，通过事件结构恢复学习视频理解。EventLens 不重建字幕，而是训练模型在受控扰动（如时间粒度变化、反向进程和事件顺序洗牌）下恢复时间结构。我们用三级时间本体论实现该框架，推导出三个可验证任务族：事件分割、进程辨别和多事件关系推理。这些任务允许确定性奖励，使任务专用 GRPO 训练无需人工偏好标签或学习奖励模型。我们还引入了多教师策略蒸馏（MT-OPD），将专业策略整合为统一模型。Qwen3-VL 骨干的实验显示，时间基础和事件敏感推理基准持续提升，同时保持了一般视频质量保证性能。学习曲线进一步表明，EventLens 实现了相当的下游传输，且与直接混合强化学习优化显著降低了训练成本。

MSC: Multi-Stage Caster-aware Control Framework with Uncertainty-Aware NMPC and RL-based Execution Layer

MSC：多阶段Caster感知控制框架，具备不确定性感知的NMPC和基于强化语言的执行层

• Authors: Jinho Park, Jeongtae Lee, Jeonghyeok Lim, Keun Ha Choi, Kyung-Soo Kim
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16199
• Pdf link: https://arxiv.org/pdf/2610.16199
• Abstract Freely swiveling casters are widely adopted in industrial AMRs/SDVs for their mechanical simplicity and load capacity. However, caster misalignment during low-speed or high-curvature maneuvers induces strong contact nonlinearities, widening the planning-execution gap between commanded $(v, \omega)$ and realized motion. Existing caster-aware strategies are highly sensitive to state-estimation reliability, causing unstable execution in uncertain transition regimes. To address this, we present Multi-Stage Caster-aware Control (MSC), a planning-execution integrated hierarchical framework featuring two core mechanisms. First, an uncertainty-aware NMPC (CU-MPC) dynamically schedules caster-aware costs and input regularization using predictive uncertainty to suppress overly aggressive maneuvers. Second, a reinforcement learning-based execution layer estimates and predicts caster states under partial observability, transforming high-level velocity commands into physically executable wheel speeds adapted to current contact conditions. Validations in Isaac Sim and real robots demonstrate that MSC offers a clear comparative advantage over existing decoupled compensation approaches by significantly improving caster-state estimation accuracy, minimizing misalignment, and enhancing tracking robustness in caster-sensitive maneuvers.
• 中文摘要 自由旋转脚轮因其机械结构简洁和载重能力被工业AMR/SDV广泛采用。然而，在低速或高曲率机动中，脚轮错位会引发强烈的接触非线性，扩大指令$（v， \omega）$与实际运动之间的规划-执行差距。现有的射手感知策略对状态估计可靠性高度敏感，导致在不确定的过渡状态下执行不稳定。为此，我们提出了多级施法者感知控制（MSC），这是一个计划-执行集成的层级框架，具有两个核心机制。首先，一个具备不确定性的NMPC（CU-MPC）通过预测不确定性动态调度施法者感知成本和输入正则化，抑制过于激进的机动。其次，基于强化学习的执行层在部分可观测性下估计并预测施法者状态，将高级别速度指令转化为适应当前接触条件的物理可执行车轮速度。Isaac Sim和真实机器人的验证表明，MSC通过显著提升施法者状态估计准确性、最小化错位以及增强施法者敏感机动中的跟踪稳健性，显著优于现有解耦补偿方法。

Look Back to Move Forward: Learning to Revisit for Agentic Compositional Image Generation

回顾前进：学会重新审视代理构图图像生成

• Authors: Zixu Wang, Xiaoran Zhao, Yahui Liu, Feng-ao Wang, Junhao Xiao, You Shu, Bing Liu, Yiting Liu, Songlei Jian, Zhiliang Tian, Dongsheng Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16224
• Pdf link: https://arxiv.org/pdf/2610.16224
• Abstract Text-to-image models have made remarkable progress in visual fidelity and instruction following, yet complex compositional generation remains challenging. Existing agentic approaches address this limitation through iterative generation and editing, adapting their actions based on interaction history. However, they typically fix the continuation state (i.e., the image from which refinement proceeds) to the latest image. Since iterative editing is not necessarily monotonic, the agent may continue refining from the latest image even when it has already degraded. In this work, we introduce GenRevisit, an agentic framework that extends action selection to joint continuation-state and action selection. The agent can not only use interaction history to adjust editing instructions but also select an image candidate to edit, jointly deciding where to continue from and how to refine. We train GenRevisit with SFT followed by agentic reinforcement learning. To address the coarse credit assignment of final outcome reward in GRPO, which assigns the same trajectory-level advantage to all actions, we further propose Normalized Action Credit Assignment (NACA) to redistribute positive advantage according to each action's contribution to reducing the remaining objective. On GenEval2, GenRevisit achieves GM scores of 84.3 and 74.4 with Qwen-Image-2512 and Z-Image-Turbo, surpassing the strongest baselines using the same generators by 12.7 and 10.9 points respectively. GenRevisit achieves state-of-the-art performance on GenEval, GenEval++, and GenEval2 among baselines.
• 中文摘要 文本转图像模型在视觉忠实度和指令跟踪方面取得了显著进步，但复杂的合成生成仍然具有挑战性。现有的代理方法通过迭代生成和编辑解决这一限制，并根据交互历史调整动作。然而，它们通常将延续状态（即进行细化的图像）固定到最新图像。由于迭代编辑不一定是单调的，代理即使最新图像已经退化，仍可能继续精炼。本研究介绍了GenRevisit，一种将动作选择扩展到联合续写状态和动作选择的智能体框架。智能体不仅可以利用交互历史调整编辑指令，还能选择候选图像进行编辑，共同决定从哪里继续和如何细化。我们用SFT训练GenRevisit，随后进行智能体强化学习。为解决GRPO中对所有行为赋予相同轨迹级优势的粗略归属问题，我们进一步提出归一化行动归属化认可分配（NACA），根据每个行为对剩余目标的贡献重新分配正向优势。在GenEval2上，GenRevisit在Qwen-Image-2512和Z-Image-Turbo中获得84.3和74.4的GM分数，分别比使用相同生成器的强基线高出12.7分和10.9分。GenRevisit在GenEval、GenEval++和GenEval2的基线表现达到最先进。

NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning

NetRL：通过NetGRPO驱动的强化学习驱动网络决策

• Authors: Lingfeng Peng, Xiaohui Xie, Yang Bao, Yong Cui
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16226
• Pdf link: https://arxiv.org/pdf/2610.16226
• Abstract Network decision-making tasks, including adaptive bitrate streaming, load balancing, and congestion control, require policies to process heterogeneous states and actions while balancing near- and long-term consequences. Existing solutions generalize poorly across network tasks, require substantial task-specific redesign, and lack reliable evaluation of long- horizon action consequences. Methods that learn through online interaction additionally face unstable optimization. We present NetRL, a two-stage LLM-based reinforcement learning framework that combines supervised fine-tuning on demonstration trajectories with online policy optimization. A universal mixed-embedding interface combines textual task descriptions with precise numerical states, while a network action head produces valid decisions. At its core, NetGRPO evaluates candidate actions over multiple future trajectories, directs deeper search toward uncertain alternatives through confidence-aware budgeting, and performs group-relative, clipped, and KL-regularized updates. NetRL achieves 23.6% higher QoE than NetLLM, 12.2% lower job completion time than Genet, and 24.1% higher network utility than Orca, yielding state-of-the-art performance among the evaluated methods. Confidence-aware budgeting reduces end-to-end training time by 27.1% without reducing validation QoE. NetRL also converges stably and generalizes to unseen network traces without retraining. Code and model weights are available at the repository linked in this paper.
• 中文摘要 网络决策任务，包括自适应码率流、负载均衡和拥塞控制，需要策略来处理异构状态和动作，同时平衡近长期后果。现有解决方案在网络任务间推广能力不佳，需要大量任务特定性重新设计，且缺乏对长期行动后果的可靠评估。通过在线交互学习的方法也面临不稳定的优化。我们提出了NetRL，一个两阶段基于LLM的强化学习框架，结合了对演示轨迹的监督微调与在线策略优化。通用混合嵌入接口将文本任务描述与精确数值状态结合，而网络动作头则产生有效决策。NetGRPO的核心是评估多个未来轨迹的候选动作，通过信心感知预算引导对不确定替代方案进行更深入的搜索，并执行群组相对、剪裁和KL正则化更新。NetRL的服务质量比NetLLM高23.6%，比Genet低12.2%的作业完成时间，比Orca高24.1%，在评估方法中实现了最先进的性能。信心感知预算可将端到端训练时间缩短27.1%，同时不降低验证服务质量。NetRL还能稳定收敛，并可推广到未见的网络痕迹，无需重新训练。代码和模型权重可在本文链接的仓库中获取。

A Survey of Flow Matching in Reinforcement Learning

强化学习中流匹配的综述

• Authors: Nabuat Zaman Nahim, Fairoz Nower Khan, Peizhong Ju
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16275
• Pdf link: https://arxiv.org/pdf/2610.16275
• Abstract Flow Matching (FM) has recently emerged as a principled and efficient generative modeling framework for reinforcement learning (RL), enabling expressive, multimodal policy parameterizations via deterministic probability transport. Compared to diffusion-based policies that rely on stochastic denoising chains, FM uses sampling based on ordinary differential equations (ODEs), with learned velocity fields, which can substantially reduce inference latency and simplify the incorporation of RL objectives. As research in flow-based RL rapidly accelerates across offline continuous control, online fine-tuning, and foundation model alignment, the literature has become highly fragmented. In this survey, we provide a comprehensive taxonomy of flow-matching approaches in reinforcement learning. We organize the literature along two axes: the target distribution being modeled (e.g., action policies, value critics, transition dynamics) and the mechanism of RL signal integration (e.g., energy-weighted regression, flow-based policy gradients, and group relative policy optimization). Furthermore, we survey emerging frontiers such as discrete and non-Euclidean action spaces, provide a systematic comparative analysis against Gaussian and diffusion baselines, and outline critical open problems. Ultimately, this survey serves as a foundational roadmap for the next generation of generative reinforcement learning and alignment.
• 中文摘要 流匹配（FM）最近作为一种原则性且高效的生成建模框架，用于强化学习（RL），通过确定性概率传输实现表达式、多模态的策略参数化。与依赖随机去噪链的基于扩散策略相比，FM采用基于常微分方程（ODE）的抽样，并结合学习速度场，从而大幅降低推断延迟并简化强化学习目标的纳入。随着基于流的强化学习研究在离线连续控制、在线微调和基础模型对齐等领域迅速加快，文献变得高度分散。本综述提供了强化学习中流匹配方法的综合分类法。我们将文献按两条轴线组织：被建模的目标分布（如行动政策、价值批评、过渡动态）和强化学习信号整合机制（如能量加权回归、基于流量的策略梯度和群体相对策略优化）。此外，我们调查了新兴前沿领域，如离散和非欧几里得行动空间，系统地对高斯和扩散基线进行比较分析，并概述了关键的未解决问题。最终，本综述成为下一代生成强化学习与对齐的基础路线图。

SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

SyncReward：视听生成的人类同步奖励

• Authors: Kaisi Guan, Ying Ba, Wenjing Wang, Xihua Wang, Chen Li, Jing Lyu, Ruihau Song
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.16338
• Pdf link: https://arxiv.org/pdf/2610.16338
• Abstract Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct \textbf{SyncReward-Bench}, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-$N$ selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at this https URL.
• 中文摘要 近年来，联合视听生成取得了快速进展。然而，视听同步仍是一大挑战，生成的语音和声音常常与视觉事件在时间上不匹配。一个关键瓶颈是缺乏人工注释数据、可靠的评估基准以及针对生成视听输出的同步感知评估模型。为解决这些局限性问题，我们进行了涵盖数据、基准测试和奖励建模的综合研究。我们首先介绍SyncReward-Data-80K，这是一个包含8个视听生成器剪辑8万专业同步评分的数据集。一个分类引导的构建流程涵盖四个声音事件族：语音、起始、乐器和环境。我们还进一步构建了\textbf{SyncReward-Bench}，这是一个由1000个剪辑组成的基准测试，每个剪辑由多位专业人士独立评分。利用该基准测试，我们系统评估了专业同步指标和全模态模型，揭示了它们与人类判断存在显著不匹配。最后，我们开发了SyncReward，一种结合真实剪辑语义-时间对比预训练与人工监督生成剪辑的奖励模型。作为评估指标，SyncReward在所有评估方法中与人类评分的一致性最强。作为奖励，它通过推断时间、$N美元最佳选择和基于强化学习的后训练提升同步性。我们将在此https URL发布数据、基准测试、模型和代码。

A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes

一个用于带有验证配方的前沿大型语言模型的生产级高效代理强化学习框架

• Authors: LoonSage Team: Bowei Song, Caiwei Zhao, Chuwen Liu, Dechuang Gu, Dong Fang, Feiyu Xie, Huaiyu Li, Hui Zhang, Jian Tian, Jian Wang, Jie Zhao, Liguang Li, Maoyun Wang, Meiyan Yang, Minghan Zhu, Pai Liu, Pandeng Yao, Renbao Tian, Ru Yang, Shuai Jia, Shuai Tan, Shuailong Li, Shuliang Li, Teng Jin, Tianze Yang, Wenbo Cui, Wenjing Bai, Wenkang Wu, Xiaowei Jiang, Xiaoyi Dong, Yanpeng Wang, Yanwen Wang, Yaohui Ma, Yawen Wang, Yihang Zhang, Yingjie Yan, Zhanbiao Lian, Zhenyu Lu, Zhida Hu, Zhihui Hao, Zhihui Hu, Zhiwen Xiao, Zhiyi Gao, Zhongwei Qi, Zhuangzhuang Wu, Ziqian Liu, Ziyue Zheng
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.16352
• Pdf link: https://arxiv.org/pdf/2610.16352
• Abstract Reinforcement learning (RL) has emerged as a critical phase in the post-training pipeline of large language models (LLMs). As the frontier of LLM capabilities shifts from static reasoning to agentic tasks--characterized by external tool invocation, multi-turn interactions, and long-horizon decision-making--existing mainstream RL frameworks have grown increasingly unwieldy, raising the overhead of adoption, usage, and debugging. To address these challenges, we present LoongSage, an agent-native, fully asynchronous RL framework with a lightweight core of roughly 14K lines of code. Built upon Megatron for distributed training and SGLang for high-throughput inference, LoongSage centralizes orchestration within a single controller process and encapsulates all GPU-resident components as Ray actors. It introduces AgentFlow, a unified execution paradigm that orchestrates both white-box and black-box agents. For asynchronous execution, it integrates sliding-window strategies that bound trajectory staleness, and employs TransferMesh, a topology-aware synchronization mechanism that routes weights after each step over the fastest transport available to each receiver. Beyond agentic RL, LoongSage incorporates a memory-efficient Teacher Manager that decouples the scaling of teacher models from GPU memory constraints, natively supporting Multi-Teacher On-Policy Distillation. Extensive evaluations across four complete training runs, spanning complex reinforcement learning and multi-teacher distillation, demonstrate that task rewards improve steadily while the train-inference gap is kept under continuous control, and the fully asynchronous pipeline removes synchronous idle bubbles, sustaining hardware utilization.
• 中文摘要 强化学习（RL）已成为大型语言模型（LLM）训练后流程中的关键阶段。随着LLM能力从静态推理转向代理任务——以外部工具调用、多回合交互和长远决策为特征——现有主流强化学习框架变得越来越笨重，增加了采用、使用和调试的开销。为应对这些挑战，我们介绍LoongSage，一个代理原生、完全异步的强化学习框架，核心轻量约14K行代码。LoongSage基于Megatron进行分布式训练和SGLang进行高吞吐量推理，将编排集中在单一控制器进程中，并将所有GPU驻留组件封装为Ray演员。它引入了AgentFlow，一种统一执行范式，能够协调白箱和黑盒代理。对于异步执行，它集成了限制轨迹陈旧的滑动窗口策略，并采用TransferMesh——一种拓扑感知同步机制，每步后权重通过每个接收端可用的最快传输方式路由。除了代理强化学习，LoongSage还集成了内存高效的教师管理器，将教师模型的扩展性与GPU内存限制解耦，原生支持多教师策略蒸馏。跨越复杂强化学习和多教师蒸馏的四次完整训练运行的广泛评估表明，任务奖励在保持列车-推理差距持续控制的同时稳步提升，完全异步流水线消除同步空闲气泡，维持硬件利用。

XAI-Guided Fourier Domain Generalisation for Pure-Simulation Sim-to-Real Deep Reinforcement Learning

XAI引导傅里叶域推广用于纯模拟模拟到实际深度强化学习

• Authors: Wei Yao, Chuyao Wang, Ziwei Wang, Nabil Aouf
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.16478
• Pdf link: https://arxiv.org/pdf/2610.16478
• Abstract Deep reinforcement learning has shown strong potential for vision-based autonomous driving, but policies trained in simulation often suffer severe performance degradation when deployed in the real world. Existing sim-to-real methods commonly address this gap through image-transfer-based domain adaptation or policy-agnostic domain randomisation. However, the former requires laborious acquisition of target-domain data and is often limited to the deployment domain, whereas the latter introduces unguided perturbations that may not reflect the underlying sim-to-real discrepancy, leading to suboptimal transfer performance. To overcome these limitations, we propose XFD, an explainable artificial intelligence (XAI)-guided Fourier domain generalisation framework for pure-simulation sim-to-real reinforcement learning. The proposed framework extracts explainability attribution maps from simulated observations, and converts these attributions into policy-conditioned Fourier-domain interventions. By perturbing appearance-sensitive spectral components while preserving task-defining spatial structures, XFD constructs robustness-enhancing visual domains entirely within simulation. Subsequently, the proposed robust policy optimisation guides the policy towards attending to task-critical features while suppressing sensitivity to irrelevant visual information. Extensive experiments in simulation and zero-shot real-world deployment demonstrate that XFD consistently improves traveled distance and reward over standard policy and randomly domain-randomised baseline, especially on unseen maps with cluttered backgrounds. Attribution-based analysis further proves that the proposed policy preserves more consistent task-relevant visual evidence across simulation and real-world observations.
• 中文摘要 深度强化学习在基于视觉的自动驾驶方面展现出强大潜力，但模拟训练的策略在实际应用时常常会严重性能下降。现有的模拟到现实方法通常通过基于图像传输的域适配或策略无关域随机化来弥补这一空白。然而，前者需要大量获取目标域数据，且通常局限于部署域，而后者则引入了无引导的扰动，可能无法反映潜在的模拟与现实差异，导致传输性能不理想。为克服这些限制，我们提出了XFD，一种由可解释人工智能（XAI）引导的傅里叶域泛化框架，用于纯模拟模拟到现实的强化学习。该框架从模拟观测中提取可解释性归因图，并将这些归因转化为策略条件的傅里叶域干预。通过扰动外观敏感的光谱分量，同时保留任务定义的空间结构，XFD完全在模拟中构建了增强鲁棒性的视觉域。随后，所提出的稳健策略优化引导策略关注任务关键特征，同时抑制对无关视觉信息的敏感性。大量模拟实验和零样本真实世界部署表明，XFD在标准策略和随机领域随机基线上持续提升旅行距离和奖励，尤其是在背景杂乱的未见地图上。基于归因的分析进一步证明，拟议策略在模拟与现实观测之间保持更一致的任务相关视觉证据。

Growing and Transferring Pathways for LLM Reinforcement Learning

LLM强化学习路径的成长与转移

• Authors: Haolin Tian, Yikang Li, Tonghan Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16489
• Pdf link: https://arxiv.org/pdf/2610.16489
• Abstract Reinforcement learning (RL) can substantially improve the reasoning capabilities of large language models, but an effective training run requires considerable resources. Reusing the computation acquired through RL across models would extend the value of that training. We propose GrowRL, an RL method that organizes learned updates into compact computational pathways through structural growth. GrowRL freezes the pretrained backbone and adds an independent causal attention branch and a feed-forward network branch at each Transformer layer. Zero-initialized output projections preserve the pretrained policy at initialization, and RL updates only the added pathways. The learned pathways remain explicit after training, allowing transfer to structurally compatible target models and subsequent adaptation with the target backbone frozen. The architecture also supports sequential transfer of additional pathways. By keeping the learned attention and feed-forward computation separate from the pretrained weights, GrowRL provides an explicit structure for preserving and reusing the outcome of an RL run. Experiments across five backbones show competitive mathematical reasoning and out-of-distribution performance with a small number of trainable parameters. Learned pathways provide immediate gains across all three transfer directions, with performance after only 10 target-side updates already close to that after 100 updates in every case. These results support computational pathways as parameter-efficient and reusable units of reinforcement learning adaptation.
• 中文摘要 强化学习（RL）可以显著提升大型语言模型的推理能力，但有效的训练运行需要大量资源。通过强化学习获得的计算在不同模型间重复使用，将提升该训练的价值。我们提出了GrowRL，一种通过结构增长将学习到的更新组织为紧凑的计算路径的强化学习方法。GrowRL冻结预训练骨干，并在每个Transformer层添加独立的因果注意力分支和前馈网络分支。零初始化输出在初始化时保持预训练策略，强化学习仅更新新增的路径。学习到的路径在训练后保持显式，允许迁移到结构兼容的目标模型，并在目标骨干冻结时进行后续适应。该架构还支持额外路径的顺序转移。通过将学习的注意力和前馈计算与预训练权重分开，GrowRL 为保存和重用强化学习运行的结果提供了明确结构。跨五个骨干链的实验显示，在少量可训练参数下，具有竞争性数学推理能力和分布外性能。学习路径在三个转移方向上均有即时提升，目标端仅10次更新后的性能在每种情况下已接近100次更新后的水平。这些结果支持计算通路作为参数高效且可重用的强化学习适应单元。

Trust-Calibrated VLA Policy Post-Training with Evolving Imagination

信任校准的VLA策略培训后，想象力不断演变

• Authors: Wenzhan Li, Yiran Qin, Heng Zhou, Huirui Wang, Yulan Guo, Ruimao Zhang
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16517
• Pdf link: https://arxiv.org/pdf/2610.16517
• Abstract Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.
• 中文摘要 视觉-语言-行动（VLA）模型通常通过模仿学习进行预训练，因此当部署超出演示覆盖范围时，强化学习（RL）在训练后非常重要。现有的精炼范式暴露出可靠性-效率张力：无模型细化避免了模拟器偏差，但交互密集;而基于模型的细化即使模拟器适应，也能利用不可靠的想象力。我们提出了模型进化强化学习（MERL）框架，这是一种信任校准框架，将策略优化与由世界模型和奖励代理头组成的在线更新模拟器结合起来。跨阶段，模拟器可靠性控制想象数据比例和推广视野;在部署中，区块级信任控制重放选择和优化权重。对于没有精确基准延续的递归想象，残差预测变量在不查询未来基层真理的情况下估计信任。在受扰动的LIBERO-PRO任务中，MERL平均成功率为79.7%，高阈值学习速度比匹配无模型对照提升10.9%，并在三个训练种子中优于同协议外部基线。每项任务50次演示的固定数据适应，在两个物理机器人任务中平均成功率达80.0%。

Reinforcement Learning with Hawkes-Structured Networks for Market Making

利用霍克斯结构化网络进行市场制造的强化学习

• Authors: Fernando Acero, Philipp Plank, Yufei Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computational Finance (q-fin.CP)
• Arxiv link: https://arxiv.org/abs/2610.16524
• Pdf link: https://arxiv.org/pdf/2610.16524
• Abstract Market making under realistic order-flow dynamics is challenging due to the clustered, self-exciting nature of order arrivals, which are naturally modeled by non-Markovian Hawkes processes. We propose a model-free reinforcement learning (RL) framework for market making with general Hawkes order-flow dynamics. We first establish non-asymptotic performance bounds that quantify the loss incurred by approximating the infinite-dimensional Hawkes state using a finite collection of exponentially weighted features. Guided by this analysis, we introduce the Hawkes-Structured Network (HSN), a novel neural architecture that encodes historical order flow through a learnable mixture-of-exponentials filter. HSN is trained end-to-end using model-free RL, jointly optimizing the feature representation and quoting policy directly from reward feedback. Experiments on single- and multi-asset market-making problems with mixture-of-exponentials, power-law, and regime-switching kernels demonstrate that HSN outperforms general-purpose neural architectures, achieving higher trading rewards while implicitly recovering the underlying order-flow intensities.
• 中文摘要 由于订单到达的聚类、自激性特性，且自然由非马尔可夫霍克斯过程建模，在真实的订单流动态下做市较具挑战性。我们提出了一个无模型强化学习（RL）框架，用于基于一般霍克斯订单流动态的市场造市。我们首先建立非渐近性能界限，通过有限指数加权特征集合近似无限维霍克斯状态，量化所发生的损失。在该分析指导下，我们引入了霍克斯结构网络（HSN），这是一种通过可学习指数混合滤波器编码历史顺序流的新型神经架构。HSN通过无模型的强化学习从端到端训练，联合优化特征表示并直接从奖励反馈中引用策略。基于指数函数、幂律和体制切换核心的单资产和多资产做市问题实验表明，HSN优于通用神经架构，在隐式恢复基础订单流强度的同时，实现更高的交易回报。

Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards

用于可验证奖励的大型语言模型强化学习的频谱驱动自适应秩优化器

• Authors: Zhipeng Chen, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16533
• Pdf link: https://arxiv.org/pdf/2610.16533
• Abstract Reinforcement learning with verifiable rewards (RLVR) is reshaping how large language models (LLMs) acquire reasoning capabilities, yet the optimizers used remain largely inherited from teacher-forcing training. This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a different optimizer? We investigate this question through the spectral structure of LLM gradients and find that gradient energy progressively concentrates into a shrinking effective subspace, which is characteristic of RLVR, as it is absent under supervised fine-tuning (SFT) and is largely removed when the verifiable reward is replaced by a random one. Moreover, the rank required to capture a fixed fraction of gradient energy steadily decreases over training, revealing that the effective dimensionality of useful updates itself evolves with training. These findings motivate a new optimization principle: rather than prescribing the update rank externally, an optimizer should adapt its rank to the evolving spectral structure of the gradient. We introduce Ares (Adaptive Rank-Effective Spectral Optimization), which estimates the effective update rank from the gradient's spectral energy distribution, tracks its evolution, and confines the gradient signal driving each update to the resulting low-rank subspace. Across LLMs of different scales, Ares improves accuracy over widely used optimizers and generalizes across RLVR algorithms and tasks. Results suggest that effective optimization may require optimizers to be designed around the geometry and dynamics of the training process. We release the code and data at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）正在重塑大型语言模型（LLM）获得推理能力的方式，但所用优化器在很大程度上仍是教师强制训练的继承。这引发了一个根本性问题：RLVR是否会诱导出一种独特的优化几何，从而需要不同的优化器？我们通过LLM梯度的谱结构研究了这一问题，发现梯度能量逐渐集中到一个缩小的有效子空间中，这也是RLVR的特征，因为在监督微调（SFT）下梯度能量不存在，且当可验证奖励被随机奖励取代时，这一过程在很大程度上被消除。此外，捕获固定比例梯度能量所需的秩在训练过程中持续下降，表明有用更新的有效维度本身也会随着训练而演变。这些发现促使了一种新的优化原则：优化器不应在外部规定更新秩，而是应根据梯度不断演变的谱结构调整其秩。我们介绍了Ares（自适应秩有效谱优化），它从梯度的谱能量分布估算有效更新秩，跟踪其演变，并将每次更新驱动的梯度信号限制在所得的低秩子空间中。在不同尺度的大型语言模型中，Ares 提高了广泛使用的优化器的准确性，并在 RLVR 算法和任务中进行了推广。结果表明，有效优化可能需要围绕训练过程的几何和动态设计优化器。我们将代码和数据发布至此 https URL。

Code-Interactive Visual Reasoning for Financial Candlestick Understanding

代码交互式视觉推理用于财务烛台理解

• Authors: Chenxi Zhang, Youwei Pang, Xiaoqi Zhao, Ziliang Gan, Rongjunchen Zhang, Shanshan Feng, Hao Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16587
• Pdf link: https://arxiv.org/pdf/2610.16587
• Abstract Financial candlestick understanding requires locating visual elements, aligning them across trading dates and panels, and combining observations through numerical and financial reasoning. Answer accuracy alone does not reveal whether predictions are supported by chart evidence. We introduce Code-Interactive Financial Candlestick Reasoning and present Fin-Thinker, a model based on Qwen3.5-9B that generates and executes code to acquire evidence and perform computations. The model uses the returned observations to guide subsequent reasoning. To train these capabilities, we construct FinKLine-32K with 32,184 replay-verified code-observation trajectories collected through teacher-guided student exploration. These include successful solutions with and without error recovery. Our three-stage training framework combines foundational visual-code fine-tuning, financial process supervision of successful reasoning and corrective continuations, and execution-feedback reinforcement learning. For evaluation, we introduce FinKLineBench with 1,200 questions across 138 financial task types to assess answer correctness, code execution, and evidence grounding separately. On FinKLineBench, Fin-Thinker improves strict accuracy by 13.1 and 33.2 percentage points over the best-performing direct-answer baseline among those evaluated and its base model with sandbox access, respectively. Training-stage ablations further show that financial process supervision and execution-feedback reinforcement learning yield successive improvements in both answer accuracy and GPT-5.6-judged grounded accuracy.
• 中文摘要 理解金融蜡烛图需要定位视觉元素，在交易日期和面板间对齐，并通过数值和财务推理结合观察结果。仅凭答案准确性并不能判断预测是否被图表证据支持。我们介绍了代码交互式金融烛台推理，并介绍基于Qwen3.5-9B的Fin-Thinker模型，生成并执行代码以获取证据和执行计算。该模型利用返回的观测数据指导后续推理。为训练这些能力，我们构建了FinKLine-32K，包含32,184条通过教师引导学生探索收集的回放验证代码观察轨迹。这些轨迹包括带错误恢复和无错误恢复的成功解决方案。我们的三阶段培训框架结合了基础的可视化代码微调、成功推理和纠正性延续的财务过程监督，以及执行反馈强化学习。在评估方面，我们引入了FinKLineBench，包含1200个问题，涵盖138种财务任务类型，分别评估答案正确性、代码执行和证据基础。在FinKLineBench上，Fin-Thinker分别将严格准确度提升了13.1个百分点，比被评估者中表现最佳的直接答案基线和其带沙盒访问的基础模型提高了33.2个百分点。训练阶段的消融进一步表明，财务流程监督和执行反馈强化学习在答案准确性和GPT-5.6判定的基础准确性上均有连续提升。

Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

Prorsa：释放预填充-解码分解在智能强化学习中的潜力

• Authors: Chiheng Lou, Xiangwei Deng, Bangjun Xiao, Xiaodong Ji, Yikai Zhao, Xin Jin, Hailin Zhang
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16592
• Pdf link: https://arxiv.org/pdf/2610.16592
• Abstract Agentic reinforcement learning (RL) is commonly used to enhance the capabilities of large language models (LLMs) in solving complex multi-turn tasks. During rollout, agentic RL exhibits a workload pattern distinct from that of traditional RL: multi-turn interactions repeatedly introduce prefill requests, causing prefill and decode workloads to coexist throughout rollout. This workload makes prefill-decode (PD) disaggregation, a design widely used for online LLM serving, promising for speeding up agentic RL rollout. This paper shows that PD disaggregation can improve rollout efficiency by (a) mitigating interference among data-parallel attention modules, (b) enabling stage-specific parallelism, and (c) reducing end-to-end trajectory completion time. However, a fixed prefill-decode allocation may fail to sustain these benefits as workload composition varies across tasks and over time. We therefore present Prorsa, an agentic RL framework for prediction-guided prefill-decode resource allocation. Prorsa observes the lifecycle of every trajectory to forecast short-horizon prefill and decode demand. These forecasts then guide worker allocation decisions and trigger role switching of rollout engines, aligning prefill and decode capacity with anticipated workload demand. Across synchronous and asynchronous agentic RL settings, Prorsa achieves end-to-end rollout speedups of 1.11x-4.05x over prefill-decode colocation.
• 中文摘要 代理强化学习（RL）常用于增强大型语言模型（LLMs）解决复杂多回合任务的能力。在展开过程中，代理强化学习表现出与传统强化学习不同的工作负载模式：多回合交互会反复引入预填充请求，使得预填充和解码工作负载在整个部署过程中共存。这种工作负载使得预填充-解码（PD）分解成为在线大型语言模型服务中广泛采用的设计，有望加快代理强化学习的推广速度。本文表明，PD拆分可以通过（a）减少数据并行注意力模块之间的干扰，（b）实现阶段特定的并行性，以及（c）缩短端到端轨迹完成时间，从而提高推广效率。然而，固定的预填充-解码分配可能无法持续这些优势，因为工作负载组合会随任务和时间变化。因此，我们提出了Prorsa，一个用于预测引导的预填充-解码资源分配的代理式强化学习框架。Prorsa观察每个轨迹的生命周期，以预测短期预填充和解码需求。这些预测随后指导工人分配决策，触发推送引擎的角色切换，使预填充和解码容量与预期工作负载需求保持一致。在同步和异步代理强化学习设置下，Prorsa实现了端到端推广加速，分别在预填充-解码共置上实现1.11x-4.05倍。

Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation

基于抽样的奖励估计的双重展开强化学习

• Authors: Haifeng Qian, David Mosallanezhad, Christian Munley, Adithya Renduchintala, Oleksii Kuchaiev
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16593
• Pdf link: https://arxiv.org/pdf/2610.16593
• Abstract Reinforcement learning with verifiable rewards (RLVR) implemented with Group Relative Policy Optimization (GRPO) has driven substantial progress in training reasoning language models. However, verifiable outcome rewards provide an imperfect measure of reasoning quality: a correct final answer may not be supported by the reasoning trace, whereas an incorrect rollout may contain a nearly correct reasoning trajectory. We present empirical evidence for this mismatch and hypothesize that the resulting reward noise contributes to instability in reinforcement learning for both reasoning accuracy and token efficiency. To address this credit assignment challenge, this paper proposes double-rollout reinforcement learning. For each sampled rollout, we truncate its reasoning trace, sample multiple continuations from the truncated prefix, and replace the original reward with the mean continuation reward, yielding a more robust estimate of reasoning quality. This double-rollout algorithm incurs only a moderate increase in compute cost and yet reaches substantially higher accuracies across math and coding tasks and across Qwen and Nemotron models than baseline algorithms with fewer RL steps. For example, using only DAPO data it trains Nemotron-3-Nano-30B-A3B to reach 95.4% on AIME 2026, versus 91.5% by GRPO and substantially above state of the art for this model size, within 30 RL steps. When combined with length-based rewards, the same algorithm further improves the accuracy-efficiency Pareto frontier over existing efficiency optimizations. These findings suggest that the double-rollout reward estimation is an effective and cost-efficient way to improve both the accuracy and token efficiency of reasoning RL.
• 中文摘要 基于群体相对策略优化（GRPO）实现的可验证奖励强化学习（RLVR）推动了推理语言模型训练的显著进展。然而，可验证结果奖励并不能完美衡量推理质量：正确的最终答案可能无法得到推理轨迹的支持，而错误的推理轨迹可能包含近乎正确的推理轨迹。我们提出了这种不匹配的实证证据，并假设由此产生的奖励噪声导致强化学习在推理准确性和代币效率上都不稳定。为解决这一学分分配挑战，本文提出了双重推展强化学习。对于每个抽样的推测，我们截断其推理轨迹，从截断前缀中采样多个延续，并用均值延续奖励替换原始奖励，从而获得更稳健的推理质量估计。该双重展开算法计算成本仅略有增加，但在数学和编码任务以及Qwen和Nemotron模型中，准确率远高于基础算法（仅有更少的强化学习步）。例如，仅使用DAPO数据，它在30步内将Nemotron-3-Nano-30B-A3B训练到AIME 2026的95.4%，而GRPO为91.5%，远高于该模型规模的先进水平。结合基于长度的奖励，该算法进一步提升了对现有效率优化的准确性-效率帕累托前沿。这些发现表明，双重展开奖励估计是提升强化学习推理准确性和代币效率的有效且成本效益高的方法。

DR-APO: Decoupled-Representation Advantage-Guided Policy Optimization from Continuous Control to Embodied Navigation

DR-APO：从连续控制到具身导航的解耦代表优势引导策略优化

• Authors: Runhua Zhang, Yingjie Niu, Chao Li
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.16618
• Pdf link: https://arxiv.org/pdf/2610.16618
• Abstract Policy representations in reinforcement learning have expanded beyond Gaussian policies to diffusion and flow-matching based generative policies. However, existing RL methods remain representation-specific, lacking a unified optimization interface for emerging generative policies. In particular, extending likelihood-based objectives to multi-step generative policies couples optimization to their internal generation process, where accumulated likelihood ratios can weaken the update signal. Meanwhile, generative-policy-specific methods often rely on multi-candidate generation and critic-based selection, improving overall return while providing limited improvement to the policy itself. We introduce DR-APO (Decoupled-Representation Advantage-Guided Policy Optimization), which decouples policy improvement from policy representation through advantage-guided target improvement and representation-compatible direct action fitting, providing a shared optimization principle for Gaussian, diffusion, and flow policies. Across five MuJoCo-v5 tasks, DR-APO remains competitive with SAC for Gaussian policies and achieves the best aggregate policy-only performance for both diffusion and flow-matching policies. NavDP post-training improves the Success Rate (SR) from 70.33% to 89.55% in simulation and from 25% to 65% in real-world navigation. These results demonstrate a unified policy-improvement principle across Gaussian, diffusion, and flow policies, from continuous-control learning to embodied-policy post-training.
• 中文摘要 强化学习中的策略表示已超越高斯策略，扩展到基于扩散和流匹配的生成策略。然而，现有的强化学习方法仍是表示特定，缺乏针对新兴生成策略的统一优化接口。特别是将基于似然的目标扩展到多步生成策略，将优化与其内部生成过程耦合，累积的似然比可能削弱更新信号。与此同时，生成策略专用方法通常依赖多候选生成和基于批评者的选择，提升整体回报，同时对策略本身的改进有限。我们引入DR-APO（解耦-表示优势引导策略优化），通过优势引导目标改进和表示兼容的直接行动拟合将策略改进与策略表示解耦，为高斯策略、扩散和流策略提供共享优化原则。在五个MuJoCo-v5任务中，DR-APO在高斯策略方面仍能与SAC竞争，并在扩散和流量匹配策略上实现最佳的单策略综合性能。NavDP在模拟中将成功率（SR）从70.33%提升到89.55%，在真实世界导航中从25%提升到65%。这些结果展示了从连续控制学习到具象策略后训练，涵盖高斯、扩散和流策略的统一策略改进原则。

StageBreaker: Shifting Video RL Post-Training Scheduling from Stages down to Data Dependencies

StageBreaker：将视频强化学习后从阶段调度到数据依赖

• Authors: Li Peng, Chenghuan Huang, Ziyang Ma, Jing Lyu, Xiang Chen, Jie Zhang
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2610.16691
• Pdf link: https://arxiv.org/pdf/2610.16691
• Abstract Reinforcement learning (RL) post-training has emerged as the key paradigm for improving autoregressive video models. Emerging vision-language reward models further strengthen post-training with accurate rewards, but their long chain-of-thought inference substantially raises the associated costs, while the pursuit of long videos prolongs policy training. Despite these rising costs, existing video RL systems still adopt coarse-grained stage-centric execution, which stalls policy training until all reward evaluation finishes, replays every trajectory twice due to a further barrier within policy training, and induces load imbalance across GPUs. To address the challenges, we propose StageBreaker, a fine-grained data-dependency-driven scheduling framework, which schedules the true dataflow of an RL step rather than its stages. Specifically, StageBreaker overlaps policy training with reward evaluation by exploiting reward-separable policy training, and merges the two trajectory replays into one by leveraging loss-independent back-propagation. It further balances GPU load by scheduling computation at the finer granularity. Our evaluation results show that StageBreaker achieves a speedup of up to 1.41$\times$ over stage-centric execution.
• 中文摘要 强化学习（RL）后训练已成为改进自回归视频模型的关键范式。新兴的视觉语言奖励模型通过精准的奖励进一步强化了训练后，但其漫长的思维链推理显著提高了相关成本，而追求长视频则延长了策略培训的时间。尽管成本不断上升，现有视频强化学习系统仍采用粗粒度的阶段中心执行，这使策略训练停滞直到所有奖励评估结束，由于策略训练中的另一个障碍，每个轨迹都重复了两次，并导致GPU间的负载不平衡。为应对这些挑战，我们提出了StageBreaker，一种细粒度的数据依赖驱动调度框架，它调度的是强化学习步骤的真实数据流，而非其阶段。具体来说，StageBreaker 利用可分奖励策略训练，将策略训练与奖励评估重叠，并通过利用损失无关的反向传播将两种轨迹重放合并为一体。它还通过在更细粒度上调度计算来平衡 GPU 负载。我们的评估结果显示，StageBreaker 在阶段中心执行时可实现高达 1.41$\times$ 的速度。

VAttnPO: Visual Attention-Guided Policy Optimization

VAttnPO：视觉引导的策略优化

• Authors: Lihao Zheng, Zhenwei Shao, Zhou Yu, Yan Yang, Haoyi Sun, Qian Wang, Ning Mao, Pan Zhou
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16735
• Pdf link: https://arxiv.org/pdf/2610.16735
• Abstract Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response. This outcome-level signal distinguishes overall response quality but leaves changes in visual-evidence use across generated tokens implicit. We introduce VAttnPO (Visual Attention-Guided Policy Optimization), which uses answer-conditioned visual attention to refine token-level credit while preserving the verifier-derived trajectory-mean advantage. A frozen old policy constructs a visual reference by teacher forcing the known correct answer and produces a visual distribution for each rollout token. VAttnPO measures their agreement in visual-attention space using Jensen--Shannon divergence, rather than reading guidance from teacher output-token distributions or introducing an attention-matching objective. Trajectory centering, relative-position weighting, and credit balancing turn this agreement into a zero-mean, position-aware correction to the GRPO sequence advantage. Token intervention, visual-index permutation, and trajectory analysis provide diagnostic evidence linking this signal to model behavior, visual correspondence, and generation position. On Qwen3-VL-8B-Instruct, VAttnPO raises six-task image AVE from 70.97 with GRPO to 73.50 and scores higher on every reported structured, video, and MME metric. Gains over GRPO also hold on Qwen2.5-VL-7B-Instruct and InternVL3-8B-Instruct.
• 中文摘要 带有可验证奖励的多模态强化学习通常为整个回答分配一个序列优势。这一结果级信号区分了整体响应质量，但隐性地隐含了生成代币间视觉证据使用的变化。我们引入了VAttnPO（视觉注意力引导策略优化），利用答案条件的视觉注意力细化代币级信用，同时保持验证者衍生的轨迹均值优势。一个冻结的旧策略通过教师强制正确答案构建视觉参考，并为每个推广代币生成视觉分布。VAttnPO通过Jensen-Shannon发散来衡量视觉-注意力空间的一致性，而非读取教师输出-代币分布的指导或引入注意力匹配目标。轨迹中心化、相对位置加权和信用平衡将该协议转化为对GRPO序列优势的零均值、位置感知修正。代币干预、视觉指数置换和轨迹分析为该信号与模型行为、视觉对应和生成位置联系起来提供了诊断证据。在Qwen3-VL-8B-Ininstruction中，VAttnPO将六任务图像AVE与GRPO从70.97提升至73.50，并在所有报告的结构化、视频和MME指标上得分更高。相比GRPO的提升也适用于Qwen2.5-VL-7B-Instruct和InternVL3-8B-Instruct。

Digital-Twin-Referenced Belief-Guided Q-Learning for Jammer Identification and Beam-Power Control in a mmWave Model

数字孪生引用信念引导Q学习用于毫米波模型中的干扰器识别和束功率控制

• Authors: Parmida Geranmayeh, Onur Günlü
• Subjects: Subjects: Information Theory (cs.IT); Cryptography and Security (cs.CR); Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2610.16770
• Pdf link: https://arxiv.org/pdf/2610.16770
• Abstract Digital Twin (DT) combined with reinforcement learning (RL) can support adaptive resource allocation and security-aware decision-making by comparing expected and observed network behavior. This paper presents a simulation-based DT reference and a belief-guided Q-learning (BGQL) for joint jammer identification and beam-power control in a site-specific 28-GHz network. For each action, the same channel simulator is evaluated under an attack-free and an attacked configuration. Receiver-level residuals are combined with spatial jamming footprints and accumulated to form a normalized suspicion vector. BGQL incorporates this vector into an aggregated state, a belief-concentration reward, and exploratory beam-selection rule. The BGQL scheme is compared with SARSA, Greedy, and Random under matched attacker sets. Across 30 paired Monte Carlo runs in four configurations, with the number of active jammers supplied to every method, BGQL achieves mean aggregate full-band link-rate scores of approximately 437.4--564.2~Mbit/s and rate-to-device-power scores of 60.37--76.79~Mbit/J. Its detection accuracy is 100\% in one-jammer configurations and approximately 78\%--81\% in two-jammer configurations. BGQL requires approximately 80--130~s per run, compared with 5--14~s for SARSA, reflecting evaluation of 625 beam pairs during exploratory steps. These findings provide a matched-model simulation proof of concept for DT-referenced belief-guided learning for security-aware beam-power control in 6G networks.
• 中文摘要 数字孪生（DT）结合强化学习（RL）可通过比较预期与观察到的网络行为，支持自适应资源分配和安全意识决策。本文提出了基于仿真的DT参考和信念引导Q学习（BGQL），用于特定28 GHz网络中的联合干扰器识别和波束功率控制。对每个动作，同一信道模拟器在无攻击和被攻击配置下进行评估。接收端级残差与空间干扰足迹结合，累积形成归一化的怀疑向量。BGQL将该向量纳入聚合状态、信念集中奖励和探索性波束选择规则中。BGQL方案与匹配攻击者集合下的SARSA、贪婪和随机进行比较。在4种配置下，30对蒙特卡洛运行，每种方法均有主动干扰器数量，BGQL实现了约437.4-564.2~Mbit/s的平均全带链路速率得分，设备功耗比为60.37-76.79~Mbit/J。单干扰器配置下的检测准确率为100%，双干扰器配置约为78%--81%。BGQL每次运行约需80--130~秒，而SARSA为5--14~秒，反映了探索步骤中对625对波束的评估。这些发现为6G网络中安全感知波束功率控制的DT参考信念引导学习提供了匹配模型仿真的概念验证。

TPPO: Tracking-Value PPO For Language Model Reasoning

TPPO：语言模型推理中的跟踪值PPO

• Authors: Futing Wang, Chenxi Li, Jianhao Yan, Yun Luo, Zhi Wang, Yizhuo Li, Runzhe Zhan, Yu Cheng, Tao Lin
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.16794
• Pdf link: https://arxiv.org/pdf/2610.16794
• Abstract Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While \emph{value evolution along trajectory} characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO (\method), a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. \method improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates. Our code is publicly available in this https URL.
• 中文摘要 近端策略优化（PPO）广泛应用于强化学习，并可验证的语言模型推理奖励。虽然沿轨迹的价值演化描述了相对的代币级信用，但标准批评者常常无法在稀疏的终端奖励下跟踪它。为建模该演变，我们分析了连续前缀值之间的关系，并证明它们在对数赔率空间中存在带有加法更新的贝叶斯分解。我们利用该加法结构构建跟踪价值PPO（\方法），这是一种累积价值参数化，将位置级标量输出相加为价值logit。这些输出通过对重构值的终端结果监督学习。\方法在Qwen3-4B-Base和Qwen3-8B-Base上分别提升了PPO的数学推理总分7.0分和8.4分，同时提升了分布外的泛化能力。提升还延伸至Qwen3-4B、Qwen2.5-Math-1.5B、Qwen3.5-9B以及倒计时和科学编码任务。进一步分析显示，价值演变跟踪更为准确，优势估计也更可靠。我们的代码在此HTTPS网址中公开发布。

X-Ragent: Tool-Augmented Reinforcement Learning for Evidence-Grounded Chest X-Ray Reasoning

X-Ragent：工具增强强化学习用于基于证据的胸部X光推理

• Authors: Yi Wang, Zhenlong Yuan, Yuchan Guo, Maolei Huang, Zijian Song, Dapeng Zhang, Rongbin Tan, Jiyuan Wang, Zhiyuan Wang, Fangfang Lin, Shuhan Song, Huawei Cao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.16811
• Pdf link: https://arxiv.org/pdf/2610.16811
• Abstract Chest X-ray (CXR) interpretation often requires integrating visual findings with task-specific evidence rather than relying on direct visual recognition alone. While tool-augmented medical agents can supply such evidence via external modules, they often lack explicit mechanisms for learning when auxiliary evidence is needed and for preventing inapplicable quantitative evidence from entering diagnostic reasoning. To address this, we propose X-Ragent, a post-training framework for tool-augmented reinforcement learning that decouples learned auxiliary-evidence seeking from validity-gated quantitative measurement acquisition for evidence-grounded CXR reasoning. We construct Rad-CoT, a 7.7K-sample corpus combining causally ordered tool-augmented trajectories with report-style diagnostic reasoning, and use it to initialize Qwen3-VL-8B via agentic supervised fine-tuning (SFT). We then optimize the policy with Group Relative Policy Optimization (GRPO) using an accuracy-gated reward that applies process-level terms for measurement consistency and auxiliary-tool cost only to correct trajectories, preventing plausible process behavior from compensating for an incorrect diagnosis. Across three CXR benchmarks, X-Ragent achieves the strongest results, including an 8.6-point improvement on the fully held-out ChestAgentBench. By seeking evidence selectively and admitting only valid measurements, X-Ragent supports more faithful and evidence-grounded CXR reasoning.
• 中文摘要 胸部X光（CXR）解读通常需要将视觉发现与任务特定证据整合，而非仅依赖直接视觉识别。虽然工具增强医疗代理可以通过外部模块提供此类证据，但通常缺乏明确机制，用于学习何时需要辅助证据，以及防止不适用的定量证据进入诊断推理。为此，我们提出了X-Ragent，一种工具增强强化学习的后培训框架，将学习到的辅助证据寻求与效度门控定量测量获取分离，用于基于证据的CXR推理。我们构建了Rad-CoT，一个7.7K样本语料库，结合了因果有序工具增强轨迹与报告式诊断推理，并利用它通过代理监督微调（SFT）初始化Qwen3-VL-8B。随后，我们采用Group Relative Policy Optimization（GRPO）优化策略，采用准确度门槛奖励，仅对测量一致性和辅助工具成本应用过程级项来修正轨迹，防止合理过程行为补偿错误诊断。在三项CXR基准测试中，X-Ragent取得了最强的结果，包括比完全保留的ChestAgentBench提升8.6分。通过选择性地寻求证据并只接受有效测量，X-Ragent支持更忠实且基于证据的CXR推理。

Efficient Resource-Shiftable RL Post-training with Generative Rewards

高效、资源可转移的强化学习后培训，伴随生成奖励

• Authors: Yanfeng Zhao, Haoyang Li, Sheng Lin, Yuming Zhou, Jiawei Jiang, Bin Cui, Fangcheng Fu
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2610.16866
• Pdf link: https://arxiv.org/pdf/2610.16866
• Abstract Reinforcement learning (RL) post-training with generative reward models (GenRM) has become a central recipe for improving large language models (LLMs), and it is also increasingly resource-intensive: policy rollout, GenRM-based reward evaluation, and policy training all compete for GPU resources. However, existing colocated and disaggregated systems rely on static resource allocation for these roles. Such allocation cannot adapt as each role's workload changes during RL and therefore leaves a role's resources underutilized. We present PivotRL, an RL post-training system that dynamically shifts resources among different roles according to their runtime demands. PivotRL introduces a resource-shiftable runtime built on a Global Parameter Storage, together with a unified mechanism that efficiently shifts role-specific context. A shift-aware scheduler enables sequence migration upon a shift, while a planner considers the aggregate throughput benefit and the shift overhead to determine when and how to shift. We evaluate PivotRL on a 128-GPU cluster using four model groups spanning dense and mixture-of-experts models. PivotRL enables resource shifting within seconds and improves end-to-end throughput by 1.81-2.05 times over colocated deployment and 1.50-1.95 times over disaggregated deployment, while preserving RL convergence.
• 中文摘要 生成奖励模型（GenRM）的强化学习（RL）后训练已成为改进大型语言模型（LLM）的核心配方，同时资源消耗也日益高涨：策略推广、基于GenRM的奖励评估和策略训练都在争夺GPU资源。然而，现有的共址和拆分系统依赖于这些角色的静态资源分配。这种分配无法随着每个角色在强化学习期间工作负荷的变化而调整，因此该角色的资源未被充分利用。我们介绍PivotRL，一种基于不同角色运行时需求动态转移资源的后期训练系统。PivotRL引入了基于全局参数存储的资源可转移运行时，并结合了高效切换角色特定上下文的统一机制。一个感知切换的调度器支持在切换时进行序列迁移，而规划者则考虑总吞吐量收益和切换开销，以决定何时以及如何切换。我们在一个128个GPU集群上，利用四个模型组评估PivotRL，模型涵盖密集模型和专家混合模型。PivotRL实现资源转移在几秒内完成，端到端吞吐量提升1.81-2.05倍，比拆分部署提升1.50-1.95倍，同时保持强化学习的收敛性。

Time Window Optimization for Motor Imagery Classification via Temporal Structure Embedded Reinforcement Learning

通过时间结构嵌入强化学习实现运动影像分类的时间窗口优化

• Authors: Huan Luo, Na Lu, Xiaopeng Wang, Ziyang Shang
• Subjects: Subjects: Neural and Evolutionary Computing (cs.NE); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.16921
• Pdf link: https://arxiv.org/pdf/2610.16921
• Abstract Online motor imagery-based brain-computer interfaces (MI-BCIs) typically decode continuous EEG using fixed-length sliding windows that may be misaligned with neural events. Existing window optimization methods often evaluate windows independently, overlooking temporal dependencies and potentially producing fragmented or physiologically implausible selections. We propose Temporal Structure Embedded Reinforcement Learning (TSE-RL), a framework that incorporates prior knowledge of MI event structure into a constrained Markov decision process for adaptive window optimization. First, TSE-RL assigns each window an explicit semantic role through Discard, Onset, Sustain, or Offset actions, extending binary retain/discard decisions to event-level alignment. Second, a finite-state machine masks invalid actions to enforce the temporal order of MI phases and promote physiologically plausible window sequences. Third, a Transformer-based history encoder captures dependencies across windows, improving training stability and computational efficiency compared with recurrent architectures. TSE-RL is plug-and-play and can be integrated with diverse MI classifiers. Experiments on multiple benchmark datasets demonstrate consistent improvements in decoding performance in both cross-session and cross-subject settings. Visualization analyses further suggest that the learned window selection policy aligns with neurophysiological principles.
• 中文摘要 在线运动图像脑机接口（MI-BCI）通常通过固定长度的滑动窗口进行连续脑电图解码，这些窗口可能与神经事件错位。现有的窗口优化方法常常独立评估窗口，忽视时间依赖关系，可能导致碎片化或生理上不合理的选择。我们提出了时间结构嵌入强化学习（TSE-RL）框架，将对时间结构事件结构的先验知识纳入受限的马尔可夫决策过程，以实现自适应窗口优化。首先，TSE-RL通过丢弃、启动、维持或偏移动作为每个窗口分配显式语义角色，将二元保留/丢弃决策扩展到事件级别的对齐。其次，有限状态机掩盖无效动作，以强制MI阶段的时间顺序并促进生理上合理的窗口序列。第三，基于Transformer的历史编码器能够捕捉跨窗口的依赖关系，相较于循环架构，提升训练稳定性和计算效率。TSE-RL即插即用，可与多种MI分类器集成。在多个基准数据集上的实验显示，在跨会话和跨受试者环境中，解码性能均有持续提升。可视化分析进一步表明，学习到的窗口选择策略与神经生理学原理相符。

WeGR: Unified Generative Retrieval for Industrial Search Systems in WeChat

WeGR：微信工业搜索系统的统一生成检索

• Authors: Jiajun Han, Guoxin Ma, Felix, Liangsheng Xu, Ruiqi Yu, Yingdi Guo, Zongbing Wang, Xinhua Ye, Chenyan Hua, Xin Xie, Pengcheng Yang, Haijun Wu
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16979
• Pdf link: https://arxiv.org/pdf/2610.16979
• Abstract Generative retrieval (GR) offers a promising paradigm for industrial search by unifying candidate generation within an autoregressive framework. However, its effectiveness is limited by two key challenges. First, existing Semantic ID (SID) tokenizers struggle to incorporate collaborative signals while maintaining effective codebook utilization, limiting their ability to distinguish items and achieve high-quality retrieval. Second, existing reinforcement learning (RL) methods suffer from severe reward sparsity under exact SID matching, leaving many sampled trajectories without effective learning signals and hindering preference optimization. To address these challenges, we propose WeGR, a unified generative retrieval framework consisting of two complementary components. Specifically, the Collaborative and Collapse-free Tokenizer (CoCo-VAE) incorporates query-to-item and item-to-item collaborative signals while mitigating codebook collapse through codebook refresh and complementary regularization, enabling more informative and balanced SID representations. Building upon these SIDs, Dynamic Horizon Policy Optimization (DHPO) progressively expands both reward evaluation and policy optimization horizons, reducing learning difficulty while leveraging behavioral reward signals at different levels of the SID hierarchy, without relying on auxiliary reward models. Extensive offline experiments demonstrate the effectiveness of WeGR in improving retrieval and ranking performance. Moreover, a two-month online A/B test in WeChat E-commerce Search demonstrates relative improvements of 2.539% in orders, 1.508% in click-through rate, and 1.381% in conversion rate, while reducing online CPU and GPU usage by 25.5% and 26.1%, respectively. The system has been successfully deployed across multiple search scenarios in WeChat, serving billions of users, generating tens of millions of PVs daily.
• 中文摘要 生成检索（GR）通过在自回归框架内统一候选生成，为工业搜索提供了有前景的范式。然而，其有效性受到两个关键挑战的限制。首先，现有的语义识别（SID）标记器难以在整合协作信号的同时保持有效码本利用，限制了其区分项目和实现高质量检索的能力。其次，现有强化学习（RL）方法在精确SID匹配下存在严重的奖励稀疏性，导致许多抽样轨迹缺乏有效学习信号，阻碍偏好优化。为应对这些挑战，我们提出了WeGR，一个由两个互补组成部分组成的统一生成检索框架。具体来说，协作无崩溃分符器（CoCo-VAE）整合了查询到项目和项目间的协作信号，同时通过代码本刷新和互补正则化来缓解码本崩溃，从而实现更具信息量和平衡的SID表示。基于这些SID，动态视野策略优化（DHPO）逐步扩展了奖励评估和策略优化的视野，降低学习难度，同时利用SID层级不同层级的行为奖励信号，而无需依赖辅助奖励模型。大量离线实验证明了WeGR在提升检索和排名性能方面的有效性。此外，微信电商搜索的两个月在线A/B测试显示订单量提升2.539%，点击率1.508%，转化率1.381%，线上CPU和GPU使用率分别降低25.5%和26.1%。该系统已成功部署于微信多种搜索场景，服务数十亿用户，日均产生数千万光鲜。

From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning

从次优到最优：离线强化学习的一步潜流策略

• Authors: Shenghua Lu, Zixuan Yang, Rongxu Zhao, Bohao Qu, Jifeng Hu, Hechang Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.16984
• Pdf link: https://arxiv.org/pdf/2610.16984
• Abstract Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval $\Delta t=1$, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from $T$ to $K \approx T/H$ and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.
• 中文摘要 离线强化学习旨在从固定数据集中学习有效的策略，而无需环境交互。尽管扩散和流匹配等生成模型提升了策略表达性，但它们通常从无信息量的高斯噪声出发，通过多步迭代生成动作，导致计算成本高昂，且未能有效利用数据集中的次优轨迹。我们提出了MeanFlow2Better（MF2B），将离线强化学习重新表述为生成决策块优化，从当前性能状态向高层潜在决策空间中的更好性能状态。MF2B通过编码器将轨迹块映射到高层潜在决策表示，将生成起点置于由行为数据雕琢的结构化决策空间中。模型在此空间中学习共享的条件平均速度场，构建了两个流映射：从高斯诱导决策源向正向行为决策区域，以及从次优决策到高价值决策区域。两种流映射均通过单位时间区间 $\Delta t=1$ 的单步常微分方程积分完成决策细化，无需多步迭代采样或时间反向传播。此外，我们提出层级块级优势，引导流向高回报区域，其中高层决策涵盖多个低层动作，从而实现序列长度从$T$压缩至$K \ 约 T/H$，计算开销显著降低。在D4RL Gym-MuJoCo基准测试中，MF2B平均得分90.91，匹配甚至超过强基线。

MaskSense: Confronting the Visual Exploration Trap in Masked Image Generation

MaskSense：直面蒙面图像生成中的视觉探索陷阱

• Authors: Yawen Shao, Jie Xiao, Kai Zhu, Yu Liu, Hongchen Luo, Xueyang Fu, Yang Cao, Wei Zhai, Zheng-Jun Zha
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.17009
• Pdf link: https://arxiv.org/pdf/2610.17009
• Abstract Reinforcement learning (RL) has shown strong potential for aligning generative models with human intent, but adapting RL to masked generative models (MGMs) remains largely underexplored. Existing approaches that directly adapt Group Relative Policy Optimization (GRPO) to MGMs leave a foundational conflict unaddressed: MGMs' exploitative decoding nature is inherently at odds with the exploratory diversity that GRPO presupposes. In this work, we reveal a structural decoding bias termed the visual exploration trap: confidence-based sampling trades the exploration of diverse subject realizations for the greedy resolution of low-uncertainty background regions, causing premature collapse of the generative exploration space and starving GRPO of the rollout diversity it depends on. To this end, we propose MaskSense, a novel RL framework for MGMs that confronts the visual exploration trap at both the sampling and optimization stages. Specifically, we introduce a semantic anchored routing sampling mechanism that leverages semantic priors to preserve high-entropy subject token exploration while amplifying intra-group reward variance through exploratory-exploitative routing to yield more discriminative advantage estimates. Furthermore, we design a global entropy transition anchoring strategy to identify the most consequential decoding steps and concentrate gradient updates on them. Extensive experiments on multiple text-to-image benchmarks demonstrate that MaskSense substantially improves upon the base model, with GenEval accuracy improving from 54% to 86% and HPS from 28.89 to 36.39, achieving state-of-the-art performance.
• 中文摘要 强化学习（RL）显示出将生成模型与人类意图对齐的强大潜力，但将强化学习适应到掩蔽生成模型（MGMs）方面仍大多未被充分探索。现有直接将群体相对策略优化（GRPO）适配到MGM的方法存在一个根本冲突：MGM的剥削性解码性质本质上与GRPO所假设的探索多样性相冲突。本研究揭示了一种结构性解码偏差，称为视觉探索陷阱：基于信心的抽样以贪婪的方式解决低不确定性背景区域，导致生成探索空间的过早崩溃，并使GRPO失去其依赖的推广多样性。为此，我们提出了MaskSense，一种面向MGM的新型强化学习框架，能够在采样和优化阶段应对视觉探索陷阱。具体来说，我们引入了一种语义锚定路由采样机制，利用语义先验保持高熵主体代币探索，同时通过探索性-利用性路由放大组内奖励方差，从而获得更具判别性的优势估计。此外，我们设计了全局熵转移锚定策略，以识别最重要的解码步骤并集中对其进行梯度更新。在多项文本到图像基准测试上的大量实验表明，MaskSense在基础模型基础上有了显著提升，GenEval的准确率从54%提升至86%，HPS从28.89提升至36.39，实现了最先进的性能。

Neos: Learning to Interleave Search and Visual Reasoning

Neos：学习交错搜索与视觉推理

• Authors: Ekko He, Jack Hong, Shilin Yan, Xintong Zhang, Yujie Zhang, Wenjie Li, Haoran Sun, Kaixun Jiang, Jiyuan Fu, Guannan Zhang, Hongwei Xue
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17043
• Pdf link: https://arxiv.org/pdf/2610.17043
• Abstract Recent multimodal deep search agents have advanced tool use and visual content retrieval, yet they still struggle to convert intermediate visual evidence into actionable guidance for subsequent search, limiting their effectiveness in long-horizon settings. To address this gap, we present Neos, a multimodal deep search framework that learns both how to acquire evidence and how to use it to guide subsequent decisions. To make this coupling learnable, Neos constructs training tasks with intermediate visual dependencies while filtering out textual shortcuts, and introduces a lightweight, trainable Multimodal Evidence Interface (MEI) that transforms retrieved content into goal-conditioned observations. At its core, an agent-environment co-evolution procedure treats MEI as a co-adaptive component of this loop. Specifically, reference-guided credit assignment uses verified successful trajectories to localize critical policy errors while preserving credit for useful decisions in failed searches, whereas rollout feedback selects informative local evidence-processing interactions for MEI supervised fine-tuning (SFT). By alternating policy reinforcement learning (RL) with MEI SFT, Neos allows the policy's evolving evidence requests to shape MEI training and the adapted MEI to reshape subsequent observations, closing the loop between evidence acquisition and evidence-conditioned action. Across ten benchmarks, Neos-9B and Neos-35B demonstrate leading performance among specialized multimodal deep-search agents.
• 中文摘要 近年来的多模态深度搜索代理具备先进的工具使用和视觉内容检索能力，但仍难以将中间视觉证据转化为后续搜索的可操作指导，限制了其在长期视野环境中的有效性。为弥补这一空白，我们提出了Neos，一个多模态深度搜索框架，既学习如何获取证据，又如何利用证据指导后续决策。为使这种耦合可学习，Neos构建了带有中间视觉依赖的训练任务，同时过滤掉文本捷径，并引入了轻量级、可训练的多模态证据接口（MEI），将检索到的内容转化为目标条件观察。其核心是代理-环境共进过程将MEI视为该循环的共适应组件。具体来说，引用引导的信用分配利用经过验证的成功轨迹定位关键策略错误，同时保留对失败搜索中有用决策的认可，而推广反馈则选择信息丰富的本地证据处理交互，用于MEI监督微调（SFT）。通过交替使用策略强化学习（RL）与MEI SFT，Neos允许策略中不断演变的证据请求塑造MEI训练，并调整后MEI重塑后续观测，实现证据获取与证据条件作用之间的循环。在十个基准测试中，Neos-9B和Neos-35B在专业多模态深度搜索代理中表现出领先性能。

Gradient Routing Localizes and Removes Unintended Behaviors in RL

梯度路由在强化学习中局部化并消除了非预期行为

• Authors: Jake Ward, Shawn Hu, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17062
• Pdf link: https://arxiv.org/pdf/2610.17062
• Abstract We propose a mitigation to reward hacking in reinforcement learning. Our setup involves a misspecified reward function that gives rise to unintended behavior, and a monitor used to detect unintended behavior. As with frontier AI systems, the monitor makes systematic errors. Consequently, using the monitor for data filtering or to apply a reward penalty fails to prevent the model from learning unintended behavior ("reward hacking"). To address this problem, we introduce GRAFT: Gradient Routing Adapter Fine Tuning. Rather than penalizing or filtering flagged samples during training, GRAFT constrains gradient updates on those samples to a sacrificial set of new parameters. This has two benefits: First, because the monitor is not part of the reward function, the model does not learn to hide misbehavior. Second, ablating the sacrificial parameters reduces or entirely stops the unintended behavior while maintaining improved performance on target tasks. We demonstrate these benefits in RL environments where reward penalties cause models to learn to reward hack in undetected ways, training models ranging from 135M to 8B parameters. Additionally, we show a similar effect at a larger scale by distilling frontier LLM trajectories containing reward hacking into Qwen3-32B, without teaching that model to reward hack. Overall, our work suggests GRAFT may be a promising approach for leveraging imperfect monitors during post-training in settings where true performance is difficult to evaluate.
• 中文摘要 我们提出了一种缓解强化学习中奖励黑客行为的方法。我们的设置涉及一个错误指定的奖励函数，导致意外行为，以及用于检测意外行为的监控器。与前沿人工智能系统类似，监控器会犯系统性错误。因此，使用监控器进行数据过滤或施加奖励惩罚，无法阻止模型学习非预期行为（“奖励黑客”）。为解决这个问题，我们引入了GRAFT：梯度路由适配器微调。GRAFT不在训练过程中惩罚或过滤标记的样本，而是将这些样本的梯度更新限制在一套牺牲的新参数中。这有两个好处：首先，由于监控器不属于奖励函数，模型不会学会隐藏不良行为。其次，削弱牺牲参数可以减少甚至完全停止非预期行为，同时保持目标任务的更好表现。我们在强化学习环境中展示了这些好处，奖励惩罚导致模型以未被发现的方式学习奖励黑客，训练模型参数范围从1.35亿到8亿亿。此外，我们还通过将包含奖励黑客的前沿大型语言模型轨迹提炼到Qwen3-32B中，未教授该模型进行奖励黑客，从而在更大尺度上展示了类似效果。总体来看，我们的研究表明，GRAFT可能是在难以评估真实性能的环境中，利用不完美的显示器在训练后利用的有前景方法。

ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

ATTEST：基于证据的强化学习用于长音频时间理解

• Authors: Ashutosh Chaubey, Jinxi Guo, Minh Tran, Michael Andersland, Ozlem Kalinli
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.17149
• Pdf link: https://arxiv.org/pdf/2610.17149
• Abstract Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.
• 中文摘要 时间基础化是音频理解的基础，但大型音频语言模型（LALMs）在长音频方面仍然薄弱，因为幻觉叠加且注释数据稀少。现有基准测试仅覆盖短片段或提供粗略合成注释。我们介绍EgoSCAPE，这是基于Ego4D手工验证的基准测试，包含700个最长5分钟的真实音频，配有密集的字幕、声音源、音频时间线和时间基础查询。我们还进一步提出了ATTEST，一种基于GRPO的强化学习框架，在不改变基础模型架构或标记词汇的情况下改善时间基础。ATTEST结合了（i）反事实接地奖励惩罚那些经声学证据消除后存活的预测，（ii）鼓励生成关注自身声称的时间跨度的注意力密度奖励，以及（iii）在代币层面给予长密集字幕的分段级认可优势。ATTEST在现有密集字幕和接地基准测试中优于以往的时间接地方法，且幻觉更少。EgoSCAPE与ATTEST共同建立了严谨的评估-训练配方，使时间基础音频理解更接近实用的长篇部署。

Size Generalization for Diffusion Routing Solvers

扩散路由求解器的规模推广

• Authors: Barak Itzchak Davidovitch, Eli Meirom, Ethan Fetaya, Gal Chechik
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17192
• Pdf link: https://arxiv.org/pdf/2610.17192
• Abstract Routing problems, like the Traveling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP), are NP-hard and slow to solve for large instances. Diffusion-based solvers can be trained in a supervised way using sets of optimal solutions, and then quickly solve new test instances. However, for large instances, optimal solutions are hard to find, and diffusion models fail to generalize to instance sizes outside their training distribution. Here we describe ResizeCO, a label free size-generalization approach that addresses both the change in statistical properties of larger instances, and the computational challenge of solving them quickly. It uses reinforcement learning on large-scale instances to adapt a diffusion model trained on small instances. Specifically, we adapt GRPO to this problem, using route length as the training signal. ResizeCO also formulates k-opt as a tree search, solved by a beam search on the GPU. In size-generalization experiments on TSP and CVRP instances with up to 10000 nodes, ResizeCO achieves a Pareto front over route quality and time that dominates learned solvers, and also outperforms other solvers at short run times, setting a new state of the art.
• 中文摘要 路由问题，如旅行推销员问题（TSP）和电容车辆路由问题（CVRP），对于大型实例来说，是NP难且求解缓慢的。基于扩散的求解器可以通过最优解集进行监督训练，然后快速求解新的测试实例。然而，对于大型实例，最优解难以找到，扩散模型也无法推广到训练分布外的实例大小。这里我们介绍ResizeCO，这是一种无标签的大小推广方法，既解决了更大实例统计属性的变化，也解决了快速解决这些实例的计算挑战。它利用大规模实例上的强化学习，适应在小实例上训练的扩散模型。具体来说，我们将GRPO适配为该问题，使用路由长度作为训练信号。ResizeCO还将k-opt表述为树状搜索，通过GPU上的束搜索解决。在对最多10000节点的TSP和CVRP实例进行规模推广实验中，ResizeCO实现了路径质量和时间的帕累托前沿，主导了学习中的求解器，并且在短时间内优于其他求解器，开创了新的技术水平。

Density-Matrix Reinforcement Learning

密度矩阵强化学习

• Authors: Chiuman Ho
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.17214
• Pdf link: https://arxiv.org/pdf/2610.17214
• Abstract What information does a reinforcement learning policy omit? At each state, a standard policy stores one probability per action. It has no place to record how actions relate to each other, such as which pairs go together and which conflict. We ask when this missing structure matters, and we give exact answers. We store the policy at each state as a density matrix, a positive semidefinite, unit-trace matrix borrowed from quantum information theory. We use it here as a purely classical data structure, computed on ordinary classical hardware. The diagonal holds the usual action probabilities. The off-diagonal entries, called coherences, are new parameters. This gives density-matrix reinforcement learning (DMRL). Coherences change outcomes in two settings. First, suppose a policy is regularized toward a coherent reference policy using matrix relative entropy. The optimum then has coherences that no diagonal reference policy produces, and its action probabilities can depend on the reference policy's coherences. This is relevant to fine-tuning of large language models (LLMs) with a Kullback--Leibler penalty. Second, suppose the agent chooses a unit-vector action and the reward is a quadratic function of that vector. This could arise when an LLM judge scores a weighted consolidation of reasoning paths, or when coupled robotic sensors each respond to a mix of actions. Then some coherent policies earn strictly more expected reward than every classical policy with the same action probabilities, and we give the margin in closed form. In a two-action example, both policies assign probability $0.5$ to each action, yet the classical one earns expected reward $0$ while the coherent one earns $1$, the maximum. Outside these two settings, DMRL matches or transfers the classical results. This is a theory paper. All results are proved, and we report no empirical evaluation.
• 中文摘要 强化学习策略省略了哪些信息？在每个状态下，标准策略为每个动作存储一个概率。它没有记录动作之间关系的地方，比如哪些配对在一起、哪些冲突。我们询问这种缺失结构何时重要，并给出准确答案。我们在每个状态将策略存储为密度矩阵，即从量子信息理论借用的正半定单位踪迹矩阵。这里我们将其作为纯经典数据结构使用，在普通经典硬件上计算。对角线包含通常的动作概率。非对角项称为相干，是新参数。这给出了密度矩阵强化学习（DMRL）。相干性在两种情况下改变结果。首先，假设策略通过矩阵相对熵正则化为相干参考策略。此时最优方案具有对角引用策略不会产生的相干性，其行动概率可依赖于参考策略的一致性。这与带有Kullback--Leibler惩罚的大型语言模型（LLM）微调相关。其次，假设代理选择单位向量动作，奖励是该向量的二次函数。当LLM评判对推理路径进行加权整合评分，或耦合机器人传感器对多种动作做出反应时，这可能发生。此时，某些相干策略获得的期望奖励严格高于所有具有相同行动概率的经典策略，我们以封闭形式给出保证空间。在双行动示例中，两种策略均赋予每个动作0.5美元的概率，但经典策略获得期望奖励$0，而连贯策略获得1美元，即最大值。在这两个环境之外，DMRL匹配或转移经典结果。这是一篇理论论文。所有结果均已证明，我们不报告实证评估。

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

CodeLift：你的测试套件其实是一套密集的软件工程课程

• Authors: Junjielong Xu, Huachao Zhu, Alyssa Wu, Zhichao Hu, Pinjia He
• Subjects: Subjects: Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.17224
• Pdf link: https://arxiv.org/pdf/2610.17224
• Abstract The pass-all-test binary reward has become a standard recipe for training software engineering agents, but its sparsity eliminates difference of partially correct patches. The test pass-rate is a natural alternative. However, we find that it is not a reliable indicator of patch quality. A functionality being tested can be covered by either one challenging test or one hundred trivial tests, where the majority might be already satisfied pass-to-pass tests. We introduce CodeLift, a reinforcement learning method that dynamically calibrates pass-rate rewards, turning the test suite into an adaptive curriculum. At each iteration, CodeLift groups tests that kill the same patches and patches that pass the same tests, then gives more weight to test groups that fewer patch groups pass. This reduces the impact of majority simple tests and focuses more on what is still challenging. Within each rollout, CodeLift applies the same scoring rule to repository snapshots, assigning credit to useful actions even when later ones introduce regressions. Experiments on SWE benchmarks show that CodeLift consistently improves pass@1/2/4/8 over binary and pass-rate baselines. Analyses show that CodeLift provides dense, stable learning signals and unlocks the inherent curriculum in the test suite.
• 中文摘要 “全测试通过”二进制奖励已成为训练软件工程代理的标准配方，但其稀疏性消除了部分正确补丁的差异。测试通过率是一个自然的替代方案。然而，我们发现它并不能可靠地反映补丁质量。被测试的功能可以被一个具有挑战性的测试覆盖，也可以被一百个琐碎测试覆盖，其中大多数测试可能已经满足于通过到通过的测试。我们引入了CodeLift，一种强化学习方法，动态校准通过率奖励，将测试套件转变为自适应课程。每次迭代，CodeLift将杀死相同补丁和通过相同测试的补丁分组，然后对通过较少补丁组的测试组给予更高权重。这减少了多数简单测试的影响，更多关注仍具挑战性的内容。在每次部署中，CodeLift对仓库快照应用相同的评分规则，即使后续快照引入回归，仍将功劳归入有用行动。SWE基准测试的实验显示，CodeLift在二进制和通过率基线上的pass@1/2/4/8持续提升。分析显示，CodeLift提供了密集、稳定的学习信号，并解锁了测试套件中固有的课程内容。

Calibrating LLM Explanations to Human Decisions with Reinforcement Learning

用强化学习校准LLM解释以适应人类决策

• Authors: Dong Shu, Ziyang Guo, Negar Kamali, Matthew Groh, Jessica Hullman
• Subjects: Subjects: Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.17306
• Pdf link: https://arxiv.org/pdf/2610.17306
• Abstract Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce \textbf{HD-RL}, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions. We apply this framework to AI-generated text detection. In our initial data collection phase, we observed that while LLMs possess strong autonomous detection capabilities, the explanations they generate for users yield near-random human accuracy (51.3\%). By applying the HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that improve human performance, achieving a GLMM-estimated human accuracy of 63.7\% in a randomized between-subjects experiment with repeated measures and outperforming RLVR, RLAIF, and other baselines by over 8\%.
• 中文摘要 LLM输出的解释被认为是使模型行为可理解的重要方式，帮助人类用户在后续任务中做出良好决策。然而，用于改进解释的默认训练方法更针对模型任务表现或人类偏好，而非人类决策质量。为弥合这一差距，我们引入了\textbf{HD-RL}，一种强化学习框架，用于优化LLM对人类决策的解释。我们的方法校准人类决策模拟器，基于观察到的人类决策，应用代理奖励模型优化LLM生成帮助人们做出良好决策的解释。我们将该框架应用于AI生成的文本检测。在初步数据收集阶段，我们观察到虽然LLM具备强大的自主检测能力，但它们为用户生成的解释几乎随机的人类准确率（51.3%）。通过将HD-RL框架应用于开源Qwen 3.6模型，我们生成了优化解释，提升人类表现，在随机受试者间实验中，重复测量实现GLMM估计的人类准确率为63.7%，并且比RLVR、RLAIF及其他基线高出8%以上。

StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions

StartLux-decision：带有强化学习的自动思考决策模型，用于校准的审议决策

• Authors: Xialie Zhuang, Zunhai Su, Jia Pan, Ngai Wong, Guanhua Chen, Chuan-Wei Kuo
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17309
• Pdf link: https://arxiv.org/pdf/2610.17309
• Abstract We introduce StartLux-Decision, decision models from 0.8B to 27B parameters that combine efficient single-pass prediction with adaptive, confidence-guided reasoning. The Instant models use bidirectional attention in full-attention layers to improve decision accuracy while keeping single-pass inference. The AutoThink models invoke explicit reasoning for uncertain decisions and answer confident cases directly. A three-stage training pipeline establishes shared decision capabilities, specializes the two variants, and equips AutoThink with reasoning through reasoning supervision and reinforcement learning. We introduce Reinforcement Learning for Calibrated Deliberative Decisions (RLCD$^2$), which uses a frozen model and proper scoring rules to reward improvements in decision probability quality and applies a soft penalty to overlong reasoning traces. RLCD$^2$ raises the share of reasoning traces that terminate within the generation limit by up to 48 percentage points and cuts average reasoning length by up to 65%, while largely preserving decision accuracy. On the public subset of Decision Index 0.3, the Instant models outperform the strongest evaluated comparable-scale open-weight decision models by 7.4-16.9 points, and the 27B AutoThink model reaches 73.4. We also introduce DecisionPivot, a programmatically verifiable benchmark of computational decisions built from contrast pairs that differ in a single answer-changing detail. On DecisionPivot, AutoThink outperforms its Instant counterparts by 11.5-18.4 points in pair accuracy and leads among evaluated systems. The models run on vLLM, this http URL and MLX, with quantized variants for lower memory use.
• 中文摘要 我们引入了StartLux决策模型，参数范围从0.8B到27B，结合了高效的单次预测和自适应、置信导向推理。即时模型在全注意力层中采用双向注意力，以提高决策准确性，同时保持单次推断。AutoThink模型为不确定决策调用显性推理，直接回答自信案例。三阶段训练流程建立了共享决策能力，专门化两种变体，并通过推理监督和强化学习为AutoThink提供推理能力。我们引入校准审议决策强化学习（RLCD$^2$），该方法使用冻结模型和适当的评分规则奖励决策概率质量的提升，并对过长推理轨迹施加软惩罚。RLCD$^2$ 将终止于生成限制内的推理痕迹比例提高最多 48 个百分点，平均推理长度缩短高达 65%，同时基本保持决策准确性。在决策指数 0.3 的公开子集上，Instant 模型的表现比最强的可比规模开放权重决策模型高出 7.4-16.9 个百分点，27B 的 AutoThink 模型达到 73.4 个百分点。我们还引入了 DecisionPivot，这是一个可编程验证的计算决策基准测试，基于单个变化细节的对比对对构建。在 DecisionPivot 上，AutoThink 在配对准确率上优于 Instant 对应者 11.5-18.4 个百分点，并在被评估系统中领先。这些模型运行于 vLLM、该 http URL 和 MLX，并有量化变体以降低内存使用。

Learning to Play: Reinforcement Learning and Games

学习玩乐：强化学习与游戏

• Authors: Aske Plaat
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17391
• Pdf link: https://arxiv.org/pdf/2610.17391
• Abstract Amazing breakthroughs in reinforcement learning have taken place. Computers teach themselves to play Chess and Go and beat world champions. There is talk about expanding application areas towards general artificial intelligence (AI). The breakthroughs in Backgammon, Checkers, Chess, Atari, Go, Poker, and StarCraft have shown that we can build machines that exhibit intelligent game playing of the highest level. These successes have been widely publicized in the media, and inspire AI entrepreneurs and scientists alike. Reinforcement learning in games has become a mainstream AI research topic. It is a broad topic, and the successes build on a range of diverse techniques, from exact planning algorithms, to adaptive sampling, deep function approximation, and ingenious self-play methods. Perhaps because of the breadth of these technologies, or because of the recency of the breakthroughs, there are few books that explain these methods in depth. This book covers all methods in one comprehensive volume, explaining the latest research, bringing you as close as possible to working implementations, with many references to the original research papers. The programming examples in this book are in Python, the language in which most current reinforcement learning research is conducted. We help you to get started with machine learning frameworks such as Gym, TensorFlow, and Keras, and provide exercises to help understand how AI is learning to play.
• 中文摘要 强化学习取得了惊人的突破。计算机自学下棋和围棋，并击败世界冠军。有人讨论将应用领域扩展到通用人工智能（AI）。西洋双陆棋、跳棋、国际象棋、雅达利、围棋、扑克和星际争霸的突破表明，我们可以制造出展现最高智能游戏水平的机器。这些成功在媒体上广泛报道，激励着人工智能创业者和科学家。游戏中的强化学习已成为主流的人工智能研究课题。这是一个广泛的课题，成功案例建立在多种多样技术之上，从精确规划算法、自适应采样、深度函数近似到巧妙的自玩方法。也许是因为这些技术的广泛性，或是突破的近代，深入解释这些方法的书籍很少。本书汇集了所有方法，汇集于一本全面的卷册，解释最新的研究，尽可能接近实际实现，并大量引用原始研究论文。本书中的编程示例均用Python语言进行，Python是大多数当前强化学习研究的语言。我们帮助你入门使用机器学习框架，如Gym、TensorFlow和Keras，并提供练习帮助理解AI如何学习玩耍。

Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL

职位：除了培训时间外，考试时间协调对于合作MARL至关重要

• Authors: Dongsu Lee, Sooraj Sathish, Joonkyung Kim, Woojun Kim, Amy Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.17409
• Pdf link: https://arxiv.org/pdf/2610.17409
• Abstract We argue that cooperative multi-agent reinforcement learning (MARL) should treat test-time coordination as part of the problem definition rather than a deployment artifact. Our focus is on long-horizon, centrally monitored cooperative systems such as warehouse robotics, automated storage-and-retrieval systems, and cloud-resource scheduling, where teams share a reward, operate under shifting objectives, and already run with global telemetry at deployment. The dominant paradigm in this regime, centralized training with decentralized execution (CTDE), uses global information only during training and provides no coordination channel at execution. This is sufficient when agents are weakly coupled or the environment is stationary, but it fails when the team must realign after conditions change. The limitation is structural rather than algorithmic, and more training cannot add a channel that does not exist at execution time. We call this the recoverability gap. To address it, we name the underexplored design space of intermittent test-time coordination as centralized training with mixed execution (CTME), in which a coordinator observes global state and emits high-level signals while agents act locally. CTME is not a particular learning algorithm, but a cooperative MARL problem setting that makes test-time information structure explicit. It specifies team-level information, high-level signals, and update frequency after training, with CTDE recovered as a limit case. We do not argue that CTDE is obsolete. We argue that removing the test-time channel is a modeling choice, not a necessity, and the wrong default when global telemetry is available and teams must recover from drift.
• 中文摘要 我们主张合作多智能体强化学习（MARL）应将测试时协调视为问题定义的一部分，而非部署产物。我们的重点是长期、集中监控的协作系统，如仓库机器人、自动存储与检索系统和云资源调度，团队共享奖励，在目标变化下运行，且部署时已具备全局遥测。该体系的主导范式是集中式训练与去中心化执行（CTDE），仅在训练期间使用全局信息，执行时不提供协调通道。当智能体耦合弱或环境静止时，这种方式已足够，但当团队在条件变化后必须重新调整时，这种方式失效。限制是结构性的而非算法的，更多训练无法增加执行时不存在的通道。我们称之为可恢复性差距。为此，我们将间歇性测试时间协调这一尚未充分探索的设计空间命名为集中训练混合执行（CTME），其中协调者观察全局状态并发出高级别信号，而代理则在局部行动。CTME不是一种特定的学习算法，而是一个协作式MARL问题设置，使测试时间信息结构显式化。它指定团队级信息、高水平信号和训练后更新频率，CTDE作为极限情况被恢复。我们不主张CTDE已过时。我们认为移除测试时间通道是一种建模选择，而非必要，且在全局遥测可用且团队需要从漂移中恢复时，这是错误的默认选择。

Reinforcement Learning In Partially Verifiable Environments for Colon Cancer Treatment Planning

部分可验证环境下的强化学习，用于结肠癌治疗计划

• Authors: Li-Ching Chen, Andrés Franco Valiente, Yuta Ishiyama, Zachary R. Chalmers, William Chen, Kelsey Natsuhara, Taema Brinjikji, Madhumita Sushil, Travis Zack, Julian C. Hong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17426
• Pdf link: https://arxiv.org/pdf/2610.17426
• Abstract Clinical decision-making is a sequential process of information collection, clinical inference, and forecasting under partial observability, often without a uniquely correct treatment decision. Large language models (LLMs) have demonstrated strong medical question-answering capabilities but extending these advances to clinical reasoning remains challenging because many clinical tasks are only partially verifiable. Recent clinical reinforcement learning (RL) work has therefore largely focused on diagnostic endpoints, where correctness can be more directly verified. Oncology treatment decision making is a particularly challenging instance of this problem that extends beyond medical question-answering in which multiple decision trajectories can be clinically defensible for the same partially observed patient state. We formulate oncology treatment planning as an interactive RL problem under partial observability and construct patient-specific, process-aware rewards from decision processes observed in real world clinical records. The agent must learn to sequentially acquire information from a structured patient representation before committing to a first-line treatment recommendation, and is evaluated on information gathering, treatment selection, and reasoning alignment. Using GRPO and GDPO to optimize Llama-3.1-8B-Instruct within this environment, RL substantially improves the base policy beyond frontier model capabilities. The GDPO-trained policy increases Mean@3 scalar clinical reward by $79.60\%$ over base Llama and exceeds GPT-5.2 by $18.82\%$ under the same interactive environment, while reducing query count during training. Overall, our results show that patient-specific, partially verifiable rewards can provide an effective learning signal for interactive clinical RL when both the patient state and the underlying clinical decision process are only partially observed.
• 中文摘要 临床决策是一个信息收集、临床推断和部分可观测性预测的顺序过程，通常没有唯一正确的治疗决策。大型语言模型（LLMs）已展现出强大的医学问答能力，但将这些进展推广到临床推理仍具挑战性，因为许多临床任务只能部分验证。因此，近期的临床强化学习（RL）工作主要聚焦于诊断终点，在那里正确性可以更直接地验证。肿瘤治疗决策是这一问题的一个特别具有挑战性的实例，超越了医学问答，在该领域，对于同一部分观察的患者状态，多条决策轨迹可以临床上合理化。我们将肿瘤治疗规划作为一个在部分可观察性下进行的交互式强化学习问题，并从现实临床记录中观察到的决策过程构建患者特定、过程感知的奖励。智能体必须学会从结构化患者代表中顺序获取信息，才能承诺一线治疗建议，并在信息收集、治疗选择和推理对齐方面接受评估。利用GRPO和GDPO在该环境中优化Llama-3.1-8B-Instruct，RL大幅提升了基础策略超越前沿模型的能力。GDPO训练策略在标量临床奖励Mean@3基础Llama上提升79.60美元，在相同交互环境中比GPT-5.2高出18.82美元，同时减少训练中的查询数量。总体来看，我们的结果表明，当患者状态和底层临床决策过程仅部分被观察时，针对患者特定且部分可验证的奖励可以为交互式临床强化学习提供有效学习信号。

The belief-action gap in language model agents

语言模型代理中的信念-行动差距

• Authors: Ayse Sila Okcu, Weisheng Wang, Tongmeng Xie, Niall Dalton, Angelos Nalmpantis, Raghu Arghal, Mario Giulianelli
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.17444
• Pdf link: https://arxiv.org/pdf/2610.17444
• Abstract Language model (LM) agents can fail for different reasons, requiring different explanations and interventions. We study how state beliefs, action values, and action preferences interact as LM agents reason and commit to decisions in text-rendered grid worlds. We estimate task-relevant beliefs using report-based and representation-based probes, and action values using inverse reinforcement learning. We find that task-relevant state and value information is often represented more reliably than it is reflected in behaviour. Tracking action preferences throughout reasoning further shows that decisions sharpen around a commitment point and reported belief uncertainty decreases near commitment, although individual belief changes do not consistently explain shifts in action preference. Finally, forced actions follow reasoning prefixes from other decisions when transplanted, whereas the tested reasoning-window activation edits provide limited control over the action. Together, these results suggest that some failures arise not from missing task-relevant information, but from how available information is translated into action. Our measurements distinguish information availability from belief-action consistency without by themselves identifying the cause of each failure. Our code is available at: this https URL
• 中文摘要 语言模型（LM）代理可能因不同原因失败，需要不同的解释和干预。我们研究了状态信念、动作值和行动偏好如何相互作用，当语言模型代理在文本渲染的网格世界中推理并承诺决策时。我们利用基于报告和表示的探针估计任务相关信念，利用逆强化学习估算行动值。我们发现与任务相关的状态和价值信息往往比行为中更可靠地表示。推理过程中的行动偏好追踪进一步表明，决策在承诺点附近变得敏锐，报告的信念不确定性在承诺附近减少，尽管单个信念变化并不总是能解释行动偏好的转变。最后，强制行动在迁移时会跟随其他决策的推理前缀，而测试的推理窗口激活编辑对动作的控制有限。综合来看，这些结果表明，有些失败并非源于缺失与任务相关的信息，而是源于可用信息如何转化为行动。我们的测量方法区分了信息可用性与信念-行动一致性，但本身无法确定每个失败的原因。我们的代码可在以下网站获取：此 https URL

Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning

组合强化学习：评估与培训协作推理的大型语言模型

• Authors: Shangbin Feng, Yulia Tsvetkov, Jan Kautz, David Acuna
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.17456
• Pdf link: https://arxiv.org/pdf/2610.17456
• Abstract Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. For a model inside a system, what matters most is not how well it performs on its own, but how much the system gains from its presence. We call this a model's compositional strength and formalize it using the Shapley value of a cooperative game over coalitions of a model pool, and use a model's leave-one-out contribution to the full coalition as a tractable proxy, requiring n+1 rather than 2^n system evaluations. Evaluating individual vs. compositional strengths across five model pools, twelve collaboration methods, and twelve datasets, we find them largely uncorrelated: in 12 of 15 (pool, method-family) settings, individual strength does not translate into compositional advantages, and the coupling is weakest among general-purpose models that practitioners are most likely to deploy. Standalone benchmarks are therefore often poor proxies for models' value in collaborative systems. We then propose compositional reinforcement learning (C-RL), which trains a policy inside a live multi-model system and rewards the outcome produced by the system as a whole, rather than the policy's correctness in isolation. The leave-one-out baseline in Shapley values cancels under a group policy gradient, so this needs no additional rollout cost beyond RL on the collaborative system's outcome and never requires running a coalition without the policy. Across three collaboration protocols and eight tasks spanning reasoning, coding, and scientific discovery, C-RL produces the strongest collaborator in 21 of 24 settings, lifting collaborative performance by +4.0 to +5.4 pts on average (~3x the gain of baselines), up to +13.1 pts.
• 中文摘要 语言模型通常单独评估，但越来越多地一起部署、在模型之间路由、进行辩论、集成或协作训练。对于系统内部的模型来说，最重要的是它自身的表现如何，而在于系统从其存在中获得多少收益。我们称之为模型的组合强度，并用合作博弈的Shapley值形式化它，并以模型对完整联盟的“一”一“贡献作为可处理的代理，需要n+1次而非2^n次系统评估。评估五个模型池、十二种协作方法和十二个数据集中的个人与组合优势，我们发现它们基本无相关性：在15个（模型池、方法-家族）环境中，12个个体优势不转化为组合优势，且在实践者最可能部署的通用模型中耦合最弱。因此，独立基准往往是模型在协作系统中价值的良好代理。随后我们提出组合强化学习（C-RL），它在实时多模型系统中训练策略，奖励系统整体产生的结果，而非单独看策略的正确性。Shapley值中的“一除外”基线在群体策略梯度下相互抵消，因此除了对协作系统结果的强化外，无需额外推广成本，且无需在无策略的情况下运行联盟。在三种协作协议和涵盖推理、编码和科学发现的八项任务中，C-RL在24个环境中产生了21个最强协作者，平均将协作表现提升+4.0至+5.4分（基线提升的~3倍），最高可达+13.1分。

RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning

中继：选择性自我提炼与自适应交接，用于基于评分标准的强化学习

• Authors: Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li, Lei Cheng, Ying Liu, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.17623
• Pdf link: https://arxiv.org/pdf/2610.17623
• Abstract Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.
• 中文摘要 基于评分标准的强化学习会根据多个书面标准评分反应，但将它们合并为一个奖励，当一组反应一致时，会舍弃评分标准所包含的细节，且当小组反应一致时，不给出GRPO奖励梯度。将评分标准解读为特权上下文的固定教师提供代币级监督，但我们观察到相比GRPO的进步处于早期平台期。RELAY利用优势和奖励选择正向优势和低得分零优势的推广，并监控蒸馏损失以决定何时教师退休。第一阶段结合了从冻结教师的选择性提炼与GRPO;然后平台检测器移除教师，第二阶段仅使用GRPO。在一对模型域中校准，并在另外三对中未变地重复使用，平台探测器在第122至171步之间触发。在四种环境下，RELAY在所有比较方法中领域得分最高，RaR-Medicine的GRPO以Qwen3-1.7B领先4.62分，Qwen3-4B的6.02分。在Qwen3-1.7B/Medicine中，将教师从共享检查点移除优于保留、削弱或刷新该检查点。精选覆盖约一半响应标记的精选推广，也优于以匹配平均系数提炼所有推广。

Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence

Open-o3-Video-v2：寻求、基础与推理，具备可验证的时空证据

• Authors: Jiahao Meng, Zhongyuan Bao, Yue Tan, Haiquan Wen, Renrui Zhang, Anna Wang, Jason Li, Yanwei Fu, Yunhai Tong, Ming-Hsuan Yang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.17656
• Pdf link: https://arxiv.org/pdf/2610.17656
• Abstract Reliable video reasoning requires models to produce correct answers and expose supporting visual evidence. Open-o3-Video enables verifiable reasoning by embedding timestamps and object-level bounding boxes into reasoning traces, but is restricted to frames from an initial sparse sampler. We introduce Open-o3-Video-v2, which formulates video reasoning as a unified agentic evidencegrounded process. The model identifies missing information, acquires additional observations, grounds the moments and objects supporting its reasoning, and writes this evidence into the trajectory before continuing or answering. To support this behavior, we construct STGR-v2-CoT135K, including 52K multi-turn agentic grounded reasoning trajectories, and STGR-v2-RL-13K for reinforcement learning. We further propose Clue-Seek GRPO (CS-GRPO), which tracks how tool calls change accumulated evidence, rewards clue coverage and step-wise gains, and penalizes excessive, repetitive, or uninformative retrieval. On V-STaR, Open-o3-Video-v2-4B achieves 40.4% mAM and 56.3% mLGM, outperforming its Qwen3-VL-4B backbone by 8.5% and 12.6%, respectively. It also delivers consistent gains across general video understanding, long-video reasoning, and temporal grounding benchmarks. These results show that active evidence acquisition and verifiable object-level grounding are complementary for reliable video reasoning. The code and models are released at this https URL.
• 中文摘要 可靠的视频推理需要模型产生正确答案并揭示支持性的视觉证据。Open-o3-Video通过在推理轨迹中嵌入时间戳和对象级边界框，实现可验证的推理，但限制在初始稀疏采样器的帧内。我们引入Open-o3-Video-v2，将视频推理构建为统一的代理证据基础过程。模型识别缺失信息，获取额外观察，为支持其推理的时刻和对象建立基础，并将这些证据写入轨迹后再继续或回答。为支持这种行为，我们构建了STGR-v2-CoT135K，包括52K多回合智能基准推理轨迹，以及STGR-v2-RL-13K用于强化学习。我们进一步提出了Clue-Seek GRPO（CS-GRPO），该工具追踪工具调用如何改变累积证据，奖励线索覆盖率和逐步收益，惩罚过度、重复或无信息检索。在V-STaR上，Open-o3-Video-v2-4B实现了40.4%的mAM和56.3%的mLGM，分别比Qwen3-VL-4B骨干网高出8.5%和12.6%。它在一般视频理解、长视频推理和时间基础基准方面也持续有显著提升。这些结果表明，主动证据获取和可验证对象级基础对于可靠的视频推理是互补的。代码和模型发布于此 https URL。

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

DeltaTTG：用代币即走进度条训练多回合特工

• Authors: Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin, Wei Lu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.17666
• Pdf link: https://arxiv.org/pdf/2610.17666
• Abstract Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps actor-training GPU-hours close to GRPO. Further experiments show improved sample efficiency and out-of-domain performance, and support the meter's ability to distinguish step quality and guide test-time action selection across actor sizes. The code is available at this https URL.
• 中文摘要 针对多回合代理的可验证奖励强化学习（RLVR）依赖稀疏的终端奖励，几乎没有说明单个步骤。组内步骤比较依赖于可靠的状态匹配和足够多样化的抽样结果，而直接步骤评分可能需要昂贵的外部评估器、人工注释或任务特定的启发式方法。我们引入了DeltaTTG，通过共享进度计量表学习可重复使用的进度信号，利用低成本监督不同策略记录轨迹，无需任务特定进度规则。其监督目标是Token-to-Go（TTG），即任务完成前剩余生成的代币数量，作为剩余完成努力的可观察代理。在演员训练过程中，冻结计量表估计的变化会在不同步骤中重新分配结果衍生优势，而不改变奖励，保持每个轨迹的优势符号和代币平均等级。在ALFWorld、PaperSearchQA和BFCLv3上，DeltaTTG将GRPO的平均得分提升了5.7分和7.2分，Qwen3-4B和Llama-3.1-8B-Ininstruction分别在平均上优于所有对比RL基线，且演员训练GPU小时数接近GRPO。进一步实验显示样本效率和域外表现有所提升，支持计量器区分步长质量和引导不同actor大小测试时动作选择的能力。该代码可在此 https 网址获取。

PrT-Flow: Probability-Tractable Flow Matching for Reinforcement Learning

PrT-Flow：强化学习中的概率可处理流匹配

• Authors: Congzhong Sun, Ning Tan
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17667
• Pdf link: https://arxiv.org/pdf/2610.17667
• Abstract Diffusion models and Flow Matching can represent flexible distributions over robot actions, but many reinforcement learning algorithms also require the probability of each action. We introduce PrT-Flow (Probability-Tractable Flow Matching), which computes probability density while retaining multimodal, multi-step action generation. Its Jacobian-Tractable Transformer (JTT) makes each sampling step invertible and enables efficient computation of its Jacobian determinant. We train these finite steps with Conditional Euler-step Training (CET), which supervises their displacement over one step during sampling rather than their instantaneous velocity. We prove that, under certain assumptions, CET has the same minimizers as fitting the Euler-solver updates of Flow Matching. PrT-Flow achieves 96% success on Robomimic Lift after 300 epochs of imitation learning, matching the diffusion policy reference, and 96% success on OGBench Cube-Triple after 600k online steps. These results demonstrate that invertible networks with tractable probability densities can learn effective robot policies and support SAC-style actor-critic training. Our code is available at this https URL .
• 中文摘要 扩散模型和流匹配可以表示机器人动作上的灵活分布，但许多强化学习算法也要求每个动作的概率。我们引入了PrT-Flow（概率可处理流匹配），该算法在保留多模、多步动作生成的同时计算概率密度。其雅可比可运变换器（JTT）使每个采样步骤可逆，并实现高效计算其雅可比行列式。我们用条件欧拉步训练（CET）训练这些有限步，该训练法监督采样过程中其在某一步的位移，而非瞬时速度。我们证明，在某些假设下，CET的最小化器与拟合流匹配的欧拉求解器更新相同。PrT-Flow在Robomimic Lift中，经过300个模拟学习阶段，匹配扩散策略参考，成功率为96%;在OGBench Cube-Triple上，在线步数达到96%，共60万步。这些结果表明，具有可处理概率密度的可逆网络可以学习有效的机器人策略，并支持SAC风格的actor-critic训练。我们的代码可在此 https URL 获取。

Cowork-Gym: Synthesizing Interactive Work Environments at Scale

Cowork-Gym：大规模综合互动工作环境

• Authors: Anni Zou, Cheng Chen, Jiaji Deng, Weikang Zhou, Xinji Mai, Yue Wu, Jianing Lin, Lipeng Xie, Lixiong Qin, Qingyue Zhao, Que Shen, Shuchang Tao, Xuchen Pan, Yaliang Li, Yunpeng Zhai, Yushuo Chen, Zhihai Wang, Zhuo Zhang, Zhaoyang Liu, Bolin Ding, Jingren Zhou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17670
• Pdf link: https://arxiv.org/pdf/2610.17670
• Abstract Cowork agents navigate dependencies across tools and files while adapting to changing user requirements and edits to shared workspaces. Scaling such environments requires maintaining consistency among task objectives, executable resources, user behavior, and success criteria. We introduce Cowork-Gym, a framework that synthesizes Cowork environments through a build-execute-revise loop that uses execution evidence and review feedback to validate and repair generated environments. To capture realistic Cowork dynamics, a stateful user simulator responds to the agent's actions through dialogue and direct edits to the shared workspace. We synthesize 64.9K environments across 14 application scenarios and use them to post-train 9B, 27B, and 35B-A3B Qwen models with supervised fine-tuning, reinforcement learning, and on-policy distillation. Averaged across six agentic benchmarks, these models outperform their corresponding base checkpoints by 8.7, 6.4, and 4.1 points, respectively. Ours-27B ranks second overall among the evaluated models and achieves the highest average scores on PawBench and ClawEval, outperforming models with over 100B total parameters on both benchmarks.
• 中文摘要 Cowork 代理在工具和文件间导航依赖关系，同时适应用户需求的变化和共享工作空间的编辑。扩展此类环境需要保持任务目标、可执行资源、用户行为和成功标准之间的一致性。我们介绍 Cowork-Gym，这是一个通过构建-执行-修订循环综合 Cowork 环境的框架，利用执行证据和反馈审查，验证和修复生成环境。为捕捉真实的 Cowork 动态，有状态用户模拟器通过对话响应代理的操作，并直接编辑共享工作空间。我们综合了 14 个应用场景中的 64.9K 环境，并用它们进行 9B、27B 和 35B-A3B Qwen 模型的后期训练，支持监督微调、强化学习和策略内蒸馏。这些模型在六个代理基准测试中平均，分别比对应的基础检查点高出 8.7、6.4 和 4.1 分。我们的27B在评估模型中排名第二，并在PawBench和ClawEval中取得最高平均分，优于两个基准总参数均超过100B的模型。

Adaptive-Group Process Rewards for GUI Agents

GUI代理的自适应组过程奖励

• Authors: Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu, Lifeng Fan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17710
• Pdf link: https://arxiv.org/pdf/2610.17710
• Abstract Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising direction for credit assignment to intermediate actions; however, accurately evaluating an action requires identifying evidence across long-horizon GUI trajectories, as its value may depend on information beyond its immediate state and observation. In this paper, we propose an adaptive-group process reward framework that dynamically partitions a GUI trajectory into semantically coherent subgoal groups, enabling process reward estimation at both the action and group levels. At the action level, a critic evaluates each action based on the visual changes and subsequent actions within the group for action correctness and contribution to the subgoal. At the group level, the framework aggregates action-level evaluations to assess subgoal completion and contribution to the overall task, and further refines these assessments by leveraging the context of all groups in the trajectory. The process reward of each action is obtained by combining its action-level evaluation with the refined group-level evaluation. We further augment GRPO's outcome advantage with the proposed process rewards to provide finer-grained guidance for GUI agent training. Experiments on process reward benchmarks and downstream RL training demonstrate the effectiveness of the proposed framework.
• 中文摘要 用强化学习训练图形用户界面（GUI）代理时，结果奖励稀疏，导致学习效率低下。过程奖励为中间动作的功劳分配提供了有前景的方向;然而，准确评估一个动作需要在长视野的图形界面轨迹中识别证据，因为其价值可能依赖于其当前状态和观察之外的信息。本文提出了一种自适应组过程奖励框架，将GUI轨迹动态划分为语义连贯的子目标组，从而实现动作和组级的过程奖励估计。在动作层面，批评者根据组内视觉变化及后续动作评估每个动作的正确性和对子目标的贡献。在群体层面，该框架汇总行动级评估以评估子目标的完成度和对整体任务的贡献，并通过利用轨迹中所有组的上下文进一步优化这些评估。每个动作的过程奖励通过将其动作级评估与细化后的组级评估结合获得。我们进一步通过提出的过程奖励增强GRPO的结果优势，为图形界面代理训练提供更细粒度的指导。过程奖励基准和下游强化学习训练的实验展示了该框架的有效性。

TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training

TSRL：基于触发的监督强化学习用于持续智能体训练

• Authors: Yuqian Fu, Kaiwen Jiang, Zijun Song, Jiawei Wang, Yuanheng Zhu, Dongbin Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17726
• Pdf link: https://arxiv.org/pdf/2610.17726
• Abstract Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient but prone to severe forgetting, while Reinforcement Learning (RL) preserves prior capabilities but converges slowly without expert guidance. To bridge this gap, we introduce Trigger-based Supervised Reinforcement Learning (TSRL), a framework that integrates the strengths of both paradigms through an adaptive, state-dependent expert supervision mechanism. TSRL employs RL as the optimization backbone, activating SFT selectively on "hard states" where the policy fails. Theoretically, we show that restricting supervision to failure states derives an upper bound on the cumulative parameter updates, thereby reducing forgetting compared to full SFT. Empirically, TSRL achieves consistent gains across three agent benchmarks (ALFWorld, WebShop, and Deep Search) at both 7B and 32B scales. In continual learning scenarios, our method retains up to 9.6 points more capability on prior tasks than SFT->RL baselines, while matching or exceeding target-task performance.
• 中文摘要 大型语言模型（LLM）代理在复杂多步任务中表现出色，但在持续学习过程中容易出现灾难性遗忘——获得新技能往往会削弱先前掌握的能力。现有的训练后范式面临一个根本性权衡：监督式微调（SFT）样本效率高但易严重遗忘，而强化学习（RL）保留了先前能力，但在没有专家指导的情况下收敛较慢。为弥合这一差距，我们引入了基于触发器的监督强化学习（TSRL），该框架通过自适应、状态依赖的专家监督机制整合了两种范式的优势。TSRL将强化学习作为优化骨干，在策略失败的“硬状态”上选择性激活SFT。理论上，我们证明将监督限制在失败状态下可以导出累计参数更新的上限，从而相比完整SFT减少遗忘。从经验中看，TSRL在3B和32B尺度上，在三个代理基准测试（ALFWorld、WebShop和Deep Search）上均取得了稳定的提升。在持续学习场景下，我们的方法在先前任务上比SFT->RL基线多保留多达9.6个百分点的能力，同时能匹配或超越目标任务的性能。

CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning

CoCA：基于评分标准的强化学习标准感知学分作业

• Authors: Guangcheng Zhu, Yingfan Ma, Xing Zheng, Haobo Wang, Shenzhi Yang, Kai Tang, Zhengqing Zang, Bowen Song, Weiqiang Wang, Gang Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17733
• Pdf link: https://arxiv.org/pdf/2610.17733
• Abstract Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose CoCA, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of +4.2% over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on open-ended tasks.
• 中文摘要 基于评分标准的强化学习（RL）通过明确标准评估反应以构建奖励，提供了一种可行的方式来训练大型语言模型进行开放式任务。然而，标准方法将标准级反馈压缩为标量奖励，并分配统一的代币优势，导致结构化评分标准的监督未被充分利用。近期研究通过将评分标准视为特权信息以推导自我提炼目标，寻求更细致的指导，但所得学习信号不基于标准结果，可能导致奖励退化。这促使我们将问题重新框定为标准感知学分分配，特权评分标准信息作为学分分配的相关性线索，而非模仿目标的来源。因此，我们提出了CoCA，它从群体相对结果中获得标准层级的学分，并根据标准诱导的分布变化在各代币间分配，从而产生基于结果和细粒度的学习信号。从实证角度看，CoCA在医学和科学基准中平均比标准基于评分标准的强化学习获得+4.2%，展示了在开放式任务中更有效的学习效果。

Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning

用记忆探索，不用记忆学习：LLM推理的上下文探索与自我提炼

• Authors: Yutong Chen, Zhichao Xu, Rui Liu, Kunpeng Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17734
• Pdf link: https://arxiv.org/pdf/2610.17734
• Abstract Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: this https URL
• 中文摘要 探索通过可验证奖励（RLVR）限制了强化学习的学习效率。目前的方法主要通过独立抽样进行探索，未充分利用过程中收集的信息，如已知错误答案和环境反馈。我们提出LEAF，在每次部署前注入动态、针对问题的记忆。记忆包含最近的响应、奖励及其他环境反馈。每次验证调用后都会更新。我们将策略更新与推广解耦，并通过自我蒸馏更新策略。在四个基准测试和两个开放模型中，LEAF相比随机抽样基线的探索速度约为4倍。在未见测试集上，LEAF在最强基线上分别提升了1.59个百分点和1.83个百分点的准确性，同时计算量低于任何强化学习基线。这些结果凸显了LEAF提升探索和学习效率的潜力。代码可在以下 https URL 获取

Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders

许多正确答案，一个培训示例：为LLM推荐者移除GRPO中的响应者噪声

• Authors: Xin Li, Hao Jiang, Jinghao Guo, Weisi Lin, Chau Yuen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.17771
• Pdf link: https://arxiv.org/pdf/2610.17771
• Abstract A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.
• 中文摘要 Reddit上的推荐请求会抽取多个回复，每个回复命名不同的标题。当LLM推荐者在这些线程上接受强化学习（RL）训练时，生成的列表应针对哪个回复进行奖励？最先进的方案Rank-GRPO为每个训练示例选择一个回复，因此优势主要记录了哪条回复被抽取（占方差的85%），而非列表的优劣。我们主张强化学习应针对请求本身训练，而非每个答案，并提出ReMaRk：它对每个列表进行一次评分，对所有回复标题的并集，并对每个请求均等采样。这使得强化学习更便宜。在七个骨干网和三个数据集上，ReMaRk在该预算下，每条回复中2000个强化步骤的验证回忆率达到25%到53%的步骤和潜在客户;训练到双方都没有进步，从不明显落后于Rank-GRPO，七对中有四对能以更少的GPU小时达到Rank-GRPO的最佳水平。同样的奖励在ProtoQA中也有帮助，大约有100人回答每个问题，这也指向一个简单的规则，包含多个有效的人类答案：共同奖励他们。

Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

超越安全执行：基于价值校准和自适应指导的自动驾驶安全在线强化学习

• Authors: Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Zhiqiang Zuo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.17799
• Pdf link: https://arxiv.org/pdf/2610.17799
• Abstract Online reinforcement learning (RL) enables driving policies to adapt to changing deployment conditions, but requires safe exploration throughout training. Safety filtering alone does not ensure stable policy improvement or reduced dependence on corrections. We propose Safe Online Adaptive Refinement (SOAR-Flow), a framework for safe and stable online fine-tuning of offline-initialized Flow-based driving policies. A control barrier function (CBF)--model predictive control (MPC) filter corrects potentially unsafe actions before execution. To support stable policy updates, we introduce Gaussian-return-aware value calibration based on calibrated Q-learning (Cal-QL), using recorded behavior returns and predicted return variability to smooth conservative critic updates and mitigate policy degradation. Paired corrective learning teaches feasible action adjustments, while an intervention-to-go predictor encourages decisions requiring fewer future corrections. Its predictions also regulate guidance strength, allowing recovery supervision to weaken as predicted dependence decreases. Together with gradually reduced calibration regularization, this supports continued task improvement while preserving hard execution constraints. We analyze conditions for safe execution at control steps and diminishing auxiliary learning influences. We evaluate SOAR-Flow in simulation and on an uncrewed ground vehicle (UGV). The results demonstrate improved driving performance and reduced reliance on safety corrections during protected online fine-tuning. The demonstration video and project overview are available at this https URL.
• 中文摘要 在线强化学习（RL）使驾驶策略能够适应不断变化的部署条件，但需要在培训过程中进行安全探索。仅靠安全过滤无法确保策略的稳定改进或减少对修正的依赖。我们提出了安全在线自适应细化（SOAR-Flow），这是一个用于安全稳定地对离线初始化的基于流的驾驶策略进行在线微调的框架。控制障碍函数（CBF）-模型预测控制（MPC）过滤器在执行前修正潜在的不安全行为。为支持稳定的策略更新，我们引入基于校准Q-学习（Cal-QL）的高斯返回感知值校准，利用记录的行为返回和预测返回变异，平滑保守批评者更新并减轻政策退化。配对纠正学习教授可行的行动调整，而干预即用预测器鼓励决策，减少未来修正次数。其预测还调节制导强度，使得随着预测依赖性降低，恢复监督得以减弱。结合逐步降低校准正则化，这支持任务持续改进，同时保持严格执行约束。我们分析控制步骤安全执行条件及辅助学习影响减弱。我们在模拟和无人地面车辆（UGV）上评估了SOAR-Flow。结果显示了在受保护的在线微调过程中，驾驶性能提升并减少了对安全修正的依赖。演示视频和项目概述可在此 https URL 访问。

Keyword: diffusion policy

DexCast: Learning Real-World Dexterous Control from Human Videos

DexCast：从人类视频中学习真实世界的灵巧控制

• Authors: Zihan Gao, Yuhong Zhang, Mengzhe Wang, Kaisheng Liu, Shengpeng Li, Siyao Xiao, Runqing Cheng, Xiao Lin, Zhifang Liu, Haoqian Wang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.14691
• Pdf link: https://arxiv.org/pdf/2610.14691
• Abstract Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.
• 中文摘要 三维手与物体交互（HOI）感知的最新进展为从人类视频学习灵巧机器人技能打开了新机遇。然而，现有灵巧模仿学习方法往往依赖昂贵且手动的机器人远程操作演示，这需要大量人力。因此，我们构建了DEXCAST框架，通过利用关节HOI重建，直接从单眼RGB-D视频学习灵巧的机器人手部控制，无需任何远程操作。此外，如何建模高维机器人手部运动并提取时间上下文以指导未来动作生成仍是关键挑战。为此，我们提出了一个DexHand运动感知模块（DMPM），用于在紧凑的潜在表征中捕捉关节间的协调和时间依赖性，从而决定我们的扩散策略。大量真实世界实验表明，DEXCAST在多种灵巧操作任务中表现出色。

Smoothness Ranks Skill, Not Success: An Audit of Published Trajectory-Smoothness Curation Metrics Under Episode-Length Controls

平滑度排名技能，而非成功：集长控制下已发布轨迹平滑度策划指标的审计

• Authors: Jaiveer Bassi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.17427
• Pdf link: https://arxiv.org/pdf/2610.17427
• Abstract Trajectory smoothness has been proposed as a cheap, training-free signal for curating imitation-learning demonstrations, most recently by rinse, which introduces Spectral Arc Length (SAL) and a contact-aware Trajectory-Envelope Distance (TED). We test whether these published metrics survive episode-length controls on real robomimic data. On can/mh, SAL separates better from worse operators with AUROC 0.952, but episode length alone achieves 0.957, and inside a duration-matched band SAL falls to 0.628 while raw mean jerk, the baseline rinse argues against, rises to 0.896. The confound replicates on square/mh. On can/mg, where every episode is exactly 150 steps, SAL is at chance (0.454) and TED is inverted (0.376) against task success, while jerk reaches 0.706. Training GMM behavior-cloning policies on each metric's curated subset (10 seeds), curating by TED costs roughly a third of downstream success relative to random selection of the same size (0.240 vs. 0.360, p=0.0001, surviving Bonferroni correction), while SAL and jerk are indistinguishable from not curating. Under a diffusion policy TED still ranks worst, though its deficit is not resolvable at five seeds, and jerk becomes significantly better than both baselines, so the curation decision does not transfer unchanged across the two policy configurations we tested. Inspecting trajectories shows why TED inverts: successful manipulation requires sustained high-deviation motion through contact, while failure is low-amplitude hovering that a smoothed envelope follows closely.
• 中文摘要 轨迹平滑性被提出作为一种廉价、无训练的信号，用于策划模仿学习演示，最近的方法是rinse，引入了谱弧长度（SAL）和接触感知轨迹-包络距离（TED）。我们测试这些已发表指标是否能在真实机器人模拟数据下的发作长度控制中存活。在can/mh上，SAL以AUROC 0.952分出较差算子，但仅发作长度可达0.957，在时长匹配带内SAL降至0.628，而原始平均抖动（Prim Rinse反对）升至0.896。混杂现象在平方/mh上重复。在 can/mg 中，每集恰好有 150 步，SAL 处于偶然率（0.454），TED 与任务成功率相反（0.376），而 jerk 则达到 0.706。在每个指标的策划子集（10个种子）上训练 GMM 行为克隆策略，由 TED 策划，相较于相同规模的随机选择，其后游成功率大约损失三分之一（0.240 对 0.360，p=0.0001，通过 Bonferroni 修正存活），而 SAL 和 jerk 与未策划无异。在扩散策略下，TED 仍排名最差，尽管其缺口无法在五个种子时解决，且 jerk 显著优于两个基线，因此策展决策不会在我们测试的两种策略配置中保持不变。检查轨迹展示了TED反转的原因：成功的操控需要通过接触持续高偏差运动，而失败则是低幅度的悬停，并紧随一个平滑包络。

PrT-Flow: Probability-Tractable Flow Matching for Reinforcement Learning

PrT-Flow：强化学习中的概率可处理流匹配

• Authors: Congzhong Sun, Ning Tan
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.17667
• Pdf link: https://arxiv.org/pdf/2610.17667
• Abstract Diffusion models and Flow Matching can represent flexible distributions over robot actions, but many reinforcement learning algorithms also require the probability of each action. We introduce PrT-Flow (Probability-Tractable Flow Matching), which computes probability density while retaining multimodal, multi-step action generation. Its Jacobian-Tractable Transformer (JTT) makes each sampling step invertible and enables efficient computation of its Jacobian determinant. We train these finite steps with Conditional Euler-step Training (CET), which supervises their displacement over one step during sampling rather than their instantaneous velocity. We prove that, under certain assumptions, CET has the same minimizers as fitting the Euler-solver updates of Flow Matching. PrT-Flow achieves 96% success on Robomimic Lift after 300 epochs of imitation learning, matching the diffusion policy reference, and 96% success on OGBench Cube-Triple after 600k online steps. These results demonstrate that invertible networks with tractable probability densities can learn effective robot policies and support SAC-style actor-critic training. Our code is available at this https URL .
• 中文摘要 扩散模型和流匹配可以表示机器人动作上的灵活分布，但许多强化学习算法也要求每个动作的概率。我们引入了PrT-Flow（概率可处理流匹配），该算法在保留多模、多步动作生成的同时计算概率密度。其雅可比可运变换器（JTT）使每个采样步骤可逆，并实现高效计算其雅可比行列式。我们用条件欧拉步训练（CET）训练这些有限步，该训练法监督采样过程中其在某一步的位移，而非瞬时速度。我们证明，在某些假设下，CET的最小化器与拟合流匹配的欧拉求解器更新相同。PrT-Flow在Robomimic Lift中，经过300个模拟学习阶段，匹配扩散策略参考，成功率为96%;在OGBench Cube-Triple上，在线步数达到96%，共60万步。这些结果表明，具有可处理概率密度的可逆网络可以学习有效的机器人策略，并支持SAC风格的actor-critic训练。我们的代码可在此 https URL 获取。