生成时间: 2026-10-28 22:57:11 (UTC+8); Arxiv 发布时间: 2026-10-28 20:00 EDT (2026-10-29 08:00 UTC+8)

今天共有 43 篇相关文章

Keyword: reinforcement learning

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

以课程为驱动的深度强化学习，用于自主远程火箭着陆

• Authors: Atin Chowdhury, Piyush Joshi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.35954
• Pdf link: https://arxiv.org/pdf/2610.35954
• Abstract Autonomous vertical landing of reusable launch vehicles is a critical yet formidable challenge in aerospace engineering, primarily due to highly nonlinear dynamics, stringent precision require- ments, and inherent environmental uncertainties. While reinforcement learning (RL) has emerged as a promising alternative to traditional model-based control, existing approaches frequently rely on flat training paradigms and highly simplified initial conditions, drastically limiting their real-world applicability. This paper proposes a comprehensive curriculum-driven deep reinforcement learning framework designed to master both far-range navigation and precision landing. By employing a three-stage curriculum augmented with a critic warm-up procedure and a rehearsal mechanism to prevent catastrophic forgetting, alongside a high-dimensional observation vector utilizing a full 3 X 3 rotation-matrix, the agent is trained to navigate from an initial horizontal distance of 1,000 to 1,500 units vastly farther than prior benchmarks before executing a soft touchdown. Crucially, we intro- duce a redesigned reward function that reduces unnecessary computational overhead and addresses limitations of earlier formulations, including poorly calibrated reward components and potentially conflicting optimization objectives. The redesigned function bounds the per-step reward magnitude by (O(1)), while its progress term accumulates the total distance closed over the trajectory. The available 100,000-episode results report a 99.98% success rate for the proposed policy and 68.31% for the baseline under the evaluation conditions. In the evaluation, the proposed policy retained 69.1% of its initial fuel capacity and achieved a mean final distance from the pad centre of 7.07 units, compared with 13.10 units for the baseline. Landing visualizations are available in Video 1 and Video 2.
• 中文摘要 可重复使用运载火箭的自主垂直着陆是航空航天工程中一项关键但艰巨的挑战，主要由于高度非线性动力学、严格的精度要求以及固有的环境不确定性。尽管强化学习（RL）已成为传统基于模型控制的有前景替代方案，但现有方法常依赖平坦训练范式和高度简化的初始条件，严重限制了其现实世界的适用性。本文提出一个全面的课程驱动深度强化学习框架，旨在掌握远程导航和精确着陆。通过采用三阶段课程，辅以批评热身程序和排练机制以防止灾难性遗忘，配合高维观察向量，使用完整的3 X 3旋转矩阵，智能体被训练能够从初始水平距离1000到1500单位远远导航，然后执行软着陆。关键是，我们引入了重新设计的奖励函数，减少了不必要的计算开销，解决了早期表述的局限性，包括奖励组件校准不当和可能冲突的优化目标。重新设计的函数将每步奖励幅度限制为（O（1）），其进度项累积轨迹上关闭的总距离。现有的10万集报告显示，在评估条件下，拟议政策的成功率为99.98%，基线成功率为68.31%。评估中，拟议政策保留了初始燃料容量的69.1%，并实现了距离发射台中心的平均最终距离7.07单位，而基线为13.10单位。着陆可视化可在视频1和视频2中提供。

Learning and Reasoning with Geometry: How Generation Helps Spatial Understanding

几何学的学习与推理：生成如何帮助空间理解

• Authors: Rui Yang, Yanwei Li, Sili Chen, Jingjia Huang, Yi Lin, Hengshuang Zhao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.36022
• Pdf link: https://arxiv.org/pdf/2610.36022
• Abstract Image and video generators can produce spatially consistent visual content, but how to use this capability to improve spatial understanding remains underexplored. We study this problem from two perspectives: geometric supervision during training and explicit geometric evidence during inference. To this end, we introduce a unified geometric understanding and generation model (GeoUG) that integrates depth and multi-view generation with spatial understanding. First, we design a multi-scale depth codec that enables metric-depth generation using a pretrained visual autoencoder, and show that joint training yields task-dependent gains in spatial understanding. Second, we introduce 3DCueBench to control geometric difficulty while balancing selected 2D cues, and find that generated depth provides larger reasoning gains on questions requiring fine-grained geometric discrimination. Building on this finding, we develop adaptive interleaved reasoning, in which GeoUG first drafts an answer and then decides whether to generate geometric evidence to confirm or revise it. We train this decision through reinforcement learning, balancing accuracy gains over retaining the draft against generation cost. GeoUG achieves circular accuracies of 97.9%, 90.8%, and 94.2% on CV-Bench-3D, 3DCueBench, and DA-2K with fewer generation calls than always generating.
• 中文摘要 图像和视频生成器可以生成空间一致的视觉内容，但如何利用这一能力提升空间理解仍未被充分探索。我们从两个角度研究该问题：训练时的几何监督和推理中的显式几何证据。为此，我们引入了一个统一的几何理解与生成模型（GeoUG），将深度和多视角生成与空间理解整合在一起。首先，我们设计了多尺度深度编解码器，利用预训练的视觉自编码器实现度量深度生成，并展示了联合训练能在空间理解上实现任务依赖性提升。其次，我们引入3DCueBench，在平衡选定二维线索的同时控制几何难度，发现生成深度在需要细粒度几何辨别的问题上提供了更大的推理提升。基于这一发现，我们开发了自适应交错推理，GeoUG 先草拟答案，然后决定是否生成几何证据以确认或修改。我们通过强化学习训练该决策，平衡保留草稿的准确性提升与生成成本。GeoUG 在 CV-Bench-3D、3DCueBench 和 DA-2K 上实现了 97.9%、90.8% 和 94.2% 的循环准确率，且生成调用次数比以往少。

Jaxnasium: Embracing PyTrees For Any-Agent, Cross-Suite Reinforcement Learning

Jaxnasium：拥抱PyTrees实现任何代理、跨套件强化学习

• Authors: Koen Ponse, Aske Plaat, Thomas M. Moerland
• Subjects: Subjects: Machine Learning (cs.LG); Digital Libraries (cs.DL)
• Arxiv link: https://arxiv.org/abs/2610.36075
• Pdf link: https://arxiv.org/pdf/2610.36075
• Abstract Hardware-accelerated reinforcement learning in JAX has gained rapid popularity in recent years and has proven to be a successful method of scaling online reinforcement learning. The resulting ecosystem, however, is fragmented: environment suites define their own interfaces, and algorithm implementations are typically written for a single suite, a single type of space, and either JAX-native or host-based environments. Here we present Jaxnasium, a library for JAX-based reinforcement learning, aiming to make the field more accessible. Jaxnasium represents observation and action spaces as PyTrees, and uses this structure to build networks and support multi-agent processing. Through this, its near-single-file implementations run on discrete, continuous, image, and composite spaces, as well as in multi-agent environments, without any changes to the algorithm code. The same files serve as readable, research-friendly algorithms, as well as importable algorithms for more applied workloads. Through a minimal Gymnasium-style interface and wrappers for 11 suites, over 300 environments are available on which a small PPO implementation with default hyperparameters is able to run. Additionally, through a simple wrapper, we can run Gymnasium and EnvPool environments with the same algorithm and training loop, with no performance loss compared to architectures specifically built for training JAX algorithms on host-based environments. Jaxnasium further provides tools for multi-seed evaluation, hyperparameter sweeps, and project scaffolding. Jaxnasium is open-sourced at this https URL
• 中文摘要 近年来，JAX中的硬件加速强化学习迅速流行，并被证明是在线强化学习扩展的成功方法。然而，最终形成的生态系统是碎片化的：环境套件定义自己的接口，算法实现通常为单一套件、单一类型空间编写，且环境要么是基于JAX原生，要么基于主机。这里介绍Jaxnasium，一个基于JAX的强化学习库，旨在使该领域更易接近。Jaxnasium将观察和动作空间表示为PyTree，利用该结构构建网络并支持多智能体处理。通过这种方式，其近单文件实现可在离散、连续、图像和复合空间以及多智能体环境中运行，且无需修改算法代码。这些文件既可读，也适合研究，同时也可导入适用于更应用工作负载的算法。通过简约的Gymnasium风格接口和11个套件的包装器，超过300个环境支持带有默认超参数的小型PPO实现。此外，通过简单的包装器，我们可以用相同的算法和训练循环运行Gymnasium和EnvPool环境，性能与专门为主机环境中训练JAX算法设计的架构相比无损耗。Jaxnasium还提供多种子评估、超参数扫描和项目支架的工具。Jaxnasium开源地址为 https

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

冻结，不克隆：在不保留离线数据的情况下，通过表达式策略微调

• Authors: Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel Castro
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.36247
• Pdf link: https://arxiv.org/pdf/2610.36247
• Abstract Offline-to-online reinforcement learning uses offline pretraining to accelerate subsequent online learning. While many methods assume the offline dataset remains available throughout fine-tuning, there are many settings where this is not the case. Prior work attributes failures in this setting primarily to critic recalibration under distribution shift. We identify a complementary actor-side failure: once the offline data disappears, continued behavioral cloning (BC) no longer preserves the offline behavior distribution and instead refits the behavioral model to the agent's limited, and nonstationary, online experience. We demonstrate that stopping online behavioral fitting while preserving the pretrained expressive prior substantially improves no-retention fine-tuning across implicit flow Q-learning (IFQL), flow Q-learning (FQL), and action-chunked RL (QC) on seven OGBench domains. We trace this failure to a behavioral feedback loop: continuing online behavioral cloning rapidly erases the pretrained prior, inducing representational collapse and action diversity loss that starves the critic of useful exploration. Freezing the prior breaks this cycle, preserving offline coverage while allowing value learning to drive continuous improvement.
• 中文摘要 离线到在线强化学习利用离线预训练加速后续在线学习。虽然许多方法假设离线数据集在微调过程中依然可用，但许多情境并非如此。先前工作将该设置的失败主要归因于分布转移下的批评者重新校准。我们识别出一个互补的行为者侧失败：一旦离线数据消失，持续行为克隆（BC）不再保留离线行为分布，而是将行为模型重新调整为代理有限且非平稳的在线体验。我们证明，在保留预训练表达先验的同时停止在线行为拟合，显著改善了七个OGBench域中隐式流Q学习（IFQL）、流Q学习（FQL）和动作块RL（QC）的无保留微调。我们将这种失败追溯到行为反馈循环：继续在线行为克隆会迅速抹去预训练的先验，导致表征崩溃和行动多样性丧失，使批评者无法获得有用的探索。冻结先前的行为打破了这一循环，保持了离线覆盖，同时让价值学习推动持续改进。

Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

桥接策略上强化学习中的探索与可塑性丧失

• Authors: Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli, Raffaello Camoriano
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.36309
• Pdf link: https://arxiv.org/pdf/2610.36309
• Abstract Deep reinforcement learning (RL) algorithms have been observed to lose the ability to adaptively learn from new data, a phenomenon termed Loss of Plasticity (LoP). While many metrics have been shown to correlate with it, the causal mechanisms and the geometric conditions from which it can arise have not been clearly identified. We analyze LoP in Proximal Policy Optimization with isotropic Gaussian policies, the state-of-the-art formulation for many continuous action spaces in RL. We report multiple metrics which characterize the effect that the policy standard deviation $\sigma$ has on LoP, and we also suggest related explanations for the effectiveness of widely adopted LoP mitigation techniques. We observe a significant effect of $\sigma$ on the extent of LoP, and accompany this with both theoretical and experimental supporting arguments. Our experimental validation consists of LoP benchmark evaluations against standard baselines as well as real-world policy deployment on a 7-DoF robotic arm.
• 中文摘要 深度强化学习（RL）算法已被观察到失去从新数据中自适应学习的能力，这一现象称为可塑性丧失（Loss of Plasticity，简称LoP）。尽管许多指标已被证明与之相关，但其因果机制和其产生的几何条件尚未被明确识别。我们在近端策略优化中分析了基于各向同性高斯策略的LoP，这是强化学习中许多连续动作空间的先进表述。我们报告了多项指标，描述了策略标准差$\sigma$对LoP的影响，并提出了广泛采用的LoP缓解技术有效性的相关解释。我们观察到$\sigma$对LoP范围有显著影响，并附有理论和实验支持论证。我们的实验验证包括对标准基线进行LoP基准评估，以及在7-DoF机械臂上的实际政策部署。

Data-Driven Hamiltonian for Neural Optimal Control

基于数据的哈密顿量用于神经最优控制

• Authors: Christopher A. Strong, Aswin Surya, Namhoon Cho, Jason J. Choi, Claire J. Tomlin
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.36378
• Pdf link: https://arxiv.org/pdf/2610.36378
• Abstract Reinforcement learning (RL) has emerged as a prominent data-driven approach to optimal control, but often requires substantial amounts of data and provides limited guarantees when encountering out-of-distribution state-action pairs. In this paper, we propose an alternative approach grounded in the well-established Hamilton-Jacobi framework in optimal control theory. Rather than requiring an explicit analytical model of the system dynamics to evaluate the Hamiltonian, we replace this with a data-driven Hamiltonian (DDH), which we generalize to a broad class of optimal control problems. In contrast to standard RL approaches, the DDH provides a principled mechanism, requiring only minimal prior system information such as a Lipschitz constant, for quantifying and accounting for uncertainty in regions not covered by the data, enabling the derivation of controllers with performance guarantees. Additional prior system knowledge, when available, can be readily incorporated to improve performance. For high-dimensional systems, we further introduce a physics-informed neural approximation of the value function, trained using a novel procedure that generates worst-case fictitious rollouts evaluated through the DDH. On an LQR problem, the DDH reaches near-optimal cost on every dataset we test, including random data on which offline RL methods remain far from optimal, and needs far fewer samples than online RL. On a nonlinear lunar lander benchmark, the DDH achieves an optimality gap to a trajectory-optimization reference more than $3\times$ smaller than CQL and FQL, and matches the performance of SAC with approximately $37\times$ fewer samples. These results establish the DDH as a promising framework for data-driven optimal control, particularly in settings where available data are limited in quantity or quality.
• 中文摘要 强化学习（RL）已成为一种重要的数据驱动最优控制方法，但通常需要大量数据，并且在遇到分布外状态-动作对时提供有限的保证。本文提出了一种基于最优控制理论中成熟哈密顿-雅可比框架的替代方法。我们不再要求显式的系统动力学分析模型来评估哈密顿量，而是用数据驱动哈密顿量（DDH）来推广到广泛的最优控制问题类别。与标准强化学习方法不同，DDH提供了一种原则性机制，只需极少的先验系统信息（如利普希茨常数），用于量化和考虑数据未覆盖区域的不确定性，从而推导出性能保证的控制器。如果有额外的系统先验知识，可以方便地整合进来提升性能。对于高维系统，我们进一步引入了基于物理的神经近似值函数，采用一种新颖的过程，通过DDH生成最坏情况虚构的展开。在LQR问题中，DDH在我们测试的每个数据集上都达到接近最优成本，包括那些离线RL方法仍远未达到最优且所需样本远少于在线RL的数据集。在非线性月球着陆器基准测试中，DDH与轨迹优化参考的最优差距比CQL和FQL小超过3美元/倍美元，且以约37美元/倍数的样本量与SAC的表现相匹配。这些结果确立了DDH作为数据驱动最优控制的有前景框架，尤其是在可用数据数量或质量有限的环境中。

Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

行为多样性强化学习中的排斥性种群政策梯度

• Authors: Saeed Hedayatian, Harsh Shah, Celine Mneimneh, Stefanos Nikolaidis
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.36425
• Pdf link: https://arxiv.org/pdf/2610.36425
• Abstract The goal of Quality-Diversity Reinforcement Learning (QD-RL) is to find a population of high-performing policies that solve a decision-making problem in diverse ways. Despite recent progress, QD-RL methods predominantly rely on evolutionary operators--including black-box optimizers, mutations, and crossovers--to facilitate exploration and induce diversity. Scaling these methods becomes increasingly costly as policies and behavior spaces grow larger. Building on the Soft QD view of quality-diversity optimization, we introduce \textbf{Re}pulsive \textbf{P}opulation \textbf{P}olicy \textbf{O}ptimization (\textbf{RePPO}), which adapts PPO to jointly optimize a compact population of diverse, high-quality policies using a population-dependent, dynamically shaped reward. On standard continuous-control tasks, RePPO achieves a strong quality-diversity tradeoff compared with state-of-the-art QD-RL methods. Ablations further characterize this tradeoff by examining how each component affects individual quality and collective diversity. To assess RePPO's scalability, we extend our evaluation to learning from pixels and fine-tuning large language models. In pixel-based tasks, where large policies make most traditional QD-RL methods infeasible, RePPO maintains quality on par with standard PPO while achieving substantially higher diversity. Its successful application to language model fine-tuning further demonstrates, for the first time, that QD-RL methods can scale to this domain. Together, these results show that directly optimizing compact populations with RePPO provides a scalable approach to QD-RL across increasingly complex domains.
• 中文摘要 质量多样性强化学习（QD-RL）的目标是找到一组高效策略，以多样化方式解决决策问题。尽管QD-RL方法近期取得进展，但主要依赖进化算子——包括黑盒优化器、突变和交叉——来促进探索和诱导多样性。随着策略和行为空间的扩大，这些方法的扩展成本也变得越来越高昂。基于软QD对质量多样性优化的观点，我们引入了\textbf{Re}pulsive \textbf{P}opulation \textbf{P}olicy \textbf{O}ptimization （\textbf{RePPO}），该方法通过调整PPO以联合优化一个紧凑且多样且高质量策略的群体，采用依赖种群、动态形状的奖励。在标准连续控制任务中，RePPO相比最先进的QD-RL方法实现了强烈的质量-多样性权衡。消融分析进一步通过考察每个组成部分如何影响个体质量和集体多样性，进一步表征这一权衡。为评估RePPO的可扩展性，我们扩展到从像素学习和微调大型语言模型。在基于像素的任务中，由于大型策略使大多数传统QD-RL方法难以实现，RePPO在实现显著更高的多样性的同时，仍保持与标准PPO相当的质量。其在语言模型微调中的成功应用首次证明了QD-RL方法能够扩展到该领域。综合来看，这些结果表明，直接用RePPO优化紧凑群体，为QD-RL在日益复杂领域提供了可扩展的方法。

Electric Vehicle Scheduling and Carbon-Aware Energy Dispatch for Charging Stations

电动汽车调度与碳意识能源调度用于充电站

• Authors: Fulong Yao, Yiming Xu, Maurizio Albano, Liana Cipcigan, Naeima Hamed, Shangyin Weng, Omer Rana
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.36426
• Pdf link: https://arxiv.org/pdf/2610.36426
• Abstract Electric vehicle (EV) charging stations integrating photovoltaic (PV) generation, battery storage, and bidirectional grid exchange must coordinate access to limited chargers with energy supply while considering both energy cost and carbon emissions. This paper proposes a solver-free greedy heuristic (SFGH) for dynamic vehicle scheduling (VS) and a twin delayed deep deterministic policy gradient (TD3) controller for carbon-aware energy dispatch (ED). SFGH assigns EVs according to reductions in service delays, and TD3 controls battery operation to balance energy costs and carbon emissions while meeting the resulting charging demand. The model allows both PV and battery exports, accounting for export revenue and carbon credits. Simulations on year-long case studies show that SFGH--TD3 achieves the lowest accumulated service delay, energy cost, and carbon emissions among the evaluated heuristic and multi-agent deep reinforcement learning (DRL) configurations.
• 中文摘要 集成光伏（PV）发电、电池储能和双向电网交换的电动汽车充电站必须协调有限充电器的使用，同时兼顾能源成本和碳排放。本文提出了无求解器的贪婪启发式（SFGH）用于动态车辆调度（VS），以及用于碳感知能源调度（ED）的双延迟深度确定性政策梯度（TD3）控制器。SFGH根据服务延迟减少分配电动汽车，TD3控制电池运行以平衡能源成本和碳排放，同时满足充电需求。该模型支持光伏和电池出口，同时考虑出口收入和碳信用。对全年案例研究的模拟显示，SFGH--TD3在评估的启发式和多智能体深度强化学习（DRL）配置中实现了最低的累计服务延迟、能源成本和碳排放。

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

CONSOR：多任务、多奖励强化学习的协调辅助奖励

• Authors: Jixuan Leng, Si Si, Fei Wang, Cho-Jui Hsieh, Jiong Zhang, Hsiang-Fu Yu, Inderjit S. Dhillon
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.36437
• Pdf link: https://arxiv.org/pdf/2610.36437
• Abstract When a Large Language Model (LLM) is trained with Reinforcement Learning (RL) on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.
• 中文摘要 当大型语言模型（LLM）在多个任务上用强化学习（RL）训练时，任务的主要奖励可以限制辅助奖励，如长度奖励：在反应失败时，辅助词不能超过主要奖励。然而，群体相对优势估计器将门控辅助词视为其他组件，因此可以反转主辅助词的顺序，只有在主项上限处才会处理指令。保留门控辅助词会损害某些任务，而放弃它们则会损害其他任务。我们提出CONCUR优势估计器，它用满足两个条件的最接近分数替换每个门控辅助词的标准化分数：主奖励较高的响应不能获得较低分数，且处于主奖励上限的反应获得相同分数。CONCUR-T在整个训练过程中应用这两个条件，CONCUR-A对与主助动顺序一致的辅助词，提升后者比在通过反应中反转主语顺序的频率更高。在两个模型家族的三个模型中，涵盖数学、代码、指令跟随和写作，这两个变体平均得分都高于所有基线，且高于放弃所有门控辅助词后。

Confident First, Reflective Second: Reinforcement Learning of Confidence-Preserving Reflection for Video Understanding

自信优先，反思其次：增强学习中保持自信的反思以促进视频理解

• Authors: Zihao Lin, Jingyuan Liu, Yucheng Zhang, Yuguang Yao, Lifu Huang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2610.36467
• Pdf link: https://arxiv.org/pdf/2610.36467
• Abstract Video models answer from sampled frames and may miss the decisive moment. A common remedy is to commit to an answer, re-examine the video, and confirm or revise it, a loop now mainly learned with reinforcement learning (RL). We find that RL erodes the initial answer, a phenomenon we term confidence erosion, where confidence means the label-verified reliability of the initial answer instead of self-reported certainty: because errors can be repaired later, a correct initial answer earns less reward, and the initial answer stalls even as the final answer's gain over it rises. We propose the principle confident first, reflective second: the initial answer must stay reliable on its own, above a floor set by a reference policy, and reflection may only add to it. Our method, CoFire, implements this principle with one mechanism for each erosion channel: reward incentives, parameter drift, and gradient interference. CoFire reaches 67.0% mean accuracy on Video-MME, MLVU, and LongVideoBench, 1.7 points above the strongest compared 7B method, and improves initial-answer and final accuracy over outcome-reward RL by 4.1 and 3.7 points on average: a stronger initial answer carries through to a stronger final one.
• 中文摘要 视频模型从采样帧中回答，可能错过决定性时刻。常见的解决方法是承诺答案，重新审视视频，并确认或修改，这一循环现在主要通过强化学习（RL）学习。我们发现，强化学习会侵蚀初始答案，这种现象称为置信度侵蚀，其中信心意味着初始答案的标签验证可靠性，而非自我报告的确定性：由于错误可以后续修复，正确初始答案获得的回报较少，初始答案在最终答案的增益增加时仍处于停滞状态。我们提出先有信心，其次反思原则：初始答案必须独立可靠，超过参考策略设定的底线，反思可能只会增加其可靠性。我们的方法CoFire通过对每个侵蚀通道采用一种机制实现该原则：奖励激励、参数漂移和梯度干扰。CoFire在Video-MME、MLVU和LongVideoBench上的平均准确率达到67.0%，比最强的7B方法高出1.7个百分点，并且在结果-奖励RL基础上平均提升了初次答案和最终准确率分别提升了4.1个和3.7个百分点：更强的初始答案会延续到更强的最终答案。

CoHARL: Co-Evolving Agent Policies and Executable Harnesses with Reinforcement Learning

CoHARL：与强化学习的共演化代理策略与可执行约束

• Authors: Jianwen Chen, Peng Xia, Kaide Zeng, Siwei Han, Zhen-Yu Zhang, Charles Fleming, Xinyu Ye, Huaxiu Yao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.36469
• Pdf link: https://arxiv.org/pdf/2610.36469
• Abstract Large language model (LLM) agents interact with their environments through executable harnesses that govern tool use and feedback. Optimizing a harness for a fixed policy overlooks how its behavior changes during learning, while training a policy under a fixed harness leaves the interface that generates its training experience unchanged. This motivates adapting the harness throughout policy training. We introduce CoHARL, a population-based reinforcement learning framework that co-evolves a shared policy and a population of executable harnesses. The policy learns from rollouts across the population, and a proposer uses its execution traces to revise harness code during training. CoHARL achieves a 27.6% mean resolved rate on SWE-rebench, improving over harness-only search by 6.4 percentage points. Experiments on SWE-bench Verified and Terminal-Bench 2.0 show gains in software engineering and terminal-based tasks. Under matched training budgets, CoHARL also outperforms searching for a harness once before RL and alternating policy updates with revisions to a single active harness.
• 中文摘要 大型语言模型（LLM）代理通过可执行的约束工具使用和反馈与环境交互。为固定策略优化框架时，忽略了其在学习过程中行为的变化，而在固定框架下训练策略则保持其训练体验的接口不变。这促使其在整个策略训练过程中不断调整框架。我们介绍了CoHARL，一种基于群体的强化学习框架，协进共享策略和可执行框架群体。该策略从总体的推广中学习，提案方在训练期间利用执行痕迹修订框架代码。CoHARL在SWE-rebench上实现了27.6%的平均解决率，优于仅辅助工具搜索6.4个百分点。在SWE-bench Verified和Terminal-Bench 2.0的实验显示，软件工程和基于终端的任务在性能上有所提升。在匹配培训预算下，CoHARL在RL前一次搜索背带以及政策更新与单一主动背带的交替修订中表现优于。

AudioRSI: Recursively Self-Improving Audio Reasoners

AudioRSI：递归自我改进音频推理器

• Authors: Jiajun Fan, Roger Ren, Jingyuan Li, Rahul Pandey, Prashanth Gurunath Shivakumar, Ankur Gandhe, Yile Gu, Ge Liu, Ivan Bulyko
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.36578
• Pdf link: https://arxiv.org/pdf/2610.36578
• Abstract Reinforcement learning has made Audio Large Language Models (Audio LLMs) better reasoners, but the reward that drives it still comes from outside the model: a final-answer check, a hand-crafted process score, or an external LLM judge that stays fixed while the policy improves. We show that an Audio LLM can supply this reward itself, and can do so recursively: each improved model becomes the critic that trains its successor, so the training signal improves along with the model. We introduce AudioRSI (Recursively Self-Improving Audio Reasoners), a reinforcement learning framework in which a single model plays two roles: as an actor it generates reasoning paths, and as a critic it compares them pairwise, turning its own judgments into a process-level reward. The critic is periodically replaced by the current policy, which closes the recursive loop. Because the critic hears the same recording as the actor, it can reject reasoning that cites acoustic evidence the audio does not contain, rather than reward reasoning that merely sounds plausible. The recursion is what sustains progress: under a frozen external judge, out-of-distribution accuracy plateaus during training, while a critic refreshed from the improving policy keeps it rising. Across four out-of-distribution benchmarks, all disjoint from the AVQA training distribution, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, ranks first on MMSU, achieves a clear lead on the challenging Multi-Audio split of MMAU-Pro, and leads the 7B open-source tier on MMAR among methods without external reward models. With no reward specifying how to reason, AudioRSI also develops structured reasoning behaviors (logical reasoning, perception-aware analysis, and reduced hallucination), corroborated by an external LLM-as-Judge evaluation and a keyword-based behavior analysis.
• 中文摘要 强化学习使音频大型语言模型（Audio LLM）成为更优秀的推理者，但驱动其的奖励仍来自模型外部：最终答案检查、手工制作的过程评分，或在策略改进时保持固定的外部LLM评判。我们展示了音频LLM能够自行提供这种奖励，并且可以递归地实现：每个改进的模型都成为训练其继任的批评者，因此训练信号与模型一同提升。我们引入了AudioRSI（递归自我改进音频推理器），这是一种强化学习框架，其中单个模型扮演两种角色：作为行为者生成推理路径，作为批评者，它成对比较这些路径，将自身判断转化为过程级奖励。批评者会周期性地被当前策略替代，从而闭合递归循环。由于批评者听到的录音与演员相同，它可以拒绝引用音频中未包含的声学证据的推理，而非仅仅听起来合理的奖励推理。递归是推动进步的关键：在冻结的外部评判下，非发行准确率在训练中趋于平稳，而从改进策略中恢复的批评者则保持准确率上升。在四个非发行基准测试中，这些基准都与AVQA训练分布不相干，AudioRSI在MMAU测试-mini上表现优于包括Audio Flamingo 3和Gemini 2.5 Pro在内的强基线，在MMU-Pro的多音频分割中取得明显领先，并在MMAR开源7B层级中领先无外部奖励模型的方法。在没有明确说明如何推理的奖励下，AudioRSI 还开发结构化推理行为（逻辑推理、感知感知分析和减少幻觉），并由外部的 LLM 作为评判者评估和基于关键词的行为分析加以支持。

InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?

InfraDev：编码代理能开发AI基础设施吗？我们如何判断？

• Authors: Yangshen Deng, Jiaqi Ruan, Yeqi Huang, Nick Lau, Guangming Sheng, Luo Mai
• Subjects: Subjects: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2610.36589
• Pdf link: https://arxiv.org/pdf/2610.36589
• Abstract Coding agents are beginning to develop the production-grade AI infrastructure that trains and serves large models, yet existing benchmarks cover only single kernels or single-GPU tasks, and grade with tests and prompts that misjudge both correct and incorrect patches. We introduce InfraDev, a benchmark of 36 tasks built from real-world pull requests to Megatron-LM, vLLM, and verl, covering infrastructure development for training, inference, and reinforcement learning. Each task asks an agent to reimplement a real change that spans many files and must run correctly across many GPUs. To grade such changes, we jointly synthesize hidden probes, which enforce the requirements the original tests leave unchecked, and a prompt that states every interface the probes call without revealing the solution. We also design a shared execution environment that reuses dependencies and artifacts across tasks, cutting storage by 52-57% and build time by 13-14% on Megatron-LM and vLLM. Across six frontier models, the best resolves only 38.9% of the tasks; the probes reject 32% of the submissions that pass every official test, and with the original pull request descriptions as prompts, no model resolves a single task. Failure analysis shows that structural and state errors dominate and that agents' own tests rarely check memory, time, or traffic. Turning these findings into an agent skill raises the resolved rate of a frontier model on the Megatron-LM tasks by 10.5%.
• 中文摘要 编码代理开始开发生产级AI基础设施，用于训练和服务大型模型，但现有基准仅覆盖单内核或单GPU任务，并通过测试和提示错误判断正确和错误补丁进行评分。我们介绍InfraDev，这是一个由真实拉取请求构建的36个任务基准测试，涵盖Megatron-LM、vLLM和verl的基础设施开发，涵盖训练、推理和强化学习的基础设施开发。每个任务都要求代理重新实现跨多个文件且必须在多个GPU上正确运行的真实变更。为对这些变更进行评分，我们联合合成隐藏探针，执行原始测试未勾选的要求，以及一个提示，说明探针调用的每个接口而不暴露解决方案。我们还设计了一个共享执行环境，跨任务重用依赖关系和伪造物，在Megatron-LM和vLLM上节省52-57%的存储空间，减少13-14%的构建时间。在六个前沿模型中，最佳模型仅解决了38.9%的任务;探测器拒绝了32%通过所有官方测试的提交，且以原始拉取请求描述为提示，没有模型能解决单个任务。失败分析显示结构性和状态错误占主导，且代理自身测试很少检查内存、时间或流量。将这些发现转化为代理技能，使前沿模型在Megatron-LM任务上的解决率提高了10.5%。

Learning Side-by-Side Navigation for Personal Mobility Vehicles from Human Drivers

从人类驾驶员那里学习个人出行车辆的并排导航

• Authors: Luis Yoichi Morales, Florent Ferreri
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.36726
• Pdf link: https://arxiv.org/pdf/2610.36726
• Abstract This paper presents an approach to achieve side-by-side autonomous motion with a pair of personal mobility vehicles (PMVs). Given a global path, two PMVs plan cooperatively so as to hold a social formation that lets the passengers hold a conversation, while keeping the ride safe and smooth. Each vehicle runs its own sampling-based model-predictive planner; the planners exchange nothing but the odometry each vehicle could observe of the other, and the joint decision is expressed as a pair of utility matrices over pairs of candidate trajectories, resolved by a common rule. The behavioural parameters are fitted by inverse reinforcement learning from 409 minutes of driving by ten pairs of people in human-driven PMVs in an indoor environment, and the lanes the pair can use are derived offline from the corridor itself rather than fixed by hand. A safety layer the cost cannot override executes a move only if the vehicle can still brake clear, measured from its whole body, and a vehicle that is stuck reverses along its own path and continues. In closed-loop simulation over 24 runs in the conditions the drivers were recorded in and 24 in four generated obstacle layouts, no vehicle entered a wall or a cone. The pair holds conversational formation for 60.5 and 68.9 percent of the time among obstacles (anticlockwise and clockwise) and 81.2 and 91.5 percent in a clear aisle, against the drivers' 34.0 and 36.0 percent, and 52.9 and 56.0 percent, and it matches their speed. It keeps one lateral formation where the drivers alternated between a close diagonal and a wide abreast one, and it varies far less. Experimental results are based on a differential drive kinematic simulator running on ROS 2.
• 中文摘要 本文提出了一种实现两辆个人出行车辆（PMV）并排自动行驶的方法。在给定全局路径的情况下，两辆PMV协作规划，以形成社会形态，使乘客能够进行对话，同时保持乘坐安全顺畅。每辆车运行自己的基于抽样的模型预测规划器;规划者交换的仅是每辆车可能观察到的里程计，联合决策通过一对效用矩阵表达，映射为对候选轨迹的效用矩阵，并通过共同规则解决。行为参数通过十对人驾驶PMV在室内环境中409分钟的逆强化学习来拟合，且配对可用的车道是从走廊本身离线推导的，而非手动固定。只有当车辆仍能从整体角度测量刹车清空时，安全层才能执行移动，且卡住车辆会倒车继续前进。在记录驾驶员的24次运行和4个生成障碍布局的闭环模拟中，没有车辆进入墙壁或锥体。这对车辆在障碍物间（逆时针和顺时针）中保持对话队形的比例为60.5%和68.9%，在清晰通道中为81.2%和91.5%，而司机的34.0%和36.0%以及52.9%和56.0%的队形，且速度匹配。它保持一个侧向队形，司机在紧密对角线和宽阔并排之间交替，变化较小。实验结果基于运行在ROS 2上的差分驱动运动学模拟器。

SetGRPO: Calibrating Grader-Induced Policy Optimization

SetGRPO：校准评分诱导的策略优化

• Authors: Jiale Huang, Zixu Li, Zhiwei Chen, Zhiheng Fu, Jinhe Bi, Yupeng Hu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.36807
• Pdf link: https://arxiv.org/pdf/2610.36807
• Abstract Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families.
• 中文摘要 基于评分标准的强化学习利用细粒度评分者反馈训练大型语言模型进行开放式任务，但评分随机性可能以单靠评分差异无法描述的方式改变GRPO更新。由于GRPO联合规范每个响应组内的奖励，并将所得训练系数与响应级梯度结合，评分差异可能导致整个组的系数发生变化，并引发策略依赖更新的变化。选择性执行进一步影响保留更新的方向反转风险。为此，我们提出了集合值组相对策略优化（SetGRPO）。该方法包含三个模块：（a）联合奖励集校准构建训练系数的联合不确定性集;（b） 几何引导安全恢复将候选系数映射到当前策略的局部更新几何体中，并确定每个响应组的最大安全恢复比率;（c） 选择性风险门控优化通过独立风险校准选择全局阈值，控制接受更新的条件性方向反转风险，同时保持原始GRPO目标。在可交换性下，SetGRPO为联合系数集提供有限样本边际覆盖，并跨候选阈值同时进行风险控制。在跨越三个领域的十个开放式基准测试中，SetGRPO在Qwen3.5和Llama3.2模型家族的所有比较训练方法中均值最高。

MVRCap: Towards Comprehensive Video Captioning via Hierarchical Multi-Dimensional Verifiable Rewards

MVRCap：通过层级多维可验证奖励实现全面视频字幕

• Authors: Weihuang Lin, Xianing Chen, Zhibin Wang, Congzhi Zhang, Yiwei Ma, Jiayi Ji, Miao Chen, Jun Song, YuCheng, Bo Zheng, Xiaoshuai Sun, Rongrong Ji
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.36838
• Pdf link: https://arxiv.org/pdf/2610.36838
• Abstract Despite significant progress, current models struggle to generate comprehensive video captions, constrained primarily by coarse-grained supervision, which often fails to capture spatial-temporal dynamics and the underlying logical intent. To address this issue, we introduce MVRCap, a novel framework that reformulates video captioning as a reinforcement learning task steered by structured verification. Central to our approach is a Fine-Grained Verifiable Reward mechanism, serving as dense supervision to enforce rigorous visual-semantic alignment. Specifically, we construct this reward system anchored by three complementary pillars: a Hierarchical Concept Coverage Reward utilizing our newly constructed MVRCap-QA dataset to ensure multi-dimensional completeness and semantic depth, a Factuality Verification Reward to enhance factual consistency, and a Conciseness Constraint to maximize semantic density. Furthermore, we curate the high-quality MVRCap-SFT dataset, providing structured annotations across six key dimensions, to serve as a robust foundational initialization for the RL policy in this training pipeline. Extensive experiments demonstrate that MVRCap achieves state-of-the-art performance on authoritative benchmarks including VDC, CAPability, and VC4VG.
• 中文摘要 尽管取得了显著进展，当前模型在生成全面的视频字幕方面仍挣扎，主要受粗粒度监督限制，常常无法捕捉时空动态和潜在的逻辑意图。为解决这一问题，我们引入了MVRCap，这是一个新框架，将视频字幕重新表述为一种由结构化验证引导的强化学习任务。我们方法的核心是细粒度可验证奖励机制，作为密集监督，强制执行严格的视觉语义对齐。具体来说，我们构建了以三个互补支柱为核心的奖励系统：利用新构建的MVRCap-QA数据集的层级概念覆盖奖励，确保多维完整性和语义深度;事实验证奖励，增强事实一致性;简洁性约束，最大化语义密度。此外，我们策划了高质量的MVRCap-SFT数据集，提供跨六个关键维度的结构化注释，作为该培训流程中强化学习策略的稳健基础初始化。大量实验表明，MVRCap在包括VDC、CAPability和VC4VG等权威基准测试上实现了最先进的性能。

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

当所有推广失败：用于困难问题推理的策略兼容联合解码

• Authors: Zehao Wang, Hanzhu Chen, Runquan Gui, Zhe Wang, Lin Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.36849
• Pdf link: https://arxiv.org/pdf/2610.36849
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) derives its learning signal from binary rewards via on-policy exploration and becomes a powerful paradigm for improving the reasoning of large language models. Yet it faces a learning cliff on problems beyond model's current ability: when every rollout fails, the samples reveal nothing about a correct solution, and group-relative estimators (e.g., GRPO, RLOO) collapse the update to zero advantage. Expert trajectories offer a remedy, but they are incompatible with the current policy and often lie in the student's low-probability regions, so the surviving updates demand large parameter shifts and decay into surface-level imitation rather than transferable reasoning. We argue that a useful learning signal on hard problems must be both verifiably correct and sufficiently compatible with the current policy. We propose Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding trajectories that are verified and policy-compatible by construction. These trajectories are optimized jointly with on-policy rollouts under a mixed-behavior objective that mitigates their distribution mismatch via importance sampling. Experiments across reasoning diverse datasets and agentic benchmarks demonstrate that PICO consistently outperforms standard RLVR and prior off-policy baselines.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过策略探索从二元奖励中获取学习信号，成为提升大型语言模型推理的强大范式。然而，它在模型当前能力之外的问题上面临学习悬崖：当每次推广失败时，样本无法揭示正确的解，群体相对估计器（如GRPO、RLOO）将更新压缩至零优势。专家轨迹提供了补救，但它们与当前策略不兼容，且通常处于学生的低概率区间，因此幸存的更新需要大幅参数转移，沦为表面模仿而非可迁移推理。我们认为，在困难问题上，有用的学习信号必须既可验证正确，又足够兼容当前策略。我们提出了策略兼容强化学习（PICO），该方法执行分布约束的联合解码：专家数据引导在符号层级解码，同时保持学生的相对代币概率，动态top-k机制将抽样分布限制在有限发散预算内，生成经过验证且策略兼容的轨迹。这些轨迹与策略上部署联合优化，基于混合行为目标，通过重要性抽样减轻分布不匹配。跨推理数据集和代理基准测试的实验表明，PICO始终优于标准RLVR及以往非策略基线。

Hyperparameter and Neural Architecture Search via Evolutionary Optimization for Reinforcement Learning: HPNAS-RL

通过进化优化进行超参数与神经结构搜索以实现强化学习：HPNAS-RL

• Authors: Nihal Acharya Adde, Hanno Gottschalk
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.36882
• Pdf link: https://arxiv.org/pdf/2610.36882
• Abstract In this study, we propose a dual-branch evolutionary framework that simultaneously performs Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) for Reinforcement Learning (RL) in Autonomous Driving (AD) tasks. While NAS has shown strong potential in automating architecture design for Convolutional Neural Networks (CNNs), its application to RL remains limited, especially in scenarios that require joint optimization of architectural and training dynamics. Building on our prior work on Evolutionary Multi-Objective NAS (EMNAS), the proposed framework evolves discrete network architectures and continuous hyperparameters in parallel under a unified multi-objective search strategy. In this setup, NAS targets the convolutional layers of the RL policy network - responsible for visual feature extraction, while HPO focuses on optimizing critical RL hyperparameters. To accelerate convergence and improve generalization, we incorporate parallel evaluation and a teacher-student policy transfer mechanism. Experiments in a high-fidelity driving simulator show that the joint NAS-HPO framework consistently outperforms standalone NAS, yielding policies with higher rewards, fewer parameters, and improved stability. These results highlight the benefits of jointly optimizing model structure and training dynamics for scalable and effective RL agent design.
• 中文摘要 本研究提出一个双分支进化框架，同时执行神经架构搜索（NAS）和超参数优化（HPO），用于自动驾驶（AD）任务中的强化学习（RL）。尽管NAS在卷积神经网络（CNN）架构设计自动化方面展现出强大潜力，但其在强化学习中的应用仍然有限，尤其是在需要架构与训练动态联合优化的场景中。基于我们之前关于进化多目标NAS（EMNAS）的研究，本框架在统一的多目标搜索策略下并行演进离散网络架构和连续超参数。在该配置中，NAS针对强化学习策略网络的卷积层——负责视觉特征提取，而HPO则专注于优化关键强化学习超参数。为加速收敛和提升泛化，我们引入了并行评估和师生策略转移机制。高精度驾驶模拟器的实验表明，联合NAS-HPO框架持续优于独立NAS，带来更高奖励、更少参数和更优稳定性的策略。这些结果凸显了联合优化模型结构和训练动态对于可扩展且高效的强化学习代理设计的益处。

Noise Steering: Exploiting Wake Steering and Acoustic Directivity for Noise-Constrained Wind Farm Control via Reinforcement Learning

噪声转向：利用尾迹转向和声学指向性，通过强化学习实现噪声约束风电场控制

• Authors: Martín de Frutos, Baptiste Corban, Claire Bizon Monroc, Ana Bušić, Donatien Dubuc, Jiamin Zhud, Esteban Ferrer
• Subjects: Subjects: Artificial Intelligence (cs.AI); Systems and Control (eess.SY); Atmospheric and Oceanic Physics (physics.ao-ph); Fluid Dynamics (physics.flu-dyn)
• Arxiv link: https://arxiv.org/abs/2610.36888
• Pdf link: https://arxiv.org/pdf/2610.36888
• Abstract Existing noise mitigation strategies for wind farms usually consist in reducing turbines power output, without accounting for the wake interactions between them and the noise emission directivity. This leads to over-conservative policies and unexploited wind power. Yet, turbine yaw can simultaneously affect wake interactions and the direction of noise emission, offering an additional degree of freedom for farm-level noise control. We therefore propose Noise Steering, a model-free reinforcement learning (RL) strategy that jointly controls turbine yaw angles and power setpoints to maximise farm power while satisfying a local noise constraint in a defined region. To enable training at a reduced computational cost, a low-fidelity acoustic model is coupled with the steady-state FLORIS simulator. The RL agent effectively learns how to redirect wakes and noise directivity away from the protected region while redistributing power across the farm. On a six-turbine test farm, the policy satisfies the 40 dB noise constraint throughout the full range of turbulent winds, with an average power deficit of only 2% relative to greedy unimpeded control, and up to 7.7% at high wind speeds. In comparison, the industry-standard noise-reduced operation (NRO) baseline here generates an average power loss of 17%. When deployed in the higher-fidelity this http URL simulator, the policy largely retains its performance under steady and turbulent inflow, despite differences in simulation fidelity. These results demonstrate the potential of combining yaw control and power derating through RL to improve the power-noise trade-off in wind farm operation.
• 中文摘要 现有风电场的噪声缓解策略通常通过减少涡轮机的功率输出，而未考虑它们之间的尾迹相互作用与噪声排放指向性。这导致了过于保守的政策和未被充分利用的风力。然而，涡轮偏航可以同时影响尾迹相互作用和噪声发射方向，为农场级噪声控制提供了额外的自由度。因此，我们提出了噪声引导，这是一种无模型强化学习（RL）策略，联合控制涡轮机偏航角度和功率设定点，以最大化农场功率，同时满足定义区域内的局部噪声约束。为了降低计算成本实现训练，低保真声学模型与稳态FLORIS模拟器结合使用。强化学习代理有效学习如何将尾迹和噪声指向从受保护区域重新定向，同时将电力重新分配到农场各处。在六台风机测试场上，该策略在整个湍流风域内满足40 dB噪声约束，平均功率赤字仅为2%，高风速时则高达7.7%。相比之下，这里的行业标准降噪运行基线产生了17%的平均功率损失。当部署于更高保真度的HTTP URL模拟器中时，该策略在稳定和湍流流入下基本保持性能，尽管仿真保真度存在差异。这些结果展示了通过强化环境结合偏航控制和功率降额，有助于改善风电场运行中的功率-噪声权衡。

Vision-Language Agents Can Learn in Their Own Latent Worlds

视觉语言代理可以在自己的潜在世界中学习

• Authors: Haoxi Li, Sikai Bai, Jianfei Ma, Qinglin Hou, Xiaocheng Lu, Tao Han, Jie Zhang, Song Guo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.36925
• Pdf link: https://arxiv.org/pdf/2610.36925
• Abstract Learning from experience is essential for building the next generation of agents that continually expand their capabilities. Pretrained vision-language models (VLMs) already internalize rich world knowledge, yet this knowledge is rarely leveraged to drive policy improvement through online interaction. Realizing this potential efficiently remains challenging, as policy updates shift the agent's interaction distribution, requiring the world model to adapt accordingly, while continually updating large pretrained models is computationally expensive. To address this challenge, we introduce Dwell, a model-based reinforcement learning (RL) framework that reframes a VLM as a recurrent latent transition operator, endowing it with the ability to predict, adjust, and act in latent space. Leveraging the VLM's own representations and its dual role as both world model and policy, the agent continually learns from real interactions to anticipate and evaluate action outcomes, and from latent imagination to improve its behavior. Across five interactive visual environments, Dwell achieves the strongest overall performance among trainable methods while substantially reducing the compute required for imagination, showing that effective world models for VLM agents need not reconstruct the future; they need to predict futures that are useful for action. Our code is publicly available at this https URL
• 中文摘要 从经验中学习对于构建不断扩展能力的下一代智能体至关重要。预训练视觉语言模型（VLMs）已经内化了丰富的世界知识，但这些知识很少被利用来推动在线交互的政策改进。高效实现这一潜力依然具有挑战性，因为策略更新会改变智能体的交互分布，需要世界模型相应适应，而不断更新大型预训练模型则计算成本高昂。为应对这一挑战，我们引入了Dwell，一种基于模型的强化学习（RL）框架，将VLM重新框架为一个循环潜在过渡算子，赋予其预测、调整和在潜空间中行动的能力。利用VLM自身的表征及其作为世界模型和政策的双重角色，智能体不断从真实互动中学习以预测和评估行动结果，并从潜在想象中改进行为。在五种交互式视觉环境中，Dwell在可训练方法中实现了最强的整体性能，同时大幅降低了想象所需的计算量，表明VLM代理的有效世界模型不必重建未来;它们需要预测对行动有用的未来。我们的代码在此 https URL 公开发布

PailitaoMMSearch2.0 Technical Report: Scaling Generative Retrieval to Unified End-to-End Multimodal Search

PailitaoMMSearch2.0 技术报告：将生成式检索扩展到统一端到端多模态搜索

• Authors: Xiaohan Ye, Zihan Gong, Jingqian Zhao, Lianyu Du, Yunmeng Shu, Xu Chen, Zesheng Zhou, Renchun You, Bocheng Chen, Jinsong Lan, Haihong Tang, Bo Zheng
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.36958
• Pdf link: https://arxiv.org/pdf/2610.36958
• Abstract Generative retrieval models have been increasingly adopted in search and recommendation systems. However, industrial generative retrieval models are commonly specialized to a fixed online scenario: they learn from scenario-specific interactions, support a narrow set of query forms and retrieval objectives. We study how to remove these couplings and build a unified end-to-end multimodal search system for open-ended image--text queries, heterogeneous shopping intents, and online scenarios with different resource constraints. We present \textbf{PailitaoMMSearch}, which directly generates product Semantic IDs from a raw query image and free-form instruction and scales along three dimensions. \textbf{Data scaling} uses a reusable toolset that can be composed into intent- and lifecycle-specific workflows for cold-start synthesis, quality control, real-traffic refinement, and continuous data production. \textbf{Intent scaling} progressively equips one model with e-commerce product knowledge, multiple retrieval relations, and adjustable reasoning budgets through continual pre-training (CPT), mixed-budget supervised fine-tuning (SFT), and unified reinforcement learning (RL). The RL stage combines intent-balanced optimization and Sample--Beam rollouts aligned with multi-candidate serving. \textbf{Scenario scaling} exposes quality--cost operating points across model capacity and reasoning budget. The system is deployed in multimodal search applications on Taobao Pailitao. A controlled online A/B test reports statistically significant improvements, including item page views $\textbf{+17.90\%}$, transactions $\textbf{+27.04\%}$, and gross merchandise value $\textbf{+8.94\%}$.
• 中文摘要 生成检索模型在搜索和推荐系统中日益被采用。然而，工业生成检索模型通常专注于固定的在线场景：它们从场景特定交互中学习，支持狭窄的查询表单和检索目标。我们研究如何消除这些耦合，构建统一的端到端多模态搜索系统，适用于开放式图像-文本查询、异构购物意图以及具有不同资源约束的在线场景。我们介绍 \textbf{PailitaoMMSearch}，它直接从原始查询图像和自由形式指令生成产品语义 ID，并在三维上进行扩展。\textbf{数据缩放}使用可重用的工具集，可组合成意图和生命周期专用的工作流程，用于冷启动综合、质量控制、真实流量优化和连续数据生产。\textbf{意图尺度}通过持续预训练（CPT）、混合预算监督微调（SFT）和统一强化学习（RL）逐步为一个模型配备电子商务产品知识、多重检索关系和可调推理预算。RL阶段结合了意图平衡优化和与多候选服务对齐的样本-光束推广。\textbf{场景扩展}揭示了模型容量和推理预算中的质量-成本操作点。该系统部署于淘宝排岛上的多模态搜索应用中。一个受控在线A/B测试报告了统计学显著的提升，包括条目页面浏览量$\textbf{+17.90\%}$、交易$\textbf{+27.04\%}$和总商品价值$\textbf{+8.94\%}$。

Omni-IGN: Onboard Image-Goal Navigation for Drones with Omnidirectional Vision

Omni-IGN：全方向视觉无人机的机载图像目标导航

• Authors: Zichen Yan, Rui Huang, Shenao Wang, Yucong Wang, Nuthasith Gerdpratoom, Lin Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.36968
• Pdf link: https://arxiv.org/pdf/2610.36968
• Abstract Image-goal navigation (ImageNav) requires an agent to reach a location specified by a goal image in an unknown environment. For drones, this task demands exploration, goal- directed navigation, and collision avoidance under limited onboard computing resources. Existing aerial ImageNav methods are built on single-view observations, requiring additional rotation and exploration to search for goal-relevant cues outside the field of view. In this paper, we present Omni-IGN, a fully onboard ImageNav system that integrates quad-fisheye perception, an attention-based reinforcement learning (RL) policy, and a collision-aware safety layer. The policy adaptively fuses goal-relevant features across multiple views to provide directional guidance for continuous navigation. The quad-fisheye RGB images support both ImageNav and obstacle avoidance without a dedicated depth sensor. On Habitat benchmarks, Omni-IGN achieves the best performance among continuous-action baselines. Moreover, flight tests in IsaacSim validate the effectiveness of integrating learning-based depth estimation with the action safety layer for collision avoidance. Finally, real-world experiments demonstrate zero-shot sim-to-real transfer without any fine-tuning or external localization. The code and model checkpoints will be released upon acceptance.
• 中文摘要 图像目标导航（ImageNav）要求智能体在未知环境中到达目标图像指定的地点。对于无人机，该任务要求在有限的机载计算资源下进行探索、目标导向导航和碰撞规避。现有的空中ImageNav方法基于单视角观测，需要额外的旋转和探索以在视野外寻找目标相关线索。本文介绍了Omni-IGN，一套全机载ImageNav系统，集成了四鱼眼感知、基于注意力的强化学习（RL）策略和碰撞感知安全层。该策略自适应地融合多个视角的目标相关特征，提供连续导航的方向引导。四鱼眼RGB图像支持ImageNav和障碍物规避，无需专用深度传感器。在Habitat基准测试中，Omni-IGN在连续动作基线中表现最佳。此外，IsaacSim中的飞行测试验证了基于学习的深度估计与动作安全层集成以避免碰撞的有效性。最后，真实世界实验展示了无需微调或外部定位即可实现零次模拟到真实的传输。代码和模型检查点将在接受后发布。

Guided-RLVR: Guided Exploration in Reinforcement Learning with Verifiable Rewards

引导式RLVR：带可验证奖励的强化学习引导探索

• Authors: Bo Yang, Zishuai Zhang, Weihua Li, Qinnan Zhang, Sijia Wen, Hainan Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.37041
• Pdf link: https://arxiv.org/pdf/2610.37041
• Abstract Reinforcement learning with verifiable rewards (RLVR) has shown strong empirical gains in improving the reasoning ability of large language models (LLMs) on tasks such as mathematics and programming. However, recent studies have shown that RLVR rollouts often collapse onto a narrow set of reasoning trajectories, limiting the explored reasoning space. Motivated by findings that thinking tokens at critical steps serve as critical structural elements in LLM reasoning pathways, we propose Guided-RLVR, a novel approach that augments RLVR with a lightweight controller. The controller strategically injects semantically meaningful steering tokens into the sampling distribution, encouraging the model to approach problems from multiple reasoning perspectives and thereby expanding the exploration space. Experiments on mathematical reasoning and code generation benchmarks show that Guided-RLVR consistently outperforms existing RLVR methods while producing more diverse reasoning trajectories.
• 中文摘要 带可验证奖励的强化学习（RLVR）在提升大型语言模型（LLMs）在数学和编程等任务上的推理能力方面已显示出显著的实证进展。然而，最新研究表明，RLVR的推广往往集中在狭窄的推理轨迹中，限制了可探索的推理空间。基于发现关键步骤中的思维标记是LLM推理路径中关键结构元素的发现，我们提出了Guided-RLVR，这是一种在RLVR基础上加入轻量级控制器的新颖方法。该控制器战略性地将具有语义意义的引导代币注入采样分布，鼓励模型从多角度思考问题，从而拓展探索空间。数学推理和代码生成基准测试的实验表明，Guided-RLVR在产生更多样的推理轨迹时，持续优于现有RLVR方法。

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

学会超越屏幕：从图形界面基础中实现可迁移的视觉观察

• Authors: Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong, Shengwu Xiong, Shuheng Shen, Changhua Meng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37093
• Pdf link: https://arxiv.org/pdf/2610.37093
• Abstract Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at this https URL and this https URL, respectively.
• 中文摘要 视觉感知正从被动识别向主动、任务导向的观察演变。我们研究GUI基础作为可扩展的空间监督来源，用于学习跨界面的观察行为。我们引入了LBS（Look Beyond Screens），这是一个通过监督初始化训练的视觉语言模型，基于教师生成的GUI基础轨迹，随后进行多回合裁剪与检查决策的强化学习。在固定推理设置下，基于Qwen3.5-35B-A3B和Qwen3.8-27B的LBS模型相较相应基线持续提升。超越监督微调的收益转移至PerceptionBench（主干网络）以及HR-Bench和ChartQAPro（Qwen3.5-35B-A3B），支持超越GUI特定模仿的观察策略。进一步分析表明这两个阶段存在互补作用：SFT建立视觉检查原语，而强化学习提升其后续效能。这些发现支持GUI基础作为可迁移主动视觉感知和更广泛多模态推理的实用监督范式。我们的代码和模型分别公开于此 https URL 和此 https URL。

VGIQA: Fine-Grained Image Quality Assessment with Visual Grounding Rewards

VGIQA：细粒度图像质量评估，附带视觉基础奖励

• Authors: Weichang Wang, Liyu Chen, Shuyue Guan, Jie Hu, Fei Gao, Wei Li, Nannan Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.37186
• Pdf link: https://arxiv.org/pdf/2610.37186
• Abstract Image quality assessment (IQA) aims to evaluate the perceptual visual quality of images. Recently, the rapid development of Multi-modal Large Language Models (MLLMs) has significantly advanced the IQA field. By integrating chain-of-thought (CoT) reasoning and reinforcement learning (RL), these models can generate more accurate and interpretable results while relying on less training data. However, existing MLLM-based methods are predominantly constrained to holistic, image-level evaluations and lack the capability to produce fine-grained, localized assessments, which substantially limits their practical applicability. To address this limitation, we propose VGIQA, a novel IQA model endowed with both local and global quality assessment capabilities. VGIQA is trained via a two-stage paradigm comprising a cold-start phase followed by reinforcement learning. Driven by carefully designed local quality assessment reward functions, our model exhibits exceptional image quality understanding on VGSR-2K, a new dataset we propose specifically for local quality evaluation, significantly outperforming existing open-source MLLMs such as Qwen2.5-VL-72B-Instruct and Qwen3-VL-32B-Instruct. Furthermore, the local quality evaluation task can effectively enhance the model's performance on the degradation perception task.
• 中文摘要 图像质量评估（IQA）旨在评估图像的感知视觉质量。近年来，多模态大型语言模型（MLLM）的快速发展显著推动了IQA领域的发展。通过整合思维链（CoT）推理和强化学习（RL），这些模型能够在减少训练数据的前提下，产生更准确、可解释的结果。然而，现有基于MLLM的方法主要受限于整体的图像级评估，缺乏产生细粒度、局部化评估的能力，这大大限制了其实际应用。为解决这一限制，我们提出了VGIQA，一种具备本地和全局质量评估功能的新型IQA模型。VGIQA通过两阶段范式训练，包括冷启动阶段和强化学习。在精心设计的局部质量评估奖励函数驱动下，我们的模型在我们专门为本地质量评估提出的新数据集VGSR-2K上展现出卓越的图像质量理解，远超现有开源MLLM，如Qwen2.5-VL-72B-Instruct和Qwen3-VL-32B-Instruct。此外，局部质量评估任务还能有效提升模型在退化感知任务中的表现。

Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning

通过延续共享进行动作-值校准，用于长视野代理强化学习

• Authors: Yingjian Chen, Lang Zhou, Zixin Zhao, Pingfan Li, Weishi Zheng, Zhilin Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37195
• Pdf link: https://arxiv.org/pdf/2610.37195
• Abstract Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at this https URL.
• 中文摘要 长视野代理的强化学习依赖于交互序列的反馈，使得给单个动作赋予功劳变得困难。现有方法如GiGPO比较共享状态下的动作以计算相对优势，但从单一抽样的延续中估计每个动作的价值。由此产生的局部功劳对后续决策保持敏感。为解决这个问题，我们提出了ConCal（续行校准），一种递归延续值估计器，结合了已存在的延续证据，出现在推广批次中。ConCal通过同一任务的访问匹配其进入状态-动作-状态的转变，将共享的延续证据与每次访问的转换估计结合，并向后传播校准值以精炼早期动作值估计。这减少了对单个抽样续写的依赖，减轻了局部信用失真，而无需额外展开。保留的回报诊断显示，组内价值差异的误差更低，且与参考动作排序的一致性比基于单个续写的估计更为一致。通过Qwen2.5-1.5B-Instruct，ConCal在ALFWorld和WebShop上分别将GiGPO的成功率提升了3.39个百分点和5.86个百分点。我们的代码可在此 https URL 访问。

Remember Where You Walk: Learning Spatial Memory for End-to-End Legged Robot Exploration

记住你走过的地方：学习空间记忆以实现端到端的腿部机器人探索

• Authors: Leon Magnus, Nico Bohlinger, Jan Peters, Oleg Arenz
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.37203
• Pdf link: https://arxiv.org/pdf/2610.37203
• Abstract Humanoids and other legged robots operating in complex environments must remember information that leaves their field of view and only becomes relevant much later. Reinforcement learning policies address this using memory networks such as LSTMs that are trained end-to-end with the policy to memorize such task-relevant aspects. Yet, such architectures often fail to preserve sufficient information over long horizons. In this work, we propose a network architecture based on Hadamard memory that specifically targets robot navigation. Our Local Spatial Matrix Memory (LSMM) enables RL agents to store generic task-relevant information at specific, position-based memory cells, enabling flexible and spatially grounded memorization. We evaluate LSMM on the task of autonomous maze exploration, where it matches the maze coverage of strong recurrent and finite-history baselines while being the only architecture that remembers the maze layout, reducing the reconstruction error threefold.
• 中文摘要 类人机器人和其他有腿机器人在复杂环境中工作时，必须记住那些超出视野、且只有在很久以后才变得相关的信息。强化学习策略通过像LSTM这样的记忆网络来应对这个问题，这些网络经过端到端训练，策略是用来记忆这些与任务相关的方面。然而，这类架构往往无法在长期时间内保存足够的信息。在本研究中，我们提出了一种基于Hadamard记忆的网络架构，专门针对机器人导航。我们的局部空间矩阵记忆（LSMM）使强化学习智能体能够将通用任务相关信息存储在特定的基于位置的记忆单元，实现灵活且空间化的记忆。我们在自主迷宫探索任务中评估LSMM，它在探索迷宫覆盖度上与强循环和有限历史基线相匹配，同时是唯一记忆迷宫布局的架构，将重建误差降低了三倍。

Uncovering Shortcut Amplification in Video RLVR through the Lens of Perceptual Uncertainty Collapse

通过感知不确定性坍缩的视角揭示视频RLVR中的捷径放大

• Authors: Jiafeng Liang, Xianhao Ou, Tao Ren, Rongchuan Mu, Runxuan Liu, Dandan Tu, Ming Liu, See-Kiong Ng, Bing Qin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.37211
• Pdf link: https://arxiv.org/pdf/2610.37211
• Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for enhancing the video reasoning capability of Large Multimodal Models (LMMs). Despite its empirical effectiveness, we show that outcome-level optimization can introduce an undesirable shortcut amplification effect, whereby models increasingly favor text-prior answers even when they are not supported by the visual evidence. To characterize this behavior, we investigate the training dynamics of video RLVR through the lens of perceptual uncertainty, analyzing how predictive uncertainty evolves under video-conditioned and text-only contexts. Our analysis identifies a perceptual uncertainty collapse phenomenon, in which RL training continuously sharpens the model's answer distribution while the uncertainty reduction attributable to visual evidence progressively diminishes. A further gradient-direction analysis reveals that such confidence enhancement becomes increasingly less aligned with visually grounded uncertainty reduction and is instead dominated by text-prior compression. Motivated by these findings, we propose Bifocal Self-distillation Policy Optimization (BSPO), a self-distillation framework that regulates policy updates with two complementary distributional signals. Specifically, BSPO derives empirical reasoning principles from intra-group correct and incorrect rollouts to identify reasoning-sensitive tokens, and introduces shortcut-aware visual perturbation to penalize predictions that remain invariant when task-relevant visual evidence is weakened. By adaptively reweighting the original RL advantage, BSPO preserves the task-solving benefits of RLVR while promoting updates that are both reasoning-sensitive and visually grounded. Extensive experiments validate the effectiveness of BSPO in improving video reasoning performance and strengthening visual faithfulness.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为增强大型多模态模型（LMM）视频推理能力的有效训练后范式。尽管其经验有效性，我们表明结果层优化可能引入不良的捷径放大效应，即模型越来越偏好文本先验答案，即使这些答案没有视觉证据支持。为描述这种行为，我们通过感知不确定性视角研究视频RLVR的训练动态，分析预测不确定性在视频条件和纯文本情境下的演变。我们的分析识别出一种感知不确定性崩溃现象，即强化学习持续锐化模型的答案分布，而视觉证据带来的不确定性减少则逐渐减弱。进一步的梯度方向分析显示，这种置信度提升越来越少依赖于基于视觉的不确定性降低，转而以文本先验压缩为主导。基于这些发现，我们提出了双焦点自蒸馏策略优化（BSPO），这是一种通过两个互补分布信号调节政策更新的自蒸馏框架。具体来说，BSPO从组内正确与错误的推导中推导出经验推理原则，以识别推理敏感的标记，并引入捷径感知视觉扰动，惩罚在任务相关视觉证据被削弱时保持不变的预测。通过自适应重新加权原始强化学习优势，BSPO保留了RLVR的任务解决优势，同时促进既具推理敏感性又具视觉基础的更新。大量实验验证了BSPO在提升视频推理性能和强化视觉忠实度方面的有效性。

RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport

RPSD：带最优传输的检索特权策略自蒸馏

• Authors: Dongyue Li, Zebang Cheng, Zhongyi Deng, Hongboxu, Yi Bin, Fei Ma
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37281
• Pdf link: https://arxiv.org/pdf/2610.37281
• Abstract Sequential recommendation predicts a user's next interaction from historical behavior. Recent LLM-based methods recast this task as generative recommendation and increasingly adopt reinforcement learning such as GRPO for post-training. However, these methods suffer from two limitations, namely credit-blind optimization, where a sequence-level reward is shared across all generated tokens, and collaborative-blind supervision, where each training signal is derived from a single user and ignores behavioral patterns shared across similar users. To address both issues, we propose Retrieval-Privileged on-policy Self-Distillation (RPSD). For credit-blind optimization, we introduce on-policy self-distillation (OPSD), in which a teacher with privileged knowledge re-evaluates the student's on-policy response, and the token-level confidence difference between the student and teacher reshapes the shared sequence-level advantage into token-specific credit. However, the privileged knowledge in vanilla OPSD is still derived from a single user, leaving collaborative-blind supervision unresolved. We therefore ground the teacher in collaborative behavioral evidence from similar users. Specifically, we use temporally aware optimal transport (OT) to compare interaction trajectories across users by jointly considering item similarity and temporal structure, and further apply diversity-aware selection to retain complementary trajectories as privileged evidence. In this way, on-policy self-distillation provides fine-grained token-level credit, while OT enriches the teacher with structured cross-user evidence. To the best of our knowledge, this is the first work to bring on-policy self-distillation to sequential recommendation and to use cross-user behavior as its privileged signal. Extensive experiments across multiple real-world datasets demonstrate that RPSD achieves state-of-the-art performance.
• 中文摘要 顺序推荐通过历史行为预测用户的下一次交互。近期基于LLM的方法将该任务重新定义为生成式推荐，并越来越多地采用如GRPO的强化学习进行训练后训练。然而，这些方法存在两个局限性：学分盲优化，即序列级奖励在所有生成的代币间共享;以及协作盲监督，即每个训练信号均来自单一用户，忽略相似用户间共享的行为模式。为解决这两个问题，我们提出了检索特权政策自我蒸馏（RPSD）。对于学分盲优化，我们引入了策略上自我蒸馏（OPSD），其中拥有特权知识的教师重新评估学生的策略响应，学生与教师之间的代币级置信度差异将共享序列层面优势重塑为代币特定学分。然而，原版OPSD中的特权知识仍源自单一用户，导致协作盲监督问题未被解决。因此，我们将教师建立在类似用户的协作行为证据之上。具体来说，我们使用时间感知最优传输（OT）比较用户间交互轨迹，结合项目相似性和时间结构，并进一步应用多样性感知选择，将互补轨迹保留为特权证据。通过策略自我蒸馏提供了细粒度的代币级信用，而OT则通过结构化的跨用户证据丰富教师。据我们所知，这是首个将策略自提炼带入顺序推荐，并将跨用户行为作为特权信号的作品。跨多个真实世界数据集的大量实验表明RPSD实现了最先进的性能。

UCVE: Graph Structured Retrieval for Compliance Validation

UCVE：用于合规验证的图结构化检索

• Authors: Emily Chen, Michelle Menachery, Himanshu Kumar, Rahul Nenavath, Amirfarrokh Iranitalab
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.37323
• Pdf link: https://arxiv.org/pdf/2610.37323
• Abstract Structured compliance validation requires determining whether every obligation in a predefined requirement space is satisfied, violated, or unaddressed by a given document. This can be described as a coverage problem that existing retrieval and classification systems, by treating requirements as independent, are not designed to solve. We introduce \textbf{\ac{UCVE}}, a domain-agnostic framework that represents compliance requirement spaces as a \ac{CRG} and performs retrieval via spectral reranking over its reinforcement learning-weighted Laplacian. Edges between requirement nodes are learned via reinforcement learning with a traversal reward grounded in violation detection, reshaping the graph's community structure such that low-pass spectral filtering distinguishes relevant from irrelevant obligation clusters. Evaluating across three compliance domains, \ac{UCVE} averages 4.3\% matched-$K$ recall lift over flat cosine retrieval and consistently outperforms strong retrieval and graph-based baselines.
• 中文摘要 结构化合规验证需要确定预定义需求空间中的每个义务是否被给定文档满足、违反或未被处理。这可以被描述为一种覆盖问题，现有的检索和分类系统通过将需求视为独立，无法解决。我们引入了 \textbf{\ac{UCVE}}，这是一个领域无关框架，将合规要求空间表示为 \ac{CRG}，并通过强化学习加权拉普拉斯算子的谱排序进行检索。需求节点之间的边通过基于违规检测的遍历奖励强化学习学习，重塑图的社区结构，使低通谱滤波区分相关与无关义务簇。在三个合规领域进行评估，\ac{UCVE} 平均匹配$K$回忆提升率为4.3%，且持续优于强检索和基于图表的基线。

Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction

多代理强化学习共适应髋外骨骼控制（MARL-Exo）以降低代谢能量

• Authors: Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.37441
• Pdf link: https://arxiv.org/pdf/2610.37441
• Abstract Objective: This study aims to determine whether a hip-exoskeleton policy trained in simulation, by co-adapting with a musculoskeletal human model under a metabolic-energy reward, reduces the metabolic cost of walking in new users without per-user tuning, and how mechanical-power shaping affects the learned assistance. Methods: We developed MARL-Exo, a multi-agent reinforcement learning framework in which a full-body human model (304 muscle actuators; randomized anatomy, gait, and speed) and a bilateral hip-exoskeleton policy are trained concurrently in two stages. The exoskeleton observes only delayed hip kinematics and its past commands; its reward includes muscle metabolic power from a physiologically based energetics model, alone (Metabolic) or with mechanical-power shaping (Metabolic + Power). The Metabolic policy was deployed and tested without retraining in eight healthy adults walking at 1.25 m/s, using indirect calorimetry. Results: In simulation, the Metabolic policy reduced modeled muscle metabolic power by 13.8% relative to no-device walking, versus 8.8% for Metabolic + Power. On hardware, MARL-Exo reduced net metabolic rate by 18.3 $\pm$ 6.6% relative to Passive in all eight participants. Relative to No-Exo, the mean reduction was 7.4 $\pm$ 10.0%; six participants more than offset the 13.1% device-wearing penalty, walking up to 24.2% below their No-Exo cost. Conclusion: A metabolically informed policy co-adapted in simulation transferred to users without per-user tuning and consistently reduced the metabolic cost of walking with the device, while explicit power shaping reduced the simulated benefit. Significance: Unlike per-user human-in-the-loop optimization, this approach yields a single population-level controller and allows assistance objectives to be screened in simulation before human testing.
• 中文摘要 目的：本研究旨在确定在模拟中训练的髋外骨骼政策，通过与肌肉骨骼人类模型在代谢能量奖励下协同适应，是否能降低新用户在无人调节的情况下行走的代谢成本，以及机械能力塑造如何影响学习辅助。方法：我们开发了MARL-Exo，一种多智能体强化学习框架，其中一个全身人体模型（304个肌肉执行器;随机解剖结构、步态和速度）和双侧髋外骨骼策略在两个阶段同步训练。外骨骼仅观察延迟髋关节运动学及其过去指令;其奖励包括来自基于生理的能量模型的肌肉代谢能力，单独（代谢）或机械动力塑造（代谢+功率）。代谢策略在八名健康成人中以1.25米/秒速度行走时，采用间接热能测试，且未重新训练。结果：在模拟中，代谢策略相较于无装置行走降低了13.8%的建模肌肉代谢能力，而代谢+动力行走则降低了8.8%。在硬件方面，MARL-Exo在所有8名参与者中净代谢率相较被动降低了18.3 $\pm$ 6.6%。与无-Exo相比，平均减少幅度为7.4 $\pm$ 10.0%;6名参与者超过抵消了13.1%的佩戴装置惩罚，步行率比无Exo成本低24.2%。结论：在模拟中共同适应的代谢知情策略，转移给用户而无需逐用户调优，且持续降低了携带设备行走的代谢成本，而显性能力塑造则降低了模拟收益。意义：与每用户的人工循环优化不同，该方法实现单一群体级控制器，允许在人体测试前先在模拟中筛选辅助目标。

Interpretable Meta-Policies for Neural Decision Making

神经决策的可解释元策略

• Authors: Raban Emunds, Jannis Blüml, Quentin Delfosse, Kristian Kersting
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.37451
• Pdf link: https://arxiv.org/pdf/2610.37451
• Abstract Deep Reinforcement Learning remains difficult to interpret and align due to the black-box nature of neural networks. While (neuro-)symbolic approaches improve transparency by incorporating symbolic reasoning, applying them directly to low-level actions results in overly complex, unreadable policies. We introduce NEXUS, a hierarchical RL framework that resolves this tension by combining neural skills with symbolic meta-policies. By shifting transparent reasoning to high-level skills rather than atomic actions, NEXUS drastically reduces symbolic complexity without sacrificing environment performance. Object-centric observations enable high-level reasoning and the design of reward functions and meta-policies by interaction with LLMs. We demonstrate that NEXUS agents learn disentangled skills, are interpretable regarding high-level goals, are better aligned to goals described in the game manual, less prone to reward hacking and more robust to game simplifications than deep RL. Finally, we isolate and evaluate the impact of different meta-policy architectures (neural, symbolic, neuro-symbolic) on environmental return and aligned task performance.
• 中文摘要 由于神经网络的黑箱特性，深度强化学习仍然难以解释和对齐。虽然（神经）符号方法通过引入符号推理提升透明度，但直接应用于低层次动作则导致策略过于复杂且难以阅读。我们介绍了NEXUS，一种分层强化学习框架，通过结合神经技能与符号元策略解决了这种张力。通过将透明推理从原子动作转向高级技能，NEXUS大幅降低了符号复杂度，同时不牺牲环境性能。以对象为中心的观察通过与大型语言模型的交互实现了高层次推理以及奖励函数和元策略的设计。我们证明了NEXUS代理学习解散技能，在高层目标上可被解释，更符合游戏手册中描述的目标，更不易被奖励，且比深度强化学习更能支持游戏简化。最后，我们分离并评估不同元政策架构（神经、符号、神经符号）对环境回报和对齐任务表现的影响。

Autonomous fixed-wing interception using reinforcement learning

利用强化学习实现自主固定翼拦截

• Authors: Daan van Hamersveld, Ewoud Smeur, Iurii Kriachko
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.37470
• Pdf link: https://arxiv.org/pdf/2610.37470
• Abstract Medium altitude, long endurance unmanned aerial systems such as Shahed-type strike drones are an unanswered threat in modern conflicts. Conventional air defense systems are too expensive to provide a feasible large-scale solution. Quadcopter interceptors are a cheap, battle-proven alternative, but their extremely limited flight time restricts them to low altitude targets at short range, whereas fixed-wing platforms have the endurance to reach higher altitude targets from longer distances. However, such interceptors are still operated manually. Recent developments in learning-based controllers show potential for automation. In this work, a fixed-wing interceptor is presented that is controlled by a neural network that maps filtered monocular vision observations directly to actuator commands. The policy was trained in a custom Python environment using reinforcement learning and proximal policy optimization. The policy was then deployed on a fixed-wing platform and tested in real interception scenarios. As a baseline, two classical guidance controllers were also developed and tested in simulation. The policy is able to intercept targets both in simulation and in real flight. In simulation, the policy outperforms the classical controllers under most conditions. The interceptor was ultimately successfully validated in real flight by intercepting maneuvering fixed-wing targets. This novel system shows the potential of artificial intelligence for autonomous control of fixed-wing interceptor drones.
• 中文摘要 中高度、长航时的无人机系统，如沙赫德型打击无人机，在现代冲突中是无可解决的威胁。传统防空系统成本过高，无法提供可行的大规模解决方案。四旋翼拦截机是一种廉价且经过实战验证的替代方案，但其极短的飞行时间限制了它们在短距离对低空目标的攻击，而固定翼平台则具备从较远距离打击更高空目标的续航能力。然而，这类拦截机仍需手动操作。基于学习的控制器的最新发展显示出自动化的潜力。在这项工作中，提出了一种由神经网络控制的固定翼拦截机，该网络将过滤后的单眼视觉观测直接映射到执行器命令。该策略在定制的Python环境中通过强化学习和近距离策略优化进行训练。该政策随后在固定翼平台上部署，并在真实拦截场景中进行了测试。作为基线，还开发并测试了两种经典制导控制器。该策略能够在模拟和实飞行中拦截目标。在模拟中，该策略在大多数条件下优于传统控制装置。拦截机最终通过拦截机动固定翼目标，成功验证了实机。该新颖系统展示了人工智能在固定翼拦截无人机自主控制方面的潜力。

Strategyproof Deep Reinforcement Learning for Online Resource Allocation

在线资源分配的策略性深度强化学习

• Authors: Fariba Dehghan, Sebastian Stein, Vahid Yazdanpanah, Stephanie Gauthier
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2610.37504
• Pdf link: https://arxiv.org/pdf/2610.37504
• Abstract Online resource allocation faces two coupled challenges: (1) allocating scarce resources efficiently under uncertain future demand, and (2) ensuring that agents report task valuations and requirements truthfully (strategyproofness). Reinforcement Learning (RL) adapts allocation decisions through experience but does not necessarily ensure strategyproofness. Existing strategyproof RL uses a linear allocation rule, which limits the resource interactions it can represent. Neural mechanism design methods offer greater expressiveness, but provide only approximate incentive guarantees. Against this background, we introduce the first strategyproof deep RL mechanism for online resource allocation. It integrates three novel components: (i) a monotonic neural network that learns nonlinear interactions by construction (higher reported value cannot reduce acceptance, while a longer reported duration or larger resource requirement cannot increase it); (ii) a generalised critical payment: the lowest accepted value, with all other inputs fixed; and (iii) training by either RL or imitation of an existing rule. Together, monotonicity and the critical payment ensure strategyproofness and individual rationality, while the neural rule provides flexibility and imitation improves training efficiency. Empirically, on a four-resource stream with two partner pairs that a linear difference cannot represent, the neural rule achieves 97.8% of an offline welfare upper bound, compared with 93.8% for a linear learner and 89.0% for first-come, first-served, closing 65% and 80% of the remaining optimality gap, respectively. Moreover, imitation reaches 97.5% in only 261 seconds, about 89 times faster than RL.
• 中文摘要 在线资源分配面临两个共同挑战：（1）在未来需求不确定的情况下高效分配稀缺资源，（2）确保代理真实报告任务估值和需求（策略可行性）。强化学习（RL）通过经验调整分配决策，但不一定保证策略性。现有策略性强化学习采用线性分配规则，限制其可表示的资源交互。神经机制设计方法更具表现力，但仅提供近似激励保证。在此背景下，我们介绍首个策略有效深度强化学习在线资源分配机制。它集成了三个新颖组件：（i）通过构建学习非线性交互的单调神经网络（报告值越高无法降低接受度，报告时长越长或资源需求越长也无法增加接受度）;（ii）广义关键支付：最低接受值，其他所有输入固定;以及（iii）通过强化学习或模拟现有规则进行训练。单调性与关键支付共同确保策略适用性和个体理性，而神经规则则提供灵活性，模仿提升训练效率。经验上，在四资源流中，两对伙伴对线性差异无法表示，神经规则实现离线福利上限的97.8%，而线性学习者为93.8%，先到先服务为89.0%，分别缩小了剩余最优差距的65%和80%。此外，模仿仅需261秒达到97.5%，约为强化学习的89倍。

Rendering-Free Event Frame Simulation for Fast Reinforcement Learning of Autonomous Robot Navigation

无渲染事件帧仿真，用于自主机器人导航的快速强化学习

• Authors: Dequan Ou, Austin Lopez, Quentin Missinne, Christophe De Wagter, Guido C.H.E. de Croon
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.37543
• Pdf link: https://arxiv.org/pdf/2610.37543
• Abstract Event cameras offer low latency, high dynamic range, and milliwatt-level power consumption, making them ideal for agile robots such as tiny drones navigating through cluttered environments. However, existing simulators generally render photorealistic images before converting them into events, making large-scale reinforcement learning prohibitively slow. We propose an analytic event simulator that exploits what event cameras fundamentally measure - moving contrast edges - to synthesize event streams directly from scene geometry and ego-motion without visual rendering. Our simulator generates a 64x64 event observation in 17 microseconds on a single GPU - over 400x faster than a state-of-the-art rendering pipeline - enabling end-to-end reinforcement learning for goal-directed obstacle avoidance to converge in under 30 minutes - three orders of magnitude faster than training on rendered events. Policies transfer zero-shot to physically based event simulations (ray-traced Isaac Sim). Our policy reaches 83% success in cluttered courses unseen during training (mid-difficulty, traversability probability approx. 0.57), at 2.4 m/s average goal velocity, and retains 70% in a simulated texture-dense indoor hall. Furthermore, we quantized the policy to a 1.2 MB int8 network running fully onboard a 56-gram Crazyflie equipped with a Prophesee GenX320 event camera and an OpenMV N6 microcontroller board. Without any fine-tuning, external processing, or positioning, the policy achieves an average success rate of 82% (5/5, 5/6, 4/6 flights) across three real-world obstacle courses of increasing difficulty. Code: this https URL
• 中文摘要 事件摄像头具备低延迟、高动态范围和毫瓦级功耗，非常适合在杂乱环境中导航的微型无人机等敏捷机器人。然而，现有模拟器通常在转换为事件前先渲染逼真的图像，使得大规模强化学习速度极慢。我们提出一种分析事件模拟器，利用事件摄像头基本测量的对比度边缘，直接从场景几何和自我运动合成事件流，无需视觉渲染。我们的模拟器在单GPU上17微秒内生成64x64事件观测，速度比最先进的渲染流快400倍以上，实现端到端强化学习，目标导向障碍避让在30分钟内收敛，比训练渲染事件快三个数量级。策略将零射击转移到基于物理的事件模拟（光线追踪Isaac模拟）。我们的策略在训练期间未见的杂乱赛道成功率达83%（中难度，穿越概率约0.57），平均目标速度为2.4米/秒，在模拟纹理密集的室内大厅中保持70%。此外，我们将策略量化为一个1.2 MB的int8网络，运行于一架配备Prophesee GenX320事件摄像机和OpenMV N6微控制器板的56克Crazyflie飞机上。在不进行任何微调、外部处理或定位的情况下，该策略在三个难度递增的真实障碍赛道中实现了82%的平均成功率（5/5、5/6、4/6飞行）。代码：此 https URL

When, Where, and What to Correct: Human-Guided Adaptive Residual Correction for Robot Policies

何时、何地及应纠正什么：机器人政策的人类引导自适应残差纠正

• Authors: Anukriti Singh, Maxime Caniot, Pratap Tokekar, Firas Abi-Farraj
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37547
• Pdf link: https://arxiv.org/pdf/2610.37547
• Abstract Imitation policies that reliably execute most of a manipulation trajectory can still fail in localized regions requiring last-mile precision. Updating the full policy or exploring residual actions everywhere treats this local failure as a global learning problem. We introduce Adaptive Residual Correction (ARC), a lightweight post-training module that uses reinforcement learning (RL) to learn a residual correction on top of a frozen policy while using sparse human corrections to structure autonomous exploration. Rather than treating human corrections as imitation targets, ARC uses pre-intervention states to determine 'when' residual control should become active and correction geometry to determine 'where' in residual action space autonomous refinement should explore; task reward determines 'what' residual is ultimately executed. Across three precision-manipulation tasks on a real robot, ARC improves average task success from 33.3% to 75.0%. The human corrections define task-dependent 2-3 dimensional subspaces of the seven-joint residual space that retain 96.0-97.7% of squared correction magnitude. Together, these results show that sparse human corrections can localize and constrain autonomous post-training without requiring the observed correction itself to serve as the final action target.
• 中文摘要 能够可靠执行大部分操作轨迹的模仿策略，在需要最后一公里精度的局部区域仍可能失败。更新完整策略或探索残余动作，将这种局部失败视为一个全局学习问题。我们引入了自适应残差修正（ARC），这是一个轻量级的后训练模块，利用强化学习（RL）在冻结策略之上学习残差修正，同时利用稀疏的人类修正构建自主探索结构。ARC不将人类修正视为模仿目标，而是利用干预前状态确定“何时”残差控制应激活，并通过修正几何确定残余作用空间中“何处”，自主细化应探索;任务奖励决定最终执行的“什么”残差。在真实机器人的三个精密操作任务中，ARC将平均任务成功率从33.3%提升至75.0%。人类修正定义了七关节残余空间中任务相关的2-3维子空间，这些子空间保留了平方修正幅度的96.0-97.7%。综合来看，这些结果表明，稀疏的人类修正可以定位并限制自主训练后，而无需将观察到的修正本身作为最终行动目标。

Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

自我强化的集体动力学：跨任务耦合如何推动无监督RLVR中的政策崩溃

• Authors: Peize Ding
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.37553
• Pdf link: https://arxiv.org/pdf/2610.37553
• Abstract Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope $\langle F'\rangle$ of the single-problem self-reinforcement drift $F$ is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by $1/N$ for $N$ training problems, races against the shared mode, which grows at a rate $\propto\lambda^2$ independent of $N$, where $\lambda$ is the strength of the shared coupling. This yields a critical number of problems $N_c\simeq\ln(1/\delta)/(\lambda^2G)$, where $\delta$ seeds the shared mode and $G$ decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, $N_c\propto\lambda^{-2}$ up to a logarithmic correction, a single critical amplification $\lambda^2NG$ across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.
• 中文摘要 无标签可验证奖励强化学习（RLVR）奖励语言模型同意其多数票。它最初提升推理能力，但最终崩溃：自我评估奖励降至1，准确率下降，通常直到模型对每个问题给出相同答案。崩溃仅在训练使用多问题时出现。我们将这种数据集大小依赖性解释为集体不稳定性。在可解多问题模型中，问题共享参数的总体会放大任何共享扰动，只要单一问题自我强化漂移$F的总体平均斜率$\langle F'\rangle}为正。当问题对答案仍不确定时，这一过程成立。每题锐化速度减慢1/N$，对于$N$训练问题，与共享模式的增长速度为$\propto\lambda^2$，独立于$N$，其中$\lambda$是共享耦合的强度。这产生了关键数量的问题 $N_c\simeq\ln（1/\delta）/（\lambda^2G）$，其中$\delta$为共享模式做种，$G$随初始置信度递减。模型的模拟验证了预测：一个锐利的转变，$N_c\propto\lambda^{-2}$，对数修正，耦合间的单一临界放大$\lambda^2NG$，对批次大小的对数依赖，且信心初始模型无崩溃。理论的核版本识别出保护耦合（锐化的推广）和边际熵加成，稳定了均匀集体模式。在小型变换器的无标签强化学习中，移除共享方向完全防止崩溃，恢复时导致崩溃，塌陷随训练集大小和基模型熵增长。

TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information

TeamGym：一个由AI代理通过分布式信息进行团队决策的健身房

• Authors: Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz, Yubin Kim, Taekyun Lee, Kaiqing Zhang, Asuman Ozdaglar
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37571
• Pdf link: https://arxiv.org/pdf/2610.37571
• Abstract Many real-world decisions require teams to integrate information distributed across participants. Yet AI agents are still primarily trained and benchmarked in isolation. We introduce TeamGym, an environment for evaluating and training language model-based agents on team decision-making problems constructed from real-world legal, finance, medical, and software engineering corpora. TeamGym introduces a method to identify a minimal set of facts, each element of which is necessary and all of which are collectively sufficient for a correct final decision, which we call pivotal information. This enables TeamGym to both control the distribution of pivotal information and track how well agents identify and integrate pivotal information when making team decisions. Examining 8 language models on 12 source corpora, we find that current AI agents often fail to effectively share information: Distributing pivotal information across participants lowers success rate by 22.9 percentage points on average relative to giving the same information to a single model (a communication gap). We also document that participants do not effectively retrieve decision-relevant information on their own: A team's success rate is lowered by a further 19.9 points when participants must identify and extract pivotal information from source material rather than receiving it directly (a retrieval gap). To help remedy these deficits, we use TeamGym to train policies through reinforcement learning on team decision loss. Across two policy-training experiments, we find that training improves final-decision quality and induces emergent coordination in participation, speaker selection, and communication, even though the reward evaluates only the final decision.
• 中文摘要 许多现实世界决策需要团队整合分散在参与者之间的信息。然而，人工智能代理仍主要在孤立环境中进行训练和基准测试。我们引入了TeamGym，这是一个用于评估和训练基于语言模型的代理的环境，解决团队决策问题，这些问题由现实世界的法律、金融、医疗和软件工程语料库构建。TeamGym引入了一种方法，用于识别一组极简的事实，每个要素都是必要且共同足以做出正确最终决策的，我们称之为关键信息。这使得TeamGym能够控制关键信息的分配，并跟踪代理在团队决策时识别和整合关键信息的能力。通过分析12个源语料库中的8个语言模型，我们发现当前AI代理常常无法有效共享信息：将关键信息分散给参与者相比，平均成功率降低22.9个百分点（沟通差距）。我们还记录了参与者无法有效自行检索决策相关信息：当参与者必须识别并提取关键信息而非直接接收时，团队成功率又下降19.9分（检索差距）。为弥补这些不足，我们使用TeamGym通过强化学习训练团队决策丢失的策略。在两项策略训练实验中，我们发现训练提升了最终决策质量，并促进参与、发言人选择和沟通中的涌现协调，尽管奖励仅评估最终决策。

TANDEM: Balancing Centralized and Decentralized Critics to Address the Information Gap in Complex Multi-Agent Environments

TANDEM：平衡中心化与去中心化批评者，以弥补复杂多智能体环境中的信息差距

• Authors: Alireza Saleh Abadi, Leen-Kiat Soh, Adam Eck, Prashant Doshi
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.37592
• Pdf link: https://arxiv.org/pdf/2610.37592
• Abstract Cooperative multi-agent reinforcement learning under centralized training with decentralized execution (CTDE) shapes decentralized policies with a centralized critic that observes the global state, then removes the critic at execution. The policies are optimized to rely on information they will not have when they act, an information gap between training and execution. It widens under partial observability, observation noise, and openness, where the set of agents and tasks present changes during an episode. Existing methods transfer centralized guidance in one direction only, removing or distilling the critic once, so the policies never rehearse acting without the critic while they can still be corrected. We propose TANDEM, which keeps one centralized critic and reads it two ways at each training step: a centralized read from the global state, and a decentralized read restricted to the information each agent will have at execution. A parameter-free gate compares the two reads for each agent and trains the policy on the decentralized read where they agree, reinstating the centralized read only where the local view is misleading. A shared target pulls the decentralized read toward the centralized one. We formalize the setting as a task- and agent-open decentralized partially observable Markov decision process (TAO-Dec-POMDP) and evaluate TANDEM on three cooperative domains, each isolating one form of the information gap: discovery for partial observability, sampling for observation noise, and openness in wildfire suppression. We compare against six CTDE baselines and ablate the gate and the shared target; the supplementary material holds the full variant sweep, diagnostics, and learning curves. We measure return at execution, once the centralized critic is removed. TANDEM is the best method on eight of the nine setups and never significantly worse than any baseline.
• 中文摘要 在去中心化执行（CTDE）集中训练下的协作多智能体强化学习，塑造了去中心化策略，中心化批评者观察全局状态，执行时移除批评者。策略优化为依赖行动时无法获得的信息，即训练与执行之间的信息差距。在部分可观察性、观察噪声和开放性下，这一差距扩大，代理和任务在一次事件中发生变化。现有方法只向单方向传递集中引导，仅移除或提炼批评者一次，因此策略在还能纠正时不会在没有批评者的情况下演练行动。我们提出TANDEM，保持一个中心化批评者，并在每个训练步骤以两种方式读取：从全局状态集中读取，以及分散式读取，限制每个执行时所掌握的信息。一个无参数门比较每个代理的两个读取，并在双方一致的分散式读取上训练策略，在局部观点误导时恢复集中式只读。共享目标将分散式读段拉向集中目标。我们将设置形式化为任务和代理开放的分散部分可观测马尔可夫决策过程（TAO-Dec-POMDP），并在三个协作域上评估TANDEM方法，每个域隔离出一种信息差距形式：部分可观测性的发现、观测噪声采样和野火抑制中的开放性。我们与六个CTDE基线比较，简化了门和共享目标;补充材料包含完整的变体扫描、诊断和学习曲线。我们测量执行时的回报，一旦中心化批评者被移除。TANDEM在九种设置中有八种是最佳方法，且从未显著差于任何基线。

EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning

EnvSched：环境感知调度，实现高效代理强化学习

• Authors: Sihan Yang, Weichen Bai, Ruichuan An, Yiming Cheng, Zhiyuan Feng, Tianyi Jiang, Renye Yan, Ning Ding, Yifan Zhang, Renrui Zhang, Yehui Tang, Wentao Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.37609
• Pdf link: https://arxiv.org/pdf/2610.37609
• Abstract As executable environments for language agents scale, generating and verifying their rollouts becomes increasingly expensive. Under finite rollout budgets, this raises an important question: can data management at the environment level make agentic reinforcement learning (RL) more efficient by organizing, evaluating, and scheduling task instances together with the environments that determine their tools, states, transitions, and verification? To address this question, we introduce EnvSched (Environment-aware Scheduling), an online scheduler that learns from rollout history. EnvSched assigns scheduling value through two complementary modeling choices: exploitation prioritizes tasks likely to provide informative reward differences, while coverage spreads each batch across underobserved environments and interaction patterns. EnvSched organizes environment--task pairs as nodes in a dual-relation graph, with within-environment edges capturing task semantics and shared tool, state, and verifier structure, and cross-environment edges capturing abstract execution and reward topology. By propagating historical rollout statistics along both edge types, it predicts within-task reward variance. These predictions guide a scheduler that allocates each batch to tasks with high predicted reward variance and underobserved environments and tasks. Environment quotas and a similarity penalty further limit concentration and redundancy, yielding informative and diverse batches without additional pilot rollouts. In RL on Qwen3-8B, EnvSched achieves an average score of 48.81 across BFCL-v3, tau-Bench, and ACEBench-Agent after 40 training steps, exceeding the highest score attained by random task selection after 120 steps. It also improves downstream performance across model sizes at the same training budget.
• 中文摘要 随着语言代理可执行环境的扩展，生成和验证其部署的成本也变得越来越高。在有限的部署预算下，这引发了一个重要问题：环境层面的数据管理是否能通过组织、评估和调度任务实例以及决定其工具、状态、转换和验证的环境，使代理强化学习（RL）更高效？为回答这个问题，我们介绍了EnvSched（环境感知调度），这是一款在线调度器，能够从部署历史中学习。EnvSched通过两种互补的建模方式赋予调度价值：利用优先处理可能带来信息性奖励差异的任务，而覆盖则将每个批次分散在未被观察到的环境和交互模式中。EnvSched 将环境任务对组织为双关系图中的节点，环境内边捕捉任务语义及共享工具、状态和验证器结构，跨环境边捕捉抽象执行和奖励拓扑。通过沿两种边类型传播历史推销统计数据，预测任务内奖励方差。这些预测引导调度器将每批分配给预测奖励方差高且环境和任务未被关注的任务。环境配额和相似性惩罚进一步限制了集中性和冗余，从而产生信息丰富且多样化的批次，无需额外的试点推广。在 Qwen3-8B 的强化学习中，EnvSched 在 BFCL-v3、tau-Bench 和 ACEBench-Agent 中完成 40 次训练后平均得分为 48.81，超过了随机任务选择在 120 步后获得的最高分。它还能在相同训练预算下提升模型规模的下游性能。

VideoLVR: Latent Visual Reasoning over Grounded Video Evidence

VideoLVR：潜在的视觉推理胜过有根据的视频证据

• Authors: Shilong Li, Xingyuan Bu, Hao Shi, Yun Xing, Weixin Feng, Qiangpeng Yang, Shilei Wen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37613
• Pdf link: https://arxiv.org/pdf/2610.37613
• Abstract Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content. Existing approaches maintain visual grounding during reasoning in two main ways: they either refer to evidence using textual coordinates, which specify symbolic locations rather than the visual content itself, or reinsert cropped regions or frames into the multimodal context, preserving visual details at the cost of substantial context expansion. To address these limitations, we introduce VideoLVR, which constructs latent visual evidence from local regions at relevant video moments, thereby providing visual support for intermediate reasoning steps. To supervise latent visual evidence generation, we introduce a visual token prediction objective that formulates visual token reconstruction as prediction over a vocabulary derived from the input video, thereby unifying visual evidence reconstruction and text generation within the same token prediction framework. This shared prediction framework enables MLLMs to reason over interleaved text tokens and latent visual states while integrating evidence from different moments. We further extend VideoLVR to reinforcement learning by treating sampled visual tokens as actions alongside text tokens and jointly optimizing the policy using final-answer rewards. By efficiently revisiting task-relevant visual information during reasoning, VideoLVR improves video understanding without substantially expanding the multimodal context. Experiments on multiple video benchmarks demonstrate its effectiveness, especially on tasks requiring fine-grained visual evidence.
• 中文摘要 思维链推理在多模态大型语言模型（MLLMs）中显著推进了视频理解。然而，基于语言的推理痕迹无法为模型重新访问相关视觉证据提供可靠的机制，导致视觉遗忘并削弱其对视频内容的推理能力。现有方法在推理过程中主要通过两种方式保持视觉基础：要么使用文本坐标来指代证据，指定符号位置而非视觉内容本身;要么将裁剪的区域或帧重新插入到多模态上下文中，保留视觉细节，但代价是大幅扩展上下文。为解决这些局限性，我们引入了VideoLVR，它从相关视频时刻的局部区域构建潜在视觉证据，从而为中间推理步骤提供视觉支持。为监督潜在视觉证据生成，我们引入了视觉代币预测目标，将视觉代币重建作为基于输入视频词汇的预测，从而在同一代币预测框架内统一视觉证据重建与文本生成。该共享预测框架使MLLM能够在交错文本符号和潜在视觉状态上推理，同时整合不同时刻的证据。我们进一步将VideoLVR扩展到强化学习领域，将抽样视觉代币与文本代币并列的动作，并共同优化策略，采用最终答案奖励。通过在推理过程中高效重访任务相关视觉信息，VideoLVR提升了视频理解，同时大幅扩展多模态上下文。多个视频基准测试的实验显示其有效性，尤其是在需要细粒度视觉证据的任务中。

ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning

ThinkWorld：世界想象力强化动力学链推理

• Authors: Chaehyun Kim, Chi-Pin Huang, Jisoo Kim, Min-Hung Chen, Yu-Chiang Frank Wang, Seungryong Kim, Fu-En Yang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.37666
• Pdf link: https://arxiv.org/pdf/2610.37666
• Abstract Vision-Language-Action (VLA) models have made substantial progress in robotic manipulation, but their generalization remains constrained by the diversity and coverage of action-labeled robot demonstrations. Human videos provide much broader behavioral and environmental experience, while learning from them faces a fundamental trade-off between scalable data and explicit action grounding. We introduce ThinkWorld, a VLA learning framework that grounds action reasoning in world dynamics and strengthens it through reinforcement learning with world imagination. ThinkWorld represents each action thought as a Chain-of-Dynamics (CoD), a sequence of discrete motion tokens that captures the intermediate dynamics connecting the current world to a future state and serves as a common action representation across embodiments. We first initialize CoD with action-labeled demonstrations, then predict the future induced by each CoD and reinforce those whose predictions better match the observed outcome. This enables videos without action annotations to provide direct supervision from observed state transitions, allowing human and robot experience to refine the same reasoning space. Experiments in simulation and on real robots demonstrate effective human-to-robot transfer and strong generalization to unseen tasks and environments.
• 中文摘要 视觉-语言-行动（VLA）模型在机器人操作方面取得了显著进展，但其推广仍受限于动作标记机器人演示的多样性和覆盖范围。人类视频提供了更广泛的行为和环境体验，而从中学习则面临可扩展数据与显式行动基础之间的根本权衡。我们介绍了ThinkWorld，一种将行动推理建立在世界动态基础上的VLA学习框架，并通过世界想象力的强化学习强化其理论。ThinkWorld将每个行动思维表示为动态链（Chain-of-Dynamics Functions，CoD），这是一系列离散运动标记，捕捉连接当前世界与未来状态的中间动态，并作为跨实体的通用行动表示。我们首先用带动作标记的演示初始化CoD，然后预测每个CoD诱导的未来，并强化那些预测更符合观察结果的行为。这使得无动作注释的视频能够直接监督观察到的状态转变，使人类和机器人的经验能够精炼相同的推理空间。模拟和真实机器人实验展示了有效的人与机器人转移以及对未见任务和环境的强推广。

Keyword: diffusion policy

From 12 to 200 Demonstrations: How ACT, Diffusion Policy, SmolVLA and $π_{0.5}$ Scale on a Low-Cost Arm

从12到200场演示：如何在低成本臂上实现ACT、扩散政策、SmolVLA和$π_{0.5}$规模

• Authors: Simon Cowling
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.37170
• Pdf link: https://arxiv.org/pdf/2610.37170
• Abstract We trained four widely used imitation-learning policies, ACT, Diffusion Policy, SmolVLA and $\pi_{0.5}$, on nested subsets of 12, 25, 50, 100 and 200 teleoperated demonstrations of one pick-and-place task with distractor objects, using one fixed training recipe per policy. Every trained model was then run on a physical SO-101 arm on a fixed set of object placements: 1,340 scored episodes in 126 blocks. ACT, trained from scratch, rose from 1 success in 40 episodes at 12 demonstrations to 37 in 40 at 100. $\pi_{0.5}$ with its action expert fine-tuned was the best policy when data was scarce (16 of 40 at 12 demonstrations, 28 of 40 at 25) and then stayed near 70%; from 100 demonstrations ACT was ahead, by a margin this study does not fully resolve. SmolVLA and Diffusion Policy peaked near 50%, and both were affected by a failure to start: SmolVLA's arm never started moving in 47% of episodes, and Diffusion Policy hardly ever started until an idle lead-in was trimmed from the demonstrations. For $\pi_{0.5}$ the fine-tuning recipe mattered more than the amount of data: low-rank adaptation succeeded in 4 of 110 episodes, against 130 of 200 for action-expert fine-tuning. From the joint and camera recordings we place every episode on a four-stage ladder (started, reached the ball, carried it, delivered it) and show that for ACT more data moves failures from reaching, to grasping, to none, while the failures $\pi_{0.5}$ retains are in grasping. We also measure how weak a ten-episode block is as a measurement (the same checkpoint tested twice differs by 2 successes in 10 on average), separate seed-to-seed variation from that noise, and audit the human scoring against the recordings. Training length was fixed within each recipe, so the amount of data and the number of times it was seen change together; we state which conclusions survive that confound and which do not.
• 中文摘要 我们训练了四种广泛使用的模仿学习策略：ACT、扩散策略、SmolVLA和$\pi_{0.5}$，分别在12、25、50、100和200个远程操作的远程操作演示中，使用一个固定训练方案，每个策略使用一个固定训练方案。每个训练模型随后在物理SO-101臂上运行，使用固定的物体配置：126个区块中1340个评分集。ACT从零训练，从40集12次演示中1次成功提升至100次时37次成功。当数据稀缺时，$\pi_{0.5}$及其动作专家微调是最佳策略（12次演示中40题中的16次，25次演示时40次中28次），随后保持在70%左右;在100个演示中，ACT领先，但该研究尚未完全解决。SmolVLA和Diffusion Policy的峰值接近50%，两者均受启动失败影响：SmolVLA的机械臂在47%的剧集中未开始移动，扩散政策几乎直到演示中删减空闲引言才开始。以$\pi_{0.5}$计价，微调配方比数据量更重要：低级改编在110集中成功4集，而动作专家微调则成功130集。通过关节和摄像机录制，我们将每集放在四阶段梯子上（启动、到达球、携带球、送球），并显示ACT中，更多数据将失败从抓取到抓取转变为无，而保留pi_的失败则在抓取中。我们还测量了十集区块的弱弱度（同一检查点测试两次，平均每10集成功中相差2次），区分种子间的差异，并对人工评分与录制进行审计。每个配方中训练时长固定，因此数据量和观察次数会共同变化;我们说明哪些结论能通过混淆，哪些不存在。