生成时间: 2026-10-30 22:55:52 (UTC+8); Arxiv 发布时间: 2026-10-30 20:00 EDT (2026-10-31 08:00 UTC+8)

今天共有 73 篇相关文章

Keyword: reinforcement learning

Echo: Scaling Visual Reinforcement Learning with Off-policy Experience

Echo：在非策略体验下扩展视觉强化学习

• Authors: Gagan Khandate
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.37865
• Pdf link: https://arxiv.org/pdf/2610.37865
• Abstract Scaling visual reinforcement learning (RL) with GPU-accelerated simulation is computationally expensive due to rendering and visual-policy inference costs. Consequently, two-stage teacher--student distillation is commonly used to train visual policies, but introduces an imitation gap. In this preliminary study, we introduce \echomethod{}, \textbf{a single-stage off-policy RL framework that scales experience collection without proportionally increasing visual computation}. Through \textbf{inverse distillation}, a state-based auxiliary policy imitates the evolving visual policy and collects inexpensive, unrendered experience. Echo leverages this experience for off-policy value learning, directly improving the visual policy through a shared privileged critic. We instantiate \echomethod{} with FastTD3 and evaluate it on three visual control tasks and two dexterous manipulation tasks in Genesis. Relative to \sdpgmethod{} under the evaluated configurations, Echo reduces time to reference returns by an average of \textbf{60\% across the three locomotion tasks} and \textbf{56\% across both hand tasks}, while improving peak mean returns by \textbf{62\% and 12\%}, respectively. Both hand tasks also achieve higher mean returns at matched simulator-transition budgets. These initial results suggest that scaling off-policy experience is a promising approach to efficient direct visual RL.
• 中文摘要 利用GPU加速仿真扩展视觉强化学习（RL）由于渲染和可视化策略推断成本，计算成本较高。因此，两阶段师生提纯常用于训练视觉策略，但会带来模仿差距。在这项初步研究中，我们引入了\echomethod{}、\textbf{一种单阶段非策略RL框架，可在不比例增加视觉计算的情况下扩展经验收集}。通过\textbf{逆蒸馏}，基于状态的辅助策略模拟不断演变的视觉策略，收集廉价的未渲染经验。Echo利用该经验进行非策略价值学习，通过共享特权批评者直接改进视觉策略。我们用FastTD3实例化\echomethod{}，并在Genesis中对三个视觉控制任务和两个灵巧操作任务进行评估。相较于 \sdpgmethod{}，在评估配置下，Echo 在三个移动任务中平均减少了 \textbf{60\%} 和两个手部任务的平均 \textbf{56\%}，同时分别提高了峰值平均回报 \textbf{62\% 和 12\%}。在匹配的模拟器-转换预算下，两个手部任务也都能实现更高的平均回报。这些初步结果表明，非策略经验的扩展是高效直接视觉强化学习的有前景方法。

Multi-Agent Reinforcement Learning for Global Climate Policy Cooperation

多智能体强化学习全球气候政策合作

• Authors: Anne Moll-Elsborg, Oskar Bohn Lassen, Francisco Camara Pereira
• Subjects: Subjects: Multiagent Systems (cs.MA); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.37870
• Pdf link: https://arxiv.org/pdf/2610.37870
• Abstract International climate policy requires coordination among countries whose mitigation decisions impose local costs but generate global benefits. Classical game-theoretic and integrated assessment approaches can study these incentives, but typically rely on prescribed behavioural assumptions or equilibrium concepts. Recent work has therefore explored Multi-Agent Reinforcement Learning as a way to model climate-policy cooperation as an emergent outcome of repeated strategic interaction. We extend an existing MARL climate framework with a multi-stage negotiation mechanism in which agents propose agreements, accept or reject commitments, and face penalties for non-compliance. This allows us to study how agreement design affects learned mitigation behaviour, compliance, and climate outcomes. In an idealized setting with four homogeneous agents, we find that non-binding agreements do little to overcome free-riding, while moderate non-compliance penalties improve cooperation and reduce warming. However, excessively strong penalties discourage ambitious commitments, revealing a trade-off between enforcement and participation. We position the framework as a methodological contribution for testing agreement design rather than a predictive model of real negotiations.
• 中文摘要 国际气候政策需要各国之间的协调，这些国家的减缓决策带来了地方成本，但产生全球利益。经典博弈论和综合评估方法可以研究这些激励，但通常依赖于规定的行为假设或均衡概念。因此，近期研究将多代理强化学习视为气候政策合作作为反复战略互动的涌现结果。我们在现有的MARL气候框架基础上扩展了多阶段谈判机制，代理提出协议、接受或拒绝承诺，违约者面临处罚。这使我们能够研究协议设计如何影响学习到的减缓行为、合规和气候结果。在拥有四个同质代理的理想环境中，我们发现非约束性协议对克服搭便车效应作用有限，而适度的不合规惩罚则改善合作并减少变暖。然而，过于严厉的处罚会阻碍雄心勃勃的承诺，暴露出执行与参与之间的权衡。我们将该框架定位为测试协议设计的方法论贡献，而非真实谈判的预测模型。

Efficient Joint Trajectory Optimization and Resource Allocation for Secure UAV Networks: A Deep Reinforcement Learning Approach

安全无人机网络的高效联合轨迹优化与资源分配：深度强化学习方法

• Authors: Abdulmalik Alwarafy, Mohamed Hayajneh, Mustafa Al Juboori, Abdulaziz Almarzooqi, Tahnoon Almazrouei, Robera Abera, Kaleab Nega
• Subjects: Subjects: Systems and Control (eess.SY); Information Theory (cs.IT)
• Arxiv link: https://arxiv.org/abs/2610.37897
• Pdf link: https://arxiv.org/pdf/2610.37897
• Abstract Unmanned Aerial Vehicles (UAVs) are increasingly deployed in wireless networks; however, ensuring secure communication and efficient energy usage remains challenging. This paper addresses the joint trajectory optimization and resource allocation problem in UAV networks in the presence of a tethered UAV flying center and a malicious ground eavesdropper. In the proposed model, UAVs are equipped with energy harvesting capabilities and can adaptively switch between transmitting, jamming, and relaying roles to deliver data to a tethered UAV center for optimized network performance. An optimization problem is formulated to maximize a utility function that balances secrecy rate, energy consumption, and interference. To address the high non-convexity of the problem and its real-time decision requirements, a Proximal Policy Optimization (PPO)-based Deep Reinforcement Learning (DRL) algorithm is proposed to solve the problem and learn the network dynamics. Extensive simulation results demonstrate that the proposed approach achieves superior secrecy performance, lower energy consumption, and reduced interference compared to conventional and DRL-based methods.
• 中文摘要 无人机（UAV）正日益被部署在无线网络中;然而，确保通信安全和高效能源使用仍然具有挑战性。本文探讨了在有系有线无人机飞行中心和恶意地面窃听者存在的情况下，无人机网络中的联合轨迹优化和资源分配问题。在所提模型中，无人机具备能量收集能力，能够自适应切换发射、干扰和中继角色，将数据传递到有线无人机中心，以实现网络性能优化。提出了一个优化问题，以最大化一个在保密率、能耗和干扰之间取得平衡的效用函数。为解决问题的高非凸性及其实时决策需求，提出了一种基于近端策略优化（PPO）的深度强化学习（DRL）算法，用于解决该问题并学习网络动态。大量模拟结果表明，所提方法相比传统和基于日程遥远的方法，实现了更优异的保密性能、更低的能耗和更小的干扰。

GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

GARDEN：通过证据导航实现通用自主奖励设计，促进合作多智能体强化学习

• Authors: Dogan Urgun
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.37926
• Pdf link: https://arxiv.org/pdf/2610.37926
• Abstract Cooperative multi-agent reinforcement learning (MARL) enables teams of agents, such as robots and autonomous vehicles, to learn coordinated behavior, but what such teams learn is largely determined by their reward function. Because task success alone provides too little guidance for learning, intermediate rewards are usually designed by hand, and this effort must often be repeated whenever the environment or the team changes. Large language models (LLMs) can automate the effort by writing reward functions as code, yet the resulting rewards are typically designed and tested in a single setting. To enable their reuse across settings, GARDEN (Generalist Autonomous Reward Design through Evidence Navigation) is proposed, in which one reward program is developed for a family of related scenarios and transferred, unchanged, to scenarios that were not available during its design. During development, environment code, training records, and agent behavior are investigated by an LLM reward engineer, and draft rewards are tested in trial training runs, while the learning algorithm and the rule for selecting the final reward are held fixed. On unseen scenarios of the Overcooked and SMAX benchmarks, task returns were increased by 58.2% and 52.5% over the default rewards on two kitchen layouts, and the mean win rate in two combat scenarios was raised from 51.0% to 89.9%. Under matched candidate and probe budgets, this exceeded the 84.2% and 79.2% obtained with fixed diagnostic feedback and Eureka-inspired reward reflection. These results show that a single autonomously engineered reward can be reused effectively across related cooperative scenarios.
• 中文摘要 协作多智能体强化学习（MARL）使智能体团队（如机器人和自动驾驶车辆）能够学习协调行为，但这些团队学习的内容主要取决于他们的奖励函数。由于任务成功本身对学习的指导不足，中间奖励通常由手工设计，且每当环境或团队变化时，这一努力常需重复。大型语言模型（LLMs）可以通过编写奖励函数代码来自动化工作，但最终的奖励通常在单一环境中设计和测试。为实现跨环境的重复使用，提出了GARDEN（通过证据导航实现通用自主奖励设计）方案，即为一系列相关场景开发一个奖励程序，并不更改地转移到设计时未可用的场景。开发过程中，环境代码、训练记录和代理行为由LLM奖励工程师调查，草稿奖励在试运行中测试，学习算法和最终奖励选择规则保持固定。在Overcooked和SMAX基准测试的未见场景中，任务回报比默认奖励提升了58.2%和52.5%，两个战斗场景的平均胜率从51.0%提升至89.9%。候选和探针预算匹配不足，超过固定诊断反馈和Eureka激励奖励反思的84.2%和79.2%。这些结果表明，单一自主设计的奖励可以在相关合作场景中有效重复使用。

Learning from Relative Action Credit for Escaping Local Minima

从相对行动中学习逃脱局部最低值的学分

• Authors: Giovanni Minelli, Giulio Turrisi, Lorenzo Amatucci, Victor Barasuol, Claudio Semini
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.38034
• Pdf link: https://arxiv.org/pdf/2610.38034
• Abstract Reinforcement learning holds the promise of robots that learn manipulation skills from interaction alone, without pretrained priors or demonstration data. In practice, on-policy methods such as PPO routinely stall on long-horizon, contact-rich tasks: dense reward shaping carries the policy through stages, yet stitching them together and discovering sparsely rewarded events rarely succeeds. We trace this failure to the missing action discrimination in state-value credit assignment: a state-value critic judges progress only by episode outcomes, so when the policy converges to a partial solution all outcomes coincide, and improvement stops because no action can prove better than another. We therefore extend PPO with an action conditioned value critic which, can compare the executed action with alternatives sampled at the same state and aid the learning signal. Combined with Generalized Advantage Estimation and a bound on exploration noise, we show how the extended algorithm escapes the plateau and completes precise peg insertion within a training budget where PPO with identical exploration fails.
• 中文摘要 强化学习有望让机器人仅通过互动学习操作技能，无需预训练的先验或演示数据。在实践中，像PPO这样的策略方法常在长期、接触丰富的任务中停滞：密集的奖励塑造推动策略通过各个阶段，但将它们拼接起来并发现奖励稀疏事件很少成功。我们将这一失败归因于状态价值赋值中缺失的行动辨别：状态值批评者仅通过事件结果来判断进展，因此当策略趋于部分解时，所有结果一致，改进停止，因为没有哪个行动能证明优于另一个。因此，我们将PPO扩展为一个动作条件值批评者，该方法可以将执行的动作与同一状态下采样的备选方案进行比较，并帮助学习信号。结合广义优势估计和探索噪声界限，我们展示了扩展算法如何突破平台期，在训练预算内完成精确的钉子插入，而PPO在相同探索方法下失败。

Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI

Nemotron 3.5 超级与闪电：面向智能人工智能的开放高效专家混合混合曼巴-变换器模型

• Authors: NVIDIA: Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahamed, Asma Kuriparambil Thekkumpate, Aswin Visva, Atefeh Sohrabizadeh, Avinash Vem, Ayush Dattagupta, Benedikt Schifferer, Benjamin Chislett, Bernard Nguyen, Besmira Nushi, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Branislav Kisacanin, Brendan Johnson, Brian Dang, Brian Yu, Bryan Catanzaro, Byung-Kwan Lee, Carlo del Mundo, Chankyu Lee, Chantal Hwang, Charles Wang, Charlie Truong, Chen Cui, Cheng Wang, Cheng-Ping Hsieh, Chenhan Yu, Chenjie Luo, Chintan Patel, Chris Alexiuk, Chris Johnson, Chris Wing, Christian Munley
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38095
• Pdf link: https://arxiv.org/pdf/2610.38095
• Abstract We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Super is a 120B-parameter, 12B-active multimodal model, while Lightning is a smaller 30B-parameter, 3B-active text model; both designed for efficient deployment. Both models combine large-scale distillation, long-context training, and native speculative decoding, while Super further incorporates Indic continued pretraining and agentic software-engineering mid-training. We develop a comprehensive post-training pipeline spanning supervised fine-tuning and reinforcement learning (RLVR and RLHF), complemented in Super by specialized teacher training and multi- teacher on-policy distillation (MOPD). This transforms Super from a text-only base model into a multimodal agent capable of reasoning over text, images, video, and interactive tool environments. We further explore test-time scaling through parallel agent execution and self-verification. Across training and deployment, we jointly optimize capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens software-engineering capabilities and robustness across agent harnesses. The post-trained Super model achieves 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra despite using approximately 4.6x fewer active parameters. Post-training MTP boosting improves speculative- decoding acceptance length by up to 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.62% median accuracy recovery relative to BF16. Inference optimizations enable up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user.
• 中文摘要 我们介绍了 Nemotron 3.5 Super 和 Nemotron 3.5 Lightning，这两款开放高效的专家混合混合 Mamba-Transformer 模型，专为推理、长上下文理解和代理工作负载设计。Super 是一个 120B 参数、12B 主动多模态模型，而 Lightning 是一个较小的 30B 参数、3B 主动文本模型;两者均设计为高效部署。两者均结合了大规模蒸馏、长上下文训练和原生推测解码，而 Super 进一步在培训中融入了 Indic 持续预训练和代理软件工程。我们开发了涵盖监督微调和强化学习（RLVR 和 RLHF）的综合培训后流程，Super 中辅以专业教师培训和多教师策略提炼（MOPD）。这使 Super 从纯文本基础模型转变为能够在文本、图像、视频和交互工具环境中进行推理的多模态代理。我们还进一步探讨了通过并行代理执行和自我验证实现测试时间扩展。在训练和部署过程中，我们共同优化能力和效率。印度语持续预训练提升了印度语言基准数据高达 9.82 个百分点，而代理式中期训练则增强了软件工程能力和代理框架间的稳健性。后训练的 Super 模型在 SWE-bench Pro 上实现了 51.78%，超过了 Nemotron 3 Ultra，尽管使用了约 4.6 倍的主动参数。训练后 MTP 提升在概率草稿抽样下，推测解码接受长度提升高达 21.8%，而 NVFP4 量化相较于 BF16 实现了 99.62% 的中位准确率恢复。推理优化支持GB200上最多4,856个输出token/s/GPU，同时至少维持50个token/s/用户。

AI-Driven Digital Twin Framework for Warehouse Logistics Automation Optimization

基于AI的数字孪生框架用于仓库物流自动化优化

• Authors: Md Mahinur Alam, Kanita Jerin Tanha, Md Tanjin Alam, Shadman Sakib, Min Gyu Kim
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38101
• Pdf link: https://arxiv.org/pdf/2610.38101
• Abstract In recent years, warehouse logistics optimization has increasingly focused on intelligent, data-driven systems to enhance operational efficiency. Conventional static product placement strategies are being progressively replaced by dynamic slotting approaches that can adapt to real-time variations in warehouse conditions. In this work, we present a novel framework that employs a reinforcement learning agent integrated within a FlexSim-based Digital Twin environment for real-time product placement optimization. The proposed agent continuously learns from streaming operational data, enabling it to reduce picking time while improving overall system throughput. The simulation environment models a four-zone warehouse layout with forklift-based material handling and leverages the Asset Administration Shell framework to ensure seamless interaction between the learning agent, physical asset, and the digital twin. Experimental evaluation demonstrates substantial performance gains, with the proposed approach achieving a 29.66\% reduction in average picking time, a 35.42\% increase in throughput, and an 18.54\% decrease in travel distance.
• 中文摘要 近年来，仓库物流优化越来越多地聚焦于智能、数据驱动系统，以提升运营效率。传统的静态产品置入策略正逐步被动态时段配置方法所取代，这些方法能够适应仓库环境的实时变化。本研究提出了一个新颖框架，采用集成在基于FlexSim的数字孪生环境中的强化学习代理，实现实时产品置放优化。该代理持续从流式运营数据中学习，从而缩短拣货时间，同时提升整体系统吞吐量。该模拟环境模拟了基于叉车物料搬运的四区仓库布局，并利用资产管理壳框架确保学习代理、实体资产和数字孪生之间的无缝交互。实验评估显示性能显著提升，拟议方法平均拾取时间减少29.66%，吞吐量提升35.42%，行程缩短18.54%。

Post-Training Large Language Models with LLMON

使用LLMON训练大型语言模型后期

• Authors: Basel Shbita, Michael Hind, Chad DeLuca, Nathan Fulton, Shubhi Asthana, Farhan Ahmed, Anisa Halimi, Yada Zhu, Mauro Martino, David Cox, Dan Gutfreund
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38120
• Pdf link: https://arxiv.org/pdf/2610.38120
• Abstract Large Language Models (LLMs) treat all context tokens uniformly, lacking explicit separation between executable instructions and passive data. This leads to prompt injection vulnerabilities, ambiguous instruction binding, and unreliable execution control. We present a post-training methodology using LLMON (LLM Object Notation), a framework that enables separating instructions from data and requires explicit identifier-based execution. We construct a large-scale structured training corpus of 4.6B tokens spanning general, RAG (Retrieval-Augmented Generation), and tool-calling data, augmented with distractor-infused variants in which competing instructions and irrelevant data segments are present in context but not referenced. Post-training two Granite-4.1 base models (3B and 8B) through supervised fine-tuning followed by reinforcement learning, we show that LLMON-structured supervision reduces attack success rates on indirect prompt-injection benchmarks from 53-100% to 0.02-5.6% while preserving general capabilities. We further introduce attention-based interpretability metrics that diagnose failure modes at individual transformer layers for structured inputs, revealing how our approach reshapes internal attention patterns. Our results demonstrate that explicit execution semantics can be taught through standard post-training without architectural modification.
• 中文摘要 大型语言模型（LLM）对所有上下文令牌一视同仁，缺乏可执行指令与被动数据的明确分离。这导致提示注入漏洞、指令绑定模糊和执行控制不可靠。我们提出了一种使用LLMON（LLM对象符号）的后训练方法论，该框架支持指令与数据分离，并要求显式基于标识符的执行。我们构建了一个大规模结构化训练语料库，包含46亿令牌，涵盖通用、检索增强生成（RAG）和工具调用数据，并辅以干扰变量，其中竞争指令和无关数据段存在于上下文中但不被引用。经过监督微调和强化学习训练两个Granite-4.1基础模型（3B和8B）后，我们表明LLMON结构化监督能将间接提示注入基准测试的攻击成功率从53-100%降至0.02-5.6%，同时保持一般能力。我们进一步引入基于注意力的可解释性指标，用于诊断结构化输入在各变换器层的故障模式，揭示了我们的方法如何重塑内部注意力模式。我们的结果表明，通过标准的后期训练，无需架构修改即可教授显式执行语义。

Zero Trust-Enabled Adaptive Incident Response using Hierarchical Reinforcement Learning

零信任驱动的自适应事件响应，采用分层强化学习

• Authors: Jisoo Lee, Bhagath Singh Bondili, Harshith Doppalapudi, Gwynith Godin, Nathaniel D. Bastian, Ankit Shah
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2610.38146
• Pdf link: https://arxiv.org/pdf/2610.38146
• Abstract Modern enterprises increasingly operate through complex information systems (IS) enabled by cloud computing, remote and hybrid work practices, and digitally integrated supply chains. These structural shifts have expanded organizational attack surfaces and weakened traditional perimeter-based trust assumptions, giving rise to new and persistent forms of cyber threats. In response, adversaries have shifted toward identity-centric, trust-based intrusion techniques that exploit legitimate access within organizational systems. This evolution has created a pronounced speed asymmetry, whereby attackers can adapt and act at machine speed while defenders remain constrained by reactive or rule-based incident response (IR) processes. Drawing on design science research, we propose ZT-AIR, an adaptive IR artifact that uses zero trust architecture as its design theory and hierarchical reinforcement learning as its kernel theory. ZT-AIR learns context-aware trust decisions, separates strategic prioritization from tactical execution, encodes zero-trust controls as actions, and jointly optimizes security effectiveness and operational continuity. Experimental results across diverse adversarial strategies, network scales, and defense environments show that ZT-AIR achieves a strong balance between security and continuity through frequent, minimally disruptive interventions. Practitioner validation further confirms the design's practical viability. This study contributes to IS research by extending computational design science from threat identification to response execution, theorizing zero trust architecture for adaptive IR, and deriving reusable design principles for adaptive decision support artifacts.
• 中文摘要 现代企业越来越多地通过云计算、远程和混合办公实践以及数字化集成供应链支持的复杂信息系统（IS）运营。这些结构性变化扩大了组织攻击面，削弱了传统的基于边界的信任假设，催生了新的、持续存在的网络威胁形式。对手转向以身份为中心、基于信任的入侵技术，利用组织系统内的合法访问。这种演变造成了明显的速度不对称，攻击者可以以机器速度适应和行动，而防御方则受制于被动或基于规则的事件响应（IR）流程。基于设计科学研究，我们提出了ZT-AI，一种自适应的IR产物，其设计理论为零信任架构，以分层强化学习为核心理论。ZT-AIR学习上下文感知的信任决策，将战略优先级与战术执行分离，将零信任控制编码为动作，并共同优化安全效能和运营连续性。在多种对抗策略、网络规模和防御环境中的实验结果表明，ZT-AIR通过频繁且干扰最小的干预，实现了安全与连续性的良好平衡。实践者验证进一步确认了该设计的实用可行性。本研究通过将计算设计科学从威胁识别延伸到响应执行，理论化自适应介入的零信任架构，并推导自适应决策支持工件的可复用设计原则，促进信息系统研究。

Estimating Implicit Expected Log Density Ratios

估算隐式预期对井密度比值

• Authors: Avi Amalanshu, Chandler Squires, Yizhou Lu, Pradeep Ravikumar
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.38151
• Pdf link: https://arxiv.org/pdf/2610.38151
• Abstract When exact probabilities cannot be computed, \emph{density ratio estimation (DRE)} is a key subroutine in the wide range of applications where density ratios appear, from experimental design to generative modelling and reinforcement learning. In this paper, we introduce the concept of an \emph{expected log density ratio (ELDR)}: log density ratios that are integrated under some third, possibly distinct measure. We show many problems with implicit densities where DRE is naïvely used are actually ELDR estimation tasks, and prove fundamental limitations on finite-sample ELDR estimation. Based on these results, we argue that the typical DRE-based approaches are unsuitable for ELDR estimation, and provide some foundational motivation towards bespoke ELDR estimators. To demonstrate, we propose a simple modification to some DRE methods that allows samples convergent to the integral distribution to appear in training data. We show through benchmarks that this can improve ELDR estimation error.
• 中文摘要 当无法计算精确概率时，\emph{密度比估计（DRE）}是密度比出现的广泛应用中的关键子程序，从实验设计到生成建模和强化学习。本文介绍了\emph{预期对数密度比（ELDR）}的概念：在某种第三种可能不同的度量下积分的对数密度比。我们展示了许多隐式密度问题，其中DRE被简单地使用，实际上是ELDR估计任务，并证明了有限样本ELDR估计的根本局限性。基于这些结果，我们认为典型基于DRE的方法不适合ELDR估计，并为定制ELDR估计量提供了一定的基础动力。为此，我们提出了对某些DRE方法的简单修改，使得与积分分布收敛的样本可以出现在训练数据中。通过基准测试，我们证明这可以改善ELDR估计误差。

Scaling Lean Theorem Proving for Real-World Code Verification

真实代码验证的缩放精益定理证明

• Authors: Debargha Ganguly, Divya Raghunathan, Dejan Jovanović, Vikash Singh, Soonho Kong, Aaron Tomb, Aman Goel, Ferhat Erata, Sam Bayless
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Logic in Computer Science (cs.LO); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.38174
• Pdf link: https://arxiv.org/pdf/2610.38174
• Abstract We present EasyProver, a Lean proving system built from recursive agents with isolated contexts, asynchronous search, parallel tactic probing, and an elastic elaboration service. Using an off-the-shelf frontier model with no additional training, it proves 669/672 PutnamBench problems and all six IMO 2026 formalizations and resolves 597/600 NTP4VC verification conditions and all 189 VERINA tasks by proof or refutation. Because these benchmarks are close to saturation, we also introduce Aegis, which converts competitive programming problems into much harder verification tasks over real implementations. The pipeline writes safe Rust solutions, translates them into Lean with Aeneas, generates specifications, and labels each frozen statement with a kernel-checked proof or refutation. It produces 2,045 differentially validated Rust-Lean program pairs and 11,533 candidate specifications. Unlike hand-written benchmarks, these tasks carry implementation-level obligations from fixed-width arithmetic, explicit failure, generated loops, casts, and functionalized mutation. On a stratified sample of 2,000 statements, EasyProver proves 1,054 (52.7%) within a 30-minute budget per statement. The same tasks can also be used to benchmark other provers and as rewards for reinforcement learning.
• 中文摘要 我们介绍EasyProver，这是一个由递归代理构建的精益证明系统，具有隔离上下文、异步搜索、并行战术探测和弹性展开服务。使用现成的前沿模型，无需额外培训，它证明了669/672个PutnamBench问题和全部6个IMO 2026形式化，并通过证明或反驳解决了597/600个NTP4VC验证条件和全部189个VERINA任务。由于这些基准测试接近饱和，我们还引入了Aegis，将竞争性编程问题转化为比实际实现更难的验证任务。该流水线编写安全的Rust解决方案，将其翻译为带有Aeneas的精益，生成规范，并为每个冻结语句标注内核检查的证明或反驳。它生成了2,045对差分验证的Rust-Lean程序对和11,533个候选规范。与手写基准不同，这些任务承担了固定宽度算术、显式失败、生成循环、铸造和泛函化变异等实现层级义务。在2,000个分层样本中，EasyProver在每个语句30分钟预算内证明了1,054个（52.7%）。同样的任务也可用于其他证明器的基准测试，并作为强化学习的奖励。

Two Birds with One Stone: Skill-Augmented Generation via Unified Retrieval and Utilization

一举两得：通过统一检索与利用实现技能增强生成

• Authors: Zihan Luo, Jiahui Kang, Wei Zhou, Qilin Fan, Mingyan Li, Chao Chen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.38204
• Pdf link: https://arxiv.org/pdf/2610.38204
• Abstract Skills, as reusable external knowledge, have demonstrated substantial potential to enhance the ability of Large Language Models (LLMs) to solve complex tasks. However, existing skill-augmented generation methods typically rely on separate models for skill retrieval and skill utilization, resulting in disconnected optimization of these two processes. To address this gap, we propose Skill-Uni, a unified reinforcement learning framework that jointly optimizes skill retrieval and utilization. Specifically, we adopt a generative skill retrieval paradigm that enables a single model to both learn how to retrieve relevant skills and leverage them for task solving. To effectively train this unified framework, we introduce a two-stage training strategy consisting of warm-up instruction tuning and GRPO, allowing the model to progressively improve its skill retrieval and utilization capabilities. Extensive experiments on five skill-related benchmarks demonstrate the effectiveness of Skill-Uni, yielding average performance gains of 66.59% and 34.82% over the base Qwen3-4B and LLaMA3.1-8B models, respectively. Besides, Skill-Uni is compatible with various skill retrieval engines, demonstrating its flexibility and generalizability. Additional in-depth analyses also provide insights into the effectiveness of Skill-Uni.
• 中文摘要 技能作为可复用的外部知识，已展现出显著潜力，能够提升大型语言模型（LLMs）解决复杂任务的能力。然而，现有的技能增强生成方法通常依赖于技能检索和技能利用的独立模型，导致这两者优化脱节。为弥补这一空白，我们提出了Skill-Uni，一个统一的强化学习框架，共同优化技能检索与利用。具体来说，我们采用了生成式技能检索范式，使单个模型既能学习如何检索相关技能，又能将其用于任务解决。为有效训练这一统一框架，我们引入了两阶段训练策略，包括热身式教学调优和GRPO，使模型逐步提升技能检索和利用能力。在五个技能相关基准测试上的广泛实验证明了Skill-Uni的有效性，平均性能分别提升了66.59%和34.82%的基础Qwen3-4B和LLaMA3.1-8B模型。此外，Skill-Uni兼容多种技能检索引擎，展现了其灵活性和通用性。更多深入分析也为Skill-Uni的有效性提供了见解。

Variational Consequence-Driven Offline Reinforcement Learning

变分后果驱动离线强化学习

• Authors: Wen Jiang, Ke Jiang, Xiaoyang Tan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38238
• Pdf link: https://arxiv.org/pdf/2610.38238
• Abstract Offline reinforcement learning (RL) typically mitigates distribution shift by imposing divergence constraints on the action distributions of the learned and behavior policies. However, this paradigm cannot ensure the underlying dynamic consistency structure: even minor action deviations can lead to drastically different transitions, inadvertently driving the agent away from the offline dataset. To address this, we explore policy constraints through their induced state-consequence distributions rather than relying on pure action-distribution matching. Nevertheless, directly enforcing such consequence-aware constraints is computationally intractable without access to the true underlying dynamics model. To bypass this issue, we propose Structural Behavior Regularization (StBR), which replaces the intractable objective with a closed-form latent divergence. Structuring this latent space to reflect transition dynamics renders the latent divergence a theoretically bounded surrogate for the true consequence-distribution divergence. By regularizing the learned policy with this latent divergence, StBR adaptively tightens the behavior constraint in dynamics-sensitive regions, effectively preventing deviations outside the offline dataset. Empirically, we demonstrate that StBR achieves superior performance across diverse offline RL benchmarks.
• 中文摘要 离线强化学习（RL）通常通过对已学策略和行为策略的行动分布施加发散约束来缓解分布偏移。然而，该范式无法保证其潜在的动态一致性结构：即使是微小的动作偏差也可能导致截然不同的转变，从而无意中使智能体远离离线数据集。为此，我们通过策略约束的诱导状态-结果分布来探讨，而非依赖纯粹的动作-分布匹配。然而，如果没有真正的底层动力学模型，直接强制执行此类后果感知约束在计算上是难以解决的。为绕过这一问题，我们提出了结构行为正则化（Structural Behavior Regularization，StBR），它用闭式潜在发散取代了难以处理的目标。将该潜在空间结构化为反映过渡动态，使潜在发散成为真实结果-分布发散的理论有界替代。通过对学习策略进行正则化，StBR在动态敏感区域自适应地收紧了行为约束，有效防止离线数据集外的偏差。通过实证，我们证明StBR在多种离线强化学习基准测试中表现优异。

Transportable Offline Reinforcement Learning in Presence of Unmeasured Confounding

在存在未测量混杂因素下的可移动离线强化学习

• Authors: Junyi Liao, Hangqi Ren, Zhuoran Yang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38253
• Pdf link: https://arxiv.org/pdf/2610.38253
• Abstract Offline reinforcement learning seeks to learn decision policies from previously collected data, but deployment environments may differ from the domains in which the data were collected, and logged variables may omit confounders that affected both historical actions and outcomes. We study offline policy learning under the joint challenges of domain discrepancy and unmeasured confounding, where multiple heterogeneous source datasets provide complementary information about a target domain. We develop a causally adjusted transportability framework combining transportability theory with instrumental variables, negative controls, and proxies to identify target-domain causal quantities. For confounded contextual bandits, identification reduces to systems of bridge-function integral equations, enabling minimax estimation from finite samples. Confidence sets for the identified quantities are then used for pessimistic policy optimization, yielding CAT-CCB. We extend the framework to episodic confounded Markov decision processes, where complementary sources identify target reward and transition mechanisms under structural and completeness conditions. Under Gaussian dynamics and linear function approximation, this yields CAT-CMDP. We establish high-probability finite-sample suboptimality guarantees that quantify the effects of approximation, estimation, source-to-target shift, offline coverage, and sample size. For bounded finite-dimensional linear classes, the contextual-bandit guarantee recovers the standard $\widetilde{\mathcal O}(N^{-1/2})$ rate. Numerical experiments support the effectiveness of the framework for learning target-domain policies from heterogeneous confounded data. Together, the results provide a principled approach when no single source dataset suffices for causal identification and policy optimization.
• 中文摘要 离线强化学习旨在从先前收集的数据中学习决策策略，但部署环境可能与数据收集的领域不同，且记录变量可能遗漏影响历史行为和结果的混杂因素。我们在领域差异和未测量混杂这两个挑战下研究离线策略学习，多个异构源数据集提供目标域的互补信息。我们开发了一个因果调整的可迁移性框架，结合可迁移性理论与工具变量、负对照和代理指标，以识别目标域因果量。对于混淆上下文强盗，识别归结为桥函数积分方程组，从而实现有限样本的极小极大估计。随后，识别出的量的置信集用于悲观策略优化，得到CAT-CCB。我们将该框架扩展到情节混淆马尔可夫决策过程，其中互补源在结构和完备条件下识别目标奖励和转移机制。在高斯动力学和线性函数近似下，这得到了CAT-CMDP。我们建立了高概率有限样本次优保证，量化了近似、估计、源到目标偏移、离线覆盖率和样本量的影响。对于有界有限维线性类，上下文强盗保证恢复了标准的$\widetilde{\mathcal O}（N^{-1/2}）速率。数值实验支持该框架在从异构混杂数据中学习目标域策略的有效性。综合而言，当单一源数据集不足以进行因果识别和策略优化时，这些结果提供了一种原则性的方法。

Does 3D Pose Improve Action Valuation in Football? An Empirical Study of Pose Representations and Learning Objectives

3D姿势能提升足球中的动作价值吗？姿势表现与学习目标的实证研究

• Authors: Rikuhei Umemoto, Kenjiro Ide, Soham Majumder, Ulf Brefeld, Keisuke Fujii
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38308
• Pdf link: https://arxiv.org/pdf/2610.38308
• Abstract Football action valuation estimates players' contributions to future scoring. Existing multi-agent deep reinforcement learning (MADRL) models use 2D positions and velocities, omitting posture information relevant to imminent actions. We investigate whether adding 3D pose to a SARSA-based model improves observed-action values relative to attack returns: goals, conceded goals or final expected possession value. Using UEFA Champions League 2022/23 event, tracking and 3D pose data, we compare six representations (no pose, raw joints, torso and head orientation, and spatial-temporal graph convolutional network (ST-GCN) embeddings) across five learning settings and three seeds. The settings vary temporal-difference targets and action-prediction loss, with Monte Carlo return regression as a reference. Across 50 on-ball and off-ball comparisons, pose robustly improved return prediction in only three, all using torso and head orientation, while worsening it in twelve, mostly with ST-GCN. Training choices had greater effects: removing the action-prediction loss raised on-ball skill of position-only values from 0.003 to 0.040 and their correlation with expected goals from 0.23 to 0.60; the improvement in return prediction persisted at matched training length. Separately, pose improved prediction of the on-ball action 0.2 s ahead (macro AUC 0.914 to 0.934 with raw joints), while the passer's pose during execution improved pass-success prediction (AUC 0.844 to 0.856). Within this training protocol, pose informed imminent actions but rarely improved observed-action values.
• 中文摘要 足球动作评估估计球员对未来得分的贡献。现有的多智能体深度强化学习（MADRL）模型使用二维位置和速度，省略与即将发生动作相关的姿势信息。我们研究向基于SARSA的模型添加三维姿势是否能提升观察到的动作值相对于进攻回攻：进球、失球或最终预期控球值。利用2022/23赛季欧冠赛事、跟踪和三维姿势数据，我们比较了六个表示方式（无姿势、原始关节、躯干和头部方向，以及时空图卷积网络（ST-GCN）嵌入），涵盖五个学习场景和三个种子。这些设置不同，时间差目标和动作预测损失，以蒙特卡洛回归为参考。在50次持球和无球比较中，仅有3次姿势显著提升，均采用躯干和头部方向，12次则较差，主要为ST-GCN。训练选择效果更大：去除动作预测损失后，位置单一的持球技能值从0.003提升至0.040，与预期进球的相关性从0.23提升至0.60;回攻预测的提升在匹配训练时仍持续。另外，0.2秒前的持球动作姿势预测改善（原始关节时宏AUC 0.914至0.934），传球手执行时姿势提升传球成功预测（AUC 0.844至0.856）。在该训练协议中，姿势告知即将发生的动作，但很少改善观察动作值。

Game-Theoretic Foundations for Undiscounted Bisimulation Metrics

无贴现双模拟度量的博弈论基础

• Authors: Matthew Zurek, Yudong Chen, Josiah P. Hanna
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.38313
• Pdf link: https://arxiv.org/pdf/2610.38313
• Abstract Bisimulation metrics offer a principled notion of task-relevant state similarity in reinforcement learning, but existing constructions and methods are largely confined to discounted MDPs, where contraction arguments ensure well-defined fixed points and downstream guarantees. We develop a unified foundation for bisimulation metrics in undiscounted settings, that is total- and average-reward problems, by replacing the standard operator-centric view with a game-centric one: we interpret prior metric operators as Bellman-Shapley operators of zero-sum turn-based games and define metrics as the corresponding game value functions, yielding unique definitions even when $\gamma=1$. This game-theoretic foundation facilitates transfer of dynamic-programming theory and algorithms. We also use this game-centric perspective to characterize how metric finiteness depends on structural properties of the underlying MDP and motivate certain directed metrics that remain finite in weakly communicating MDPs. We further introduce an abstract framework for modifying the games' action spaces, allowing general transition coupling sets, that recovers many existing metrics and guides new constructions. Finally, we prove that these metrics bound total-reward and, when the optimal gain is state-independent, optimal relative value differences. We also provide error and suboptimality guarantees for aggregation-based planning in total- and average-reward settings.
• 中文摘要 双模拟度量为强化学习中任务相关状态相似性提供了原则性概念，但现有构造和方法主要局限于折现MDP，其中收缩论证确保了明确定义的不动点和下游保证。我们通过用以博弈为中心的观点取代标准的算子中心视角，为未贴现环境下的双模拟度量（即全奖励问题）建立统一基础：我们将先前的度量算子解释为零和回合制游戏的Bellman-Shapley算子，并将度量定义为对应的博弈价值函数，即使$\gamma=1$时也能产生唯一定义。这一博弈论基础促进了动态规划理论和算法的转移。我们还利用这一博弈中心视角来描述度量有限性如何依赖底层MDP的结构性质，并激励某些在弱沟通MDP中保持有限的有向度量。我们进一步引入了一个抽象框架，用于修改游戏的动作空间，允许通用的过渡耦合集，恢复许多现有指标并指导新的构建。最后，我们证明这些指标限制了总奖励，当最优收益与状态无关时，则限制了最优相对价值差异。我们还为基于聚合的规划提供了总奖励和平均奖励环境下的错误和次优保证。

CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained Optimization

CSO：通过技能分布对齐和技能粒度优化优化机器人政策

• Authors: Zhiyuan Xiang, Xiang Deng, Wanjie Tao, Xu Chen, Zhao Jielun, Weili Guan
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38357
• Pdf link: https://arxiv.org/pdf/2610.38357
• Abstract Discretizing continuous actions into discrete skills using methods like VQ-VAE has emerged as a powerful paradigm for robotic manipulation. However, quantization errors yield a suboptimal distribution for the skill prior, degrading its performance. While reinforcement learning offers a path for refinement, its direct application faces unstable encoder updates and a granularity dilemma in importance sampling. To address these challenges, we introduce Cascaded Skills Optimization (CSO), a two-stage post-training framework. First, Rejection-Sampling Supervised Fine-tuning aligns the model's observation-to-skill mapping with the distribution of successful online trajectories via supervised fine-tuning. Second, with the observation encoder frozen, Skills Policy Optimization computes an independent, clipped importance ratio for each complete skill, enabling stable and efficient updates. With a lightweight QueST-style policy, CSO raises overall LIBERO success from 80.1% to 97.5%, improves performance on RoboTwin2.0, and achieves 61.0% success across five physical single-arm tasks. Additional comparisons with FAST, discrete BEAST, and fixed-buffer IQL further validate the effectiveness of our two-stage post-training strategy.
• 中文摘要 利用VQ-VAE等方法将连续动作离散化为离散技能，已成为机器人操作的强大范式。然而，量化误差会导致前述技能分布不优，降低其性能。虽然强化学习提供了精炼路径，但其直接应用面临不稳定的编码器更新和重要性采样的粒度困境。为应对这些挑战，我们引入了级联技能优化（CSO），这是一个两阶段的后训练框架。首先，拒绝-抽样监督微调将模型的观察到技能映射与通过监督微调成功在线轨迹的分布对齐。其次，在观察编码器冻结后，技能策略优化为每个完整技能计算独立且截断的重要性比，实现稳定高效的更新。采用轻量级QueST式政策，CSO将整体LIBERO成功率从80.1%提升至97.5%，提升RoboTwin2.0性能，并在五个单臂物理任务中实现61.0%。与FAST、离散BEAST和固定缓冲IQL的进一步比较进一步验证了我们两阶段训练后策略的有效性。

SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning

范围：能动强化学习的选择性关键步骤策略自我蒸馏

• Authors: Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38391
• Pdf link: https://arxiv.org/pdf/2610.38391
• Abstract Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.
• 中文摘要 基于结果的强化学习为长视野LLM代理提供粗糙监督：失败的轨迹可能包含正确的步骤，而成功的轨迹可能包含错误。轨迹级技能还可能在无关步骤改变教师偏好，引入偏离目标的提炼信号。我们介绍了SCOPE，一个选择性关键步骤策略自蒸馏框架。在监督热身后，该策略分析自身轨迹，生成步骤特定的事后诸葛亮技能，并仅在选定错误步骤提炼其指导。有界的带符号师生系数稳定辅助更新，同时保持正强化和负向纠正。成功相对的成对长度奖励进一步鼓励高效完成任务。ALFWorld和WebShop上的实验显示，多个模型骨干间持续有持续的改进。在Qwen2.5-3B中，SCOPE在ALFWorld上实现了97.7%的成功率，在WebShop上达到了85.2%，分别比SEED高出5.9%和6.3个百分点。在未公开的ALFWorld任务中，SCOPE的成功率达到89.6%，比GRPO提升23.2个百分点，且交互次数减少了46.7%。扩展到Qwen3-32B后，SCOPE在已见ALFWorld任务中成功率为99.2%，在未公开任务中成功率为94.0%。

Vision-Based Density Estimation for Adaptive Traffic Control

基于视觉的密度估计用于自适应交通控制

• Authors: Mohammad Usaid, Deepak Silaych, Hrithik Mhatre, Archak Mittal
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.38497
• Pdf link: https://arxiv.org/pdf/2610.38497
• Abstract Traffic congestion is becoming a critical challenge in rapidly growing urban centers, particularly in developing nations where rapid urbanization and inadequate transport infrastructure place increasing strain on transport networks. In highly heterogeneous and chaotic traffic flows, camera-based traffic-state estimation is limited by severe vehicle occlusion and overlap. In this paper, we propose a framework to optimize traffic flow directly from stochastic visual observations. The architecture is structurally decoupled into three primary components: (a) vehicle density estimation, (b) predictive dynamics modeling, and (c) policy optimization. The proposed methodology introduces a convolutional neural network (CNN) that learns the mapping from camera feeds to spatial density maps, utilizing an adaptive k-nearest-neighbor (k-NN) Gaussian kernel to handle varying congestion levels. This feeds into an ST3DNet predictive module for spatiotemporal forecasting, followed by a hierarchical Graph-Transformer Multi-Agent Reinforcement Learning (MARL) algorithm to optimize signal phases. The density estimation stage was evaluated on 40 annotated intersection images from Rambagh 4, Jaipur, India, with a reported five-fold validation mean absolute error (MAE) of $3.32 \pm 0.95$ vehicles and a mean absolute percentage error (MAPE) of 14.5%. These results pertain to the perception component; empirical training and evaluation of the forecasting and control stages remain future work.
• 中文摘要 交通拥堵正成为快速发展的城市中心中的一个关键挑战，尤其是在发展中国家，快速城市化和交通基础设施不足给交通网络带来了日益加大的压力。在高度异质和混乱的交通流中，基于摄像头的交通状态估计受限于严重的车辆阻塞和重叠。本文提出了一个框架，直接通过随机视觉观测优化交通流。该架构结构上分为三个主要组成部分：（a）车辆密度估计，（b）预测动力学建模，（c）策略优化。提出的方法引入了卷积神经网络（CNN），利用自适应的k-NN高斯核，从摄像头画面学习到空间密度图的映射，以应对不同的拥堵程度。该模块输入ST3DNet预测模块进行时空预测，随后采用分层图-变换器多智能体强化学习（MARL）算法优化信号相位。密度估计阶段基于印度斋浦尔Rambagh 4的40张带注释交叉图像进行评估，报告五重验证平均绝对误差（MAE）为3.32 \pm 0.95美元车辆，平均绝对百分比误差（MAPE）为14.5%。这些结果涉及感知部分;预测和控制阶段的实证训练和评估仍将持续进行。

Force-Driven Hook Disassembly via Constraint-Aware Object-Centric Reinforcement Learning

通过约束感知对象强化学习实现力驱动钩子拆解

• Authors: Nadeen Elhadad, Takuya Kiyokawa, Kensuke Harada
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.38513
• Pdf link: https://arxiv.org/pdf/2610.38513
• Abstract The growing volume of end-of-life products has intensified the need for automated disassembly, yet hook-type connections remain challengin: their orientations vary, they deform under load, and their geometry is hidden, so vision alone cannot identify a feasible non-destructive disassembly direction. This work proposes a constraint-aware reinforcement learning framework that infers feasible extraction directions online from contact forces. The policy is trained in a robot-agnostic, object-centric simulation on a simplified peg-in-hole pair and deployed zero-shot on a physical robot, where it transfers to real connections with unseen, multi-stage constraint structures. Because instantaneous contact forces are noisy and locally ambiguous, the policy is coupled with a particle filter belief that provides the policy with temporally integrated directional information and reduces redundant exploratory motion. The approach is validated on four real-world tasks, from peg-in-hole pairs to the inner propeller and front cover of an end-of-life air-conditioning unit. The full method achieves success rates of 100%, 100%, 90%, and 75% with the highest path efficiency on every task, and raises the success rate on the air-conditioning cover from 45% to 75% relative to a policy trained without the belief. These results indicate that force-driven reinforcement learning can enable disassembly without product-specific models or demonstrations.
• 中文摘要 报废产品的数量不断增加，自动化拆解的需求加剧了需求，但钩式连接依然充满挑战：它们的方向变化，受载时变形，几何形状隐藏，仅凭视觉无法确定可行的非破坏性拆解方向。本研究提出了一种约束感知强化学习框架，能够从接触力中在线推断可行的提取方向。该策略在机器人无关、以物体为中心的模拟中训练于简化的孔中钉对，并在物理机器人上部署零射击，随后转移到带有未见多级约束结构的真实连接。由于瞬时接触力噪声大且局部模糊，策略与粒子滤波器信念结合，为策略提供时间整合的方向信息，减少冗余的探索运动。该方法在四个实际任务中得到了验证，从孔中钉对到空调设备的内螺旋桨和前盖。完整方法在每个任务中实现了100%、100%、90%和75%的成功率，路径效率最高，并且空调覆盖的成功率从45%提升到75%，相较于无信念训练的策略。这些结果表明，力驱动强化学习能够实现无需特定产品模型或演示即可拆解。

A Minimally Informed Black-Box Fake-Report Attack on DRL Traffic-Signal Control

对DRL交通信号控制的有限信息黑匣子伪报告攻击

• Authors: Amro Saleh, Loukas Lazos, Mohammad J. Abdel-Rahman
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2610.38557
• Pdf link: https://arxiv.org/pdf/2610.38557
• Abstract Deep reinforcement learning (DRL) traffic-signal controllers may use vehicle-to-infrastructure (V2I) reports, exposing their perception layer to false-data injection. Existing demonstrations typically assume either white-box access to the controller's weights or full observability of the traffic state. We study the impact of a weaker and more realistic adversary who has no access to the RL model and whose traffic observations are limited to her own approach. This black-box, minimally informed attacker trains an imitation surrogate of the controller. She then crafts bounded fake reports, restricted to inflating vehicle counts on her own approach, that transfer to the real controller. We demonstrate that despite the loss of accuracy relative to an attacker who can observe the entire state and action space, this limited attacker can significantly increase the average waiting time for other approaches while barely changing the waiting time on her own approach, i.e. externalizing the added delay onto the other approaches. Because the only realizable move is to inflate one's own reported queue, we find that a zero-knowledge attacker that simply reports the maximum feasible queue is as damaging as the surrogate-based one, while an unstructured random inflation is not: the surrogate is not required to reproduce the degradation, and the exposed vulnerability is the V2I sensing channel itself. The surrogate's role is directional control, selecting a disruptive or a selfish objective at the same budget. The attack persists under +/-20% uniform demand variation. Finally, on a common per-lane budget we place the attacker on a knowledge ladder against white-box adversaries, and find the black-box surrogate matches a realizable white-box attacker that uses the victim's own gradients; only abandoning realizability raises the damage further.
• 中文摘要 深度强化学习（DRL）交通信号控制员可能使用车辆到基础设施（V2I）报告，使其感知层暴露于虚假数据注入的风险。现有演示通常假设对控制者权重的白盒访问或交通状态的完全可观测性。我们研究了一个较弱且更现实、无法访问强化学习模型且交通观察受限于自身方法的对手的影响。这个黑匣子、信息有限的攻击者训练了一个控制者的模拟替代者。她随后制作有界的假报告，限制在自己进近时膨胀车辆数量，并将这些报告传递给真实控制者。我们证明，尽管相较于能观察整个状态和行动空间的攻击者，该受限攻击者可以显著增加其他进近的平均等待时间，同时几乎不改变自身进近的等待时间，即将额外的延迟外化到其他进近进近。由于唯一可实现的举措是膨胀自己报告队列，我们发现仅报告最大可行队列的零知识攻击者与基于替代者的攻击者同样有害，而非结构化的随机膨胀则不然：替代者无需重现退化，暴露的漏洞正是V2I感知通道本身。代理人的角色是方向控制，在相同预算下选择干扰目标或自私目标。攻击在+/-20%的均匀需求变化下持续存在。最后，在统一的每条路预算下，我们将攻击者置于对白盒对手的知识阶梯上，发现黑盒替代者匹配于使用受害者自身梯度的可实现白盒攻击者;只有放弃实现性才会进一步提升伤害。

Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

通过能力意识共同演进政策和评分标准，实现开放式生成

• Authors: Chengsheng Zhang, Wenshen Xu, Yuchuan Wu, Zhiyuan Yu, Junfeng Fang, Tianlong Wang, Bingxue Qiu, Yongbin Li, Xiang Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38566
• Pdf link: https://arxiv.org/pdf/2610.38566
• Abstract Rubric-based reinforcement learning (RL) offers interpretable, fine-grained supervision for aligning large language models (LLMs) on open-ended generation tasks. However, static rubric criteria may become saturated or remain beyond the policy's exploration, rendering the reward signals insufficiently discriminative. While recent efforts seek to guide sampling exploration or evolve rubrics, they fail to explicitly factor in how rubric difficulty should track the evolving policy's capabilities, resulting in a persistent mismatch between the optimization objective and the policy's current capability frontier. In this paper, we propose \textbf{Co}-Evolving Policy and Rubric via \textbf{C}apability \textbf{A}wareness (\textbf{CoCA}), a closed-loop framework that jointly adapts the rubric and policy through criterion-level diagnosis. Specifically, CoCA contrasts query-only and rubric-guided rollouts to characterize each criterion relative to the current policy, recalibrates criteria with mismatched difficulty, and identifies guidance-revealed capability gaps for frontier-based learning. Based on this insight, CoCA either expands rubric-conditioned frontiers through guided optimization or transfers qualified gains through on-policy self-distillation. As the policy updates, CoCA improves the capability diagnosis and enables more reliable distillation, thereby completing the co-evolution loop. Extensive experiments across diverse benchmarks and LLMs demonstrate that CoCA consistently improves performance over rubric-based RL methods. Moreover, CoCA promotes the internalization of guidance-revealed capabilities into query-only behavior while mitigating capability degradation.
• 中文摘要 基于评分标准的强化学习（RL）为大型语言模型（LLMs）在开放式生成任务上的对齐提供了可解释、细粒度的监督。然而，静态评分标准可能过于饱和或超出策略探索范围，导致奖励信号的区分力不足。尽管近期努力试图指导抽样探索或演进评分标准，但它们未能明确考虑评分标准难度如何追踪策略能力，导致优化目标与策略当前能力前沿之间存在持续不匹配。本文提出通过 \textbf{C}apability \textbf{A}意识（\textbf{CoCA}）演化策略和评分标准，这是一个闭环框架，通过标准层面诊断共同调整评分标准和策略。具体来说，CoCA对比了仅查询和评分标准引导的推广，以区分每个标准相对于当前策略的特征，重新校准难度不匹配的标准，并识别了基于前沿学习的指导揭示能力差距。基于这一洞察，CoCA要么通过引导优化扩展评分标准条件前沿，要么通过策略上的自我蒸馏转移合格收益。随着策略更新，CoCA提升能力诊断，实现更可靠的蒸馏，从而完成共演化循环。跨多种基准测试和大型语言模型的大量实验表明，CoCA在性能上持续优于基于评分标准的强化学习方法。此外，CoCA促进了指导揭示能力内化为仅查询行为，同时减轻能力退化。

Latent Decision Flow Rewards for Clinical Guideline Reasoning

临床指南推理的潜在决策流奖励

• Authors: Gaurav Bhatt, Marinka Zitnik
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38709
• Pdf link: https://arxiv.org/pdf/2610.38709
• Abstract Clinical practice guidelines encode evidence-based recommendations as conditional decision flows, which makes them a natural source of process supervision for training reasoning models. Two properties of clinical reasoning make this supervision difficult to use. A decision flow consistent with a guideline does not by itself determine the correct answer for a given patient, which we call the path--answer gap, and several valid decision flows can support the same clinical decision, which we call path multiplicity. Current training approaches address parts of this problem. Answer rewards supervise the final decision, and process rewards computed against a reference flow supervise one valid decision flow per case. Supervised fine-tuning on annotated flows teaches guideline-consistent reasoning but constrains the model to one trajectory per case and can reduce answer accuracy. We introduce GUIDER, a method that jointly optimizes a decision flow controller and a reasoning policy. The controller learns to generate plausible decision flows from guideline-derived supervision, and the policy learns to generate a decision flow and a final answer for each patient case. GUIDER computes the process reward for the policy over several controller-generated flows, which accounts for path multiplicity, and gates this reward on answer correctness, which addresses the path--answer gap. Joint training allows the process supervision to evolve with the policy. On MedGUIDE, GUIDER with Qwen3-14B achieves 82.5 weighted accuracy and 73.5 guideline adherence, 3.3 and 3.1 points above the strongest clinical reinforcement learning baseline. With Qwen3-14B and Gemma4-12B backbones, GUIDER also improves over CRPO, MRPO, and FaithMed on the out-of-domain benchmarks AMEGA-LLM, RealMedQA, and CancerGUIDE.
• 中文摘要 临床实践指南将循证建议编码为条件决策流程，因此成为训练推理模型的过程监督的自然来源。临床推理的两个特性使得这种监督难以使用。与指南一致的决策流程本身无法确定给定患者的正确答案，我们称之为路径-答案差距，多个有效的决策流程可以支持同一临床决策，我们称之为路径多重性。当前的培训方法解决了这一问题的部分问题。答案奖励监督最终决策，而基于参考流程计算的过程奖励则监督每个案例一个有效的决策流程。对注释流程进行监督微调教授指南一致性推理，但限制模型每个案例只能遵循一个轨迹，且可能降低答案准确性。我们引入了GUIDER，一种联合优化决策流程控制器和推理策略的方法。控制者学习从指南衍生的监督中生成合理的决策流程，策略则学习为每个患者病例生成决策流程和最终答案。GUIDER计算策略在多个控制器生成流程中的过程奖励，考虑路径多重性，并根据答案正确性进行奖励，解决路径-答案差距。联合培训使流程监督能够随着策略演进。在MedGUIDE中，GUIDER配合Qwen3-14B实现82.5加权准确率和73.5指南依从性，比最强临床强化学习基线高出3.3和3.1分。凭借Qwen3-14B和Gemma4-12B骨干，GUIDER在域外基准测试AMEGA-LLM、RealMedQA和CancerGUIDE上也优于CRPO、MRPO和FaithMed。

Value Function-Guided MPPI for Hierarchical Planning and Control in Off-Road Autonomous Navigation

价值函数引导MPPI，用于越野自主导航中的层级规划与控制

• Authors: Anoushka Alavilli, Raymond Song, Guanya Shi, Jeff Schneider
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.38731
• Pdf link: https://arxiv.org/pdf/2610.38731
• Abstract Autonomous navigation in off-road terrain is commonly achieved using hierarchical control and planning architectures that rely on explicit cost maps and one-way information flow from a high-level planner to a low-level controller, decoupling planning from the true execution cost experienced during control. We propose a hierarchical planning and control framework in which information flows bidirectionally, allowing the low-level controller to guide trajectory selection at the planning level. The highest level of the hierarchy is a Model Predictive Path Integral (MPPI) control-style planner that determines optimal trajectories and passes these waypoints to a pretrained low-level reinforcement learning (RL) controller. Rather than drawing from an explicit cost map, our MPPI planner leverages the value function of the RL controller as an implicit cost representation. In this formulation, MPPI serves as a trajectory exploration mechanism, while the low-level controller's value function guides the search. To account for the distribution shift seen by the value function between training and evaluation, we propose an online policy evaluation process that adapts the value function to unseen driving experiences, and we demonstrate this adaptation in unseen off-road simulation environments. We evaluate our approach in a high-fidelity off-road driving simulator across a range of challenging terrain conditions and compare its performance against a baseline MPPI controller. The results demonstrate that our method achieves safer and more effective navigation to goal locations while eliminating the need for explicit cost maps and enabling tighter coupling between planning and control.
• 中文摘要 越野地形中的自主导航通常通过依赖明确成本图和从高级规划器到低级控制器的单向信息流实现的分层控制与规划架构，将规划与控制过程中实际执行成本脱钩。我们提出了一个分层规划与控制框架，信息双向流动，使低级控制器能够在规划层面指导轨迹选择。层级结构的最高层是模型预测路径积分（MPPI）控制风格规划器，它确定最优路径，并将这些路径点传递给预训练的低级强化学习（RL）控制器。我们的MPPI规划器不是从显式成本地图中提取，而是利用RL控制器的价值函数作为隐式成本表示。在这种表述中，MPPI作为轨迹探索机制，而低级控制器的价值函数则指导搜索。为了考虑价值函数在训练与评估之间观察到的分布偏移，我们提出了一种在线政策评估流程，将价值函数适配到未见的驾驶体验，并在未见的越野模拟环境中展示了这一适应。我们在高保真度越野驾驶模拟器中评估了我们的方法，适用于各种具有挑战性的地形条件，并将其性能与基线MPPI控制器进行比较。结果表明，我们的方法实现了更安全、更高效的目标导航，同时消除了对明确成本图的需求，并实现了规划与控制之间的紧密耦合。

RuleForge: Inferring and Instantiating Latent Visual Rules as Executable Programs

RuleForge：推断并实例化潜在的可视化规则作为可执行程序

• Authors: Weifeng Xu, Yaxiong Wang, Zhedong Zheng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.38737
• Pdf link: https://arxiv.org/pdf/2610.38737
• Abstract Visual rule reasoning asks whether multimodal models can recover an abstract concept from a few examples. Existing studies either infer rules from visual examples but evaluate only classification or verbalization, or supervise executable generation with rules provided as input. Consequently, the relationship between visual rule understanding and reliable rule instantiation remains insufficiently explored. We present Bongard-Forge100k, a controlled benchmark, together with RuleForge, a backbone-agnostic framework that jointly learns visual rule understanding and rule-conditioned executable generation. The benchmark contains 108,797 problems: 86,000 for training, 19,297 for testing familiar rules, and 3,500 for testing held-out rule compositions. The framework progresses through rule-understanding training, joint understanding and scene-program supervision, and refinement via execution-grounded reinforcement learning with Understanding Replay. Executing generated programs under multiple random seeds provides verifiable feedback on executability, rule satisfaction, diversity, and robustness. With UniLIP as the primary backbone, RuleForge improves multiple-choice accuracy over the base model by 20.93 percentage points on familiar rules and 28.42 percentage points on held-out rule compositions, while increasing scene-level rule satisfaction on familiar rules from 2.91% to 70.99%. Additional experiments on InternVL3.5, Qwen3-VL, and Show-o2 demonstrate the effectiveness of the proposed training data and framework across diverse multimodal architectures.
• 中文摘要 可视化规则推理探讨多模态模型是否能从少数示例中恢复抽象概念。现有研究要么从视觉示例推断规则，但仅评估分类或口述，要么监督以规则作为输入的可执行文件生成。因此，视觉规则理解与可靠规则实例化之间的关系尚未充分探讨。我们介绍了Bongard-Forge100k，一个受控基准测试，以及RuleForge，一个骨干无关框架，结合视觉规则理解和规则条件可执行文件生成。基准测试包含108,797个问题：86,000个用于训练，19,297个用于测试熟悉规则，3,500个用于测试保留规则组合。该框架通过规则理解训练、联合理解和场景程序监督，以及通过执行基础强化学习（配合Understanding Replay）进行完善。在多个随机种子下执行生成程序，可以获得可验证的可执行性、规则满意度、多样性和鲁棒性的反馈。以UniLIP为主要骨干，RuleForge在熟悉规则上使基础模型的多项选择准确率提升了20.93个百分点，在保留规则组合上提高了28.42个百分点，同时将熟悉规则的场景级规则满意度从2.91%提升至70.99%。InternVL3.5、Qwen3-VL和Show-o2上的额外实验展示了所提训练数据和框架在多种多模态架构中的有效性。

Skillful and realistic precipitation forecasting with reinforced diffusion

增强扩散的技术性且真实的降水预报

• Authors: Ozan Sener, Anand Avati, David Hafner, Jierui Lin, Laura Manduchi, Xiaoying Pang, Langwen Huang, Torsten Hoefler, Russ Schumacher, Vladlen Koltun
• Subjects: Subjects: Machine Learning (cs.LG); Atmospheric and Oceanic Physics (physics.ao-ph)
• Arxiv link: https://arxiv.org/abs/2610.38775
• Pdf link: https://arxiv.org/pdf/2610.38775
• Abstract Accurate precipitation forecasting remains elusive as physics-based models struggle to resolve mesoscale convection and machine learning often yields physically inconsistent, blurred results that consistently underestimate extreme events. We present FitzRoy, the first system that simultaneously achieves state-of-the-art predictive skill and physical realism over the continental United States. Through a reinforced diffusion framework, where a conditional discrete diffusion model is refined via reinforcement learning, FitzRoy effectively resolves the long-standing trade-off between pointwise accuracy and structural fidelity. Across lead times up to 24 hours, FitzRoy consistently matches or exceeds state-of-the-art models while preserving structural realism and capturing extreme events. Its low-latency inference further enables deployment in real-time operational settings. In case studies such as the 2025 Texas Hill Country floods and the 2026 Missouri floods, FitzRoy signals record-breaking accumulations that existing models largely missed. By bridging the gap between realism and skill, FitzRoy establishes a new frontier for high-fidelity precipitation forecasting.
• 中文摘要 准确的降水预报仍然难以实现，因为基于物理的模型难以解析中尺度对流，机器学习常常产生物理不一致、模糊的结果，持续低估极端事件。我们介绍FitzRoy，这是首个同时实现美国大陆最先进预测能力和物理真实性系统的系统。通过强化扩散框架，通过强化学习完善条件离散扩散模型，FitzRoy有效解决了点状准确性与结构真实性之间的长期权衡。在长达24小时的交货时间内，FitzRoy始终匹配或超越最先进模型，同时保持结构真实性并捕捉极端事件。其低延迟推断进一步支持在实时操作环境中的部署。在2025年德克萨斯丘陵地区洪水和2026年密苏里州洪水等案例中，菲茨罗伊预示了现有模型大多未察觉的破纪录积水。通过弥合现实与技能之间的差距，菲茨罗伊为高保真降水预报开辟了新前沿。

Discovering Jailbreaks via Curriculum of Models

通过模型课程发现越狱

• Authors: Michał Burzyński, Noah T. Rossi, Shomit Basu, Andy Arditi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38833
• Pdf link: https://arxiv.org/pdf/2610.38833
• Abstract Automated jailbreak discovery aims to find prompts that elicit prohibited behaviors from safety-trained language models, helping developers identify vulnerabilities before deployment. One class of approaches uses reinforcement learning to train an investigator model to generate prompts that elicit a specified behavior from a target model. However, training directly against a safety-trained target is difficult because nearly all prompts are refused, leaving the investigator with a sparse reward signal. Prior work addresses this with the propensity bound (PRBO), a proxy reward estimated by sampling responses from a more compliant target, and then adjusting for how likely those responses would have been under the original target. We introduce Curriculum of Models (CoM), an alternative method that trains the investigator against a sequence of modified versions of the target, progressing gradually from a version that readily exhibits the specified behavior to the original target. For jailbreak discovery, training begins against a target version that rarely refuses harmful requests, and advances to the next, harder version once the investigator reliably elicits the behavior from the current one. We evaluate CoM on elicitation tasks involving CBRN materials, explosives, and illicit drugs, using two open-weight targets, gemma-4-31b-it and gpt-oss-120b. We compare against PRBO and direct training on the original target without a curriculum. When the investigator is trained to elicit a single behavior, CoM finds successful jailbreaks more reliably than either baseline. When trained on multiple behaviors simultaneously, the CoM-trained investigator can elicit held-out behaviors, though PRBO-trained investigators achieve higher success rates. Our results suggest that a simple curriculum over target models offers a promising approach for addressing sparse rewards in rare-behavior elicitation.
• 中文摘要 自动越狱发现旨在寻找能引发安全训练语言模型禁止行为的提示，帮助开发者在部署前识别漏洞。一类方法利用强化学习训练调查员模型，生成能引发目标模型特定行为的提示。然而，直接针对安全训练对象进行训练较为困难，因为几乎所有提示都被拒绝，导致调查者获得稀疏的奖励信号。先前的研究通过倾向界限（processity bound， PRBO）来解决这个问题，这是一种代理奖励，通过抽样更顺从的目标的反应，然后调整这些反应在原始目标下出现的可能性来估算。我们引入了模型课程（Curriculum of Models，CoM），这是一种替代方法，通过一系列修改后的目标进行训练，从一个容易表现出指定行为的版本逐步推进到原始目标。对于越狱发现，训练从一个很少拒绝有害请求的目标版本开始，一旦研究者可靠地从当前版本中引出该行为，训练则进入下一个更难的版本。我们使用两个开放权重目标gemma-4-31b-it和gpt-oss-120b，评估涉及CBRN材料、爆炸物和非法药物的诱发任务中的CoM。我们与PRBO和对原始目标的无课程直接训练进行比较。当研究者被训练以引发单一行为时，CoM发现成功的越狱比任何基线都更可靠。当CoM训练的调查员同时针对多种行为进行训练时，可以引发长期保留的行为，尽管PRBO训练的调查员成功率更高。我们的结果表明，基于目标模型的简单课程为解决罕见行为诱发中奖励稀疏提供了有前景的方法。

Discovery of Diverse and Realistic Financial Tail-Risk Using Generative Flow Networks

利用生成流网络发现多样且现实的金融尾风险

• Authors: Adir Serruya, Gilad Katz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38838
• Pdf link: https://arxiv.org/pdf/2610.38838
• Abstract Accurate modeling of rare, high-impact financial events is essential for robust risk management. Sampling-based techniques such as Sequential Markov Chain Monte Carlo (SMCMC) generate diverse scenarios but inefficiently explore high-risk regions, while Deep Reinforcement Learning (DRL) converges on narrow sets of scenarios, limiting diversity. We present GRID, which uses Generative Flow Networks (GFlowNets) -- samplers that draw objects in proportion to a reward -- to systematically generate diverse and realistic financial tail-risk scenarios. Building on continuous GFlowNets, GRID introduces a flexible mixture-policy architecture with variable-specific distribution families for high-dimensional, sequential macro-financial data, together with plausibility constraints that keep generated trajectories realistic. A central practical advantage is robustness to reward-proxy imperfection: because GFlowNets sample proportionally rather than maximize, they remain broadly exploratory even when the oracle's out-of-sample accuracy is low -- an unavoidable reality in financial forecasting. On three real-world crises, GRID improves scenario diversity, realism, and tail-risk coverage over RL and MCMC baselines, and ablations over oracle architectures and distribution families yield practical insights into continuous GFlowNet behavior under noisy rewards.
• 中文摘要 对罕见且高影响力的金融事件进行准确建模对于稳健的风险管理至关重要。基于抽样的技术如顺序马尔可夫链蒙特卡洛（SMCMC）生成多样场景，但低效地探索高风险区域;而深度强化学习（DRL）则收敛于狭窄的场景集，限制了多样性。我们介绍了GRID，它使用生成流网络（GFlowNets）——根据奖励比例绘制对象的采样器——系统生成多样且真实的金融尾风险场景。基于连续GFlowNet，GRID引入了灵活的混合策略架构，具备高维、顺序宏观金融数据的变量特定分布族，并结合合理性约束，确保生成的轨迹真实。一个核心的实际优势是对奖励代理不完美问题的鲁棒性：由于GFlowNets采用比例抽样而非最大化，即使预言机的样本外准确率较低，GFlowNet仍保持广泛的探索性——这是金融预测中不可避免的现实。在三种现实世界危机中，GRID提升了基于强化学习和MCMC基线的情景多样性、现实性和尾风险覆盖率，针对预言机架构和分布家族的消融则提供了关于噪声奖励下GFlowNet连续行为的实用洞见。

Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training

仅凭参考样本学习推理：现代大型语言模型后培训中的强化对抗性学习

• Authors: Albert Bou, Javier Valdes-Aleman, Jon M. Laurent, Samuel G. Rodriques, Andrew D. White, Michaela Hinks, Siddharth Narayanan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38851
• Pdf link: https://arxiv.org/pdf/2610.38851
• Abstract In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks where an explicit correctness signal is available. Prior adversarial approaches to training reasoning LLMs either complement verifiable rewards with co-trained critics or require reference reasoning traces. We ask whether a co-trained LLM judge can turn reference answers alone into the sole reward for on-policy post-training of reasoning LLMs. We call this method Reinforced Adversarial Learning (RAL). We first validate RAL on three verifiable tasks (integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation), using the verifiers only for evaluation. With sufficiently capable judges and enough training, RAL matches direct RLVR. We then apply RAL to scientific abstract generation in two settings. In the first setting, RAL guides the generator to produce abstracts in the scientific domain of the reference abstracts, even though the prompt leaves that domain unspecified. In the second setting, we generate abstracts for U.S. National Institutes of Health grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier-LLM baselines. These results suggest a path toward post-training reasoning models in scientific and other high-value domains, where increasingly capable judges infer hard-to-specify reward functions from example answers.
• 中文摘要 在许多需要LLM后训练的环境中，高质量的答案可识别，但通过显式奖励难以得分。带可验证奖励的强化学习（RLVR）仅限于有显式正确信号的任务。以往对抗式训练推理LLM的方法要么用共同训练的批评者补充可验证奖励，要么要求引用推理痕迹。我们询问共训练的LLM评判是否能仅将引用答案作为推理LLM策略后训练的唯一奖励。我们称此方法为强化对抗性学习（RAL）。我们首先在三个可验证任务（整数乘法、基于代码的算术表达式综合和SMILES转公式转换）上验证RAL，仅使用验证器进行评估。在足够有能力的评判和足够训练的情况下，RAL可匹配直接RLVR。然后我们将RAL应用于两种科学摘要生成。第一种情况下，RAL引导生成器生成参考摘要的科学领域中的摘要，尽管提示未明确说明该领域。在第二种情况下，我们根据资助标题生成美国国立卫生研究院（NIH）资助申请的摘要。人类科学家和一批大型语言模型评委发现，我们的世代比前沿大型语言模型基线的世代更难区分真实摘要。这些结果暗示了科学及其他高价值领域中训练后推理模型的路径，越来越有能力的评审员能从示例答案推断难以具体说明的奖励函数。

Aligning Autoresearch: Co-Improvement between Agents and Humans

自动研究的对齐：智能体与人类的协同改进

• Authors: Tianjian Li, Chuanyang Jin, Seungone Kim, Tong Chen, John Hoffman, Yoram Bachrach amd Pedro Rodriguez, Ping Yu, Eryk Helenowski, Swarnadeep Saha, Jack Lanchantin, Weizhe Yuan, Ilia Kulikov, Jason Weston
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38882
• Pdf link: https://arxiv.org/pdf/2610.38882
• Abstract Autoresearch agents can now attempt long-horizon AI research end-to-end, i.e., self-improvement. However, best results are still obtained by research agents interacting with human researchers, and improving AI's ability to perform this co-research will only reinforce this effect---a process termed co-improvement (Weston and Foerster, 2025). Further, aligning with human expectations is likely a safer path for research agents, especially given recent agent reward hacking phenomena (OpenAI, 2026b). In this work we address these issues in four ways: (i) we empirically study the role of humans in directing autoresearch agents, focusing on the domain of benchmark creation, and find humans still play a pivotal role particularly in the ideation phase; (ii) we also develop a benchmark, HackBench, that shows current autoresearch agents are not aligned and are unable to achieve hack localization sufficiently; (iii) we present a method to train agents for improved human interaction by using human simulators as a proxy during reinforcement learning (RL); and finally (iv) we develop a method to improve model text generation, and hence agent communication, by RL with rubrics aligned with human experts. Together, our training and evaluation recipes provide a path toward more capable and aligned (co-)research agents.
• 中文摘要 自研究代理现在可以尝试从端到端的长期AI研究，即自我提升。然而，最佳结果仍来自研究代理与人类研究者的互动，提升AI进行这种共同研究的能力只会强化这一效应---这一过程称为共改进（Weston 和 Foerster，2025）。此外，鉴于近期代理奖励黑客现象，符合人类期望可能是研究代理更安全的路径（OpenAI，2026b）。本研究通过四种方式解决这些问题：（i）实证研究人类在指导自我研究代理中的作用，重点关注基准创建领域，发现人类在创意阶段仍扮演关键角色;（ii）我们还开发了基准HackBench，显示当前自研究代理尚未对齐，无法充分实现黑客定位;（iii）我们提出一种方法，通过在强化学习（RL）期间使用人类模拟器作为代理，训练代理以提升人类互动能力;最后（iv）我们开发一种通过强化学习改进模型文本生成及代理沟通的方法，评分标准与人类专家保持一致。我们的训练和评估方案共同为培养更有能力且更契合的（共同）研究代理提供了一条路径。

Geometric Time Score Estimators: Bridging Self-Predictive Representations and Generative Occupancy Models For Reinforcement Learning

几何时间分数估计器：连接自我预测表示与生成占有模型于强化学习

• Authors: Aravind Venugopal, Grace Liu, Jeff Schneider, Benjamin Eysenbach
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.38901
• Pdf link: https://arxiv.org/pdf/2610.38901
• Abstract In reinforcement learning (RL), temporal reasoning is essential for estimating the density of future outcomes under a policy's occupancy measure. While self-predictive representations have been a dominant paradigm for occupancy density estimation, we find that they yield surprisingly poor estimates. On the other hand, generative occupancy models can produce remarkably accurate samples from the occupancy measure. Yet, they cannot easily recover occupancy densities, limiting their use for control despite their expressivity. In this paper, we propose Geometric Time Score Estimators (GTSE), a framework that bridges the two paradigms by learning occupancy density estimators along probability paths. We analyze how GTSE implicitly induces better occupancy estimators through improved representation learning and show how to directly extract self-predictive representations from this framework. Through experiments across several locomotion and manipulation tasks, we show that GTSE improves performance over state-of-the-art methods by $2\times$ to $6\times$ in goal-conditioned RL settings, and that the representations learned using our framework transfer effectively to downstream RL tasks. Website: this https URL
• 中文摘要 在强化学习（RL）中，时间推理对于估计政策占用度量下未来结果的密度至关重要。虽然自我预测表征一直是占用密度估计的主导范式，但我们发现它们的估计效果出人意料地差。另一方面，生成式占用模型可以从占用度量中产生极为准确的样本。然而，它们难以恢复占用密度，尽管表现力强，但限制了其控制的使用。本文提出了几何时间分数估计器（GTSE），这是一个通过沿概率路径学习占用密度估计量来连接两种范式的框架。我们分析GTSE如何通过改进表征学习隐式诱导更优的占用率估计器，并展示了如何直接从该框架中提取自我预测表征。通过在多个移动和操作任务中的实验，我们表明GTSE在目标条件化强化学习设置中，较最先进方法的性能提升了2至6倍，且通过我们框架学习的表示有效转移至下游强化学习任务。网站：此 https URL

HEDGE: Gated Hypergraph Memory for Scalable Multi-Agent Debate

HEDGE：用于可扩展多代理辩论的门控超图存储器

• Authors: Amirhossein Abaskohi, Mahdi Mostajabdaveh, Zirui Zhou
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.38902
• Pdf link: https://arxiv.org/pdf/2610.38902
• Abstract Multi-agent debate systems require agents to retain and revisit arguments across multiple rounds, yet limited context windows and unstructured memories make relevant information increasingly difficult to recover. Existing flat and pairwise-graph memories also struggle to capture higher-order relations, including multi-agent agreement, conflicting evidence, and evolving consensus. We introduce HEDGE, a selective memory framework that represents debate trajectories as semantic hypergraphs. Hyperedges connect related claims across agents, while intra-hyperedge relational graphs preserve fine-grained argumentative structure. A learned memory selection module (MSM), trained through supervised learning and reinforcement learning, routes each memory candidate to STORE, BACKUP, or OMIT, retaining useful information while limiting redundancy. Across seven diverse benchmarks, HEDGE consistently outperforms strong single-agent, multi-agent, and memory baselines, improving accuracy by 5.0 points on average over the strongest multi-agent system with GPT-4o-mini. The gains reach 8.2 points on conflict-heavy tasks, while HEDGE reduces memory usage by 48% and the number of debate rounds by 34%.
• 中文摘要 多智能体辩论系统要求代理在多轮中保留和反复回顾论点，但有限的上下文窗口和非结构化记忆使相关信息越来越难以恢复。现有的平面和成对图记忆也难以捕捉高阶关系，包括多智能体一致、冲突证据和不断演变的共识。我们引入了HEDGE，一种选择性记忆框架，将辩论轨迹表示为语义超图。超边连接代理间相关主张，而超边内部关系图保持细粒度的论证结构。通过监督学习和强化学习训练的学习记忆选择模块（MSM）将每个记忆候选路由到STORE、BACKUP或OMIT，保留有用信息同时限制冗余。在七个多样化基准测试中，HEDGE始终优于强的单代理、多智能体和内存基线，平均比最强的多智能体系统GPT-40-mini提升准确率5.0分。在冲突密集的任务中，准确率提升达到8.2分，而HEDGE则减少了48%的内存使用，辩论回合减少了34%。

Reinforcement Learning for the Game of Balatro: A Fast Simulator and Two-Policy Training Approach

巴拉特罗游戏的强化学习：快速模拟器和双策略训练方法

• Authors: Ajay Vikram Periasami, Anjaneya Gupta, Ethan Cheng, Ivan Rosales
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.39049
• Pdf link: https://arxiv.org/pdf/2610.39049
• Abstract Balatro, a 2024 roguelike deckbuilder, presents three difficulties that have historically derailed reinforcement learning methods: a 3,000-action state-dependent action space, horizons of several hundred decisions with sparse rewards, and pervasive stochasticity in both card draws and shop generation. Published RL results on the game remain scarce. We introduce an end-to-end pipeline comprising (i) a high-throughput Rust simulator that is automatically state-diffed against the live game client via BalatroBot, (ii) a condensed observation-action interface with dynamic per-step masking of legal moves, and (iii) a multi-agent PPO architecture that decomposes play into a tactical blind policy and a strategic shop policy that share the full game state but act in disjoint phases. Long-horizon credit assignment is partially addressed by per-agent reward routing with explicit cross-agent credit. On the main joint run, training metrics show a sharp phase transition in which the first-blind clear rate converges above 99% and mean episode return increases by roughly an order of magnitude, plateauing at a mean maximum ante of 3 out of 8; single-agent baselines under identical rewards oscillated between greedy and hoarding failure modes and did not stably clear the first blind. These results establish the two-policy decomposition as a strong reactive baseline for Balatro and localise the remaining gap to long-horizon strategic planning rather than tactical blind play.
• 中文摘要 Balatro是一款2024年的Roguelike卡组构建游戏，提出了三个历史上阻碍强化学习方法的难题：3000个动作依赖状态的动作空间、数百个决策且奖励稀疏的视野，以及在抽牌和商店生成中普遍存在随机性。游戏中公开的强化学习结果仍然稀少。我们引入了端到端的流程，包括（i）通过BalatroBot自动与实时游戏客户端进行状态差异的高吞吐量Rust模拟器，（ii）一个紧凑的观察-动作接口，具有动态每步合法走法的掩盖，以及（iii）一个多代理PPO架构，将游戏分解为战术盲策略和战略商店策略，共享完整游戏状态但处于不相交阶段。长视野的信用分配部分通过每个代理的奖励路由和显式的跨代理信用来解决。在主联合运行中，训练指标显示出现明显的相位转变，首次盲清算率收敛于99%以上，平均集数回归率约增加一个数量级，平均最大前端为8次中3个;单代理基线在相同奖励下在贪婪和囤积失败模式间波动，未能稳定通过第一次盲注。这些结果确立了双策略分解作为Balatro强有力的反应基准，并将剩余差距定位为长期战略规划而非战术盲注。

Robotic Data Inoculation

机器人数据接种

• Authors: Qiyang Li, Zhiyuan Zhou, Charles Xu, Sergey Levine
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.39054
• Pdf link: https://arxiv.org/pdf/2610.39054
• Abstract Modern robot learning systems feed on high-quality demonstration data, and their performance hinges on coverage. Without good coverage, small errors compound and the policy can quickly go into out-of-distribution states from which it cannot recover. Yet there is no principled way of collecting high-coverage data, and data collection often relies on heuristic instructions to human operators. Purely online reinforcement learning (RL) explores at the cost of safety and sample efficiency, while DAgger-style interventions require the policy to fail by chance before a human can correct it. In this paper, we ask instead whether we can modify how the robot behaves during data collection to collect more useful data. We propose Data Inoculation (DI), a shared-control framework in which a human provide closed-loop teleoperation commands while a residual policy perturbs these commands. The residual policy is trained online, on the growing dataset, with the same reward-maximizing objective later used for policy extraction, so its perturbations steer data collection toward the deviations a policy trained on the data would make. These perturbations expose the robot to a wider range of states, while the human's continuous corrections provide recovery demonstrations, "inoculating" the dataset against the learner's future mistakes while the human remains in control. On three challenging real-world robotic tasks, DI consistently improves or matches the success rate of policies trained with both behavioral cloning and offline RL, showing DI as an effective data collection strategy for improving downstream policy performance.
• 中文摘要 现代机器人学习系统依赖高质量的演示数据，其性能依赖于覆盖率。如果覆盖率不高，小错误会累积，策略可能迅速进入无法恢复的分发状态。然而，收集高覆盖率数据没有原则性方法，数据收集通常依赖于对人工操作员的启发式指令。纯在线强化学习（RL）以安全性和样本效率为代价进行探索，而DAgger式干预则要求策略在人类纠正前偶然失效。本文则探讨是否可以修改机器人在数据收集过程中的行为，以收集更有价值的数据。我们提出数据接种（Data Inoculation，DI）框架，一种共享控制框架，由人类提供闭环远程操作指令，剩余策略扰动这些指令。残余策略在不断增长的数据集上在线训练，目标与后续策略提取相同的奖励最大化目标，因此其扰动引导数据收集朝向基于数据训练的策略可能出现的偏差方向。这些扰动使机器人暴露于更广泛的状态范围，而人类的持续修正则提供了恢复演示，使数据集“免疫”于学习者未来的错误，而人类则保持控制。在三个具有挑战性的现实机器人任务中，DI持续提升或匹配使用行为克隆和离线强化学习训练策略的成功率，表明DI是提升下游策略性能的有效数据收集策略。

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

欣赏教会未经授权的执行：自我进化削弱了个人代理的稳健性，转而成为间接的即时注入

• Authors: Xinyan Wang, Xiaogeng Liu, Jun Shao, Chaowei Xiao
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.39122
• Pdf link: https://arxiv.org/pdf/2610.39122
• Abstract Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to read external content and complete the task described in it, a common request in daily use. The agent then learns from the user's feedback and improves itself as designed. In the test phase, a third party writes a similar task into new external content, but the user's test request does not authorize the injected task. Using this methodology, we build multiple tasks with diverse test instances in different domains for a real personal agent. Our extensive experiments reveal that self-evolution substantially degrades robustness to IPI. Specifically, compared with the base agent, which does not evolve from user feedback, the average attack success rate rises by 14.0--48.8 percentage points under skill evolution and by 3.0--39.3 percentage points under reinforcement learning, posing safety risks. The code is available at this https URL
• 中文摘要 自我进化的个人代理为用户行动，并从用户反馈中学习。然而，如果代理服从不信任第三方隐藏的内容指令，第三方将获得对代理的控制权。这种间接提示注入（IPI）存在严重风险。我们调查自我进化是否会削弱个人代理对IPI的稳健性。为研究这个问题，我们设计了一种受控的两阶段方法论。在适应阶段，用户要求代理读取外部内容并完成其中描述的任务，这是日常使用的常见请求。代理随后从用户反馈中学习，并按设计自我改进。在测试阶段，第三方将类似任务写入新的外部内容，但用户的测试请求并未授权注入的任务。利用该方法，我们为真实个人代理构建多个任务，并在不同领域使用不同的测试实例。我们广泛的实验显示，自我进化会显著降低对IPI的鲁棒性。具体来说，与不依赖用户反馈进化的基础代理相比，技能进化的平均攻击成功率提高了14.0个百分点——48.8个百分点，强化学习中提升了3.0个百分点——39.3个百分点，存在安全风险。该代码可在此 https URL 获取。

STRATA: Reinforcement-driven Active Sampling with Tri-paradigm Adaptation for Efficient 3D Spatial Transcriptomics

STRATA：强化驱动的主动采样与三范式适应，实现高效的三维空间转录组学

• Authors: Junchao Zhu, Ruining Deng, Chongyu Qu, Zhengyi Lu, Yanfan Zhu, Yuechen Yang, Yu Wang, Shilin Zhao, Haichun Yang, Yuankai Huo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.39151
• Pdf link: https://arxiv.org/pdf/2610.39151
• Abstract Spatial transcriptomics (ST) enables joint characterization of tissue morphology and gene expression. However, 3D volumetric ST acquisition remains expensive. AI-driven methods offer a low-cost alternative but remain limited in accuracy, while recent 2D section-wise ST acquisition still rely on dense grid sampling, resulting in redundant measurements and undersampling of biologically informative regions. To bridge the gap, we propose STRATA, a reinforcement-driven active sampling framework with tri-paradigm adaptation for reconstructing dense 3D transcriptomic maps from sparse voxel-wise ST acquisition. STRATA constructs a 3D graph integrating H&E morphology with single-cell priors and employs a reinforcement learning agent to estimate morphological sparsity and transcriptomic informativeness, enabling budget-aware selection of informative spots. Given these sparse measurements, an adaptive tri-paradigm predictor combines retrieval, regression, and cross-layer imputation through lightweight fusion, dynamically weighting their contributions. Experiments across datasets demonstrate improved sampling efficiency and accuracy, particularly under low-budget settings. Code is available at this https URL.
• 中文摘要 空间转录组学（ST）使组织形态和基因表达的联合表征成为可能。然而，三维体积ST采集仍然昂贵。AI驱动的方法提供了低成本的替代方案，但准确性有限;而近期的二维切片ST采集仍依赖密集网格采样，导致测量重复且生物信息区采样不足。为弥合这一差距，我们提出了STRATA，一种基于强化驱动的主动采样框架，具有三范式适应，用于从稀疏的体素ST采集中重建密集的三维转录组图谱。STRATA构建了一个将H&E形态与单细胞先验整合的三维图，并采用强化学习代理估计形态稀疏度和转录组信息量，实现预算意识地选择信息量。鉴于这些稀疏测量，自适应三范式预测器通过轻量级融合结合了检索、回归和跨层补值，动态加权其贡献。跨数据集的实验显示抽样效率和准确性有所提升，尤其是在低预算环境下。代码可在此 https URL 获取。

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

推理还是模式利用？来自强化学习训练语言模型的机制性洞见

• Authors: Aishik Nagar, Cheston Tan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.39182
• Pdf link: https://arxiv.org/pdf/2610.39182
• Abstract Large language models (LLMs) are increasingly described as acquiring "reasoning" skills after reinforcement learning with verifiable rewards. However, whether these gains reflect genuine reasoning or more superficial adaptations remains underexplored. In this work, we probe LLMs trained in a finite and exhaustively analyzable logical domain, namely Tic-Tac-Toe, and trace how internal representations evolve across reinforcement learning (RL) with Group Relative Policy Optimization (GRPO). Quantitatively, RL yields high accuracy and robustness across prompt variations, suggesting the emergence of game reasoning. Mechanistic interpretability, however, paints a different picture: features extracted with sparse autoencoders (SAEs) reveal that models primarily adapt to rely more heavily on information already explicit in the prompt, such as whose turn it is, game progression and board occupancy. In comparison, high-level concepts like board symmetries and strategic forks remain weakly represented, raising concerns that reasoning benchmarks overestimate model abilities. This suggests that RL-driven "reasoning" may conflate task-specific adaptations to input information with reasoning ability. We provide, for the first time: (i) a systematic interpretability pipeline tracing representation dynamics across RL training in small-scale LLMs; (ii) an extension of SAE-based feature discovery to hypothesis-driven testing in a finite logical domain; and (iii) an interpretability-based demonstration that RL post-training neither adds nor removes concept information but reorganizes which features dominate the representation, with causal interventions locating the decision downstream of it, in the late-layer state.
• 中文摘要 大型语言模型（LLMs）越来越多地被描述为在强化学习后获得“推理”技能，并获得可验证的奖励。然而，这些提升究竟反映了真实的推理还是更表面的适应，目前尚未被充分探讨。在本研究中，我们探讨了在有限且可详尽分析的逻辑领域——井字棋（Tic-Tac-Toe）中训练的LLM，并追踪内部表征在强化学习（RL）与群体相对策略优化（GRPO）中的演变。定量上，RL在提示变异中表现出高准确性和鲁棒性，表明博弈推理的出现。然而，机制性可解释性则呈现出不同的图景：使用稀疏自编码器（SAE）提取的特征显示，模型主要适应更依赖提示中已明确的信息，如轮到谁、游戏进程和棋盘占有率。相比之下，板对称性和战略分叉等高层概念代表性仍然较弱，引发了推理基准高估模型能力的担忧。这表明强化学习驱动的“推理”可能将任务特定的输入信息适应与推理能力混为一谈。我们首次提供了：（i） 系统化的可解释性流水线追踪表征动态，涵盖小规模大型语言模型中强化学习训练;（ii） 基于SAE的特征发现扩展到有限逻辑域中的假设驱动测试;以及（iii） 基于可解释性的演示，证明强化学习后训练既不添加也不删除概念信息，而是重组主导表示的特征，并通过因果干预将其决策定位在其下游，处于后期阶段。

Event2Reward: Learning Robotic Rewards from Progress Relations

Event2Reward：从进展关系中学习机器人奖励

• Authors: Junnan Nie, Zitian Fan, Jiachen Zhang, Junyi Lao, Likui Zhang, Chenghao Liu, Xiaoyang Lu, Yunbo Wang, Yixue Zhang, Linbo Zhai, Jiayi Li, Mengzhe Wang, Xichang Wu, Bin Wang, Chang Liu, Tianle Zhang, Songfang Huang
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.39212
• Pdf link: https://arxiv.org/pdf/2610.39212
• Abstract Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.
• 中文摘要 通过机器人演示学习任务进展，为构建机器人强化学习的密集塑形奖励提供了实用方法。一种常见方法将归一化演示时间作为进展标签，隐含假设任务进度单调增加。然而，在策略探索过程中，实际任务进展可能停滞甚至倒退。此类模型可能将回归误认为进展，将正向塑造奖励分配给导致进展损失的行为。真实的次优和失败执行提供了纠正经验，但赋予其状态绝对进展值较为困难。我们的关键观察是，停滞、回归和恢复即使绝对进展未知，也能建立局部相对进展关系。我们引入Event2Reward，利用归一化的时间标签锚定进展尺度，并通过轨迹内关系约束局部变化，结合绝对和相对进展监督。其进展预测作为奖励塑造的潜力。我们从约10万条原始真实机器人执行记录中策划事件注释数据集，涵盖成功、次优和失败执行，无需帧级绝对进度标签。在相同训练预算下，Event2Reward仅用稀疏任务奖励，在40个LIBERO套件中平均策略成功率提升18.16个百分点。在真实机器人上，30分钟墙钟训练后，三个精确插入任务的平均成功率达到60.0%，三个对象操作任务达到90.0%，分别在同一预算下，较稀疏奖励基线高出36.7个百分点和23.3个百分点。消融研究进一步表明，绝对进展监督与相对进展关系互为互补，联合监督带来更好的下游策略学习。

A Sequential Decision Framework for Evaluating LLM-Guided Search

评估LLM引导搜索的顺序决策框架

• Authors: Siqing Huo, Andrew Parry, Debasis Ganguly, Mark Smucker, Charles L. A. Clarke
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.39231
• Pdf link: https://arxiv.org/pdf/2610.39231
• Abstract Recent systems applying retrieval-augmented generation (RAG) and "deep research" iteratively generate queries, retrieve documents, and examine their contents, using the resulting information to guide subsequent search steps. When this process is carried out autonomously rather than by a human searcher, the system must make explicit decisions about which search trajectories to explore and what objective the search is intended to optimize. We introduce a simple framework for reasoning about such search policies, providing a principled approach to evaluation. The framework models the search environment as a set of ranked lists and represents behavior using two primitive actions: select, which advances judgment through an existing ranking, and spawn, which generates a new ranking through query generation or reformulation. Each action produces a document to inspect, yields information that updates the search state, and contributes a reward for a task-specific objective. Within this abstraction, a variety of existing retrieval strategies can be expressed as policies operating over a common environment, while common retrieval objectives can be represented through reward and weighting functions over the resulting search trajectory, and the resulting formulation admits a natural reinforcement learning interpretation. We illustrate the approach using high-recall information retrieval (HRIR), where the goal is to identify as many relevant documents as possible with limited judging effort, showing how the framework provides a unified way to describe and reason about strategies for both classical retrieval tasks and emerging LLM-guided search systems. Through experiments, we demonstrate that both the underlying retrieval method and the "deep research" policy employed can materially affect performance.
• 中文摘要 近期应用检索增强生成（RAG）和“深度研究”的系统，通过迭代生成查询、检索文档并检查其内容，并利用所得信息指导后续搜索步骤。当这一过程由人类搜索者自主执行时，系统必须明确决定探索哪些搜索轨迹以及搜索旨在优化的目标。我们引入了一个简单的框架来推理此类搜索策略，提供有原则的评估方法。该框架将搜索环境建模为一组排名列表，并通过两种原始动作表示行为：select（通过现有排序推进判断）和 spawn（通过查询生成或重构生成新排名）。每个动作都生成一份可供检查的文档，生成更新搜索状态的信息，并为特定任务目标贡献奖励。在该抽象中，多种现有检索策略可以表示为在共同环境中运行的策略，而共同检索目标则通过对结果搜索轨迹的奖励和加权函数表示，且该表述具有自然的强化学习解释。我们用高回忆信息检索（HRIR）说明该方法，目标是在有限判断努力下识别尽可能多的相关文档，展示了该框架如何统一地描述和推理经典检索任务和新兴LLM引导搜索系统的策略。通过实验，我们证明了底层检索方法和所采用的“深度研究”策略都能实质性影响性能。

Grounding Multimodal Reasoning with Evidence-Ablated Negatives

用证据消融否定为多模态推理奠定基础

• Authors: Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Heng Huang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.39237
• Pdf link: https://arxiv.org/pdf/2610.39237
• Abstract Reinforcement learning with verifiable rewards (RLVR) has become a powerful tool for improving multimodal reasoning. However, standard outcome supervision suffers from credit assignment ambiguity, as a model may receive positive rewards by exploiting language priors or dataset shortcuts rather than grounding its reasoning in the requisite visual evidence. This makes high-quality negative responses critical, since effective negatives should remain plausible under the current policy while exposing the shortcut that should be suppressed. In this paper, we propose EVAN, a multimodal reinforcement learning framework utilizing EVidence-Ablated Negatives to train LMMs against their own language-prior failures. EVAN first identifies highly visually dependent examples using a scaled image information gain metric and preserves data diversity with a mixed-ranking strategy. It then constructs evidence-ablated images by masking the most discriminative image patches, eliciting plausible but visually ungrounded responses from the current policy. During policy optimization, these negative responses are incorporated as bounded auxiliary signals under the clean image, penalizing shortcut learning while maintaining RL stability. Experiments across challenging multimodal reasoning benchmarks demonstrate that EVAN mitigates modality imbalance and improves over strong RLVR baselines. Our code is available at this https URL.
• 中文摘要 带有可验证奖励的强化学习（RLVR）已成为提升多模态推理的有力工具。然而，标准的结果监督存在学分分配的歧义，模型可能通过利用语言先验或数据集捷径而获得正向奖励，而非基于必要的视觉证据来推理。这使得高质量的负面反应变得至关重要，因为在当前策略下，有效的否定应保持合理性，同时揭示应被抑制的捷径。本文提出了EVAN，一种利用EVidence-Ablated负面来训练LMM自身语言先验失败的多模态强化学习框架。EVAN首先通过缩放图像信息增益指标识别高度依赖视觉的实例，并通过混合排名策略保持数据多样性。随后，它通过掩蔽最具辨别性的图像片段构建证据消融图像，从当前策略中引出合理但视觉上缺乏依据的响应。在策略优化过程中，这些负面响应作为有界辅助信号被纳入干净图像下，惩罚捷径学习，同时保持强化学习的稳定性。在具有挑战性的多模态推理基准测试中的实验表明，EVAN 能够缓解模态不平衡，并且在强 RLVR 基线下有所改善。我们的代码可在 https URL 访问。

Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps

在线密集强化学习，使用规划-接地代理用于实时iOS应用的UI导航

• Authors: Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio, Emmanuel Abbe
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.39264
• Pdf link: https://arxiv.org/pdf/2610.39264
• Abstract While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.
• 中文摘要 虽然前沿视觉语言模型在用户界面（UI）导航方面表现出强大能力，但其高推理成本限制了日常使用。相反，通过监督微调（SFT）或离线强化学习训练的小型模型常常记忆静态轨迹，导致执行脆弱且无法从错误中恢复。为解决这些限制，我们提出了一种新型在线强化学习框架，直接在实时iOS应用上训练UI导航代理。我们的框架将代理拆分为两个专业组件：规划代理和UI基础模型。我们依赖LLM作为评判者提供全面的奖励信号，结合密集的阶级反馈评估中间进展与终端奖励以完成整体任务。为有效利用这种双重奖励结构，我们采用近端策略优化（PPO）和阶级优势估计对代理进行优化。此外，我们引入了GiGPPO，一种新颖的邻居条件值估计方法，通过利用视觉相似状态下的奖励信号，提升批评稳定性。为大规模支持这一在线训练范式，我们引入了高效基础设施，协调数百台并发虚拟iOS设备。对包括UINavBench、iOSWorld和AndroidWorld在内的严谨基准测试的评估显示，我们的在线强化学习代理在训练收敛速度上显著优于SFT和仅限终端的强化学习基线。值得注意的是，我们的8B强化学习代理在非分布推广至Android环境方面比SFT基线提升近七倍，同时在真实iOS应用中表现优于Gemini-3.5-Flash，与Gemini-3.1-Pro持平。

There Is No Free Lunch in RLVR: Every pass@1 Gain Has a pass@k Price

RLVR没有免费午餐：每pass@1收益都有pass@k代价

• Authors: Yinuo Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.39295
• Pdf link: https://arxiv.org/pdf/2610.39295
• Abstract Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at large sampling budget k. Is this a failure of RL to add reasoning, or a transient that more training removes? We settle the question with a theorem. The reversal is a price, and we compute its exact rate. Modeling RLVR as an exponential tilt of a base policy over strategies shared across prompts, we prove that under exclusive specialization every update that raises pass@1 strictly lowers the success of some prompt; a strictly positive price on prompts (Stiemke duality) is the exact condition, and being merely the best strategy on a prompt is provably not enough. When the loss reaches the hardest prompts, it forces a pass@k crossover, and the number of crossings is at most the sign variation of the shift in per-prompt success, a totally positive bound that is tight and that recovers the known single-crossover result as its order-preserving case. The crossover is not a bug to patch: a constrained update traces the exact pass@1 versus pass@k frontier, and removing the crossover entirely costs the entire pass@1 gain. The same tilt identity yields entropy collapse and the entropy-performance law with an identifiable exponent, unifying two literatures under one mechanism. Reinforcement cannot widen the reasoning frontier; it can only move along it, at a price we make measurable by a diagnostic that reads the bound, the order-preserving test, and the price from any evaluation log. There is no free lunch, only a price.
• 中文摘要 带有可验证奖励的强化学习（RLVR）使语言模型在首次尝试中解决更多问题，而在多次尝试时解决问题减少：它提高了pass@1，而基础模型在大采样预算k时超过了它。这是强化学习未能增加推理，还是更多训练消除的暂态？我们用定理来解决这个问题。反转是一个价格，我们计算其精确率。将RLVR建模为基础策略对跨提示词共享策略的指数倾斜，我们证明在排他专用下，每次提升pass@1的更新都会严格降低某个提示的成功率;提示词的严格正价格（Stiemke对偶性）是确切条件，仅仅是提示词上的最佳策略是不可证明的。当丢失到达最难提示时，会强制pass@k交叉，交叉次数最多是每个提示成功率的符号变化，是一个完全正的界限，紧密且恢复已知单交叉结果作为保持顺序的情况。交叉点不是需要修补的漏洞：受限更新会追踪精确的pass@1与pass@k边界，完全移除交叉点会损失全部pass@1收益。相同的倾斜恒等式会产生熵坍缩和带有可识别指数的熵性能律，将两部文献统一在一个机制下。强化无法拓宽推理边界;它只能沿着边界前进，代价通过诊断读取界限、保持顺序测试和任何评估日志的价格来衡量。没有免费午餐，只有价格。

MRC-MMAgent: Reinforcement Learning for Multi-Role Collaboration in Multimodal Tool-Using Agents

MRC-MMAgent：多模态工具使用代理中多角色协作的强化学习

• Authors: Shengqin Wang, Jie Jin, Xiaofan Li, Wentao Yan, Yu Cheng, Qingsong HuYihang Chen, Weilin Luo, Jingyu Gong, Xin Tan, Yuan Xie, Zhizhong Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.39324
• Pdf link: https://arxiv.org/pdf/2610.39324
• Abstract Multimodal tool-using agents augment vision-language models with access to external information and capabilities, enabling them to solve knowledge-intensive and visually complex tasks that cannot be addressed using visual context alone. However, existing reinforcement learning pipelines largely train single-role agents to search and answer using terminal rewards, overlooking how humans inspect retrieved evidence, reformulate queries, and revise conclusions during search. Consequently, they fail to optimize this self-corrective process over complete interaction trajectories. We propose MRC-MMAgent, a multi-role collaborative reinforcement learning framework that incorporates trajectory assessment and continuation directly into the executable agent loop. First, the framework establishes a multi-role collaborative loop that integrates solving, inspection, and continuation, enabling the agent to acquire evidence through tools, evaluate the preceding trajectory, and resume problem solving when the current answer is insufficiently supported. Second, we introduce a role-coupled learning objective that combines task success, interaction validity, and successful correction with full-trajectory policy optimization across the decisions generated by all roles. Extensive experiments show that, using only the original baseline training set without constructing additional datasets, our method consistently outperforms the corresponding base models and achieves leading or highly competitive performance across benchmarks.
• 中文摘要 多模态工具使用代理增强视觉语言模型，使其能够访问外部信息和能力，使其能够解决仅靠视觉上下文无法解决的知识密集型且视觉复杂任务。然而，现有的强化学习流程大多训练单角色代理使用终端奖励进行搜索和回答，忽视了人类如何检查检索到的证据、重新表述查询以及在搜索过程中修订结论。因此，它们未能在完整交互轨迹上优化这种自我纠正过程。我们提出了MRC-MMAgent，一种多角色协作强化学习框架，将轨迹评估和延续直接整合进可执行代理循环。首先，该框架建立了一个多角色协作循环，整合了求解、检查和延续，使智能体能够通过工具获取证据，评估前一轨迹，并在当前答案不足时继续解决问题。其次，我们引入角色耦合学习目标，结合任务成功、交互有效性和成功纠正，并在所有角色生成的决策中实现全轨迹策略优化。大量实验表明，仅使用原始基线训练集而未构建额外数据集，我们的方法始终优于相应的基础模型，并在基准测试中取得领先或高度竞争的性能。

Safe Multi-Objective Reinforcement Learning: Zero Constraint Violation and Primal-Averaging Convergence

安全的多目标强化学习：零约束违规与原始平均收敛

• Authors: Chenxi Li, Yao Shu, Siyang Gao, Shuang Qiu, Guangxin Jiang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.39409
• Pdf link: https://arxiv.org/pdf/2610.39409
• Abstract Many applications of multi-objective reinforcement learning (MORL) require policies that balance conflicting objectives while satisfying safety constraints. We study safe MORL with zero constraint violation throughout learning, covering both cumulative and stepwise constraints. A key challenge is to explore sufficiently under unknown dynamics and constraint costs without violating the constraints. For a given preference over objectives, we develop an algorithm to learn a safe Pareto optimal policy by combining optimistic reward estimates with pessimistic cost bounds. With high probability, every policy executed over $T$ episodes incurs zero constraint violation. The algorithm returns a safe Markov policy induced by the averaged occupancy measure and achieves an $\widetilde{O}(1/\sqrt{T})$ suboptimality gap under smooth Tchebycheff scalarization. We further develop a preference-free algorithm that first explores safely without a prespecified preference and then computes a safe policy for any given preference without further interaction. Over $T$ exploration episodes and $K$ planning iterations, it retains the same safety guarantee and achieves an $\widetilde{O}(1/\sqrt{T}+1/\sqrt{K})$ suboptimality gap. Both algorithms require knowledge of a policy that strictly satisfies the constraints and its costs. We then relax these requirements, incurring at most constant cumulative violation. To our knowledge, we provide the first sample-efficient MORL algorithms that guarantee safety for every executed policy.
• 中文摘要 多目标强化学习（MORL）的许多应用需要在满足安全约束的同时平衡冲突目标的策略。我们研究在学习过程中零约束违规的安全MORL，涵盖累积约束和逐步约束。一个关键挑战是在未知动力学和约束成本下充分探索，同时不违反约束。针对特定目标偏好，我们开发了一种算法，通过将乐观奖励估计与悲观成本界限结合，学习安全的帕累托最优策略。高概率下，每次执行$T$集的策略都不发生约束违规。该算法返回由平均占用度量诱导的安全马尔可夫策略，并在平滑切比谢夫标量下实现$\widetilde{O}（1/\sqrt{T}）$次最优差。我们进一步开发了一种无偏好的算法，先在没有预设偏好的情况下安全探索，然后为任意给定偏好计算安全策略，无需进一步互动。经过$T$次探索和$K$的规划迭代，它保持相同的安全保障，并实现$\widetilde{O}（1/\sqrt{T}+1/\sqrt{K}）$次最优差距。这两种算法都需要知道严格满足约束及其成本的策略。然后我们放宽这些要求，最多产生恒定累积违规。据我们所知，我们提供了首个保证每个执行策略安全的样本高效MORL算法。

Timbre: Reliability-aware harmonic consensus for label-free multimodal test-time rexinforcement learning

Timbre：用于无标签多模测试时间强化学习的可靠性感知和谐共识

• Authors: Haotian Lyu, Haoyu Wang, Ru Wang, Feng Ji, Ziwen Li, Jiaxian Guo, Mingming Gong
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.39439
• Pdf link: https://arxiv.org/pdf/2610.39439
• Abstract Label-free test-time reinforcement learning adapts vision--language models using rewards derived from their own predictions. However, misinterpreted visual evidence and sensitivity to textual formulation can produce a consistent but incorrect consensus that majority voting reinforces. Cross-view aggregation can expose such errors, but image and text transformations may themselves degrade task-relevant evidence to different extents. We introduce \method, a reliability-aware harmonic consensus framework for multimodal test-time reinforcement learning. \method constructs a $2\times2$ grid combining original and perturbed images with direct-solving and reframing modes, selects pseudo-labels through weighted harmonic aggregation, and adapts modality-level weights using batch-averaged, temporally smoothed shifts in pseudo-label confidence as a heuristic reliability signal. We formulate consensus as a weighted $\chi^2$-barycenter problem whose unique minimizer is the normalized weighted harmonic aggregate, without requiring correct-answer invariance. An exact support--dispersion decomposition characterizes when harmonic aggregation preserves or corrects arithmetic rankings, while a sensitivity analysis establishes how modality reweighting changes pairwise ranking margins, informing interpretation of the heuristic updates. Across six benchmarks and three Qwen3-VL model scales, \method outperforms TTRV in 16 of 18 benchmark--model settings on adaptation instances and 17 of 18 settings on held-out instances under matched estimation and optimization rollout counts. These results demonstrate improved target-task adaptation and transfer to unseen instances without further updates.
• 中文摘要 无标签测试时间强化学习通过视觉-语言模型的奖励，调整其自身预测的奖励。然而，误解的视觉证据和对文本表述的敏感性可能产生一致但错误的共识，而多数投票则强化了这种共识。交叉视图聚合可以暴露此类错误，但图像和文本转换本身也可能在不同程度上削弱任务相关证据。我们介绍了\method，一种基于可靠性的谐和谐共识框架，用于多模态测试时间强化学习。\方法构建了一个价值$2\times2$的网格，结合原始图像和扰动图像，配合直接求解和重构模式，通过加权谐波聚合选择伪标签，并通过批次平均、时间平滑的伪标签置信度变化作为启发式可靠性信号调整模态级权重。我们将共识表述为加权$\chi^2$-重心问题，其唯一最小化子为归一化加权调和聚合，无需正确答案不变性。精确支持-离散分解描述调和聚合是否保持或修正算术排序，而敏感性分析则确定模态重权重如何改变两两排序间距，从而指导启发式更新的解释。在六个基准测试和三个Qwen3-VL模型尺度中，\方法在18个基准测试中优于16个基准测试——在匹配估计和优化推广计数下，在未被保留实例中17个设置中优于TTRV。这些结果展示了目标任务适应和未见实例转移能力的提升，无需进一步更新。

Bayesian Offline Multi-Agent Reinforcement Learning with Latent World Models

基于潜在世界模型的贝叶斯离线多智能体强化学习

• Authors: Xi Xiong, Lu Liu, Yiming Li, Jakob Foerster
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.39576
• Pdf link: https://arxiv.org/pdf/2610.39576
• Abstract Model-based offline multi-agent reinforcement learning (MARL) provides a promising approach to multi-agent decision-making under partial observability, where the global state is unavailable from offline data. However, limited dataset coverage can lead to substantial out-of-distribution (OOD) model errors, which may mislead policy optimization and degrade downstream performance. Existing model-based offline MARL methods typically rely on access to global states or mitigate model errors through pessimistic or conservative policy optimization, leaving the reliability and refinement of learned world models less explored. We propose a Bayesian offline MARL framework that infers latent global states from decentralized observation-action histories and iteratively refines a posterior world model using only offline data. Building on posterior information loss, we extend this criterion to the multi-agent setting as multi-agent posterior information loss (MPIL), which provides a reliability measure for world model refinement and offline evaluation. Theoretically, we establish an upper bound on policy regret that is monotonic in MPIL and further characterize its dependence on the size and state-joint-action coverage of the offline dataset. Experiments on partially observable multi-agent variants of D4RL MuJoCo tasks demonstrate competitive policy performance across diverse offline data conditions, while MPIL-guided refinement improves world model prediction and downstream policy performance. These results support a principled connection between world model reliability, offline data coverage, and policy performance in partially observable offline MARL.
• 中文摘要 基于模型的离线多智能体强化学习（MARL）为部分可观测性（即离线数据无法获得全局状态）提供了一种有前景的多智能体决策方法。然而，有限的数据集覆盖可能导致显著的非分布（OOD）模型错误，误导策略优化并降低下游性能。现有基于模型的离线MARL方法通常依赖于访问全局状态，或通过悲观或保守策略优化来缓解模型错误，这使得学习世界模型的可靠性和精炼性较少被探索。我们提出了一个贝叶斯离线MARL框架，从分散式观察-行动历史推断潜在全局状态，并仅使用离线数据迭代完善后验世界模型。基于后验信息丢失，我们将该准则扩展至多智能体环境，称为多智能体后验信息丢失（MPIL），为世界模型细化和离线评估提供了可靠性度量。理论上，我们建立了MPIL中单调的政策遗憾上限，并进一步表征其对离线数据集规模和状态-联合行动覆盖的依赖性。在部分可观测的多智能体D4RL MuJoCo任务中实验显示，在多种离线数据条件下具有竞争力的政策表现，而MPIL引导的细化则提升了世界模型预测和下游政策表现。这些结果支持了世界模型可靠性、离线数据覆盖率与部分可观测离线MARL政策表现之间的原则性联系。

Incentivizing Self-Learning in Large Language Models via Reinforcement Learning

通过强化学习激励大型语言模型中的自学

• Authors: Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen, Wenwei Zhang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.39671
• Pdf link: https://arxiv.org/pdf/2610.39671
• Abstract Large language models (LLMs) draw on parametric knowledge and retrieval-augmented generation (RAG) to solve knowledge-intensive tasks. Yet problem solving in unfamiliar domains calls for tightly coupled knowledge acquisition and reasoning, beyond factual retrieval and aggregation or learning solely from a preassembled context. We study self-learning -- autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly. To investigate this capability, we formulate a textbook-based mathematical problem-solving task. LLMs must navigate textbook exposition, learn and apply relevant principles, and construct proofs for exercises, with reference solutions withheld. We introduce a BookNavigator interface to support this human-inspired study process and incentivize self-learning through reinforcement learning (RL) with fine-grained rewards. Experiments show that our method improves performance across in-domain and out-of-domain textbooks and subjects, raising Intern-S2-Preview-35B's overall proof accuracy from 51.5% to 79.5% and outperforming RL-trained textbook-free and fixed-retrieval baselines. These gains extend to complex long-context benchmarks involving different task types, with improvements over the base model of 1.7 and 2.3 percentage points on BrowseComp-Plus and CL-bench, respectively. Together, these findings suggest that RL can foster a domain-agnostic self-learning strategy, highlighting the potential of self-learning from long-form corpora to strengthen broader problem-solving capabilities on complex, long-context tasks.
• 中文摘要 大型语言模型（LLMs）依赖参数化知识和检索增强生成（RAG）来解决知识密集型任务。然而，在陌生领域解决问题需要紧密结合的知识获取和推理，超越事实检索和聚合或仅从预组语言环境中学习。我们研究自我学习——即实时自主检索、学习和应用长篇参考资料中的知识。为探讨这一能力，我们制定了一个基于教材的数学问题解决任务。LLM必须驾驭教科书说明，学习并应用相关原则，构建练习证明，且不包含参考解答。我们引入了BookNavigator接口，支持这种以人为本的学习过程，并通过强化学习（RL）激励自学，提供细粒度奖励。实验显示，我们的方法提升了跨域和域外教材及学科的表现，将Intern-S2-Preview-35B的整体证明准确率从51.5%提升至79.5%，并优于强化学习训练的无教材和固定检索基线。这些提升延伸至涉及不同任务类型的复杂长上下文基准，较BrowseComp-Plus和CL-bench的基础模型分别提升了1.7个百分点和2.3个百分点。综合这些发现表明，强化学习能够促进一种领域无关的自学策略，凸显了从长格式语料库自学在复杂长上下文任务中增强更广泛问题解决能力的潜力。

Federated Agentic AI for Interference-Aware Adaptive Spectrum Coordination

联合智能人工智能用于干扰感知的自适应频谱协调

• Authors: Tomer Baziza Paz, Yaniv Cohen, Kobi Cohen
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.39680
• Pdf link: https://arxiv.org/pdf/2610.39680
• Abstract Emerging intelligent wireless networks increasingly rely on autonomous AI agents to adapt spectrum usage to dynamic and heterogeneous operating conditions. However, collective learning is challenging when agents experience different local interference environments and cannot exchange raw observations due to communication-overhead and privacy constraints. We propose Federated Agentic Spectrum Coordination with Interference Awareness (FASCIA), a federated multi-agent deep reinforcement learning framework for adaptive spectrum coordination. In FASCIA, each network manager acts as an autonomous AI agent that perceives its local interference environment, accumulates experience in a persistent local memory, and learns an adaptive spectrum coordination policy through local interactions. Coordination among agents is achieved through periodic federated aggregation of locally trained models, without exchanging raw sensing data or experience. Since spectrum decisions made by one agent alter the environments observed by others, local learning is inherently heterogeneous and coupled. FASCIA employs proximal regularization to mitigate the resulting model drift and explicitly accounts for inter-carrier interference arising from partially overlapping channels. FASCIA is the first federated agentic AI framework to address this challenging interference-coupled spectrum coordination setting. Extensive simulations demonstrate that FASCIA achieves performance close to the centralized benchmark while avoiding raw-experience exchange and maintaining effectiveness across heterogeneous network configurations.
• 中文摘要 新兴的智能无线网络越来越依赖自主人工智能智能体来适应动态和异构的运行条件来调整频谱使用。然而，当智能体经历不同的局部干扰环境，且由于通信开销和隐私限制无法交换原始观测数据时，集体学习就变得具有挑战性。我们提出了联合智能体频谱协调与干扰感知（FASCIA），这是一个联邦多智能体深度强化学习框架，用于自适应频谱协调。在FASCIA中，每个网络管理者作为自主的人工智能智能体，感知其局部干扰环境，积累经验于持久的本地记忆中，并通过局部交互学习自适应频谱协调策略。代理之间的协调通过定期联合聚合本地训练模型实现，无需交换原始的传感数据或经验。由于一个智能体做出的频谱决策会改变其他智能体观察到的环境，局部学习本质上是异构且耦合的。FASCIA 采用近端正则化来缓解模型漂移，并明确考虑部分重叠信道引起的载波间干扰。FASCIA 是首个解决这一复杂干扰耦合频谱协调设置的联邦智能人工智能框架。大量模拟表明，FASCIA 在避免原始经验交换的同时，在异构网络配置中保持有效性，实现接近中心化基准的性能。

Heterogeneous Co-Evolution Learning for Tool-Efficient Search Agents with Zero Labels

工具效率高的零标签搜索代理的异构共进化学习

• Authors: Qin-Wen Luo, Sheng Ren, Xiang Chen, Rui Liu, Jun Fang, Naiqiang Tan, Jun Wang, Hua Chai, Sheng-Jun Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.39682
• Pdf link: https://arxiv.org/pdf/2610.39682
• Abstract Deep search agents extend language models by interleaving reasoning with external retrieval. However, effective search-agent training requires addressing two challenges beyond final-answer optimization. First, the capability frontier of agents is largely determined by training questions, whereas fixed or labeled task collections cannot adapt as agents improve. Second, outcome-only reinforcement learning treats all successful trajectories equally, even when some correct answers rely on redundant searches, creating an inherent tension between answer accuracy and tool efficiency. To address these issues, we introduce \textsc{ZEST-Search}, a zero-label and tool-efficient framework for search-agent training via heterogeneous co-evolution. Starting from unlabeled reference questions, \textsc{ZEST-Search} constructs validated search tasks online through an answer-first and evidence-path-guided generation process. A heterogeneous proposer adaptively constructs training tasks around the solver's evolving capability boundary, while the solver learns search-augmented reasoning through reinforcement learning. To jointly optimize task difficulty and search efficiency, \textsc{ZEST-Search} employs an intermediate-difficulty reward for the proposer and a task-local tool-aware reward for the solver, encouraging informative tasks and correct answers with fewer redundant searches. Experiments on seven general-domain and multi-hop question answering benchmarks demonstrate that \textsc{ZEST-Search} consistently improves search-agent performance while achieving higher tool efficiency without labeled search-training data or expert search trajectories. Further analyses show that heterogeneous generation, adaptive difficulty control, and tool-aware optimization enable more effective search-agent training.
• 中文摘要 深度搜索代理通过交错推理与外部检索来扩展语言模型。然而，有效的搜索代理训练需要解决超越最终答案优化的两个挑战。首先，代理的能力边界主要由训练问题决定，而固定或带标签的任务集合无法随着代理的提升而调整。其次，仅结果强化学习对所有成功轨迹一视同仁，即使部分正确答案依赖冗余搜索，这在答案准确性与工具效率之间产生了固有的张力。为解决这些问题，我们引入了 \textsc{ZEST-Search}，这是一个零标签且工具高效的搜索代理训练框架，通过异构共进化实现。从无标签参考问题出发，\textsc{ZEST-Search} 通过以答案为先、证据路径引导的生成过程构建经过验证的在线搜索任务。异构提案者自适应地围绕求解器不断演化的能力边界构建训练任务，而求解者则通过强化学习学习搜索增强推理。为共同优化任务难度和搜索效率，\textsc{ZEST-Search} 为提议者提供中等难度奖励，为求解者提供任务局部工具感知奖励，鼓励信息性任务和正确答案，减少冗余搜索。在七个广域和多跳问答基准测试上的实验表明，\textsc{ZEST-Search} 在无标签搜索训练数据或专家搜索轨迹的情况下，持续提升搜索代理性能，同时实现更高的工具效率。进一步分析显示，异构生成、自适应难度控制和工具感知优化使搜索代理训练更有效。

Sandman: Building and Operating Sandboxes for Agentic Reinforcement Learning on Shared Kubernetes

Sandman：在共享 Kubernetes 上构建与运行代理强化学习沙盒

• Authors: Yiming Yan, Hongyang Wang, Wenbo Chen, Xuan Wang, Jingguo Liu, Xian Li, Chen Qi, Leo Fan, Siwei Li, Qianli Wu, Yichuan Li, Ruochen Wen, Qian Qu, Zheshen Wang, Benoit Dumoulin
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.39711
• Pdf link: https://arxiv.org/pdf/2610.39711
• Abstract Agentic reinforcement learning (RL) often requires repeated tool calls in task-specific environments. A training job may create thousands of these environments in bursts while sharing a cluster with other jobs. We present Sandman, a sandbox service built and operated at Amazon for agent tool execution and evaluation. It serves multiple production agentic RL workloads on shared Kubernetes. A training framework supplies a task image and uses commands and file operations in the returned sandbox. Files and processes persist across calls until the framework releases the environment. Sandman manages creation, readiness and cleanup. Each sandbox is a Kubernetes pod with a small daemon mounted into the unmodified image. In a scale test, Sandman held 11,858 concurrent tiny sandboxes. Four production SWE-bench Verified training campaigns created 3.1 million sandboxes in October 2026. We report the design trade-offs and operational lessons from handling creation bursts, preparing task environments, reclaiming abandoned sandboxes, and running evaluation under controlled conditions. We will open-source Sandman's daemon, provider and pool.
• 中文摘要 代理强化学习（RL）通常需要在任务特定环境中重复调用工具。一个训练任务可能在与其他作业共享集群的同时，以爆发方式创建数千个此类环境。我们介绍Sandman，亚马逊开发并运营的沙盒服务，用于执行和评估代理工具。它在共享的Kubernetes上服务多个生产代理强化学习工作负载。训练框架提供任务镜像，并在返回的沙盒中使用命令和文件操作。文件和进程在调用间持续存在，直到框架发布环境。Sandman负责创建、准备和清理。每个沙盒是一个Kubernetes Pod，未修改的镜像中安装了一个小型守护进程。在一次规模测试中，Sandman同时拥有11,858个并发的微型沙盒。四个生产型SWE工作台验证培训活动在2026年10月创建了310万个沙盒。我们报告了处理创建突发、准备任务环境、回收废弃沙箱以及在受控条件下进行评估时的设计权衡和运营经验。我们将开源Sandman的守护进程、提供者和池。

Deep Reinforcement Learning for Blind Passive Beamforming in RIS-Assisted MIMO Systems

RIS辅助MIMO系统盲被动波束成形的深度强化学习

• Authors: Oumaïma Bounhar, Thomas Chêne, Ghaya Rekaya-Ben Othman, Oussama Damen
• Subjects: Subjects: Information Theory (cs.IT); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.39827
• Pdf link: https://arxiv.org/pdf/2610.39827
• Abstract Reconfigurable Intelligent Surfaces (RIS) are a promising solution for improving millimeter-wave (mmWave) communications by controlling signal propagation. Yet, selecting the optimal RIS configuration remains challenging due to the high-dimensional search space and signaling overhead. We propose a Deep Reinforcement Learning (DRL) framework for blind passive beamforming, where a Deep Q-Learning (DQL) agent learns an adaptive policy to select near-optimal RIS configurations while sounding only few codewords, without channel state information. Results show that our approach outperforms Q-Learning (QL) and conventional baselines in convergence, performance, and overhead reduction.
• 中文摘要 可重构智能表面（RIS）是一种有前景的解决方案，能够通过控制信号传播来改善毫米波（mmWave）通信。然而，由于高维搜索空间和信号开销，选择最优RIS配置仍然具有挑战性。我们提出了一种深度强化学习（DRL）框架用于盲被动波束成形，其中深度Q学习（DQL）智能体学习自适应策略，在仅发音少量码字的同时选择接近最优的RIS配置，且不涉及信道状态信息。结果显示，我们的方法在收敛、性能和开销减少方面优于Q-Learning（QL）和传统基线。

Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

将医学视觉语言模型与熵自适应规律基础对齐能力边界

• Authors: Hongyu Bai, Jiaqi Zeng, Chunzheng Zhu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.39835
• Pdf link: https://arxiv.org/pdf/2610.39835
• Abstract Medical diagnostic reasoning requires verifiable links between visual evidence and clinical knowledge, yet current medical vision language models are aligned with objectives that reward the wrong behavior. Uniform supervised fine tuning lets easy cases dominate updates while clinically critical boundary cases saturate prematurely. Sparse rewards based only on final-answer correctness cannot assess anatomical grounding or clinical logic, inviting reward hacking through answer shortcuts. We propose MedCBR, a two-stage alignment framework coupling boundary-aware preference optimization with rubric-grounded reinforcement learning. First, we construct counterfactual boundary preferences at the model's capability frontier using empirical pass rates and preference pairs that share reasoning prefixes but diverge at decisive diagnostic branches. Difficulty Weighted Boundary Preference Optimization (DBPO) concentrates learning on this calibrated boundary region. Second, we distill reusable clinical rubrics from boundary trajectories and activate them according to prediction entropy. Entropy Gated Grounded Policy Optimization (EGPO) rewards answer correctness, anatomical grounding, and reasoning fidelity while suppressing rubric noise on reliable samples. A reference policy anchored to the boundary-aligned model couples population-level boundary priors with instance-level uncertainty, making diagnostic reasoning more accurate and clinically grounded. Experiments on six medical benchmarks show consistent improvements over strong general, specialized, and reinforcement learning baselines, reaching 61.2% average accuracy with the largest gains on challenging differential diagnosis tasks. Ablations show super-additive gains: combining the two stages exceeds the sum of their individual improvements.
• 中文摘要 医学诊断推理需要视觉证据与临床知识之间可验证的联系，但当前医学视觉语言模型与奖励错误行为的目标保持一致。统一监督微调使简单案例主导更新，而临床关键边界案例过早饱和。仅基于最终答案正确性的稀疏奖励无法评估解剖基础或临床逻辑，导致通过答案捷径进行奖励黑客。我们提出了MedCBR，一个两阶段对齐框架，将边界感知偏好优化与基于评分标准的强化学习结合起来。首先，我们利用经验通过率和偏好对构建模型能力前沿的反事实边界偏好，这些对共享推理前缀但在决定性诊断分支处存在分歧。难度加权边界偏好优化（DBPO）将学习集中于该校准边界区域。其次，我们从边界轨迹中提取可重复使用的临床规律，并根据预测熵激活它们。熵门控基础策略优化（EGPO）奖励答案正确性、解剖学基础和推理准确性，同时抑制可靠样本上的评分规包噪声。锚定于边界对齐模型的参考策略将群体层面边界先验与实例级不确定性结合，使诊断推理更准确且更具临床依据。六个医学基准测试显示，在强通用、专业和强化学习基线上均有持续提升，平均准确率达到61.2%，在具有挑战性的鉴别诊断任务中获得最大提升。消融显示超加性提升：两阶段结合超过各自改进的总和。

Learning Calibrated Confidence for Multi-Agent Collaboration

多智能体协作的校准信心学习

• Authors: Chenxiao Fan, Chongming Gao, Gangyi Zhang, Yaxin Gong, Leyang Shen, Jiakai Wang, Dong Wang, Yang Liu, Haoyan Liu, Xinting Hu, Fuli Feng, Xiangnan He
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.39927
• Pdf link: https://arxiv.org/pdf/2610.39927
• Abstract Complex information seeking on the Web often requires collecting evidence from multiple documents and reasoning across them, as in multi-hop question answering. Multi-agent systems address such tasks by dividing work among specialized agents, where upstream agents produce intermediate answers and supporting evidence that downstream agents build upon. However, upstream answers vary in correctness, and most systems provide no quantitative signal about it, leaving the downstream agent to judge whether to trust or verify what it receives based solely on the textual evidence. In this paper, we propose MASCAL. Each agent learns to produce a correctness probability for its answer by jointly optimizing task performance and confidence calibration in reinforcement learning; the upstream agent passes this probability at handoff, giving the downstream agent a quantitative basis for its decisions, and the final answer also carries a calibrated probability, quantifying the reliability of the system output. To guide the downstream agent in using this signal during training, we introduce confidence-aware weighting, which increases the learning weight at positions responsive to the upstream probability, so that the agent learns more from decisions where it actively uses the signal. Across three collaboration architectures spanning Web retrieval and reasoning tasks, MASCAL improves task accuracy and calibration quality for both agents, and generalizes to deeper collaboration chains and different model scales. Intervention analysis confirms that upstream confidence affects downstream decision behavior, and the learned final confidence supports selective answering and targeted rechecking.
• 中文摘要 在网络上寻求复杂信息通常需要从多份文档中收集证据并跨越这些文档进行推理，如多跳问答。多智能体系统通过将工作分配给专业智能体来解决此类任务，上游智能体提供中间答案和支持证据，下游智能体则在此基础上构建。然而，上游答案的正确性各不相同，大多数系统不提供定量信号，下游智能体只能根据文本证据判断是信任还是验证所接收到的信息。本文提出MASCAL方法。每个智能体通过强化学习中共同优化任务性能和置信度校准，学会为其答案产生正确概率;上游智能体在交接时传递该概率，为下游智能体的决策提供定量基础，最终答案也带有校准概率，量化系统输出的可靠性。为了指导下游智能体在训练中使用该信号，我们引入了信心感知加权，提高对上游概率响应的位置的学习权重，使智能体从主动使用该信号的决策中获得更多学习。在三种跨越网络检索和推理任务的协作架构中，MASCAL 提高了两个智能体的任务准确性和校准质量，并推广到更深层次的协作链和不同模型尺度。干预分析证实上游置信会影响下游决策行为，而学习到的最终置信支持选择性回答和有针对性的重新检查。

H$^2$R: Hierarchical Hint-Based Reward for Mathematical Reasoning via Reinforcement Learning

H$^2$R：基于强化学习的基于层级提示的数学推理奖励

• Authors: He Tu, Siye Li, Hongyi Wen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.39934
• Pdf link: https://arxiv.org/pdf/2610.39934
• Abstract Reinforcement Learning with Verifiable Rewards has shown great promise in enhancing mathematical reasoning in large language models. However, existing approaches provide limited intermediate supervision for complex, multi-step reasoning tasks. To bridge this gap, we propose \textbf{Hierarchical Hint-based Reward}, a framework that leverages the intrinsic sub-question structure of mathematical problems as structured intermediate conditioning signals during RLVR training. Unlike chain-of-thought prompting, these hierarchical hints serve as logically grounded intermediate milestones that effectively reduce the reasoning horizon toward the final answer. To validate this framework, we introduce \textbf{HiMath}, a benchmark of hierarchically structured mathematical problems. Across diverse LLMs, hierarchical hints consistently improve reasoning performance even without task-specific training. When integrated into RL training via H$^2$R, they boost Qwen2.5-Math-7B accuracy on HiMath from 30.4\% to 39.1\% using hierarchical hints, and up to 45.6\% with Oracle Hierarchical Hints. This pronounced improvement demonstrates that incorporating reliable intermediate checkpoints into reward design is essential for sustaining complex mathematical reasoning.
• 中文摘要 带可验证奖励的强化学习在提升大型语言模型中的数学推理方面展现出巨大潜力。然而，现有方法对复杂多步推理任务提供的中间监督有限。为弥合这一差距，我们提出了\textbf{层级提示奖励}框架，利用数学问题的内在子问题结构作为RLVR训练中的结构化中间条件信号。与思维链提示不同，这些层级提示作为逻辑基础的中间里程碑，有效缩小了通往最终答案的推理视野。为验证该框架，我们引入了\textbf{HiMath}，这是层级结构数学问题的基准。在多种大型语言模型中，层级提示即使在没有任务特定训练的情况下，也能持续提升推理表现。通过H$^2$R将Qwen2.5-7B的HiMath准确率从30.4%提升至39.1%，使用Oracle Hierarchical Hints则提升至45.6\%。这一显著提升表明，将可靠的中间检查点纳入奖励设计对于维持复杂数学推理至关重要。

Mastering Multi-State Dynamics: Continuum Damage and Viscoplastic MPM for RL-Based Dexterous Manipulation

掌握多态动力学：基于强化学习的连续损伤与粘塑性MPM在灵巧操作中的应用

• Authors: Yusen Feng, Shangbo Luan
• Subjects: Subjects: Graphics (cs.GR)
• Arxiv link: https://arxiv.org/abs/2610.39975
• Pdf link: https://arxiv.org/pdf/2610.39975
• Abstract Robotic manipulation of multi-state materials needs a simulator that handles large deformation, topology change, and two-way contact between tools and materials while staying fast enough to drive online reinforcement learning. We present MG-CD-MPM, a tool-centered localized extension of CD-MPM. The implicit Ginzburg-Landau phase-field damage system is solved on a local patch by preconditioned conjugate gradients using a geometric-multigrid V-cycle as the preconditioner. The background MPM discretization remains uniform: only the damage solve is localized. Two-way rigid-tool coupling converts MPM contact impulses into forces and torques on the tool, exposing material resistance directly to the policy. The resulting Gym-like simulator covers soft solids, brittle materials, granular media, and viscoplastic or semi-fluid media within a unified per-particle constitutive layer. We demonstrate cutting, scooping, and shoveling in six scenes, combining scripted tool motions with PPO policies trained on two cutting tasks.
• 中文摘要 多态材料的机器人操作需要一个模拟器，能够处理大变形、拓扑变化以及工具与材料之间的双向接触，同时保持足够速度以驱动在线增强学习。我们介绍MG-CD-MPM，这是CD-MPM的以工具为中心的局部扩展。隐含的Ginzburg-Landau相场损伤系统通过预处理共轭梯度在局部补丁上求解，使用几何多栅点V-循环作为预处理器。背景MPM离散化保持均匀：只有损伤解是局部化的。双向刚性工具耦合将MPM接触冲量转换为工具上的力和扭矩，直接暴露材料抵抗于策略。最终的Gym类模拟器覆盖软固体、脆性材料、颗粒介质以及粘塑性或半流体介质，形成统一的每颗粒构件层。我们在六个场景中演示切割、铲雪和铲土，结合了脚本化工具动作与基于两个切割任务训练的PPO政策。

Humanoid Dodgeball: Learning Fast Dynamic Obstacle Avoidance Motions for Humanoids

类人躲避球：学习快速动态障碍避让动作

• Authors: Aoqian Zhang, Chunzheng Wang, Haoran Chen, Annan Tang, Cheng Xiang, Fan Shi, Shuzhi Sam Ge
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.40003
• Pdf link: https://arxiv.org/pdf/2610.40003
• Abstract Humanoids have become increasingly agile and capable in real-world environments. To operate safely in dynamic scenarios, they must rapidly evade fast-moving obstacles while maintaining stability and physically feasible whole-body motion. Learning such behaviors remains challenging due to the high-dimensional dynamics of humanoid systems and the need to jointly balance reactive obstacle avoidance, motion naturalness, and control robustness. In this work, we present Humanoid Dodgeball, a reinforcement learning (RL) framework for agile dynamic obstacle avoidance in humanoids. Our method enables highly dynamic whole-body evasive maneuvers against incoming projectiles through threat-aware reward shaping, while multiple direction-specific adversarial motion priors regularize the learned behaviors toward stable and diverse human-like motions. Extensive simulation and hardware experiments demonstrate that the proposed framework substantially improves dynamic obstacle avoidance performance across diverse obstacle speeds and launch distances, while preserving robust and physically plausible humanoid motions.
• 中文摘要 类人生物在现实环境中变得越来越敏捷和具备能力。为了在动态场景中安全操作，他们必须在保持稳定性和物理可行的全身运动的同时，快速躲避快速移动的障碍物。由于类人系统具有高维动态性，需要共同平衡反应性障碍避让、运动自然性和控制鲁棒性，学习此类行为仍然具有挑战性。在本研究中，我们提出了人形躲避球，一种用于人形生物敏捷动态避障的强化学习（RL）框架。我们的方法通过威胁感知的奖励塑造，实现了对来袭投射物的高度动态全体规避机动，同时多方向特定的对抗运动先验规范了学习到的行为，朝向稳定且多样化的类人类动作。广泛的仿真和硬件实验表明，所提框架在不同障碍速度和发射距离下显著提升了动态障碍避让性能，同时保持了稳健且物理合理的类人生物动作。

Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information

从一个数量：信息的价值中获得廉价检索和观察信用分配

• Authors: Daniel Strauss, Nikolaos Sotirakis, Zhenshan Bing, Alois Knoll
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.40030
• Pdf link: https://arxiv.org/pdf/2610.40030
• Abstract Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.
• 中文摘要 强化学习中的记忆引发了两个问题。注意力通过对整个存储进行评分来找到相关项目，这对在延迟和能量限制下的机器人来说成本较高。当检索到的记忆获得回报时，获得该记忆的功劳必须跨越整个序列。我们认为两者都受一个量控制，即信息的价值，并建议通过学习世界模型从存档的展开中估算信息价值。我们得到，当在观察时读取时，它给出了获取动作的最优值可证明的下界，从而对其TD目标定下了底，并将该底线归因于获得该目标的观察。在决策处读取，它定义了最优检索集。一个试点检索系统，将用该信号进行训练和改进，解决上下文较少的记忆任务。

Reward-Adaptive Co-Evolution of Policy and Rubric

奖励-适应性政策与评分标准的共同演进

• Authors: Yizhu Chen, Jianguang Jiang, Jiameng Huang, Chuyu Fang, Bojun Zhang, Zhenyu Jiao, Xing Yuan, Tong Lin
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.40096
• Pdf link: https://arxiv.org/pdf/2610.40096
• Abstract Rubric-based rewards provide fine-grained, interpretable supervision for open-ended reinforcement learning, where holistic scalar rewards struggle to capture multidimensional response quality. Static rubrics can lose discriminative power as the policy distribution evolves, while existing policy--rubric co-evolution methods rely on globally prescribed update schedules that ignore heterogeneous reward informativeness. We introduce RACE, a Reward-Adaptive Co-Evolution framework. At the group level, per-rollout-group training-signal routing classifies groups as informative, diversity-collapsed, or rubric-failed, updating the policy only with informative groups while caching rubric failures. At the phase level, event-triggered phase switching enters rubric optimization upon persistent reward saturation and returns when rubric-reward improvement plateaus. Rubric phases train the generator on cached pairs using a frozen judge and a scale-normalized pairwise-margin reward. Across both levels, online threshold adaptation adjusts routing and switching thresholds through proportional control. Under a unified evaluation protocol, RACE achieves the highest average performance and outperforms the strongest fixed-alternation baseline on all eight tasks, while improving rubric discrimination.
• 中文摘要 基于评分标准的奖励为开放式强化学习提供了细粒度且可解释的监督，而整体标量奖励难以捕捉多维响应质量。静态评分标准随着策略分布的演变可能失去判别力，而现有的策略——评分标准共演化方法依赖于全球规定的更新计划，忽视异质奖励信息量。我们介绍了RACE，一种奖励自适应共进化框架。在组级，按推广组的训练信号路由将组分为信息型组、多样性崩溃组或评分标准失败组，仅更新策略，缓存评分表失败。在阶段层面，事件触发阶段切换在持续奖励饱和时进入评分标准优化，当评分标准-奖励改善趋于平稳时回归。评分标准阶段通过冷冻评判和量表归一化的两两边际奖励训练缓存对的生成器。在这两个层级中，在线阈值适应通过比例控制调整路由和切换阈值。在统一评估协议下，RACE在所有八项任务中均获得最高的平均性能，并优于最强的固定交替基线，同时提升了评分标准的辨别能力。

Dual-Agent Reinforcement Learning for Vehicle Scheduling and Multi-Energy Dispatch in Electric-Hydrogen-Transport Systems

电力-氢运输系统中车辆调度和多能调度的双智能体强化学习

• Authors: Fulong Yao, Yiming Xu, Maurizio Albano, Liana Cipcigan, Naeima Hamed, Bo Zou, Omar Rana
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.40116
• Pdf link: https://arxiv.org/pdf/2610.40116
• Abstract Electric-hydrogen-transport systems (EHTSs) require coordinated scheduling of electric and hydrogen vehicles together with multi-energy dispatch. These operations form a sequential decision process: upstream vehicle scheduling (VS) determines service allocation and order, thereby endogenously shaping the electricity and hydrogen demands faced by downstream energy dispatch (ED). This paper develops a dual-agent reinforcement learning framework tailored to the heterogeneous decision structures. The vehicle-scheduling agent combines proximal policy optimisation (PPO) with a Plackett-Luce policy to learn vehicle-priority permutations, which are then mapped to feasible charger and refueller assignments through a greedy allocation. The energy-dispatch agent employs twin delayed deep deterministic policy gradient (TD3) for continuous storage and energy-flow control. The directional coupling between the two decision layers is analysed, revealing that updates to the upstream scheduling policy can shift the demand distribution experienced by the downstream dispatcher. A three-stage training mechanism is therefore introduced, combining layer-wise pretraining with concurrent fine-tuning to mitigate the effects of scheduling-induced distribution shifts on downstream ED learning. Year-long case studies show that the proposed framework achieves the lowest vehicle-service delay and energy cost among the 11 evaluated methods, demonstrating effective coordination between vehicle scheduling and multi-energy dispatch.
• 中文摘要 电力-氢气运输系统（EHTS）需要电力和氢能车辆的协调调度，以及多能调度。这些操作构成了一个顺序决策过程：上游车辆调度（VS）决定服务分配和顺序，从而内生塑造下游能源调度（ED）所面临的电力和氢气需求。本文开发了针对异构决策结构的双代理强化学习框架。车辆调度代理结合了近端策略优化（PPO）和Plackett-Luce策略，学习车辆优先级排列，然后通过贪婪分配映射到可行的充电器和加油车分配。能源调度代理采用双延迟深度确定性策略梯度（TD3）实现连续储存和能量流控制。分析了两层决策层之间的方向耦合，发现上游调度策略的更新可以改变下游调度员所经历的需求分布。因此引入了三阶段训练机制，结合层级预训练与并发微调，以减轻调度诱发的配电偏移对下游ED学习的影响。为期一年的案例研究表明，所提出的框架在11种评估方法中实现了最低的车辆服务延迟和能源成本，展示了车辆调度与多能源调度之间的有效协调。

Escaping Path Mirages in Offline Goal-Conditioned Reinforcement Learning

离线目标条件强化学习中的逃离路径幻象

• Authors: Junhyeon Bae, Jaebak Hwang, Gwanwoo Choi, Minung Kim, Seungyul Han
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.40130
• Pdf link: https://arxiv.org/pdf/2610.40130
• Abstract Offline goal-conditioned reinforcement learning remains challenging in stochastic long-horizon settings, where compounding value estimation errors hinder reliable goal reaching. While prior methods have sought to address this challenge through various approaches, a key challenge remains path mirage, where agents overcommit to spuriously successful trajectories in offline datasets induced by stochasticity, often leading to failure on difficult tasks. To address this issue, we propose Branch-aware Graph Planning (BGP), which captures stochastic branching structures by identifying high-variability points as graph nodes and designing goal-conditioned edges with segmented learning to avoid unreliable branches and reduce unnecessary stochasticity along paths. As a result, BGP enables reliable goal reaching even in highly stochastic long-horizon environments, and experiments on diverse OGBench tasks show that it substantially outperforms prior state-of-the-art offline goal-conditioned RL methods. Our implementation is available at this https URL.
• 中文摘要 离线目标条件强化学习在随机长视野环境中依然具有挑战性，因为叠加的值估计误差阻碍了可靠的目标达成。虽然以往方法通过多种方法试图解决这一挑战，但关键挑战仍是路径幻象，即代理在离线数据集中因随机性诱导而过度承诺错误成功的轨迹，常导致困难任务失败。为解决这一问题，我们提出了分支感知图规划（BGP），该方法通过识别高变异点为图节点，并通过分段学习设计目标条件边，避免不可靠的分支并减少路径上的不必要随机性，从而捕捉随机分支结构。因此，BGP即使在高度随机的长视野环境中也能实现可靠的目标达成，且在各种OGBench任务上的实验显示，其性能远超以往最先进的离线目标条件强化学习方法。我们的实现可在此 https 网址获取。

DYNAS: A Dynamic Network Penetration Testing Framework Using Deep Hierarchical Reinforcement Learning

DYNAS：一个基于深度层级强化学习的动态网络渗透测试框架

• Authors: Md. Rakibul Hassan, Majedul Islam, Md. Ashraful Alam, Muhammad Iqbal Hossain, Swakkhar Shatabda
• Subjects: Subjects: Cryptography and Security (cs.CR); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.40214
• Pdf link: https://arxiv.org/pdf/2610.40214
• Abstract As network scale and uncertainty complicate automated penetration testing, reinforcement learning (RL) environments must support controlled evaluation under defensive and non-stationary conditions. We present DYNAS (Dynamic Network Attack Simulator), which combines parameterized large-scale scenario generation with stateful intrusion detection responses, explicit scan false-positive and false-negative noise, temporary service unavailability, and duration-based action timeouts. We further introduce Pen-DHRL, a recurrent manager--worker policy that selects persistent subgoals using IDS-conditioned global context. We compare Pen-DHRL with GNN, Invariant, and Attention baselines across four scenario variants and multiple training regimes. Under IDS-disabled training followed by IDS-enabled evaluation, Pen-DHRL obtains approximately 64\% and 24\% higher reward per step on Bridge and Dynamic, respectively, and approximately 26\% and 30\% more captured hosts on VarA and VarB than the next-best baseline. With all defensive dynamics active from the beginning of training, Pen-DHRL achieves positive mean episode returns in all four scenarios, whereas the flat baselines produce negative mean returns in all 12 architecture--scenario combinations; staged curriculum training produces positive mean returns across all 16 evaluated combinations.
• 中文摘要 随着网络规模和不确定性使自动化渗透测试复杂化，强化学习（RL）环境必须支持在防御性和非平稳条件下的受控评估。我们介绍了DYNAS（动态网络攻击模拟器），它结合了参数化的大规模场景生成、有状态入侵检测响应、显式扫描的假阳性和假阴噪声、临时服务不可用以及基于持续时间的动作超时。我们还进一步介绍了Pen-DHRL，这是一种循环管理者-工人策略，利用IDS条件的全局上下文选择持久子目标。我们将Pen-DHRL与GNN、不变和注意力基线在四种场景变体及多种训练模式中进行比较。在IDS障碍训练和IDS激活评估下，Pen-DHRL在桥接和动态模式中每步奖励约高出64%和24%;在VarA和VarB捕获的宿主数量比次优基线多约26%和30%。从训练开始起所有防御动力学都激活，Pen-DHRL在所有四个场景中均均值为正，而平坦基线在所有12种架构场景组合中均值为负;分阶段课程训练在所有16种评估组合中均值均为正。

EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation

EnvOS：世界变迁时审计计算机使用代理：基于现象的任务构建、认证与循证评估

• Authors: Ankush Kadu, Athira Gopal, Niharika Singh, Arpit Jain, Aswanth Krishnan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.40246
• Pdf link: https://arxiv.org/pdf/2610.40246
• Abstract Computer-use agents built on large language models (LLMs) work in software that can change while they act: an appointment moves, a policy is revised, a record is updated. Testing agents under such changes is harder than it looks. When an agent fails, the cause may be the agent, but it may also be a change that did not fire as intended, an update the agent never received, or a grader that scored the episode wrongly. We introduce EnvOS, a framework for building and certifying environments that change mid-task. Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence. We apply it to stale information, where a value the agent has already seen changes before the agent acts on it. In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown; the gain over the reminder holds after correcting for all tests (p = 0.00293). In insurance-claim approval, Opus 5 re-checks on its own and succeeds on 10/10, whereas Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests. Whether an agent caught a change thus varied with the model and the interface. We also stress-test the grading itself: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks. Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.
• 中文摘要 基于大型语言模型（LLM）构建的计算机代理在操作过程中可能变化的软件中工作：约会移动、策略修订、记录更新。在这些变化下测试代理比看起来更难。当代理失败时，原因可能是代理本身，也可能是未按预期触发的更改、代理未收到的更新，或评分错误的评分器。我们引入了EnvOS，一个用于构建和认证任务中途变化环境的框架。在任何代理运行前，EnvOS检查每个变更是否正确触发，任务是否可解决，评分器是否正确与错误行为;每次运行后，单独代码从归档证据中重新核对判决。我们对陈旧信息应用，即代理在执行前已观察到的值变更。在诊所预约中，当没有显示额外信息时，Claude Opus 5成功率为0/16，提醒重新检查来源时成功率为2/16，更新值显示时成功率为15/16;在所有测试校正后，提醒值的增益依然成立（p = 0.00293）。在保险理赔审批中，Opus 5会自行重新检查并于10/10成功，而Claude Sonnet 5在再次显示当前保单时，从1/10升至9/10，这在其预注册测试中获得显著收益，但修正所有测试后并非如此。因此，代理人是否发现变化取决于模型和界面。我们还对评分本身进行了压力测试：对于规则盲设计的损坏事件，EnvOS流水线在26个中没有一个被接受为干净，而在外部生成的集上，它捕获了39个中的25个，而简单日志检查则捕获了4个。由于每个EnvOS环境都是从种子生成并由审计的自动验证器评分的，因此这些环境自然成为训练代理的对象，并通过可验证的奖励进行强化学习。

When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications

当需求相互作用：通过相互依赖的规范扩展长视野编码

• Authors: Yishuo Cai, Minzheng Wang, Liang Zhao, Linghao Zhang, Lei Li, Shicheng Li, Xu Sun, Fuli Luo
• Subjects: Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.40256
• Pdf link: https://arxiv.org/pdf/2610.40256
• Abstract To succeed in real-world long-horizon software development, coding agents must satisfy multi-dimensional task specifications, such as interface compatibility and error handling, and do so without breaking existing dependencies. However, current works lack specification in both quantity and diversity, often emphasizing primary goals while ignoring complex constraints. Synthesizing tasks to bridge this gap presents two challenges: naively adding specifications risks reducing tasks to trivial checklists rather than compelling joint reasoning, and verifying these subtle interactions risks overlooking hidden flaws or rejecting valid alternatives. To address these challenges, we introduce CodeSpecWeave, a framework for synthesizing tasks with rich, interdependent specifications. Grounded in real developer seeds, a synthesis agent iteratively develops both coupled task requirements and executable tests, while a review agent audits solution trajectories from multiple models to guarantee test reliability. Using this pipeline, we construct 4,670 training tasks across 2,865 repositories. Applying Reinforcement learning on these tasks significantly improves performance over the base model on DeepSWE-v1.1, largely outperforming existing baselines. Furthermore, we curate CodeSpecWeave-Bench, a rigorous benchmark comprising 100 tasks and 1,044 specifications, which clearly separates frontier models and reveals significant headroom in handling interdependent engineering constraints.
• 中文摘要 为了在现实世界的长期软件开发中取得成功，编码代理必须满足多维任务规范，如接口兼容性和错误处理，并且不破坏现有依赖关系。然而，当前作品缺乏数量和多样性的规范，常常强调主要目标，忽视复杂约束。综合任务以弥合这一差距存在两个挑战：天真添加规范可能导致任务简化为琐碎的检查清单，而无法推动共同推理;验证这些微妙交互则可能忽视隐藏缺陷或拒绝有效替代方案。为应对这些挑战，我们引入了CodeSpecWeave，一个用于综合具有丰富且相互依赖规范任务的框架。基于真实开发者种子，综合代理反复开发耦合任务需求和可执行测试，而审查代理则审计多个模型的解决方案轨迹，以确保测试可靠性。利用该流程，我们构建了4670个训练任务，覆盖2865个仓库。在这些任务上应用强化学习显著提升了DeepSWE-v1.1基础模型的性能，远超现有基线。此外，我们策划了CodeSpecWeave-Bench，这是一个包含100个任务和1044个规范的严谨基准测试，明确区分前沿模型，展现出处理相互依赖工程约束的巨大优势。

Learned Terminal Costs for Short-Horizon Model Predictive Control of Autonomous Trucks in Interactive Highway Traffic

在交互式高速公路交通中预测控制自动驾驶卡车的短期模型终端成本学习

• Authors: Hooman Hamze, Benjamin Houshmand, Deepthi Pathare, Erik Börve, Morteza Haghir Chehreghani, Nikolce Murgovski
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.40257
• Pdf link: https://arxiv.org/pdf/2610.40257
• Abstract Autonomous highway driving for heavy-duty trucks is a challenging control problem. The greater mass, dimensions and intricate dynamics increase the requirements for cooperation from surrounding traffic in critical maneuvers, requiring a careful balance between safety constraints and efficient long-horizon decision making. Model Predictive Control (MPC) provides rigorous constraint satisfaction but relies on terminal cost which can limit its effectiveness at short, computationally affordable horizons. Reinforcement Learning (RL) captures long-term driving behavior but offers no constraint guarantees and tends toward overly conservative policies when trained under sparse collision penalties. In this paper, we propose a hybrid MPC and RL framework for highway driving in autonomous trucks, where a finite-horizon MPC, solved via dynamic programming, uses a pre-trained RL model for terminal cost estimation. We evaluate the framework in a simulation environment with reactive surrounding traffic and compare it against pure MPC with zero terminal cost, rollout with terminal cost estimation using a base policy, and pure RL. Results show that the hybrid approach achieves zero collisions with the lowest cost of any configuration.
• 中文摘要 重型卡车的自动驾驶是一个具有挑战性的控制问题。更大的质量、尺寸和复杂的动态增加了关键机动中周围交通的协作需求，需要在安全约束与高效长视野决策之间取得谨慎平衡。模型预测控制（MPC）提供严格的约束满足，但依赖终端成本，这可能限制其在短期且计算可负担视野内的有效性。强化学习（RL）捕捉长期驾驶行为，但不提供约束保证，且在稀疏碰撞惩罚下训练时倾向于过于保守的策略。本文提出了一个混合MPC和RL框架，用于自动驾驶卡车的高速公路驾驶，其中通过动态规划求解有限视野MPC，利用预训练的RL模型进行终端成本估算。我们在有反应式环绕流量的仿真环境中评估该框架，并将其与纯MPC无终端成本、基于基础策略的终端成本估计推广以及纯强化学习进行比较。结果表明，混合方法实现了零碰撞，成本是所有配置中最低的。

SCARL: Sequence Confidence-Aligned Reinforcement Learning

SCARL：序列信心对齐强化学习

• Authors: Zhenghong Ke, Jingbo Zhou, Hua Wu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.40407
• Pdf link: https://arxiv.org/pdf/2610.40407
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning by scoring sampled responses with a rule-based verifier and reinforcing those that pass. Where a classifier's training couples predicted probability to labels, nothing ties an RLVR policy's confidence in a response to its reward: aligned training would drive responses with the same advantage toward equal confidence, yet the advantage is blind to confidence. Under this sequence-confidence misalignment, a success the policy barely found earns the same update as one it produces with near certainty, and a confident error is corrected no harder than an exploratory one. We propose SCARL (Sequence Confidence-Aligned Reinforcement Learning), which reshapes, but never re-signs, each advantage with the alignment gap between the response's group-relative confidence and the outcome its advantage certifies, normalized into shares of each class's advantage budget. Misaligned samples absorb the force, responses with the same advantage are driven toward equal confidence, and repairing the misalignment doubles as a remedy for the entropy collapse that erodes exploration. SCARL provably reduces to its base estimator when a class's confidences agree, adds no extra passes and one hyperparameter. On Qwen3-4B-Base it lifts average Pass@32 by 2.4 points over DAPO and is best among nine compute-matched methods on both coverage and per-draw accuracy; applied unchanged across four estimators and four backbones, it improves 14 of 16 settings while preserving policy entropy and widening the confidence-correctness separation. Our code is available at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过用规则验证器对抽样反应评分，并强化通过的回答，从而提升语言模型推理能力。当分类器的训练将预测概率与标签耦合时，RLVR策略对响应的信心与其奖励没有任何联系：对齐训练会推动具有相同优势的反应趋向于同样的置信度，但优势却盲于信心。在这种序列-置信度错位下，策略勉强找到的成功获得的更新与其几乎确定性产生的成功是相同的，且信心错误的纠正不比探索性错误更难。我们提出SCARL（序列置信度对齐强化学习），它重塑但永不重新签名每个优势，并利用反应的群体相对置信度与其优势所证明的结果之间的对齐差距，归一化为每个类别优势预算的份额。错位样本吸收了力量，具有相同优势的响应被引导至置信度相等，修复错位同时也是对熵崩溃的缓解，从而侵蚀探索。当类别置信度一致时，SCARL可证明归约为其基础估计量，无需额外经过和一个超参数。在Qwen3-4B-Base上，SCARL比DAPO提升平均Pass@32 2.4个百分点，在九种计算匹配方法中覆盖率和每次抽取准确率均为最佳;在四个估计量和四个骨干链上保持不变，它在保持策略熵并扩大置信度与正确性之间的前提下，改进了16个设置中的14个。我们的代码可在此 https 网址获取。

Uncertainty-aware Feedback Loop Diagnosis: integrating reinforcement learning and conformal prediction for reliable wafer defect classification

不确定性感知反馈环诊断：整合强化学习与共形预测，实现可靠的晶圆缺陷分类

• Authors: Aleksandr Dekhovich, Carlas Smith, Oleg Soloviev
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.40421
• Pdf link: https://arxiv.org/pdf/2610.40421
• Abstract Semiconductor manufacturing is a complex, multi-step process that requires both high precision and low operating costs. Therefore, detecting defects in a high-volume setting is an important component of wafer map analysis for reliable production. However, most state-of-the-art deep learning approaches require a full scan of the surface, which hinders the development of high-throughput inspection systems. To address this problem, we propose Uncertainty-aware Feedback Loop Diagnosis (UFLD), a novel algorithm that employs reinforcement learning for adaptive measurement acquisition and provides a prediction uncertainty level after each step, stopping the process once sufficient information is obtained. Unlike existing methods that rely on ensemble models or auxiliary confidence estimation deep learning models, UFLD utilizes a more parameter-efficient conformal prediction framework to determine the optimal stopping time. The proposed UFLD reduces acquisition time by about $40\%$ and achieves $98.4\%$ inspection accuracy, without requiring an additional learned confidence-estimation network. We validated our approach on the MixedWM38 dataset, demonstrating its advantages over existing adaptive acquisition procedures and standard convolutional neural networks.
• 中文摘要 半导体制造是一个复杂、多步骤的过程，需要高精度和低运营成本。因此，在大批量环境中检测缺陷是晶圆图谱分析中实现可靠生产的重要组成部分。然而，大多数最先进的深度学习方法要求对表面进行全面扫描，这阻碍了高通量检测系统的开发。为解决这一问题，我们提出了一种新颖算法，利用强化学习实现自适应测量获取，并在每一步后提供预测不确定性水平，一旦获得足够信息便停止过程。与依赖集合模型或辅助置信估计深度学习模型的现有方法不同，UFLD采用更高效的参数效率共形预测框架来确定最佳停止时间。拟议的UFLD将采集时间减少约40美元，并实现98.4%%的检测准确率，无需额外的学习置信估计网络。我们在MixedWM38数据集上验证了我们的方法，展示了其相较于现有自适应采集程序和标准卷积神经网络的优势。

Reinforcement learning for control of partial differential equations

用于偏微分方程控制的强化学习

• Authors: Nicolò Botteghi, Matteo Tomasetto, Andrea Manzoni
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.40449
• Pdf link: https://arxiv.org/pdf/2610.40449
• Abstract Controlling dynamical systems governed by partial differential equations (PDEs) remains a challenging task in scientific computing, owing to high-dimensional state spaces, distributed actuation, nonlinear dynamics, and stringent real-time constraints. Reinforcement learning (RL) has emerged as a promising paradigm for synthesizing feedback controllers that adapt to varying operating conditions and can be deployed efficiently in real-time. This review provides a unified perspective on RL and PDE-constrained optimization, and surveys model-free, model-based, and multi-agent RL approaches to PDE control, with particular emphasis on sample efficiency, generalization across parametric variations, and scalability to high-dimensional problems. Two demonstrative examples, the stabilization of a chaotic system and a flow control problem, illustrate the performance and pitfalls of RL agents in practice. Finally, we discuss evaluation and benchmarking practices, current limitations, and open research directions toward reliable, data-efficient, and scalable RL for scientific computing.
• 中文摘要 由于高维状态空间、分布式驱动、非线性动力学和严格的实时约束，控制偏微分方程（PDE）驱动的动力系统仍然是科学计算中的一项挑战。强化学习（RL）已成为合成适应不同操作条件且可高效实时部署反馈控制器的有前景范式。本综述提供了关于强化学习和偏微分方程约束优化的统一视角，综述了无模型、基于模型和多代理的强化学习方法中的偏微分方程控制方法，特别强调样本效率、参数变异泛化及高维问题的可扩展性。两个示范性例子——混沌系统的稳定化和流量控制问题——展示了强化学习代理在实际中的性能和陷阱。最后，我们讨论了评估和基准测试实践、当前局限性以及面向科学计算中可靠、数据高效和可扩展的强化学习的开放研究方向。

Training language models in simulation for psychiatric intake

精神病学接收模拟中的语言模型训练

• Authors: Samuel Schmidgall, Tim Strother, Alex Bijamov, Min Woo Sun, Anil Palepu, Akshay Goel, Ulysse Marteau-Ferey, Théo Moutakanni, Yubin Kim, Joseph Breda, Christina Chen, Justin Chen, Paul Jhun, Andrew Marshall, Sarah Montgomery-Taylor, Eun Kyoung Hong, Shashank Ravi, Kameron C. Black, David J. H. Wu, Austin Schoeffler, John Lyon Havlik, Elizabeth C. Stade, Jared Moore, Johannes C. Eichstaedt, Logan Pierce, Jeff Jopling, Johannes Thrul, Thomas H. McCoy, Neguine Rezaii, Mark S. Chen, Ellen E. Fitzsimmons-Craft, Shirley Wang, Desmond C. Ong, Katharina Schultebraucks, Torrey Creed, Bhavna Daryani, Aleksandra Faust, S. Sara Mahdavi, Caroline Figueroa, Eric Lin, Kate H. Bentley, Ernest Lim, Zhihan Jiang, Xuhai "Orson" Xu, Daniel McDuff, John Torous, Yun Liu, Po-Hsuan Cameron Chen, Mike Schaekermann, Joelle Barral, Alvin Rajkomar, Tao Tu, Lin Yang, Isaac Galatzer-Levy, David Racz, Valentin Liévin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.40476
• Pdf link: https://arxiv.org/pdf/2610.40476
• Abstract In psychiatric practice, the clinical intake interview remains a primary diagnostic instrument, requiring the clinician to establish rapport while disentangling complex, overlapping presentations. Here, we show that multi-turn reinforcement learning (RL) in simulation environments improves language model performance on structured text-based psychiatric intake, spanning history-taking, differential diagnosis, and initial management planning. In a randomized, double-blind crossover study ($N = 60$ matched scenario pairs, 120 encounters) comparing the RL-trained agent against its identically prompted base model, 32 board-certified psychiatrists role-played standardized patients while a separate panel of 15 independent, board-certified psychiatrists scored the de-identified transcripts. The trained agent outperformed the base model in 76.7% of matched scenario pairs ($p < 0.0001$), with gains on an adapted clinical-examination rubric ($d_z = 0.77$, $p < 0.0001$), patient-centered communication (82.0% vs. 65.2%, $p < 0.001$), and differential diagnosis appropriateness (68.3% vs. 33.3%, $p < 0.001$). Across 127 psychiatrist-validated simulated intake scenarios, the trained agent nearly doubled its base model's clinical rubric score (0.706 vs. 0.379; composite reward 0.564 vs. 0.294, $p < 0.0001$). To systematically probe conversational safety, we established two Delphi-consensus taxonomies, of clinical safety risks and relational competencies, from which 256 challenging adversarial presentations were derived; we find that training in simulation environments more than halved under-triage of acute safety risks (12.9% vs. 30.5%, $p < 0.0001$). Simulation environments thus provide a controlled testbed for training and stress-testing psychiatric conversational agents, with gains that transferred to psychiatrist-enacted encounters, though residual gaps require safeguards and clinician oversight.
• 中文摘要 在精神科实践中，临床入诊访谈仍是主要的诊断工具，要求临床医生在理清复杂重叠的表现时建立关系。本研究显示，模拟环境中的多回合强化学习（RL）提升了语言模型在结构化文本式精神病学接收中的表现，涵盖病史采集、鉴别诊断和初期管理规划。在一项随机、双盲交叉研究（$N = 60$匹配情景对，120次会诊）中，32名认证精神科医生扮演了标准化患者，另有15名独立认证精神科医生对去标识化的文字记录进行了评分。训练有素的特工在匹配情景对中表现优于基础模型（$p < 0.0001$），在调整后的临床检查评分标准（$d_z = 0.77$，$p < 0.0001$）、以患者为中心的沟通（82.0% 对 65.2%，$p < 0.001$）和鉴别诊断适宜性（68.3% 对 33.3%，$p < 0.001$）方面均有提升。在127个精神科医生验证的模拟入院情景中，训练过的特工几乎将基础模型的临床评分翻倍（0.706对0.379;复合奖励0.564对比0.294，$p <0.0001$）。为系统性探究会话安全性，我们建立了两种Delphi共识分类法，分别是临床安全风险和关系能力，从中推导出256条具有挑战性的对抗性表现;我们发现，模拟环境中的培训将急性安全风险的分诊不足减少了一半以上（12.9%对30.5%，$p <0.0001美元）。模拟环境因此为精神科会话代理的训练和压力测试提供了受控测试平台，其益处转移至精神科医生主动的遭遇，尽管残留的差距需要保障措施和临床监督。

Training Foundation Models with a Single Algorithm

用单一算法训练基础模型

• Authors: Yuda Song, Alvin Zhang, Fahim Tajwar, Daman Arora, Jeff Schneider, Daniel Khashabi, Ruslan Salakhutdinov, J. Andrew Bagnell, Aarti Singh, Andrea Zanette
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.40564
• Pdf link: https://arxiv.org/pdf/2610.40564
• Abstract Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm, unified training. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy-compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.
• 中文摘要 基础模型通常按阶段训练：预训练、中期训练、监督微调（SFT）和强化学习（RL），数据和学习算法均有变化。我们表明这种算法分离并非根本性。通过推导合适的强化学习目标作为模型生成目标分布的最大似然拟合，我们将所有这些阶段置于单一基于似然的训练算法统一训练之下。这一观点将基础模型训练生命周期转变为选择训练对象、训练比例和时间的问题，传统分阶段训练是一个特例。我们将这一框架实例化为统一训练系统，将静态数据和强化学习生成混合在单一训练器中。在受控和现实实验中，统一训练提升了准确性与计算权衡和任务覆盖率，同时保留了更广泛的能力，并为持续学习提供了更强的起点。这项工作为基础模型训练的统一科学奠定了基础，连接数学原理、培训基础设施和模型能力的发展。

PIVOT: Progressive Intervention via Online Teacher Correction

PIVOT：通过在线教师纠正进行渐进式干预

• Authors: Zhihao Yang, Boyuan Zheng, Puhui Tan, Yukun Chen, Kuan Zhang, Longze Chen, Jiaming Li, Renke Shan, Lu Wang, Xianpei Han
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.40576
• Pdf link: https://arxiv.org/pdf/2610.40576
• Abstract Reinforcement learning from verifiable rewards (RLVR) is limited by sparse outcome-level feedback on hard reasoning tasks: when all rollouts in a group fail, rewards provide no learning signal. Stronger teachers can provide additional guidance, but this raises a central question: how can such signals be incorporated into RLVR to support successful rollouts while remaining learnable for the student? We distinguish two complementary mechanisms: trajectory-level intervention, which improves access to reward-bearing trajectories, and token-level supervision, which provides dense targets to guide student learning. Our controlled experiments show that either mechanism alone can remain insufficient, motivating their combination. We therefore introduce PIVOT, which inserts teacher-generated corrections into student rollouts and applies token-level KL supervision to these segments alongside the RLVR objective. Beyond their combination, we systematically analyze where to intervene and how to supervise, yielding a recipe for effective guidance. Across hard logic games and mathematical reasoning benchmarks, PIVOT consistently outperforms existing methods, nearly doubling mean avg@8 on logic games from 19.5% for the best-performing baseline to 35.6%.
• 中文摘要 可验证奖励强化学习（RLVR）受限于难推理任务中结果层级的稀疏反馈：当小组中所有推广失败时，奖励将不提供学习信号。更优秀的教师可以提供额外指导，但这引出了一个核心问题：如何将此类信号纳入RLVR，以支持成功的推广，同时保持学生可学习性？我们区分了两种互补机制：轨迹级干预，改善对奖励轨迹的访问;代币级监督，提供密集目标指导学生学习。我们的对照实验表明，单靠这两种机制都可能不足，从而激励它们的组合。因此，我们引入PIVOT，将教师生成的纠正插入学生的学习，并在RLVR目标的同时对这些片段应用代币级的知识点学习监督。除了它们的结合外，我们还系统分析了干预地点和监督方式，得出了有效指导的方案。在硬逻辑游戏和数学推理基准测试中，PIVOT持续优于现有方法，逻辑游戏的平均分avg@8几乎翻倍，从最佳基线的19.5%提升至35.6%。

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

VeriCUA：作为计算机代理探索的验证

• Authors: Qiushi Sun, Da Yu, Alexander Knop, Amer Sinha, Lingpeng Kong, Ben Kao, Ravi Kumar, Pritish Kamath
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.40578
• Pdf link: https://arxiv.org/pdf/2610.40578
• Abstract Building and scaling computer-using agents (CUAs) across desktop, mobile, and web environments requires reliable task-completion verification for data curation, evaluation, and reinforcement learning. Typical automated approaches prompt vision-language models (VLMs) to judge trajectories or generate programs to inspect application state. However, both rely on partial views of complex digital environments. VLM judges are restricted to supplied observations, while generated programs depend on accessible state. These views can omit decisive evidence or leave completion judgments unsupported. Tailoring separate verifiers to different platforms and access settings adds further engineering cost. We present VeriCUA, a platform-agnostic framework for verification through agentic exploration. It treats verification as dynamic evidence gathering and assessment within a shared procedure for online and offline settings across platforms and observation modalities. After inspecting the available context to construct a task rubric, the verifier selects GUI or code checks for unresolved requirements and combines their findings under explicit evidence rules. Experiments show improved agreement with reference labels over the compared baselines across these settings, with the largest offline gains on hard cases and stronger rejection of false completion claims online. Further analyses connect these gains to complementary evidence and test-time exploration.
• 中文摘要 构建和扩展跨桌面、移动和网页环境的计算机代理（CUA）需要可靠的任务完成验证，用于数据整理、评估和强化学习。典型的自动化方法会提示视觉语言模型（VLM）判断轨迹或生成检查应用状态的程序。然而，这两者都依赖于对复杂数字环境的部分视图。VLM评判仅限于提供的观察数据，而生成的程序依赖于可访问状态。这些视图可能遗漏决定性证据或使完成判断无法支持。针对不同平台和访问设置定制独立验证器增加了更多工程成本。我们介绍VeriCUA，一个平台无关的智能体探索验证框架。它将验证视为在线和线下平台和观察模式共享流程中的动态证据收集和评估。在检查可用上下文构建任务评分标准后，验证者选择未解决需求的图形界面或代码检查，并在显式证据规则下将结果合并。实验显示，在这些设置下的对比基线中，参考标签的一致性有所提升，难题案例的离线收益最大，在线错误完成声明的拒绝更为强烈。进一步分析将这些成果与互补证据和测试时间探索联系起来。

ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender

ShallowDefend：利用浅层安全对齐为多转弯防御器打造

• Authors: Qiyao Ma, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Chengquan Guo, Jialin Song, Carolyn Buractaon, Zhe Zhao, Junshan Zhang, Sandeep Atluri, Christopher White, Jianfeng Gao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.40627
• Pdf link: https://arxiv.org/pdf/2610.40627
• Abstract Safety alignment has enabled large language models to reliably refuse harmful requests directly, yet the same request distributed across a multi-turn conversation still succeeds far too often. This brittleness has a mechanistic root: Alignment is shallow, with safety behavior concentrated in a model's first few output tokens, and existing defenses inherit the weakness, supervising the surface form of the response without ever giving the model an explicit safety decision to learn. We take the opposite view of shallowness: if a short span determines the trajectory of an entire response, that span is the highest-leverage place to intervene, and the defender can occupy it first. We propose Assess $\to$ Decide $\to$ Respond, a framework in which the defender assesses the user's intent, commits to an explicit, parseable safety decision, and only then generates a reply conditioned on it, with the commitment trained by reinforcement learning at the payload turn. Because the decision is a token span, the reward can be a rule: a self-verified reward checks the committed label against the conversation's known nature, requiring no judge and no annotation, while the same format also supports an LLM-as-judge reward over a graded decision space with safe redirection. Across two model families and fourteen benchmarks spanning safety, utility, and helpfulness, our defenders achieve the strongest multi-turn safety in a controlled comparison, maintaining their lead in safety while preserving utility at a modest cost to helpfulness. A per-token KL analysis and interventional probes confirm why this works: training concentrates its effect on the brief deliberation that sets a response's trajectory, and the decision committed there is a causal control point. The very shallowness attackers exploit becomes the mechanism of defense.
• 中文摘要 安全对齐使大型语言模型能够可靠地直接拒绝有害请求，但同一请求在多回合对话中仍然成功率过高。这种脆弱性有机械的根源：对齐较浅，安全行为集中在模型的前几个输出令牌中，现有防御继承了弱点，监督响应的表面形式，却从未给模型明确的安全决策可学习。我们持相反的浅表观点：如果短跨度决定了整个响应的轨迹，那么该跨度是干预的杠杆最高处，防御者可以优先占据该区域。我们提出“评估$\to$ 决定$\to$ 响应”框架，防御者评估用户意图，承诺明确且可解析的安全决策，然后基于该意图生成回复，承诺由有效载荷回合的强化学习训练。由于决策是代币跨度，奖励可以是规则：自我验证的奖励会将提交标签与对话已知性质进行核对，无需裁判和注释;同一格式也支持在分级决策空间下以LLM为评判的奖励，且安全重定向。在两个模型家族和涵盖安全性、效用性和帮助性的十四个基准中，我们的辩护方在受控比较中实现了最强的多回合安全，保持安全领先优势，同时保持效用性，且对帮助性的代价适中。每标记KL分析和介入探针证实了其有效性：训练集中于设定响应轨迹的简短审议，而在此所做决策是因果控制点。攻击者利用的浅薄性成为防御机制。

Keyword: diffusion policy

CuraEval: Evaluating Demonstration Curation on Mixed-Skill Robot Data

CuraEval：评估混合技能机器人数据的演示策展

• Authors: Jiuyi Xu, Meida Chen, Yang Ye, Yangming Shi
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.38221
• Pdf link: https://arxiv.org/pdf/2610.38221
• Abstract Robot imitation learning uses demonstrations from operators of varying skill, but selecting training data remains challenging. Existing methods score demonstrations using smoothness, diversity, or estimated influence on policy performance, but differences in selection and evaluation protocols make it difficult to determine whether they outperform simple baselines. We introduce \textbf{CuraEval}, a framework combining mathematical analysis of demonstration scores with controlled comparisons against fixed size-matched random subsets and full-data training on mixed-skill robomimic data using Diffusion Policy and BC-RNN. We show that the evaluated PSD score depends only on episode length and squared position magnitudes, and produces nearly the same ranking as episode length on \textit{square} ($\rho=0.999$). With Diffusion Policy, selecting the shortest 50\% of demonstrations improves \textit{square} success from 70.7\% under random selection to 76.5\%. Compared with random selection, our adapted geometric-entropy LOO selector reduces success at 25\% retention from 94.7\% to 74.0\% on \textit{can} and achieves 41.7\% on \textit{square}, below the mean success rates of all three independently sampled random subsets (59.8--62.5\%). On the same 3 \textit{square} Diffusion Policy training seeds, length-based selection at 75\% retention improves success over full-data training from 76.7\% to 86.7\% under the original protocol, but lowers it from 80.5\% to 79.5\% after checkpoint reporting, rollout count, and evaluation seed change jointly. No evaluated curated condition significantly outperforms full-data training after Holm correction, although confidence intervals still permit useful gains. Code is available at this https URL.
• 中文摘要 机器人模仿学习依赖不同技能操作员的演示，但选择训练数据仍具挑战性。现有方法通过平滑度、多样性或估计对政策表现的影响来评分演示，但选择和评估协议的差异使得判断其是否优于简单基线变得困难。我们引入了 \textbf{CuraEval}，这是一个结合了演示分数的数学分析、对固定大小匹配随机子集的受控比较以及使用扩散政策和 BC-RNN 对混合技能机器人模拟数据进行全数据训练的框架。我们表明，评估后的 PSD 分数仅依赖于集数长度和位置大小的平方，且其排名几乎与 \textit{square} 上的集数长度相当（$\rho=0.999$）。使用扩散策略，选择最短50%的演示将_textit{square}成功率从随机选择下的70.7%提升至76.5%。与随机选择相比，我们改良的几何熵LOO选择器在\textit{can}上25%保留率从94.7%降至74.0%，在\textit{square}上实现41.7%，低于三个独立抽样随机子集的平均成功率（59.8-62.5%）。在同一3个\textit{square}训练种子中，基于长度的选择保留率75%使全数据训练成功率从76.7%提升至86.7%，但在检查点报告、推广计数和评估种子合并变更后，保留率从80.5%降至79.5%。没有任何经过评估的策划条件在霍尔姆校正后显著优于全数据训练，尽管置信区间仍允许有益的提升。代码可在此 https URL 获取。