Arxiv Papers of Today

生成时间: 2026-07-16 18:10:38 (UTC+8); Arxiv 发布时间: 2026-07-16 20:00 EDT (2026-07-17 08:00 UTC+8)

今天共有 35 篇相关文章

Keyword: reinforcement learning

Autonomous UAV Route Planning for Coverage Maximization in Environmental Monitoring: A Systematic Literature Review

自主无人机航线规划以实现环境监测覆盖最大化：系统文献综述

Authors: Sebastian Jouannet-Contreras, Carola Figueroa-Flores
Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13054
Pdf link: https://arxiv.org/pdf/2607.13054
Abstract Environmental monitoring with unmanned aerial vehicles (UAVs) requires route planning methods that maximize covered area while handling energy limits, operational constraints, and geometric complexity. This paper reports the protocol and preliminary results of an ongoing systematic literature review (SLR) on autonomous UAV route planning for coverage-oriented environmental monitoring. The review follows the PRISMA 2020 framework and searches Scopus and Web of Science for studies published between 2015 and 2026. The protocol focuses on path planning, coverage path planning, and informative path planning, with emphasis on algorithmic families, coverage and energy metrics, obstacle handling, geometric environment representations, and environmental constraints. At the current stage, 562 records have been identified, 161 duplicates have been removed, and 401 unique records have been screened by title, abstract, and keywords. From these, 247 studies were retained for full-text eligibility assessment (235 eligible and 12 borderline records to be resolved during full-text review). A preliminary analysis of the retained studies suggests strong concentration on coverage-oriented formulations, multi-UAV coordination, and energy-aware optimization, while fewer studies explicitly address weather, uncertainty, or obstacle-rich environments. Most retained studies rely on simulation-based validation, highlighting a potential simulation-to-reality gap, and recent publications show increasing interest in reinforcement learning, hybrid optimization, and geometry-aware planning. These early findings indicate an active but fragmented research landscape and support the need for a structured synthesis to identify mature techniques and unresolved gaps for realistic environmental monitoring missions.
中文摘要 无人机（UAV）的环境监测需要在应对能量限制、操作限制和几何复杂性的情况下，最大化覆盖面积的路线规划方法。本文报告了一项关于自主无人机航线规划用于覆盖导向环境监测的系统性文献综述（SLR）的方案和初步结果。本综述遵循PRISMA 2020框架，并在Scopus和Web of Science中检索2015年至2026年间发表的研究。该协议侧重于路径规划、覆盖路径规划和信息性路径规划，重点关注算法族、覆盖度和能量指标、障碍物处理、几何环境表示以及环境约束。目前已识别562条记录，删除161条重复记录，并按标题、摘要和关键词筛选了401条独立记录。其中保留247项研究进行全文资格评估（其中235项符合条件，12项边缘记录待全文审查解决）。对保留研究的初步分析显示，重点关注覆盖导向的表述、多无人机协调和能量感知优化，而明确关注天气、不确定性或障碍丰富环境的研究较少。大多数保留研究依赖基于仿真的验证，凸显了模拟与现实之间潜在的差距，近期出版物显示对强化学习、混合优化和几何感知规划的兴趣日益增长。这些早期发现表明研究环境活跃但分散，支持结构化综合的必要性，以识别成熟技术和未解决的空白，以实现现实的环境监测任务。

AI in Cyberpsychology: A systematic literature review of Cybersecurity enhancement by using AI for analyzing psychology of Victims, Attackers, and Defenders

网络心理学中的人工智能：利用人工智能分析受害者、攻击者和防御者心理的网络安全增强的系统文献综述

Authors: Georg Thamer Francis, Malek Malkawi, Sevim Eyüpoğlu, Reda Alhajj, Selim Akyokuş
Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13123
Pdf link: https://arxiv.org/pdf/2607.13123
Abstract Cybersecurity is the practice of protecting systems, networks, and data from digital attacks. Cyberpsychology (CPSY) is defined as the use of psychology to enhance cybersecurity applications. Since the early 2010s, the evolution of Artificial Intelligence (AI) has increasingly integrated with CPSY, leveraging advanced data analysis to decode the distinct personality traits and behavioral patterns of victims, attackers, and defenders. In this systematic literature review (SLR), we carefully analyze 34 collected research studies of AI usage in cyberpsychology (AI-CPSY) using the preferred reporting items for systematic reviews and meta-analyses (PRISMA) methodology. The review presents a comprehensive taxonomy of the cyber-security applications, the AI methodologies used, and the psychological concepts employed across the studies . We sort the research studies into four cybersecurity applications: Anomaly Detection (AD), Vulnerability Risk Prediction (VRP), Security Awareness Training (SAT), and Authentication/Identity Verification (AIV). Within each application area, studies are further sorted according to the AI method used including machine learning (ML), deep learning (DL), natural language processing (NLP), and reinforcement learning (RL). Furthermore, the review identifies the most commonly utilized psychological concepts, quantify the datasets used in the field, and present their current implementation and deployment status. At last, it detect research gaps, present open challenges, and deduce the trending and most effective and emerging methodologies used across the AI-CPSY landscape.
中文摘要 网络安全是保护系统、网络和数据免受数字攻击的实践。网络心理学（CPSY）被定义为利用心理学来增强网络安全应用的学科。自2010年代初以来，人工智能（AI）的发展越来越多地与CPSY集成，利用先进的数据分析来解码受害者、攻击者和防御者的独特人格特征和行为模式。在这份系统文献综述（SLR）中，我们认真分析了34项关于网络心理学（AI-CPSY）中人工智能使用的研究，采用了系统评价和荟萃分析（PRISMA）首选的报告项目。综述对网络安全应用、所用人工智能方法论及研究中所采用的心理学概念进行了全面的分类。我们将研究分为四个网络安全应用：异常检测（AD）、漏洞风险预测（VRP）、安全意识培训（SAT）和身份验证/身份验证（AIV）。在每个应用领域内，研究会根据所使用的人工智能方法进一步分类，包括机器学习（ML）、深度学习（DL）、自然语言处理（NLP）和强化学习（RL）。此外，综述识别了最常用的心理学概念，量化该领域使用的数据集，并展示了其当前的实施和部署状态。最后，它发现了研究空白，提出了未解的挑战，并推断了人工智能-计算机科学领域中流行、最有效和新兴的方法论。

Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models

通过世界模型学习安全代理行为，从人类偏好和理由中学习

Authors: Ilias Kazantzidis, Timothy J. Norman, Yali Du, Christopher T. Freeman
Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.13172
Pdf link: https://arxiv.org/pdf/2607.13172
Abstract We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.
中文摘要 我们解决了在环境动态未知且无合适奖励函数的情况下，安全训练代理策略并部署良好且安全策略的问题。在安全关键环境中，我们认为传统的强化学习不切实际，转而依赖人类投入资源。我们引入了DROPJ，这是一种以人为本的方法，既安全培训，也用于部署。我们首先从先前的现实轨迹数据集中学习一个世界模型（学习过的模拟器）。随后，人类在这个学习型模拟器中进行游戏，提取出多个有用的模拟轨迹。从这些中，我们抽样成对模拟轨迹段，并从人类那里推导出对这些段的偏好，以及选择这些段的理由（理由）。然后，我们从这些正当偏好中训练出奖励模型，并结合世界模型，直接部署代理，采用模型预测控制。通过进行真实用户实验，我们发现从用户生成有信息量的模拟轨迹相比其他策略显著降低了训练中的计算成本，并且还能提升部署时的性能。在学习模拟器中的训练背景下，我们展示了使用偏好而非其他类型的反馈，显著提升了部署期间的性能。我们还进一步证明，伴随偏好的安全理由可以显著提升安全性，或优先考虑用户在部署期间与之相关的安全措施。

SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancy

SteinGate：通过Stein差异实现尾敏安全强化学习

Authors: Yassine Chemingui, Chenhua Fan, Honghao Wei, Janardhan Rao Doppa
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13175
Pdf link: https://arxiv.org/pdf/2607.13175
Abstract Safe reinforcement learning typically enforces safety by bounding expected cumulative costs, a criterion that often fails to detect rare but catastrophic tail events. To overcome these limitations, this paper introduces SteinGate, a boundary-aware distributional safety certificate that replaces fragile tail fitting with a robust consistency check using Kernelized Stein Discrepancy while accounting for boundary atoms induced by clipped costs. SteinGate evaluates whether observed policy rollout costs remain consistent with a safe reference distribution, providing a non-parametric safety certificate. This certificate is used to dynamically adapt the learning regime: favoring reward-improving policy updates when rollouts remain consistent with the safe reference and switching to recovery behavior when the cost tail deviates. Experiments on continuous-control benchmarks demonstrate that SteinGate significantly reduces both the frequency and severity of constraint violations during training while maintaining competitive returns relative to state-of-the-art baselines.
中文摘要 安全强化学习通常通过限制预期累计成本来强制执行安全，而这一标准常常无法检测到罕见但灾难性的尾部事件。为克服这些限制，本文引入了SteinGate，一种边界感知型分布安全证书，利用核化斯坦差异（Kerine Stein Discrepancy）替代脆弱的尾拟合，同时考虑由截波成本引起的边界原子。SteinGate评估观察到的政策推广成本是否与安全参考分布一致，提供非参数安全证书。该证书用于动态调整学习机制：当推广与安全参考保持一致时，优先更新奖励改进的政策;当成本尾部偏差时切换到恢复行为。连续对照基准测试的实验表明，SteinGate在保持相较于最先进基线的竞争性回报的同时，显著降低了训练期间约束违规的频率和严重程度。

Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners

解构演员-批评者：为从业者设计组成部分进行的大规模实证研究

Authors: Haseeb Shah, Lingwei Zhu, Adam White, Martha White
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13274
Pdf link: https://arxiv.org/pdf/2607.13274
Abstract Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.
中文摘要 强化学习越来越多地被考虑用于控制现实世界的系统，从聚变等离子体和自动驾驶车辆到药物发现和饮用水处理，这些领域可靠性至关重要且调优预算有限。演员-批评算法共享一组设计决策，如策略如何更新、如何表示对动作的分布、如何估计梯度，以及相对于价值估计器的更新频率。利用源自真实水处理厂的对照任务，我们分析了33,000多个实验，以确定这些成分如何影响不同运行的变异性以及对超参数的敏感性。常见默认配置，如带有路径梯度估计的高斯作用分布，是最不可靠的配置之一，而带有自适应更新计划的有界分布则在多种环境中依然稳健。这些发现为科学和工程领域的从业者提供了实证指导，帮助他们理解并在将actor-critic方法适应新现实控制环境时做出组件级决策。

Meta-Learning Preferences for Multilingual LLM Alignment

多语言LLM对齐的元学习偏好

Authors: Jiaying Lin, Seongho Son, Nam Phuong Tran, Long Tran-thanh, Ilija Bogunovic, Debmalya Mandal
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.13315
Pdf link: https://arxiv.org/pdf/2607.13315
Abstract Unequal availability of human preference data across languages poses a significant challenge for aligning large language models in multilingual settings. To address the lack of sufficient data in low-resource language alignment, we propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization. By leveraging preference data from other languages, our framework learns a transferable initialization that enables effective adaptation to a target language with minimal data. We provide theoretical guarantees for both the meta-reward modeling and meta-policy optimization settings, and empirically demonstrate the effectiveness of our approach on multilingual benchmarks. In an extremely low-resource setting with only 100 target-language preference samples, our approach achieves up to $28\%$ win-rate improvements over baseline methods, and consistently outperforms baselines across multiple target languages and model scales. Our approaches retain these advantages across different combinations of meta-training languages and varying linguistic distances from the target languages.
中文摘要 不同语言中人类偏好数据的可用性不均，是多语言环境中大型语言模型协调的重大挑战。为解决低资源语言对齐数据不足的问题，我们提出了一个基于人类反馈和直接偏好优化的强化学习元学习框架。通过利用其他语言的偏好数据，我们的框架学习了可转移的初始化，使得能够以最少的数据有效适应目标语言。我们为元奖励建模和元策略优化设置提供了理论保证，并通过实证证明了我们方法在多语言基准上的有效性。在资源极度匮乏、仅有100个目标语言偏好样本的情况下，我们的方法比基线方法提升了高达28%的胜率，并且在多种目标语言和模型尺度上始终优于基线。我们的方法在不同元训练语言组合和与目标语言的不同语言距离中都保持了这些优势。

Safe Overtaking for Autonomous Racing Using Hierarchical Optimization and Learning-Based Control

利用层级优化和基于学习的控制，为自动驾驶赛车提供安全超车

Authors: Hassan Jardali, Kai Yin, Lantao Liu
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13348
Pdf link: https://arxiv.org/pdf/2607.13348
Abstract Autonomous racing overtaking requires balancing competitive performance with safety under nonlinear vehicle dynamics and real-time constraints. Model Predictive Control (MPC) combined with Control Barrier Functions (CBFs) provides a principled mechanism for certifying forward invariance of a safe set. However, commonly used fixed-decay discrete-time CBF formulations can become overly conservative in interactive racing scenarios, limiting overtaking performance and requiring manual tuning across track conditions. This paper proposes a hierarchical overtaking framework that explicitly separates maneuver-level decision making from safety-certified trajectory control, reducing conservatism while preserving safety. A high-level Mixed-Integer Quadratic Program (MIQP) resolves the combinatorial passing-side selection problem by selecting a feasible overtaking topology, while a nonlinear Frenet-frame MPC enforces vehicle dynamics and safety through embedded discrete-time CBF constraints. This decomposition isolates the combinatorial complexity of maneuver selection from the continuous trajectory optimization. To further mitigate the sensitivity of fixed-decay barrier constraints, a reinforcement learning policy adapts the discrete-time CBF decay parameter online, enabling context-dependent modulation of safety margins without directly controlling vehicle inputs. Simulation and scaled-hardware experiments show that no single fixed decay parameter achieves uniformly strong performance across tracks, whereas the adaptive strategy attains the highest aggregate success rate and consistently strong safety--performance trade-offs without per-track tuning, improving robustness to environment variation while maintaining safety constraint satisfaction in nominal operation.
中文摘要 自动驾驶超车需要在非线性车辆动力学和实时限制下，平衡竞争性能与安全性。模型预测控制（MPC）结合控制障碍函数（CBF）提供了一种有原则的机制来认证安全集的前向不变性。然而，常用的固定衰减离散时间CBF配方在互动赛车场景中可能过于保守，限制超车性能，并需要在赛道条件下手动调校。本文提出了一个层级超车框架，明确将机动级决策与安全认证的轨迹控制分离，降低保守性同时保持安全。高阶混合整数二次规划（MIQP）通过选择可行的超车拓扑来解决组合通过侧选择问题，而非线性Frenet帧MPC则通过嵌入的离散时间CBF约束来强制车辆动力学和安全。这种分解将机动选择的组合复杂性与连续轨迹优化分离出来。为进一步降低固定衰减障碍约束的敏感性，强化学习策略在线调整离散时间CBF衰减参数，实现上下文依赖性安全裕度调制，而无需直接控制车辆输入。仿真和缩放硬件实验表明，没有单一固定衰减参数能在各轨道间实现均匀强效性能，而自适应策略则能获得最高的总成功率和持续强劲的安全——无需逐轨调校，从而提升对环境变化的鲁棒性，同时保持标称运行中的安全约束满足。

Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback

强化学习的训练后计算应该去哪里？模型规模、搜索、学习与反馈

Authors: Patrick Wilhelm, Odej Kao
Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.13389
Pdf link: https://arxiv.org/pdf/2607.13389
Abstract Reinforcement Learning (RL) post-training is increasingly used to adapt foundation models for reasoning, planning, and feedback-driven robot-learning pipelines, but constrained post-training resources are often summarized by a single total FLOP budget. We study the fixed-budget decision problem behind this practice: under the same post-training budget, should one use a larger policy, train a smaller policy longer, generate more rollout search, or spend compute on stronger reward feedback? We introduce a FLOP-accounting framework for GRPO post-training that decomposes compute into rollout/search, policy-update/learning, and reward- or feedback-model evaluation. Across LoRA-adapted Qwen2.5 policies, we find conditional allocation frontiers: the best observed allocation changes with model size, compute budget, reward system, and evaluation target. Same-FLOP model-size comparisons show that model choice and training allocation are coupled because larger policies consume more per-token compute and therefore buy fewer updates or rollouts under the same budget. Reward systems also change the accounting: rule-based rewards spend nearly all non-update compute on policy rollouts, while PRM-style feedback allocates a visible part of the budget to reward-model inference. We present RACE as a diagnostic pilot-grid protocol, not a guarantee of held-out improvement, for identifying allocation regimes before expensive validation runs; our results suggest that RL post-training papers should report total FLOPs together with how compute is divided among model size, search, learning, and feedback.
中文摘要 强化学习（RL）后训练越来越多地被用来调整基础模型以实现推理、规划和反馈驱动的机器人学习流程，但受限的训练后资源通常被总结为单一的总FLOP预算。我们研究了这一做法背后的固定预算决策问题：在同一培训后预算下，应该使用更大的策略、更长时间的培训、生成更多的推广搜索，还是将计算资源投入到更强的奖励反馈上？我们引入了一套用于GRPO培训后的FLOP会计框架，将计算分解为推广/搜索、策略更新/学习以及奖励或反馈模型评估。在LoRA适配的Qwen2.5策略中，我们发现了条件分配前沿：在模型规模、计算预算、奖励系统和评估目标下观察到的最佳分配变化。同FLOP模型规模比较显示，模型选择和训练分配是耦合的，因为较大的策略在每个令牌计算量上占用更多，因此在同一预算下购买的更新或推广数量较少。奖励系统也改变了核算方式：基于规则的奖励几乎将所有非更新计算用于政策推广，而PRM式反馈则将预算中可见的一部分分配给奖励模型推断。我们将RACE介绍为一种诊断性试点网格协议，而非保证持续改进，用于在昂贵验证运行前识别分配机制;我们的结果表明，强化学习训练后的论文应报告总FLOP数，以及计算如何在模型规模、搜索、学习和反馈中分配。

GFlowRL: Scaling Distribution-Matching RL to Large Language Models

GFlowRL：分布匹配强化学习扩展到大型语言模型

Authors: Xiaodong Liu, Michael Xu, Jack W. Stokes, Paul Smolensky, Doug Burger, Jianfeng Gao
Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.13394
Pdf link: https://arxiv.org/pdf/2607.13394
Abstract Generative Flow Networks (GFlowNets) offer a promising alternative to reward-maximizing reinforcement learning (RL) for large reasoning models, encouraging diverse reasoning paths by matching reward distributions rather than collapsing to dominant modes. Recent work shows promise on math and code, but scaling GFlowNet-style RL to modern post-training pipelines remains difficult: as model size, rollout horizon, reward noise, and distributed-systems complexity grow together, a learned prompt-conditional partition function becomes a source of gradient instability and engineering overhead rather than a useful normalizer. Through systematic analysis, we find that the learned partition function, previously treated as essential, can be replaced by an in-batch Monte Carlo estimate computed from the rollout group already required for training. We propose GFlowRL, a streamlined GFlowNet-style RL algorithm that removes the auxiliary partition network entirely while preserving the reward-distribution-matching objective, completed by two stabilizers: importance-sampling correction for rollout/trainer drift and asymmetric flow-gap clipping for outlier residuals. GFlowRL exceeds all counterparts on math, code, and adversarial red-teaming benchmarks, reaching a Codeforces rating of 2048 at the 14B scale (within 25 Elo of o3-mini) and attaining the highest average ASR@1 on AdvBench and HarmBench, outperforming the previous SOTA multi-turn attacker in a regime where FlowRL, a prior GFlowNet-style method, diverges. The same recipe transfers to all evaluated MoE configurations up to 235B parameters, where FlowRL again fails to converge. To our knowledge, GFlowRL is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures. Code will be at: this https URL
中文摘要 生成流网络（GFlowNets）为大型推理模型提供了一种有前景的奖励最大化强化学习（RL）替代方案，通过匹配奖励分布而非崩溃到主导模式，鼓励多样化的推理路径。近期研究在数学和代码方面展现出潜力，但将GFlowNet式强化学习扩展到现代训练后流程仍然困难：随着模型规模、推广范围、奖励噪声和分布式系统复杂度的增加，学习到的提示条件划分函数反而成为梯度不稳定性和工程开销的来源，而非有用的归一化器。通过系统分析，我们发现此前被视为必备的学习分区函数可以被已从训练所需的展开组计算出的批量蒙特卡洛估计值所替代。我们提出了GFlowRL，这是一种简化的GFlowNet风格强化学习算法，完全去除辅助划分网络，同时保持奖励分布匹配目标，并辅以两个稳定器：对rollout/trainer漂移进行重要性抽样校正，以及对异常残差进行非对称流间隙裁剪。GFlowRL在数学、代码和对抗红队基准上均领先所有对应者，在14B尺度上达到2048的Codeforces评分（距离o3-mini不到25个Elo），并在AdvBench和HarmBench中获得最高平均ASR@1，在FlowRL这种之前的GFlowNet风格方法出现分歧的环境中，表现优于之前的SOTA多回合攻击者。同样的原理适用于所有评估的 MoE 配置，最高可达 235B 参数，但 FlowRL 再次未能收敛。据我们所知，GFlowRL 是首个能够稳定扩展到密集和稀疏架构的 GFlowNet 风格 RL 算法。代码将位于：这个 https URL

Ego-Dynamics-Augmented World Model for Autonomous Driving with Zero-Shot Cross-Chassis Adaptation

自我动力增强世界模型，适用于零射点交叉底盘适应的自动驾驶

Authors: Zhidong Wang, Jingsong Liang, Zirui Li, Zhan Chen, Han Yu, Chen Lv
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13410
Pdf link: https://arxiv.org/pdf/2607.13410
Abstract World model (WM)-based reinforcement learning enables sample-efficient end-to-end autonomous driving learning by imagining long-horizon trajectories in latent space. However, most driving WMs operate on bird's-eye-view (BEV) representations that are inherently egocentric: the transition between consecutive frames entangles the ego vehicle's own motion with scene dynamics. As a result, the WM devotes significant capacity to recovering ego-motion from warped observations, at the cost of scene modeling fidelity and imagination accuracy. This work proposes DynaDreamer, a dynamics-augmented Dreamer-style reinforcement learning method to address this problem by augmenting the WM with an explicit ego-dynamics prior. A physics-informed ego-dynamics encoder-decoder extracts the ego-state history into a compact and identifiable context, which modulates a causal Transformer WM to condition both its prior and posterior latents. During imagination, the ego-dynamics predictor propagates this context forward to keep the ego-dynamics prior synchronized with the rollout. An information-theoretic analysis shows that conditioning on this context reduces both the predictive entropy of the observation transition and the prior--posterior Kullback--Leibler divergence, confining the WM's modeling burden to the scene dynamics beyond ego-motion. An additional benefit is zero-shot cross-chassis adaptation: the ego-dynamics context depends on identifiable chassis parameters, so that a vehicle with previously unseen dynamic characteristics can adapt the WM to the new chassis without retraining. Experiments demonstrate that DynaDreamer improves task success rates over the strongest baseline by 28% and 61% in urban and highway driving scenarios, respectively, with the advantage rising to 73% when extrapolating to unseen chassis.
中文摘要 基于世界模型（WM）的强化学习通过想象潜空间中的长视野轨迹，实现高效的端到端自动驾驶学习。然而，大多数驾驶型白人车运行在鸟瞰视角（BEV）表征上，这种表现本质上是自我中心的：连续帧之间的过渡使自我车辆自身的运动与场景动态纠缠在一起。因此，WM投入大量能力从扭曲的观察中恢复自我运动，代价是场景建模的真实性和想象力的准确性。本研究提出了DynaDreamer，一种动态增强型Dreamer式强化学习方法，通过先验显式自我动力学来增强WM来解决这一问题。基于物理的自我动力学编码-解码器将自我状态历史提取到紧凑且可识别的上下文中，调制因果变换器 WM，以调制其前后潜在因素。在想象过程中，自我动力学预测器会将这一情境传递到前方，以保持自我动力学与展开同步。信息论分析表明，基于此上下文的条件化既降低了观测转变的预测熵，也降低了先验——后验库尔巴克-莱布勒散度，使 WM 的建模负担限制在超越自我运动的场景动态中。另一个好处是零发子弹跨底盘适配：自我动力学背景依赖可识别的底盘参数，因此具有此前未见动态特性的车辆可以在无需重新训练的情况下将机车适应新底盘。实验表明，DynaDreamer在城市和高速公路驾驶场景中，较最强基线的任务成功率分别提升了28%和61%，其中在未见底盘上推算时，优势提升至73%。

Joint On-and-Off Policy Learning for Vision-and-Language Navigation

视觉与语言导航的联合开关政策学习

Authors: Qingrong He, Lin Zhao, Kevin Zheng, Liang Lin
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13461
Pdf link: https://arxiv.org/pdf/2607.13461
Abstract Vision-and-Language Navigation (VLN) necessitates an embodied agent to navigate in the physical world by adhering to natural language instructions. Recent advancements in Vision-Language Models (VLM) have propelled the development of VLM-based VLN methods with two predominant paradigms: (1) imitation learning (IL) on expert demonstrations, followed by the Dataset Aggregation (DAgger) algorithm to bolster error recovery capabilities; (2) reinforcement learning (RL) driven by verifiable rewards to enhance reasoning and exploration. A notable gap is the absence of integration between these two distinct paradigms. This paper introduces JOP-VLN, a novel VLN framework that synergistically combines off-policy imitation learning and on-policy exploration within a three-stage training pipeline. Initially, IL is employed on expert demonstrations to acquire basic navigation skills. Subsequently, the DAgger algorithm is utilized to generate heuristic exploration trajectories, which are then used for imitation learning to improve error recovery capabilities. Finally, a joint on-and-off policy learning framework is implemented, featuring high-entropy trajectory sampling to enhance RL training efficiency and an error-correction-prioritized trajectory sorting strategy for effective error correction. Extensive experiments demonstrate the efficacy of JOP-VLN, achieving success rates of 69.9% and 68.0% on the VLN-CE R2R and RxR benchmarks, respectively, setting a new state-of-the-art on R2R. Project page: this https URL.
中文摘要 视觉与语言导航（VLN）要求具身代理通过遵循自然语言指令在物理世界中导航。视觉语言模型（VLM）的最新进展推动了基于VLM的VLN方法的发展，采用两种主要范式：（1）基于专家演示的模仿学习（IL），随后是数据集聚合（DAgger）算法以增强错误恢复能力;（2）由可验证奖励驱动的强化学习（RL），以增强推理和探索能力。一个显著的空白是这两种截然不同的范式之间缺乏整合。本文介绍了JOP-VLN，这是一种新型VLN框架，协同结合了非策略模仿学习与三阶段训练流程中的非策略模拟学习。最初，IL受雇于专家示范，以掌握基本导航技能。随后，DAgger算法被用于生成启发式探索轨迹，进而用于模仿学习以提升错误恢复能力。最后，实施了联合开关策略学习框架，采用高熵轨迹抽样以提高强化学习训练效率，并采用以纠错为优先的轨迹排序策略，实现有效的纠错。大量实验证明了JOP-VLN的有效性，分别在VLN-CE R2R和RxR基准测试中达到69.9%和68.0%，为R2R树立了新的尖端水平。项目页面：这个 https URL。

EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal

EgoHTR：以自我为中心的四维人体地形穿越演示

Authors: Alex Brandes, Haig Conti Georges Sajelian, Manthan Patel, Dominik Hollidt, Chenhao Li, Matthias Heyrman, Oliver Hausdoerfer, Manuel Kaufmann, Xi Wang, Jonas Frey, Angela P. Schoellig, Christian Holz, Marc Pollefeys, Marco Hutter
Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
Arxiv link: https://arxiv.org/abs/2607.13472
Pdf link: https://arxiv.org/pdf/2607.13472
Abstract Deploying humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness. The restricted motion synthesis is a direct consequence of existing dataset pipelines failing to capture human-scene sequences in challenging environments. To bridge this gap between humanoid learning and scene reconstruction, we introduce the Egocentric Human-Terrain Reconstruction (EgoHTR) dataset. We develop and open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup of egocentric wearables and a portable 3D scanner. The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis. Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions. Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build foundational, context-aware robots that reliably traverse uneven terrain.
中文摘要 在无结构地形中部署类人机器人仍是一个悬而未决的问题。虽然经典强化学习在现实世界互动的复杂性中苦苦挣扎，但利用人类先验的更有前景的方法仍限于缺乏情境意识的模型。受限运动合成是现有数据集流程未能在复杂环境中捕捉人文场景序列的直接结果。为了弥合类人生物学习与场景重建之间的差距，我们引入了以自我为中心的人地重建（EgoHTR）数据集。我们开发并开源了一条重建流程，利用多传感器的自我中心穿戴设备和便携式3D扫描仪，捕捉55个场景对齐的4D人体运动序列，分布在多样复杂环境中。最终数据集包含超过15万帧，我们用动作捕捉的真实数据进行评估，展示了最先进的精度，并为人体运动分析与综合树立了严谨的基准。此外，我们利用这些数据训练感知运动策略，演示了在Unitree G1上重建参考运动的硬件部署。我们的流程支持社区驱动的数据集扩展，并对问题进行分析，帮助研究人员构建能够可靠穿越崎岖地形的基础性、具上下文感知的机器人。

Factorized Spectral Representations for Reinforcement Learning

强化学习中的分解谱表示

Authors: Junyi Wu, Dan Li
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.13498
Pdf link: https://arxiv.org/pdf/2607.13498
Abstract Learning a compact model of the world from interaction data is central to sample-efficient deep reinforcement learning. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state-action pairs on one side and next states on the other, and learning a low-rank factorization through self-supervised contrastive objectives. We take this view one step further. The transition kernel is naturally a three-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next-state encoders that together form a single spectral representation. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions. Empirically, FaStR delivers its largest gains on high-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained.
中文摘要 从交互数据中学习一个紧凑的世界模型是样本高效深度强化学习的核心。谱表示方法已成为连续控制中表示学习的领先范式，通过将跃迁核视角为矩阵视角，一侧为状态-作用对，另一侧为下一态，并通过自监督对比目标学习低秩分解。我们将这一观点进一步推演。转移核自然是对状态、动作和下一状态的三模态张量，CP分解给出每个模态一个特征映射。我们提出了FaStR，它通过噪声对比物镜拟合了该分解，产生独立的状态、作用和下一状态编码器，这些编码器共同形成一个单一的光谱表示。分解形式产生的假设类更小，表示学习所需的样本量缩小一个因子，该因子随状态维度和动作维度中较小者而缩小。从经验上看，FaStR在与分解结构动态一致的高维运动任务上获得最大收益，且学习到的状态编码器在执行器位移过程中完整转移，而仅动作编码器被重新训练。

LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning

LAPO：多回合搜索推理中自生成过程奖励的留一回合归因

Authors: Qiang Zhu, Jiajun Wu
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13501
Pdf link: https://arxiv.org/pdf/2607.13501
Abstract Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LAPO, a self-generated process-supervision method based on backward leave-one-turn attribution. For each search turn, LAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder and measures the resulting change in the current policy's mean log-likelihood of the gold answer. This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context. LAPO further applies sign-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores. The method requires no additional reward model, teacher, verifier, or LLM-as-a-Judge. Across seven knowledge-intensive question-answering datasets with local retrieval, LAPO achieves an average exact-match score of 0.326, outperforming the strongest step-reward baseline, IGPO, by 0.053. Ablations show complementary benefits from backward attribution and sign-consistency gating, demonstrating that policy-derived retrospective attribution can provide effective process supervision for multi-turn search agents.
中文摘要 多回合搜索推理的强化学习通常依赖终端结果奖励，无法区分有用的、冗余和有害的中间交互。我们提出了LAPO，这是一种基于反向一轮归属的自生成过程监督方法。对于每个搜索回合，LAPO用固定的[DELETE]占位符替换该回合及其检索观察，并衡量当前策略金答案的平均对数似然变化。该答案似然增益在保留所有后续交互的前提下估计该回合的贡献，使早期证据能够在完整的推理背景下进行评估。LAPO进一步应用符号一致性门控，仅保留方向与其原始归因分数一致的归一化过程优势。该方法无需额外的奖励模型、教师、验证者或作为评判的LLM。在七个具本地检索的知识密集问答数据集中，LAPO的平均精确匹配得分为0.326，比最强的阶梯奖励基线IGPO高出0.053。消融显示出反向归因和符号一致性门控的互补益处，表明基于策略的追溯归因可以为多回合搜索代理提供有效的过程监督。

ThinkBLOX: 3D Indoor Scene Generation with Progressive Reasoning

ThinkBLOX：带有进步推理的3D室内场景生成

Authors: Yuan Xiao, Can Wang, Xiangyu Kong, Jing Liao
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Graphics (cs.GR)
Arxiv link: https://arxiv.org/abs/2607.13539
Pdf link: https://arxiv.org/pdf/2607.13539
Abstract While traditional graphics methods often synthesize 3D indoor scenes autoregressively or hierarchically, recent vision-language model (VLM)-based generators predominantly adopt a one-shot paradigm where the full layout is planned at once. This one-shot approach often requires global re-optimization or complete reconstruction during interactive editing (e.g., inserting or moving objects) and can lead to physically or semantically poorly organized arrangements. To address these challenges, we propose ThinkBLOX, a VLM-based progressive reasoning framework that iteratively designs and refines 3D scenes. ThinkBLOX treats layout generation as a state-conditioned, step-by-step reasoningand-action process. To power this, we construct the ThinkBLOX-Data-200K dataset, containing 224,757 procedural placement pairs annotated with multi-view scene context, explicit Chain-of-Thought (CoT) rationales, and structured JSON layouts. Through supervised fine-tuning (SFT) on this dataset, the VLM learns to bridge the reasoning-action gap under incremental updates. Furthermore, recognizing that scene synthesis is inherently a multisolution task where SFT suffers from reward conflict, we introduce Tier-Decoupled GDPO. This reinforcement learning scheme organizes heterogeneous rewards into distinct tiers, stabilizing policy optimization across physical validity, semantic plausibility, and reasoning-action consistency. Extensive experiments show that ThinkBLOX significantly outperforms recent one-shot and iterative baselines in physical plausibility, semantic alignment, and interactive editability. Additionally, we show that it supports diverse applications, including both global and local generation and rearrangement of 3D scenes.
中文摘要 传统图形方法通常以自回归或层次方式合成3D室内场景，而近期基于视觉语言模型（VLM）的生成器主要采用一次性设计范式，即一次性规划完整布局。这种一次性方法通常需要在交互式编辑（例如插入或移动对象）时进行全局重新优化或完全重建，可能导致物理或语义上的排列不够清晰。为应对这些挑战，我们提出了ThinkBOX，一种基于VLM的渐进推理框架，能够迭代设计和完善三维场景。ThinkBLOX将布局生成视为一个状态条件化的、逐步推理和操作的过程。为此，我们构建了ThinkBLOX-Data-200K数据集，包含224,757对程序放置对，注释多视图场景上下文、明确的思维链（Chain-of-Thought，CoT）理据和结构化JSON布局。通过对该数据集进行监督微调（SFT），VLM学会在增量更新下弥合推理与行动之间的差距。此外，鉴于场景合成本质上是多解决方案任务，SFT存在奖励冲突，我们引入了分层解耦GDPO。该强化学习方案将异质奖励组织为不同层级，稳定策略优化，涵盖物理效度、语义合理性和推理-行动一致性。大量实验表明，ThinkBLOX在物理可信度、语义对齐和交互式编辑方面显著优于近期的单次和迭代基线。此外，我们展示了它支持多种应用，包括全球和局部的3D场景生成与重排。

Flow-aware Optimal Navigation in Unsteady Flows through Reinforcement Learning

通过强化学习实现非稳态流中的流动感知最优导航

Authors: Andrea Maria Braghin, Nicolò Botteghi, Matteo Tomasetto, Andrea Manzoni, Gabriele Cazzulani
Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Systems and Control (eess.SY)
Arxiv link: https://arxiv.org/abs/2607.13553
Pdf link: https://arxiv.org/pdf/2607.13553
Abstract Autonomous robotic navigation in nonstationary time-varying fluid flows remains a fundamental challenge due to partial observability and the unpredictability of realistic environments. While classical optimal control frameworks employed in robotics require unrealistic a-priori global flow knowledge, biological systems are able to navigate successfully by exploiting localized sensory cues. In this work we present a reinforcement learning approach using the TD3 algorithm to train autonomous agents to reach arbitrary targets within a parametric, chaotic double-gyre flow. To investigate optimal sensory mechanisms, we evaluate five bio-inspired observation strategies based on relative position, local velocity or local vorticity measures, and short-term memory variants. Additionally, we analyze the impact of providing agents with explicit global flow parameters. Numerical results demonstrate that an agent that is able to sense and remember a set number of flow velocity measures achieves the highest performance. The experiments reveal a trade-off in sensor utility: velocity-aware agents optimize energy efficiency, whereas vorticity sensors provide superior structural mapping and achieve better target proximity. Incorporating explicit global flow parameters is shown to decrease navigation performance. This behavior suggests that reinforcement learning-based autonomous systems develop more robust and general policies when restricted to implicit flow representations. The presented results offer insights for improving the transition of bio-inspired robotic navigation from simulation to real-world environments.
中文摘要 由于部分可观测性和现实环境的不可预测性，自主机器人导航在非静止的时变流体流中依然是一个根本性的挑战。虽然机器人学中使用的经典最优控制框架需要不切实际的全局流动知识，但生物系统能够通过利用局部感官线索成功导航。本研究提出了一种利用TD3算法的强化学习方法，训练自主智能体在参数化、混沌的双环流中达到任意目标。为了探究最佳感官机制，我们基于相对位置、局部速度或局部涡度测量以及短期记忆变体，评估了五种仿生观察策略。此外，我们还分析了为代理提供明确的全局流量参数的影响。数值结果表明，能够感知并记住一定数量的流速测量的代理能获得最高性能。实验揭示了传感器效用上的权衡：速度感知剂优化能量效率，而涡度传感器则提供更优越的结构映射并实现更佳的目标接近。明确的全局流量参数被证明会降低导航性能。这种行为表明，基于强化学习的自主系统在限制于隐式流表示时，会制定更稳健、更通用的策略。所呈现的结果为提升仿生机器人导航从模拟向现实环境的转变提供了见解。

Multi-Agent Collaborative Reasoning with Tool-Augmented Evidence for Urban Region Profiling

多智能体协作推理与工具增强证据用于城市区域画像

Authors: Xixuan Hao, Yutian Jiang, Jiabo Liu, Yihang Yang, Guangyin Jin, Song Gao, Yuxuan Liang
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13558
Pdf link: https://arxiv.org/pdf/2607.13558
Abstract Urban region profiling constitutes a core problem in urban computing, supporting applications such as population estimation, economic assessment, and environmental monitoring. Existing methods typically formulate this task as multimodal representation learning, fusing heterogeneous urban data, e.g., satellite imagery, points of interest, textual descriptions, and 3D building information, into latent embeddings for prediction. However, these approaches are largely correlation-driven, assume cross-modal consistency, and rely on static pipelines, which limit their robustness in heterogeneous or unseen urban regions. We propose UrbanAgent, an agentic framework that reframes urban region profiling as a reasoning-driven inference problem. UrbanAgent instantiates an independent agent for each data modality and performs structured multi-agent collaborative reasoning to explicitly address cross-modal inconsistencies rather than absorbing them into a single representation. In addition, UrbanAgent extends indicator prediction as a closed-loop process of active evidence acquisition and iterative reasoning, enabling agents to verify uncertain inferences through tool-augmented retrieval of external knowledge optimized via reinforcement learning. Extensive experiments on global urban datasets for Carbon emissions, GDP, and Population estimation show that UrbanAgent consistently outperforms existing baselines, achieving an average improvement of 8.1% in R2, and exhibiting strong generalization performance in unseen-city settings.
中文摘要 城市区域剖析构成城市计算的核心问题，支持人口估计、经济评估和环境监测等应用。现有方法通常将该任务表述为多模态表示学习，将异构城市数据（如卫星影像、兴趣点、文本描述和三维建筑信息）融合为潜在嵌入以供预测。然而，这些方法大多依赖相关性，假设跨模态一致性，并依赖静态管道，这限制了它们在异质或看不见的城市区域中的鲁棒性。我们提出了UrbanAgent，一种将城市区域画像重新定义为推理驱动的推理问题的智能体框架。UrbanAgent为每个数据模态实例化一个独立代理，并进行结构化多代理协作推理，明确解决跨模态不一致，而非将其吸收到单一表示中。此外，UrbanAgent将指标预测扩展为一个闭环过程，包括主动证据获取和迭代推理，使智能体能够通过工具增强的外部知识检索，并通过强化学习优化，验证不确定的推断。在全球城市数据集上的碳排放、GDP和人口估计实验显示，UrbanAgent持续优于现有基线，R2平均提升8.1%，并在未见城市环境中展现出强劲的泛化性能。

Structured Reinforcement Learning for Bayesian Persuasion : Application to Intelligent Interactive Driving

贝叶斯说服的结构化强化学习：智能交互式驾驶的应用

Authors: Merlin Paul, Anup Aprem
Subjects: Subjects: Machine Learning (cs.LG); Signal Processing (eess.SP)
Arxiv link: https://arxiv.org/abs/2607.13576
Pdf link: https://arxiv.org/pdf/2607.13576
Abstract Interactive driving, wherein an intelligent lead vehicle equipped with real-time traffic data coordinates route choices of connected vehicles, offers a promising approach to dynamic traffic management. To address the challenge of harmonising decisions, this paper considers the strategic information revealing framework of Bayesian persuasion. Here, the principal (lead vehicle) aims to guide the agent's (connected vehicle) partially observable sequential decision making towards its own objectives by selectively revealing information, such as real-time traffic ahead, using signals. However, the agent's farsighted response to maximize its long-term reward, renders the principal's signaling strategy design computationally challenging. We propose an online structured reinforcement learning framework to synthesize computationally efficient signaling strategy which is persuasive for a far-sighted agent. The main contributions of the paper are as follows: (i) For a monotonic agent with approximate best response, we propose MAPL, a structured policy learning algorithm for faster online learning, (ii) Identification of sufficient conditions for the supermodular structure of the Q function of the principal for a monotonic agent, (iii) Identification of sufficient conditions to ensure the persuasiveness of the principal's signaling strategy, (iv) Supermodular Q learning for Principal (SQP), which leverages the supermodular structure of principal's action value to synthesize computationally efficient signaling strategy that is persuasive for a monotonic learning agent, (v) Numerical analysis considering a real-time application of Bayesian persuasive driving for lane selection demonstrates that the proposed method is 30% cost efficient for optimising travelling rewards of both the lead and connected vehicle compared to the existing methodologies for signaling strategy design.
中文摘要 交互式驾驶，即配备实时交通数据的智能引导车辆协调连接车辆的路线选择，为动态交通管理提供了有前景的方法。为了解决决策协调的挑战，本文探讨了贝叶斯说服中的战略信息揭示框架。在这里，主导车辆（主导车辆）旨在通过信号选择性揭示信息（如前方实时交通）来引导代理（联网车辆）部分可观测的顺序决策，朝其自身目标方向发展。然而，代理人为最大化长期回报而做出的远见性反应，使得委托人的信号策略设计在计算上具有挑战性。我们提出了一种在线结构化强化学习框架，用于综合计算高效的信号策略，对远见智能体具有说服力。论文的主要贡献如下：（i）对于具有近似最佳响应的单调代理，我们提出MAPL，一种结构化策略学习算法，用于加快在线学习;（ii）识别单调代理Q函数超模结构的充分条件，（iii）识别确保主体信号策略具有说服力的充分条件，（iv）主体超模Q学习（SQP），利用主体动作值的超模块结构，合成对单调学习代理具有说服力的计算高效信令策略;（v）考虑实时应用贝叶斯说服驾驶用于车道选择的数值分析表明，该方法在优化领先车辆和联挂车辆行驶奖励方面，与现有车辆相比具有30%的成本效益信号策略设计方法论。

Agile perceptive multi-skill locomotion for quadrupedal robots in the wild

野外四足机器人的敏捷感知多技能运动

Authors: Jun-Gill Kang, Jaehyun Park, Tae-Gyu Song, Joon-Ha Kim, Seungwoo Hong, Hae-Won Park
Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.13579
Pdf link: https://arxiv.org/pdf/2607.13579
Abstract Enabling quadrupedal robots to traverse complex terrains-from rugged outdoor environments to urban landscapes-requires seamless integration of multiple motor skills, smooth transitions between gaits, and high-speed perceptive locomotion using only onboard sensors. We present APT-RL (Action Pretrained Transformer-based Reinforcement Learning), a unified framework that enables multi-skill locomotion to achieve high-speed traversal in complex environments through autonomous skill transitions utilizing only onboard perception and computation. Our approach generates large-scale, feature-rich 2D motion datasets through trajectory optimization with simplified dynamics. These datasets enable training of diverse, reusable locomotion skills that transfer effectively to a real quadruped robot operating on complex uneven terrains. The resulting high-quality skills serve as strong priors for efficient learning of complex downstream tasks and extend naturally to 3D environments, enabling smooth, high-speed multi-skill locomotion in deployed policy. Real-world experiments demonstrate the framework's capabilities: the robot performs agile maneuvers through complex indoor obstacles and outdoor wild environments, including dynamic drop-down maneuvers that reach instantaneous peak speeds of up to 6 meters per second. A single onboard policy enables robust traversal of diverse obstacles, including stairs, hurdles, stepping stones, gaps, and fallen branches, demonstrating the versatility and effectiveness of our approach.
中文摘要 使四足机器人能够穿越复杂地形——从崎岖的户外环境到城市景观——需要多种运动技能的无缝整合，步态间的平滑过渡，以及仅用车载传感器实现高速感知运动。我们介绍APT-RL（基于Transformer的行动训练强化学习），这是一个统一框架，使多技能移动能够通过仅依靠车载感知和计算实现的自主技能转移，实现复杂环境中的高速穿越。我们的方法通过轨迹优化和简化动力学生成大规模、特征丰富的二维运动数据集。这些数据集使得多样化且可重复使用的移动技能能够有效转移到真实四足机器人在复杂崎岖地形上运行。由此产生的高质量技能为高效学习复杂下游任务提供了强有力的先验，并自然延伸到三维环境中，实现了部署策略中流畅、高速的多技能操作。真实实验展示了该框架的能力：机器人能够灵活地穿越复杂的室内障碍和户外野生环境，包括瞬时最高达到6米每秒峰值的动态下坠动作。单一的车载政策使我们能够稳健地穿越各种障碍，包括楼梯、障碍、踏脚石、缝隙和倒枝，展示了我们方法的多样性和有效性。

OvisOCR2 Technical Report

OvisOCR2技术报告

Authors: Shiyin Lu, Yinglun Li, Yu Xia, Yuhui Chen, An-Yang Ji, Jun-Peng Jiang, Qing-Guo Chen, Jianshan Zhao, En Lin, Haijun Li, Cheng Qin, Zhao Xu, Weihua Luo
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13639
Pdf link: https://arxiv.org/pdf/2607.13639
Abstract We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at this https URL.
中文摘要 我们引入了 OvisOCR2，一个 0.8B 的文档解析模型。OvisOCR2 设计为端到端解析器：给定文档页面图像，生成自然阅读顺序的 Markdown 表示，涵盖文本、公式、表格和视觉区域。我们构建了一个数据引擎，将过滤的真实文档注释与合成页面结合，其渲染图像和Markdown目标源自同一HTML源。训练方案包括监督微调、在4B分支上的强化学习，采用多元奖励设计，策略中提炼至0.8B模型，以及模型融合。在 OmniDocBench v1.6 版本中，OvisOCR2 取得了最先进的整体得分 96.58，将端到端模型置于此前由流水线方法主导的排行榜顶端，凸显了端到端文档解析的潜力。在PureDocBench上，OvisOCR2还获得了最高的Avg3分数75.06。除了这两个公开基准外，我们还在内部基准上评估OvisOCR2，该基准旨在涵盖更广泛的长尾和挑战性场景。OvisOCR2在比较方法中整体表现最佳，进一步证明了其泛化性和鲁棒性。OvisOCR2 可在此 https 网址获取。

Consensus as Privileged Context for Label-Free Self-Distillation

共识作为无标签自我蒸馏的特权语境

Authors: John Gkountouras, Josip Jukić, Ivan Titov
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.13643
Pdf link: https://arxiv.org/pdf/2607.13643
Abstract Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.
中文摘要 采样多重解并返回多数答案是提高无标签大型语言模型推理准确性的最可靠方法之一，越来越多的方法将共识信号转化为训练监督。然而，现有方法仅以有限形式使用共识：作为筛选解决方案进行微调的过滤器、作为答案偏好，或作为强化学习的标量奖励，丢弃了大多数共识解决方案中包含的信息。我们介绍CANON（共识-ANchored自蒸馏），这是一种无标签训练方法，将共识转化为密集的代币级监督。对于每个无标签提示，CANON采样多个解，提取多数答案，并对到达该解的模型进行冻结快照;这位以共识为基础的教师随后在每个代币上监督模型的独立推广。数学和科学推理基准测试显示，CANON在计算量的七分之一时，pass@1提升多达12分，比无标签强化学习高出6分，且接近以黄金解为条件的教师;它基于未标记的合并数据进行训练，并会转移到持有的基准测试，与使用金标签的训练方法相匹配。分析显示，这些改进并非纯粹的分布锐化：训练后，模型在32次尝试中解决了此前未曾解决的问题，其多数票本身也变得更加准确。

Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation

探索性、沟通性和可部署性：面向开放世界移动操作的愿景驱动具身代理

Authors: Boyu Mi, Mengchen Ma, Yifei Yao, Xing Gao, Junting Chen, Yangzi Li, Zihou Zhu, Guohao Li, Zhenfei Yin, Tai Wang, Yao Mu, Jiangmiao Pang, Hanqing Wang
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13653
Pdf link: https://arxiv.org/pdf/2607.13653
Abstract Real-world deployment of embodied agents requires active exploration, visual grounding, and interactive intent disambiguation. However, existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges. To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent environment APIs without oracle perception and integrates a simulated user to enable human-in-the-loop interaction. Within this environment, we design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning, systematically optimizing agent performance. To comprehensively evaluate this approach, we introduce REAL-Bench, a benchmark spanning 241 tasks across active exploration, visual distraction, articulated manipulation, and interactive disambiguation. Experimental results demonstrate that our trained agent outperforms leading commercial closed-source VLMs on interactive tasks with a 56.9% success rate. Further empirical analysis reveals that our hierarchical training pipeline successfully aligns the model's tool-use capabilities while maintaining robust open-vocabulary reasoning under extended exploration horizons. Finally, we deploy and evaluate our framework on a physical dual-arm mobile robot, where it achieves a 78.3% end-to-end success rate over 60 real-world episodes. These physical trials demonstrate robust zero-shot transferability to unseen household scenarios, validating that our sim-to-real-consistent design successfully bridges the reality gap for long-horizon mobile manipulation. Code is available at this https URL.
中文摘要 实体智能体的现实部署需要主动探索、视觉基础和互动意图消歧。然而，现有框架通常依赖特权模拟器状态或假设完整的指令，绕过了现实中的部署挑战。为了弥合这一差距，我们提出了REAL，一个用于开放世界移动操作的代理框架。REAL 建立了无需预言机感知的模拟与现实一致环境 API，并集成了模拟用户，实现了人机参与的交互。在此环境中，我们设计多样化的任务组合，推动数据收集、监督式微调和在线强化学习，系统性优化代理性能。为了全面评估这一方法，我们引入了REAL-Bench，这是一个涵盖主动探索、视觉分散、可表达操作和交互消歧的241项任务的基准测试。实验结果显示，我们训练有素的代理在交互任务中以56.9%的成功率优于领先的商业闭源VLM。进一步的实证分析显示，我们的分层训练流程成功地对齐了模型的工具使用能力，同时在更长的探索视野下保持了强健的开放词汇推理能力。最后，我们将框架部署并评估在一台实体双臂移动机器人上，在60个真实世界演示中实现了78.3%的端到端成功率。这些物理试验展示了对未见家庭场景的强健零机会迁移能力，验证了我们的模拟到真实一致性设计成功弥合了长期移动操作的现实差距。代码可在此 https URL 访问。

Explaining Reinforcement Learning Agents via Inductive Logic Programming

通过归纳逻辑编程解释强化学习代理

Authors: Celeste Veronese, Edoardo Zorzi, Daniele Meli, Alessandro Farinelli
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.13655
Pdf link: https://arxiv.org/pdf/2607.13655
Abstract Explainable Reinforcement Learning (XRL) seeks to make Reinforcement Learning (RL) policies more transparent and interpretable, a key requirement in safety-critical and human-centric scenarios. However, it is mostly based on user studies, thus targeting the needs of a specific audience and lacking shared evaluation metrics. On the other hand, logic-based approaches within eXplainable Artificial Intelligence (XAI) provide compact, human-readable abstractions of decision-making. However, the systematic quantification of the explainability degree of logical representations remains an open problem. This work aims to advance the state of the art in XRL by introducing objective and planning-oriented metrics for policy explainability in RL settings. At the same time, it contributes to the field of logic for XAI by providing a principled way to quantify the explainability of logical rules, moving beyond common-sense assessments and simple propositional fragments. We employ Inductive Logic Programming (ILP) to extract symbolic representations of RL policies and define a novel set of explainability metrics, including activation rate, feature coverage, syntactic distance and semantic distance. These metrics quantify alignment between symbolic rules and agent behavior, the role of features in decision-making, and the evolution of policies during training and across agents in single and multi-agent RL. Experiments across different RL domains show that the proposed metrics highlight action-specific learning dynamics beyond global return, provide fine-grained insights into domain features beyond classical approaches for global feature importance estimation, and uncover coordination, specialization, and adaptation patterns in MARL. Moreover, they provide crucial insights for the transfer and generalization of action-specific policies.
中文摘要 可解释强化学习（XRL）旨在使强化学习（RL）政策更加透明和可解释性，这是安全关键和以人为中心场景中的关键要求。然而，它主要基于用户研究，针对特定受众的需求，缺乏共享的评估指标。另一方面，基于逻辑的可解释人工智能（XAI）方法提供了简洁、人类可读的决策抽象。然而，逻辑表征可解释度的系统性量化仍是一个未解之谜。本研究旨在通过引入客观且以规划为导向的强化学习环境中政策可解释性指标，推动XRL的技术进步。同时，它通过提供一种有原则的方式量化逻辑规则的可解释性，超越了常识评估和简单命题片段，为XAI的逻辑领域做出了贡献。我们采用归纳逻辑编程（ILP）提取强化学习策略的符号表示，并定义一套新的可解释性指标，包括激活率、特征覆盖率、语法距离和语义距离。这些指标量化了符号规则与代理行为的对齐、功能在决策中的作用，以及单代理和多代理强化学习中策略在培训和代理间的演变。跨不同强化学习领域的实验表明，所提出的指标突出了超越全局回报的动作特异性学习动态，提供了超越传统全局特征重要性估计方法的细致领域特征洞见，并揭示MARL中的协调、专精和适应模式。此外，它们为行动特定政策的转移和推广提供了关键见解。

DAGR: State-Conditioned Goal Representations via Difference-Aware Goal Cross-Attention

DAGR：通过差分感知目标交叉注意力实现状态条件目标表示

Authors: Xing Lei, Wenyan Yang, Xuetao Zhang, Donglin Wang
Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Arxiv link: https://arxiv.org/abs/2607.13731
Pdf link: https://arxiv.org/pdf/2607.13731
Abstract Goal-conditioned reinforcement learning hinges on how the goal is encoded. Contrastive, metric, temporal-distance, and information-theoretic encoders differ in objective. They still share one trait. None of them sees the current state. Such a state-independent embedding cannot mark which part of the goal still needs action. The policy must then recover that cue by inverting both encoders. We propose DAGR. It refines the static embedding of any late-fusion encoder into a state-conditioned one through multi-scale gated cross-attention. A near-identity gated residual preserves the base representation. Difference-aware Goal Cross-Attention then biases the attention scores using a per-token state-goal discrepancy map. On OGBench, DAGR improves navigation. Our ablations trace the gain to the gated residual, not to the difference bias that names the method. On manipulation and puzzle tasks it matches or falls below the base. DAGR is a structured refinement, not a universal improvement.
中文摘要 目标条件强化学习依赖于目标的编码方式。对比编码器、度量编码器、时间距离编码器和信息理论编码器在目标上有所不同。他们仍然有一个共同点。他们都看不到现在的状态。这种状态独立的嵌入无法标明目标的哪部分仍需采取行动。策略必须通过反转两个编码器来恢复该提示。我们提出DAGR。它通过多尺度门控交叉注意力，精细将任何晚聚变编码器静态嵌入状态条件编码器。近单位门控残差保持基表示。差值感知目标交叉注意力（Difference aware Goal Cross-Attention）则使用每个代币的状态-目标差异图对注意力得分进行偏向。在OGBench上，DAGR改进了导航。我们的消融方法是将增益追溯到门控残差，而不是命名方法的差偏差。在操控和解谜任务中，它会匹配或低于基础。DAGR是一种结构化的改进，而非普遍改进。

Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape Pre-, Intra-, and Post-CoT Calibration

训练后转移置信度：SFT、RL和OPD如何影响训练前、训练中和后CoT校准的三阶段分析

Authors: Shuhao Li, Guodong Du, Anhao Zhao, Wanyu Lin, Tianyu Yuan, Xiaoyu Shen
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.13753
Pdf link: https://arxiv.org/pdf/2607.13753
Abstract Large language models have made strong reasoning gains through supervised fine-tuning, reinforcement learning, and on-policy distillation, yet these post-training methods are usually evaluated only by final-answer accuracy. We study how they reshape confidence during reasoning. We introduce a three-stage calibration framework that evaluates confidence before, during, and after chain-of-thought generation, corresponding to difficulty estimation, early termination, and answer aggregation. Through a controlled comparison on mathematical reasoning benchmarks, we find that OPD provides the most useful pre-reasoning confidence, SFT gives the strongest online signal for early stopping, and RL produces the most reliable trace-level signal for aggregation. We further show that confidence reliability is position-dependent: RL confidence becomes informative after a path-commitment phase, while OPD confidence is useful early but can become inversely calibrated later. Based on this observation, we propose PosConf, a position-aware confidence strategy that uses confidence only from reliable relative-position intervals. PosConf improves RL answer aggregation by 6.1 points over majority voting and consistently improves OPD early stopping under tight token budgets, with gains up to 4.3 points by avoiding its later inverse-calibration region, showing that \emph{confidence in reasoning models should be used both stage-wise and position-awarely}. Our code is available at this https URL.
中文摘要 大型语言模型通过监督微调、强化学习和策略提炼取得了显著的推理进步，但这些训练后方法通常仅通过最终答案的准确性来评估。我们研究它们如何在推理过程中重塑自信。我们引入了三阶段校准框架，评估在思维链生成前、中、后三阶段的置信度，分别对应难度估计、提前终止和答案聚合。通过对数学推理基准的受控比较，我们发现OPD提供了最有用的预推理置信，SFT提供了最强的早期停止在线信号，而RL则产生最可靠的迹级信号用于聚合。我们进一步证明置信度依赖位置：强化学习置信度在路径承诺阶段后变得具有信息性，而OPD置信度在早期有用，但后期可反向校准。基于这一观察，我们提出了PosConf，这是一种仅使用可靠相对位置区间置信度的位觉置信策略。PosConf使强化学习答案聚合比多数投票提高了6.1个百分点，并且在代币预算紧张下，OPD早期停止率持续提升，通过避免后期反校准区可提升最多4.3个百分点，表明推理模型的置信度应在阶段和位置意识层面上均适用。我们的代码可在此 https URL 访问。

Vision-Based Obstacle Separation for Strawberry Harvesting in Clusters Using Hierarchical Reinforcement Learning

基于视觉的障碍物分离，用于簇中采摘草莓，利用层级强化学习

Authors: Teng Li, Hanfei Shi, Chunjiang Zhao, Ya Xiong
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13799
Pdf link: https://arxiv.org/pdf/2607.13799
Abstract Selective harvesting in clustered strawberry environments is challenging because ripe fruits are often occluded by surrounding unripe fruits, making direct grasping unreliable. To address this problem, this paper proposes a hierarchical reinforcement learning framework, termed VGPA, which integrates a vision-guided decision mechanism and a Progressive Adaptive Exploration Strategy (PAES) for vision-based obstacle separation and harvesting. The task was decomposed into two sequential stages: obstacle separation and target grasping. At the high level, the vision-guided mechanism improved option selection and accelerated policy convergence. At the low level, PAES improved exploration efficiency and training stability during continuous control learning. In simulation experiments, the learned policy achieved a success rate of 96.7%. In addition, sim-to-real transfer experiments on a self-developed parallel robot showed that the proposed method achieved success rates ranging from 71.7% to 88.3%, outperforming direct picking while requiring only 1.22~s more average harvesting time. These results verified the effectiveness, generalization ability, and practical potential of the proposed method for robotic harvesting in complex clustered environments.
中文摘要 在成簇草莓环境中选择性采收具有挑战性，因为成熟的果实常被周围未成熟的果实遮挡，导致直接采集不可靠。为解决这一问题，本文提出了一种分层强化学习框架，称为VGPA，它集成了视觉引导决策机制和渐进自适应探索策略（PAES），用于基于视觉的障碍分离和收获。该任务被分解为两个连续阶段：障碍物分离和目标抓取。在高层面上，愿景引导机制改善了选项选择和加速政策趋同。在低层次，PAES提高了持续控制学习过程中的探索效率和训练稳定性。在模拟实验中，该学习策略的成功率为96.7%。此外，在自开发的并行机器人上进行的模拟到实物转移实验显示，该方法成功率在71.7%至88.3%之间，优于直接采摘，且平均收获时间仅多1.22~秒。这些结果验证了该方法在复杂集群环境中机器人采集的有效性、泛化能力及实用潜力。

Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learning

结合智能强化学习的机器人操作中的稳健执行学习

Authors: Xiaopeng Zhang, Yueyang Weng, Qi Liu, Yongjin Mu, Yanjie Li
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13818
Pdf link: https://arxiv.org/pdf/2607.13818
Abstract Robotic manipulation poses fundamental challenges due to uncertainty, long-horizon execution, and compounding errors, which can easily destabilize execution and lead to task failure. Although recent vision-language-action (VLA) models exhibit strong generalization, they typically lack explicit mechanisms to assess execution stability and to recover when execution deviates from its nominal behavior. In this paper, we propose: (1) two complementary metrics to assess execution quality at runtime, and (2) an agentic reinforcement learning framework that learns to restore effective execution through high-level decision-making rather than directly learning low-level actions. In this framework, an agentic policy reasons over recent execution history and selects among a small set of execution modes to regulate the execution process. Under execution degradation, it triggers appropriate recovery mechanisms to restore the robot to previously visited nominal states, enabling the task to continue. We evaluate the proposed method on the LIBERO benchmark, achieving up to a 13.7% improvement in success rate under standard settings and up to a 39.2% improvement under disturbance settings, demonstrating substantially enhanced execution robustness.
中文摘要 机器人操作面临根本性挑战，原因包括不确定性、长期执行和累积错误，容易破坏执行稳定性并导致任务失败。尽管最新的视觉-语言-动作（VLA）模型表现出强烈的泛化性，但它们通常缺乏显式机制来评估执行稳定性，以及在执行偏离名义行为时进行恢复。本文提出：（1）两种互补的指标用于在运行时评估执行质量，以及（2）一种通过高层决策而非直接学习低层次动作来恢复有效执行的代理强化学习框架。在该框架中，代理策略推理近期执行历史，并从少数执行模式中选择以调节执行过程。在执行降级时，它会触发适当的恢复机制，将机器人恢复到先前访问的正常状态，使任务得以继续。我们在LIBERO基准测试上评估了该方法，在标准设置下成功率提升了最多13.7%，在干扰条件下提升了最高39.2%，显著提升了执行鲁棒性。

SPyCE: Skill-Policy Co-evolution for Multimodal Agents

SPyCE：多模态代理的技能-策略共演化

Authors: Ru Zhang, Weijie Qiu
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.13854
Pdf link: https://arxiv.org/pdf/2607.13854
Abstract Multimodal agents that think with images iteratively manipulate visual evidence and invoke tools across many steps. Existing reinforcement learning methods reduce trajectories to scalar rewards, forcing the policy to discover reusable tool-use patterns from scratch on every new task; memory-based alternatives retain past experience, yet they rely on test-time retrieval, without updating the policy to absorb reusable patterns from that experience. Our key insight is that multimodal reasoning trajectories should be distilled into reusable skills that co-evolve with the policy during training, rather than being consumed as rewards or retrieved from a static store. To this end, we propose SPyCE (Skill-Policy Co-evolution), a framework that distills trajectories into a hierarchical skill library and updates it throughout reinforcement learning. Execution skills capture local visual operations, while workflow skills encode high-level priors that orchestrate tool use. During training, the policy model conditions on retrieved skills to guide its rollouts, while the skill library evolves using valuable rollouts generated by the policy. This creates a closed loop in which improved policies yield better skills, and the evolving skill library, in turn, provides stronger priors for policy rollouts. Experiments across eight benchmarks demonstrate that SPyCE consistently outperforms both RL-based and memory-based baselines. Further analysis reveals that both the hierarchical skill design and the co-evolution mechanism are critical to our design. These results suggest joint skill-policy optimization as a promising paradigm for building capable multimodal agents.
中文摘要 多模态智能体通过图像迭代操作视觉证据，并在多个步骤中调用工具。现有的强化学习方法将轨迹简化为标量奖励，迫使策略在每个新任务中从零开始发现可重复使用的工具使用模式;基于内存的替代方案保留了过去的经验，但它们依赖测试时的检索，而不更新策略以吸收可重复使用的模式。我们的核心见解是，多模态推理轨迹应被提炼为可重复使用的技能，在培训过程中与策略共同演进，而不是作为奖励消费或从静态存储中提取。为此，我们提出了SPyCE（技能-政策共进化）框架，该框架将发展轨迹提炼成层级技能库，并在强化学习过程中不断更新。执行技能捕捉局部视觉操作，而工作流技能则编码高层次先验，协调工具的使用。在培训过程中，政策模型以检索到的技能为指导其推广，而技能库则通过策略生成的有价值推广而演进。这形成了一个闭环，政策改进带来更强技能，技能库不断演变，为政策推广提供了更强的先验。八个基准测试的实验表明，SPyCE始终优于基于强化学习和基于内存的基线。进一步分析显示，层级技能设计和共进化机制对我们的设计至关重要。这些结果表明，联合技能-策略优化是构建多功能多模态智能体的有前景范式。

Task-Oriented Sensing and Covert Transmissions for Collaborative Multi-AUV Systems

任务导向感测与协作多AUV系统的隐蔽传输

Authors: Xueyao Zhang, Chenyang Yan, Bo Yang, Xuelin Cao, Zhiwen Yu, Bin Guo, George C. Alexandropoulos, Merouane Debbah, Chau Yuen
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.13880
Pdf link: https://arxiv.org/pdf/2607.13880
Abstract In underwater covert cooperative missions, autonomous underwater vehicles (AUVs) often cannot rely on active sonar to continuously obtain complete information, since active sensing and frequent communications increase the risk of exposure. As a result, AUVs primarily rely on passive observation, an approach that yields incomplete local perception and limited task efficiency. Although underwater acoustic communications can mitigate this limitation through information sharing, they are simultaneously constrained by long delays, severe interference, low reliability, and the risk of covert exposure. Existing communications-oriented multi-agent reinforcement learning (MARL) studies often model communication as an ideal information flow, whereas traditional communication optimization primarily focuses on link-level performance. However, both are insufficient to characterize the actual contribution of perceptual information to cooperative tasks under realistic conditions of covert physical communications. This paper proposes a Sensed Information Value Realization Multi-Agent Reinforcement Learning (SVR-MARL) framework that leverages practical information to characterize the utility of information for cooperative tasks and learns distributed cooperative policies under realistic communication and covert constraints. Through a case study of covert multi-AUV cooperative localization and tracking, the potential of the proposed framework to improve collaborative task efficiency while reducing unnecessary communication and exposure risks is demonstrated.
中文摘要 在水下隐秘合作任务中，自主水下载具（AUV）通常无法依赖主动声纳持续获取完整信息，因为主动探测和频繁通信增加了暴露风险。因此，AUV主要依赖被动观察，这种方法导致局部感知不完整且任务效率有限。虽然水下声学通信可以通过信息共享缓解这一限制，但同时也受到长时间延迟、严重干扰、低可靠性以及隐蔽暴露风险的限制。现有面向通信的多智能体强化学习（MARL）研究通常将通信建模为理想的信息流，而传统通信优化主要关注链路级性能。然而，这两者都不足以描述在隐秘物理通信的现实条件下，感知信息对协作任务的实际贡献。本文提出了一种感知信息价值实现多智能体强化学习（SVR-MARL）框架，利用实用信息来描述信息在协作任务中的效用，并在现实通信和隐蔽约束下学习分布式合作策略。通过隐蔽多AUV协作定位与跟踪的案例研究，展示了该框架在降低不必要通信和暴露风险的同时，提升协作任务效率的潜力。

SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

SIVA-RL：多模态强化学习中的敏感性-不变视觉对齐

Authors: Cheng Tang, Junzhi Ning, Min Cen, Wei Li, Xinyi Zeng, Pinxian Zeng, Rongbin Li, Qiming Zhu, Yuqiang Li, Junjun He, Yirong Chen, Ming Hu
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
Arxiv link: https://arxiv.org/abs/2607.13931
Pdf link: https://arxiv.org/pdf/2607.13931
Abstract Reinforcement learning with verifiable rewards (RLVR) drives multimodal reasoning, but answer-level correctness does not guarantee that a vision-language model grounds its predictions in visual evidence. Existing visual-intervention methods contrast policy behavior on original and modified images, yet assign supervision by the type of intervention rather than its observed effect. This assumption fails: identical operators produce heterogeneous outcomes across samples. We propose SIVA-RL, a Sensitivity-Invariance Visual Alignment framework that replaces operator-conditioned regularization with sample-wise, outcome-conditioned supervision. SIVA-RL constructs localized interventions through token-aligned, distance-constrained within-image PatchSwap. A frozen audit policy then scores each clean-intervention pair, and the observed reward drop becomes soft routing weights. Large-drop pairs drive sensitivity alignment, low-drop pairs drive clean-anchored invariance alignment, and ambiguous pairs are down-weighted. This design decouples intervention construction from supervision assignment and is compatible with both GRPO and DAPO backbones. Across nine multimodal reasoning benchmarks spanning mathematical, logical, and vision-dependent tasks, SIVA-RL improves 3B and 7B models over matched RL baselines in every setting. It yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.
中文摘要 带有可验证奖励的强化学习（RLVR）推动多模态推理，但答案层级的正确性并不保证视觉语言模型的预测基于视觉证据。现有的视觉干预方法对原始图像和修改图像的政策行为进行了对比，但监督是根据干预类型而非观察到的效果来分配。这一假设不成立：相同的算符在样本间产生异质结果。我们提出了SIVA-RL，一种敏感性-不变视觉对齐框架，用样本分样、结果条件化监督取代操作员条件正则化。SIVA-RL通过标记对齐、距离受限的图像内PatchSwap构建局部干预。冻结审计策略随后对每个干净干预对进行评分，观察到的奖励下降变成软路由权重。大降对驱动灵敏度比对，低落差对驱动干净锚定的不变性比对，歧义对则下加权。该设计将干预构建与监督分配分离，兼容GRPO和DAPO骨干。在九个涵盖数学、逻辑和视觉依赖任务的多模态推理基准测试中，SIVA-RL在所有环境中改进了3B和7B模型，优于匹配的强化学习基线。在视力依赖推理方面提升8.79个百分点，且在四种基于GRPO和DAPO的配置中，整体相对提升最高达14.9%。

Discriminative Barrier Functions for Safe Adversarial Imitation Learning from Observation

安全对抗模仿学习的判别障碍函数

Authors: Anubhav Vishwakarma, Bhaumik Mehta, Caleb Hsu, Byron Boots, Karen Leung, Tyler Han
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.13938
Pdf link: https://arxiv.org/pdf/2607.13938
Abstract Inverse Reinforcement Learning (IRL) algorithms are powerful tools for learning from and generalizing expert demonstrations, but they often rely on unconstrained exploration, rendering them unsafe for real-world deployment. Meanwhile, Control Barrier Functions (CBFs) can guarantee the safety of control systems, but the analytical design of CBFs can be time-consuming and esoteric. In this work, we address these limitations jointly by constraining reward function candidacy during IRL to the space of CBFs, yielding a formulation that exhibits safe online control with continuous experiential improvement. Crucially, this framework enables the data-driven recovery of barrier functions directly from unlabeled expert observations. We demonstrate that the recovered barrier function is robust to unsafe states entirely absent from the expert data. Furthermore, we benchmark our method against standard IRL baselines in a simulated navigation environment, demonstrating improved safety performance. Finally, we investigate the trade-offs of planning-based versus policy-based IRL methods across both simulation and a real world obstacle avoidance task.
中文摘要 逆向强化学习（IRL）算法是从专家演示中学习和泛化的强大工具，但它们通常依赖于不受限制的探索，因此不适合实际应用。与此同时，控制屏障功能（CBF）可以保证控制系统的安全性，但CBF的分析设计可能耗时且晦涩难懂。本研究通过将现实生活中的奖励功能候选权限制在CBF空间内，共同解决这些局限性，形成一种具有安全在线控制和持续体验改善的方案。关键是，该框架能够直接从未标记的专家观察数据驱动恢复屏障函数。我们证明了恢复的屏障函数对专家数据中完全缺失的不安全状态具有鲁棒性。此外，我们将方法与模拟导航环境中的标准现实基线进行基准测试，展示了安全性的提升。最后，我们探讨了基于规划与基于政策的IRL方法在模拟和现实障碍避让任务中的权衡。

TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

TRACE：通过信用估算对长期视野代理进行回合级奖励分配

Authors: Leitian Tao, Baolin Peng, Wenlin Yao, Tao Ge, Hao Cheng, Mike Hang Wang, Jianfeng Gao, Sharon Li
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.13988
Pdf link: https://arxiv.org/pdf/2607.13988
Abstract Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.
中文摘要 多轮代理通过长时间的工具交互序列解决复杂任务，最终给出最终答案，这使得学分分配成为培训后的基本挑战。结果奖励为短期推理提供了可靠的监督，但随着工具调用数量增加到数十甚至数百次，结果奖励会变得稀疏且方差性高。它们也可能具有误导性：一次失败的推广可能包含许多有用的动作，使代理更接近目标，但结果训练却赋予他们与最终错误相同的负面优势。我们提出了TRACE（通过信用估计进行回合级奖励分配），这是一种用于能动强化学习的密集学分分配方法。TRACE将部署表示为工具调用边界的状态转换，从冻结的参考模型中获得黄金答案对数概率，转换为对数比状态值，并根据这些值的时间差值变化推导每次动作奖励。这无需额外的批评或工艺标签培训，其一步对数比TD组件在冗余工具调用中实现了显著的伸长效应。在长期复杂搜索中，TRACE通过纯强化学习大幅提升了基础模型工具的使用能力，无需冷启动监督微调阶段、代理式中期训练阶段或实时网络数据训练。在闭门网页的BrowseComp-Plus基准测试中，Qwen3-4B从7.2美元涨至35.6美元，Qwen3-30B-A3B从8.4美元涨至42.6美元。所学的搜索行为也可转化为开放网络基准测试，学习曲线在强化学习训练中更早提升并更快收敛。

Lyapunov Exponent as Physics-Informed Dense Reward: RL Discovery of Stabilization Beyond the Kapitza Pendulum

李雅普诺夫指数作为物理学基础的密集奖励：现实学习发现超越卡皮察摆的稳定化

Authors: Slava Andrejev
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.14001
Pdf link: https://arxiv.org/pdf/2607.14001
Abstract We suggest using the Lyapunov characteristic exponent (LCE) as a dense reward signal for the reinforcement learning problem of stabilizing the inverted pendulum with vertical motion. With LCE, the agent not only successfully found the oscillatory motion known as the Kapitza pendulum but also damped the pendulum's pivoting, leaving it in a strictly upright position.
中文摘要 我们建议使用李雅普诺夫特征指数（LCE）作为强化学习问题的密集奖励信号，用于稳定倒摆垂直运动的问题。使用LCE时，该探剂不仅成功找到了被称为卡皮察摆的振荡运动，还抑制了摆的旋转，使其保持严格直立状态。

Lighthouse RL: Sample-Efficient Circuit Optimization via Strategic Reset Points

Lighthouse RL：通过战略重置点实现采样高效电路优化

Authors: Mustafa Emre Gürsoy, Stefan Uhlich, Ryoga Matsuo, Yağız Gençer, Arun Venkitaraman, Chia-Yu Hsieh, Andrea Bonetti, Eisaku Ohbuchi, Lorenzo Servadei
Subjects: Subjects: Machine Learning (cs.LG); Hardware Architecture (cs.AR)
Arxiv link: https://arxiv.org/abs/2607.14008
Pdf link: https://arxiv.org/pdf/2607.14008
Abstract In this paper, we introduce Lighthouse RL, a sample-efficient reinforcement learning (RL) approach for analog circuit sizing. Traditional methods lack generalization across different performance targets, while standard RL approaches waste resources exploring unpromising regions. Our method addresses these inefficiencies through a strategic reset strategy that initializes episodes from high-performing configurations discovered during training, called "lighthouses". These states, which are closer to the target objectives, guide exploration toward promising regions. When compared to RL and Bayesian optimization methods from the literature, we demonstrate the effectiveness of our approach on a 2D benchmark problem and on two analog circuits, showing significant improvements in sample efficiency (up to 1.72x faster), optimization performance (100% vs. 0-87% success rate), generalization (75% vs. 0-50% extrapolation success), and objective maximization. This efficiency is particularly valuable for computationally expensive black-box optimization problems, and our reset strategy can be used as a plug-and-play enhancement for any RL-based optimization approach.
中文摘要 本文介绍了Lighthouse RL，一种用于模拟电路尺寸的样本高效强化学习（RL）方法。传统方法缺乏跨越不同性能目标的泛化能力，而标准强化学习方法则浪费资源探索不利领域。我们的方法通过一种战略重置策略解决这些低效问题，该策略初始化训练中发现的高效能配置（称为“灯塔”）的发作。这些更接近目标的州，引导勘探前往有前景的区域。与文献中的强化学习和贝叶斯优化方法相比，我们展示了该方法在二维基准问题和两个模拟电路上的有效性，在样本效率（最高快1.72倍）、优化性能（100%对0-87%成功率）、推广（75%对0-50%外推成功率）和目标最大化方面均有显著提升。这种效率对于计算量高的黑箱优化问题尤为重要，我们的重置策略可以作为任何基于强化学习的优化方法的即插即用增强。

Keyword: diffusion policy

Industrial Dexterity Benchmark: A Hardware-Software Benchmarking Platform for Industrial Dexterous Manipulation

工业灵巧基准：一个用于工业灵巧操作的硬件-软件基准测试平台

Authors: Honglu He, Jacob Laufer, Zhiwu Zheng, David Elkan-gonzalez, Raman Goyal, Xinyi Li, Su Lu, Mishek Musa, Berke Saat, Nicolas Tan, Colm Prendergast
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.14021
Pdf link: https://arxiv.org/pdf/2607.14021
Abstract Dexterous manipulation remains a critical bottleneck in industrial automation; tasks such as cable routing, connector insertion, and precision assembly still rely heavily on manual labor despite decades of robotics research. This work presents a progression from classical, modular robotics pipelines toward an end-to-end multimodal imitation-learning framework for industrial dexterous manipulation. As a part of this work, we introduce three key contributions: a set of Industrial Dexterity Benchmark (IDB) boards aimed to mimic datacenter cable management, automotive cable harnesses, and gearbox assembly tasks; a scalable imitation learning framework (DAG-ROS); and a multimodal diffusion-based policy framework (AG-iDP3) that creates models fusing RGB images, point clouds, joint positions, and wrist-frame wrench data. Focusing on the datacenter cable manipulation board, we evaluate the performance of a task involving cleaning a single cable over variations of an end-to-end AI policy using 48 trials per configuration. The best performing configuration, a multimodal expansion Diffusion Policy (DP), includes a multi-view RGB image source passed through an R3M encoder and reaches a 78% grasp and insert combined task success rate. This performance marks a significant improvement over the 36% observed from the single-camera RGB DP baseline. Each of the tested configurations requires only approximately 100 teleoperated demonstrations per task phase. These results indicate that the correct learned policy can outperform classical vision and control robotic methods in robustness, generalization, and deployment efficiency, justifying a shift toward scalable robotic automation for high up-time industrial environments.
中文摘要 灵巧的操作仍然是工业自动化中的关键瓶颈;尽管机器人研究已有数十年，诸如电缆布线、连接器插入和精密组装等任务仍然高度依赖人工劳动。这项工作展示了从经典模块化机器人流水线向工业灵巧操作端到端多模态模仿学习框架的进展。作为这项工作的一部分，我们介绍了三项关键贡献：一组工业敏捷基准（IDB）板，旨在模拟数据中心电缆管理、汽车电缆线束和变速箱组装任务;可扩展的模仿学习框架（DAG-ROS）;以及基于多模扩散的策略框架（AG-iDP3），该框架生成融合RGB图像、点云、关节位置和腕框扳手数据的模型。我们聚焦数据中心电缆操作板，评估了一项涉及清理单根电缆任务的性能，且每个配置使用48次尝试，采用了端到端AI策略的多种变体。表现最佳的配置是多模态扩展扩散策略（DP），包含多视图RGB图像源通过R3M编码器，任务的抓取与插入合并成功率达到78%。这一性能较单摄像机RGB DP基线的36%有了显著提升。每种测试配置每个任务阶段只需约100次远程操作演示。这些结果表明，正确学习的策略在鲁棒性、泛化性和部署效率方面可以优于传统的视觉与控制机器人方法，这为高运行时间工业环境中向可扩展机器人自动化的转变提供了正当性。