生成时间: 2026-09-21 22:55:50 (UTC+8); Arxiv 发布时间: 2026-09-21 20:00 EDT (2026-09-22 08:00 UTC+8)

今天共有 36 篇相关文章

Keyword: reinforcement learning

COAL-SQL: Coverage-Guided Augmentation and Failure-Driven Learning for Text-to-SQL Post-Training

COAL-SQL：覆盖引导增强与失败驱动的文本转SQL后培训学习

• Authors: Qifeng Cai, Xuanguang Pan, Hao Liang, Chang Xu, Wentao Zhang
• Subjects: Subjects: Computation and Language (cs.CL); Databases (cs.DB)
• Arxiv link: https://arxiv.org/abs/2609.20842
• Pdf link: https://arxiv.org/pdf/2609.20842
• Abstract Text-to-SQL translates natural-language questions into executable SQL queries, but open-source large language models still require task-specific post-training for complex, real-world SQL generation. Effective post-training requires both training data that cover the capabilities demanded by the target task and a learning strategy that enables the model to acquire them. Existing datasets provide valuable supervision but incompletely cover SQL structures, while augmentation methods typically expand data without identifying structural gaps. Moreover, supervised fine-tuning (SFT) or reinforcement learning (RL) alone cannot dynamically address weaknesses exposed during training. We propose COAL-SQL, a unified framework combining Coverage-Guided Augmentation (CGA) and Failure-Driven Learning (FDL). CGA uses greedy selection to identify SQL structures missing from the original dataset and constructs complementary examples, improving structural coverage. FDL retains GRPO as the main optimization objective while supplying targeted supervision for unsolved examples. At the step level, it applies SFT to verified reasoning traces generated by a strong LLM for accumulated failures. At the epoch level, it retrieves structurally related examples based on accumulated failures to create targeted practice, helping the model acquire the corresponding SQL capabilities. With only 12,600 distinct post-training examples, COAL-SQL achieves 64.9% execution accuracy on the BIRD development set and outperforms baselines trained at comparable scale. The code is available at this https URL.
• 中文摘要 文本转SQL将自然语言问题转换为可执行的SQL查询，但开源大型语言模型仍需针对任务的后期训练以生成复杂的真实世界SQL。有效的后期训练既需要涵盖目标任务所需能力的训练数据，也需要学习策略使模型能够获得这些能力。现有数据集提供了有价值的监督，但未能完全覆盖SQL结构，而增强方法通常扩展数据却未发现结构性缺口。此外，仅靠监督微调（SFT）或强化学习（RL）无法动态解决训练过程中暴露的弱点。我们提出了COAL-SQL，一个结合覆盖引导增强（CGA）和失败驱动学习（FDL）的统一框架。CGA通过贪婪选择识别原始数据集中缺失的SQL结构，构建互补示例，提升结构覆盖率。FDL保留GRPO作为主要优化目标，同时为未解决的实例提供有针对性监督。在步骤层面，它对强大型语言模型生成的累积失败推理痕迹应用SFT。在纪元层面，它基于累积失败检索结构相关示例，创建针对性练习，帮助模型获得相应的SQL能力。仅有12,600个不同的训练后实例，COAL-SQL在BIRD开发集上实现了64.9%的执行准确率，并且优于同等规模训练的基线。代码可在此 https 网址获取。

Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Traces

通过自发的深度研究推广，提升深度研究和长上下文的能力

• Authors: Zihan Wang, Hao Wang, Boyuan Jiang, Yiqun Zhang, Shi Feng, Xiaocui Yang, Yiwen Ye, Jianghang Lin, Xiaozhong Ji, Jinghao Lin, Kai Wu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.20844
• Pdf link: https://arxiv.org/pdf/2609.20844
• Abstract Deepresearch (DR) agents interact with real-world web environments through multi-turn search and visit, causing their contexts to grow rapidly over time. We observe that, even after DR Agentic Reinforcement Learning (DR-RL), 61.6% of the model's remaining prediction errors can still be attributed to insufficient long-context understanding, including longcontext hallucination and failures in cross-document evidence integration. It motivates us to further break the bottleneck of DR-RL by strengthening the model's long-context ability. However, effective LongContext training requires more than simply increasing context length. To bridge the data gap, we propose `DR Rollouts to LongContext-QA (DR-to-Long)'. The method repurposes DR-RL trajectories, which naturally contain search histories, visited webpages, evidence snippets, and final-answer supervision. It then replaces the compact snippets and webpage summaries in each trajectory with the full contents of their corresponding URLs, producing substantially longer multi-document contexts while preserving the original evidence relationships. Building on DR-to-Long, we introduce DLD (DR -> LongQA -> DR)-RL. DLD-RL first performs a short DR-RL stage to collect rollout trajectories, which are then converted into LongQA instances at zero annotation cost. The model is subsequently optimized with LongQA-RL to strengthen LongContext ability, followed by full DR-RL to continue improving its DR capability. Experiments show that DLD-RL outperforms standard DR-RL by 7.3% on three Deepresearch benchmarks and improves performance by 13.5% on three long-context benchmarks.
• 中文摘要 深度研究（DR）代理通过多回合搜索和访问与现实世界网络环境互动，使其上下文随时间快速增长。我们观察到，即使在DR智能强化学习（DR-RL）之后，模型剩余的预测误差中仍有61.6%可归因于长期上下文理解不足，包括长上下文幻觉和跨文档证据整合失败。这激励我们进一步打破DR-RL的瓶颈，增强模型的长上下文能力。然而，有效的LongContext训练不仅仅需要增加上下文长度。为弥合数据差距，我们提出“DR向LongContext-QA（DR到Long）的推广”。该方法重新利用DR-RL轨迹，后者自然包含搜索历史、访问网页、证据片段和最终答案监督。然后，它用对应URL的完整内容替换每个轨迹中的紧凑片段和网页摘要，从而在保留原始证据关系的同时，产生更长的多文档上下文。基于DR-to-Long，我们引入了DLD（DR -> LongQA -> DR）-RL。DLD-RL首先执行一个短的DR-RL阶段收集推广轨迹，然后以零注释成本转换为LongQA实例。随后模型经过LongQA-RL优化以增强LongContext能力，接着进行完整的DR-RL，以持续提升DR能力。实验显示，DLD-RL在三个Deepresearch基准测试中比标准DR-RL高出7.3%，在三个长上下文基准测试中性能提升13.5%。

BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence

BI-Agent与BI-Bench：迈向自动化端到端商业智能

• Authors: Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Databases (cs.DB)
• Arxiv link: https://arxiv.org/abs/2609.20886
• Pdf link: https://arxiv.org/pdf/2609.20886
• Abstract Business intelligence (BI) is a cornerstone of enterprise decision-making and is widely used by enterprise users in software such as Power BI and Tableau. In traditional BI workflows, users need to prepare data by (1) identifying relevant tables, (2) performing data transformations, and (3) building join relationships, before they can (4) answer their business questions. These steps can be complex and time-consuming, making BI challenging. Given the strong capabilities of large language models (LLMs) in working with data, we study their ability to answer BI questions end-to-end, without requiring users to manually perform the tedious preparation steps. To do this, we harvest a large collection of real-world BI projects from public sources, and manually extract pairs of (questions, ground-truth answers) from real user dashboards. The resulting benchmark, BI-Bench, is the first benchmark to systematically study LLMs' ability on end-to-end BI. We find that even frontier LLMs perform poorly on BI-Bench, with less than 50% accuracy. To address their limitations, we design a tool-augmented BI-Agent that decomposes BI workflows into subtasks on structured data, such as search, join, and transform, and orchestrates specialized data management methods across BI stages. Furthermore, we develop a post-training framework that synthesizes training trajectories from real BI projects, enabling BI-Agent to be further post-trained using both supervised fine-tuning (SFT) and reinforcement learning (RL). BI-Agent achieves substantial accuracy gains of up to 40 percentage points with vanilla LLMs, and post-trained BI-Agent yields gains of up to 30 points. Our results highlight the importance of combining tool-augmented reasoning with domain-specific post-training in complex BI workflows, and point to promising directions for future research.
• 中文摘要 商业智能（BI）是企业决策的基石，被企业用户广泛应用于Power BI和Tableau等软件中。在传统BI工作流中，用户需要通过（1）识别相关表、（2）执行数据转换和（3）建立连接关系来准备数据，才能（4）回答业务问题。这些步骤可能复杂且耗时，使BI具有挑战性。鉴于大型语言模型（LLMs）在处理数据方面的强大能力，我们研究它们在端到端回答BI问题的能力，而无需用户手动完成繁琐的准备步骤。为此，我们从公共来源收集大量真实BI项目，并手动从真实用户仪表盘中提取成对（问题、真实答案）。最终的基准测试BI-Bench是首个系统性研究LLM在端到端BI能力的基准测试。我们发现即使是前沿的大型语言模型在BI-Bench上的表现也不佳，准确率不到50%。为解决其局限性，我们设计了一个工具增强的BI-Agent，将BI工作流分解为结构化数据的子任务，如搜索、连接和转换，并在BI阶段协调专门的数据管理方法。此外，我们开发了一个后培训框架，综合了真实BI项目的训练轨迹，使BI-Agent能够通过监督微调（SFT）和强化学习（RL）进一步进行后期训练。BI-Agent在普通LLM中实现高达40个百分点的显著准确性提升，后训练BI-Agent则提升高达30个百分点。我们的结果强调了在复杂BI工作流中将工具增强推理与领域特定后训练结合的重要性，并为未来研究指明了有前景的方向。

Continuous Delayed-Memory Stochastic Gradient Descent and Continuous-Time Reinforcement Learning from History of Astrophysical Time Series Studies

连续延迟记忆随机梯度下降与连续时间强化学习，源自天体物理时间序列研究的历史

• Authors: Debartha Paul, Juncheng Yi
• Subjects: Subjects: Machine Learning (cs.LG); Applications (stat.AP)
• Arxiv link: https://arxiv.org/abs/2609.20906
• Pdf link: https://arxiv.org/pdf/2609.20906
• Abstract Quasars are luminous objects in the universe that exhibit stochastic brightness variations encoding information about the supermassive black holes powering them, and modeling these variations from ground-based survey data time series, known as light curves, is a statistical challenge. This paper reviews how stochastic differential equations (SDEs) have been adapted with neural network parameterizations to overcome this challenge in history. We create the Continuous-Delayed-Memory Stochastic Gradient Descent which depend on the past state of the discrete iteration process. We performed the simulation on some 2-dimensional landscape and observed some wider-exploration and more precise convergent behavior compared to Vanilla SGD by adjusting hyperparameters. Besides, we proposed a reinforcement learning structure with continuous time policy gradients for exploratory policies without solving HJB PDE, and we show that its optimality conditions recover the Gibbs policy of previous works.
• 中文摘要 类星体是宇宙中明亮的天体，它们表现出随机亮度变化，编码了驱动其的超大质量黑洞信息，而从地面巡天数据时间序列（称为光变曲线）中建模这些变化是一项统计学上的挑战。本文回顾了随机微分方程（SDEs）如何通过神经网络参数化被改造以克服历史上的这一挑战。我们创建了依赖离散迭代过程过去状态的连续延迟记忆随机梯度下降。我们在一些二维地景上进行了模拟，通过调整超参数观察到相比原版SGD更广泛的探索和更精确的收敛行为。此外，我们提出了一种具有连续时间策略梯度的强化学习结构，用于探索性策略而不求解HJB偏微分方程，并证明其最优条件能够恢复之前工作的Gibbs策略。

Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specifications

高效贝叶斯自适应强化学习与时间逻辑规范

• Authors: Jonathan Hau, Alessandro Abate
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.20954
• Pdf link: https://arxiv.org/pdf/2609.20954
• Abstract We present a novel end-to-end model-based Reinforcement Learning (RL) algorithm for efficient policy synthesis under given Linear Temporal Logic (LTL) specifications (e.g., safety or reachability) in unknown environments. To do so, a Limit-Deterministic B{ü}chi Automaton (LDBA) representation of the LTL task is synchronised with a Bayes-Adaptive Markov Decision Process (BAMDP) representation of the environment, which allows us to leverage an enhanced exploration-exploitation trade-off that is achieved via Bayesian RL, as opposed to traditional non-Bayesian approaches. We further propose a novel Bayes-Adaptive Monte-Carlo Planning (BAMCP) algorithm to allow for approximate Bayes-optimal strategy synthesis in the synchronised BAMDP construct. A range of finite- and infinite-horizon task experiments demonstrate the effectiveness of our approach in terms of both property satisfaction and sample efficiency, when compared to traditional model-free approaches. Additional ablation studies also successfully highlight the value of the novel BAMCP algorithm in comparison to classical BAMCP for LTL task satisfaction. Finally, we also showcase a successful application of our approach for \textit{cautious} RL, namely to reduce the number of task violations incurred during policy training.
• 中文摘要 我们提出了一种新颖的端到端基于模型的强化学习（RL）算法，用于在未知环境中，在给定的线性时序逻辑（LTL）规范（如安全性或可达性）下高效综合策略。为此，将LTL任务的极限确定性Bü}chi自动机（LDBA）表示与环境的贝叶斯自适应马尔可夫决策过程（BAMDP）表示同步，使我们能够利用通过贝叶斯RL实现的增强探索-利用权衡，而非传统的非贝叶斯方法。我们还提出了一种新颖的贝叶斯自适应蒙特卡洛规划（BAMCP）算法，以实现同步BAMDP构造中的近似贝叶斯最优策略综合。一系列有限和无限视野任务实验展示了我们方法在性质满足度和样本效率方面的有效性，相较于传统无模型方法。其他消融研究也成功凸显了新颖BAMCP算法相较于经典BAMCP算法在LTL任务满足度方面的价值。最后，我们还展示了我们方法在\textit{cautious}强化学习中的成功应用，即减少策略培训过程中任务违规数量。

ASGARD: Action-Space Guard for UAV Resilience via Reinforcement Learning

阿斯加德：通过强化学习实现无人机韧性行动空间守卫

• Authors: Mohsen Salehi, Karthik Pattabiraman
• Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.20982
• Pdf link: https://arxiv.org/pdf/2609.20982
• Abstract Reinforcement learning (RL) controllers have been recently adopted for Unmanned Aerial Vehicles (UAV) navigation and control. However, they are susceptible to action-space attacks that overwrite the action commands after the policy generates them and before the actuators execute them. While most existing defenses target attacks on the policy's inputs, those addressing action-space attacks retrain the policy at training time and are not resilient to corrupted actions at runtime. We propose ASGARD, a two-phase teacher-student pipeline for making RL-based UAV control resilient to action-space attacks. In the teacher phase, an encoder combines the UAV's physical state with action-attack-related privileged information to produce an action-attack-aware latent that trains the RL control policy and a monitor that outputs corrected action commands to the actuators. In the student phase, both the encoder and the monitor are trained via supervised learning from their teacher counterparts to run on-board using only the UAV's physical state history. We evaluate ASGARD across attack scenarios targeting different action commands on UAV. We find that ASGARD is resilient to action-space attacks and completes the missions despite the attack. We further find that ASGARD generalizes to unseen attacks and remains resilient against stealthy attacks.
• 中文摘要 强化学习（RL）控制器最近被无人机（UAV）导航和控制所采用。然而，它们容易受到行动空间攻击的攻击，这些攻击在策略生成后、执行器执行前会覆盖行动命令。虽然大多数现有防御手段针对策略输入的攻击，但针对行动空间攻击的防御系统在训练时重新训练策略，且在运行时对破坏行为缺乏弹性。我们提出了ASGARD，这是一个两阶段的师生流程，旨在使基于RL的无人机控制对行动空间攻击具备韧性。在教师阶段，编码器将无人机的物理状态与与行动攻击相关的特权信息结合，生成一个动作攻击感知潜在器，训练强化学习控制策略，并输出更正动作指令给执行器。在学生阶段，编码器和监视器都通过教师的监督学习，仅利用无人机的物理状态历史进行机载运行。我们评估了针对无人机不同动作指令的攻击场景。发现ASGARD对行动空间攻击具有韧性，尽管遭受攻击仍能完成任务。我们还发现ASGARD可推广到看不见的攻击，并对隐形攻击保持韧性。

DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

DEXTERA：从单一图像到可部署的灵活操作，通过真实到模拟再到真实

• Authors: Jin Wu, Lianjie Yuan, Zeyan Sun, Yuanyuan Lei, Disi A, Bicheng Han, Fangzhou Xia
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21045
• Pdf link: https://arxiv.org/pdf/2609.21045
• Abstract Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures.
• 中文摘要 收集真实机器人数据进行灵巧操作既昂贵又耗时。虽然高精度物理模拟器支持可扩展的数据综合和策略学习，但手动构建部署型数字孪生仍需大量劳动力，且残留的视觉、几何和动力学差距阻碍了可靠的模拟到真实传输。我们介绍DEXTERA，一个自动化的实物到模拟再实的框架，将单幅RGB图像转换为可部署的策略，实现四个统一阶段的灵活操作：（1）将单图像场景分解为静态高斯背景和交互式刚性或关节资产，并结合VLM推断物理参数;（2）度量场景全局对齐、对象规范化及形态平衡机器人校准;（3）可扩展模拟器任务原始构建、虚拟现实远程操作和以对象为中心轨迹综合;以及（4）支持模仿学习和强化学习的共享多模态策略接口。我们评估了13对任务-具身、2个灵活机器人平台和6种策略架构的DEXTERA。实验结果表明，DEXTERA相比生成基线实现了更优的视觉真实性和3D几何重建，而跨域轨迹重放验证了强的物理交互一致性。此外，仅模拟训练策略支持可行的零样本真机器人部署，而模拟-真实共训练则显著提升了多种策略架构中物理策略的平均成功率，从29.2%提升至61.9%。

Design of Adaptive PID Controller Based On Asynchronous Advantage Actor Critic Learning Method for QuadCopter Control

基于异步优势演员批评者学习方法设计的自适应PID控制器，用于四旋翼控制

• Authors: Ali Jokar, Aria Alasty
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.21082
• Pdf link: https://arxiv.org/pdf/2609.21082
• Abstract Quadcopters offer great utility in many applications, but their nonlinear nature and disturbance sensitivity present great control challenges. Basic PID controllers are generally not sophisticated enough to cope with these complexities. This paper suggests a control system that integrates the Asynchronous Advantage Actor-Critic (A3C) algorithm with a PID controller for quadcopter attitude and trajectory tracking. The A3C controller uses parallel agents to optimize PID parameters dynamically using a neural network. A system identification module for the complementary system makes predictions about system states for optimal control policy. The proposed framework was compared with a standard actor-critic (A2C) model. Simulation results verify that they both track accurately. However, the A3C-based controller converges much more for the loss function, as evidenced by reward figures and loss curves, demonstrating better parameter optimization. This shows that A3C-based approach results in improved performance for the control of quadcopter, effectively integrating reinforcement learning and traditional control to achieve higher adaptability.
• 中文摘要 四旋翼在许多应用中具有极大实用价值，但其非线性特性和扰动敏感性带来了巨大的控制挑战。基础的PID控制器通常还不够复杂，无法应对这些复杂性。本文建议了一种将异步优势演员-批判者（A3C）算法与PID控制器集成的控制系统，用于四旋翼机的姿态和轨迹跟踪。A3C控制器利用并行代理动态优化PID参数，利用神经网络实现。为互补系统的系统识别模块对系统状态进行预测，以实现最佳控制策略。所提出的框架与标准的演员-批判者（A2C）模型进行了比较。模拟结果验证了两者都准确跟踪。然而，基于A3C的控制器在损失函数上的收敛度更大，这一点通过奖励数据和损耗曲线可见一斑，显示出更好的参数优化。这表明基于A3C的方法提升了四旋翼控制性能，有效整合了强化学习与传统控制，实现了更高的适应性。

Dynamics-Induced Commitment in Learning-Based Robotic Penalty Kicks

基于学习的机器人点球中的动力学诱导承诺

• Authors: Ruize Geng, Hao E. Zhang, Yisen Li, Yikai Wang, H. Eric Tseng, Ding Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21100
• Pdf link: https://arxiv.org/pdf/2609.21100
• Abstract Learning in robotic games is constrained not only by strategic information but also by what the body can still execute. We study this coupling in a hierarchical humanoid-quadruped penalty system in which game-level self-play policies command fixed soccer whole-body controllers (S-WBCs). The humanoid shooting skill is initialized from self-collected motion-capture data, whereas the quadruped saving skill is learned by reinforcement learning. We introduce dynamics-induced commitment mapping (DIC-Map), a body-grounded analysis that estimates continuation capability, identifies the first persistent loss of a terminal alternative, and tests whether the remaining interaction admits a reduced zero-sum game. For symmetric terminal alternatives, the reduced game yields a closed-form bound on optimal strategy concentration determined by the responder's value of deferring. We further show that, when the responder acts through an estimator, equal response values eliminate the direct terminal-allocation gradient and leave an estimator-mediated first-order learning channel. Experiments locate commitment about 0.29 s before contact, and changing only ball speed shifts deferral coverage. Across four responder policies, replacing the estimator raises save rate from 0.240 to 0.472, whereas a comparable gain in read accuracy obtained by waiting raises it only to 0.246. Posterior analysis is used for the equilibrium comparison because the available coverage terms are observational proxies. Project website: this https URL
• 中文摘要 机器人游戏中的学习不仅受战略信息限制，还受身体仍能执行的内容限制。我们在一个层级类人生物四足惩罚系统中研究这种耦合，其中游戏级自玩策略控制固定的全身足球控制器（S-WBC）。类人生物射击技能由自我收集的动作捕捉数据初始化，而四足救援技能则通过强化学习学习。我们引入了动力学诱导承诺映射（DIC-Map），这是一种基于身体的分析，估计持续能力，识别终端替代方案的首次持续丢失，并测试剩余互动是否允许一个简化零和博弈。对于对称终端替代方案，简化博弈给出由响应者延迟值决定的最优策略集中度的闭合界限。我们还进一步表明，当响应者通过估计器行动时，相等的响应值消除了直接终端分配梯度，留下由估计器介导的一阶学习通道。实验在接触前约0.29秒定位承诺，仅改变球速会改变延迟覆盖率。在四种响应者策略中，替换估计器可将保存率从0.240提升至0.472，而等待获得的读取准确率相当提升仅为0.246。均衡比较采用后验分析，因为可用的覆盖项为观测代理。项目网站：此 https URL

REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement

REFINEPPO：通过迭代动作精炼学习持续控制策略

• Authors: Sachini Weerasekara, Sagar Kamarthi, Jacqueline Isaacs
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.21108
• Pdf link: https://arxiv.org/pdf/2609.21108
• Abstract Deep reinforcement learning (DRL) has achieved strong performance across a wide range of continuous-control problems. These continuous-control policies, however, are often defined as direct mappings from an observed state to an action or action distribution, requiring a single feed-forward network to construct an optimal control decision in one pass. While effective, this formulation leaves little opportunity for the policy to reconsider or progressively improve an action once an initial prediction has been formed. In this work, we explore an alternative approach: rather than learning only to directly predict an action, can a policy learn to iteratively improve one, and can this iterative process provide advantages during policy learning? We introduce Iterative Action Refinement (IAR), an iterative action-construction method that constructs control actions through a sequence of learned residual corrections. Starting from an initial proposal, a shared refinement network repeatedly conditions on the observed state and the current action proposal, allowing each refinement step to revise the action constructed by preceding steps. The final refined proposal is then used to determine the action executed by the agent. We integrate this iterative action-construction mechanism with Proximal Policy Optimization (PPO), yielding REFINEPPO. We evaluate REFINEPPO across 14 benchmark control tasks, complemented by controlled ablations of refinement depth and update schedules and analyses aimed at understanding why iterative refinement is effective. Across these environments, REFINEPPO matches or exceeds the performance of standard PPO while demonstrating faster convergence on several tasks.
• 中文摘要 深度强化学习（DRL）在广泛的连续控制问题中取得了优异表现。然而，这些连续控制策略通常被定义为从观察到的状态直接映射到动作或动作分布，需要单一前馈网络在一次内构建最优控制决策。虽然有效，但该表述在形成初始预测后，几乎没有给策略重新考虑或逐步改进动作的机会。本研究探讨了另一种方法：策略不仅仅学习直接预测动作，还能否学会迭代改进，这种迭代过程是否能在策略学习中带来优势？我们引入了迭代动作细化（IAR），这是一种通过一系列学习残差修正构建控制动作的方法。从初始提案开始，共享细化网络反复对观察到的状态和当前动作提案进行条件，允许每个细化步骤修正前一步构建的动作。最终的精炼提案用于确定代理执行的动作。我们将迭代动作构建机制与近端策略优化（PPO）整合，生成REFINEPPO。我们评估REFINEPPO在14个基准控制任务中的应用，辅以精细深度的受控消融以及更新计划和分析，旨在理解迭代精细为何有效。在这些环境中，REFINEPPO性能与标准PPO相当甚至超过，同时在多个任务上展现出更快的收敛速度。

MetaPusher: Meta Learning and Planning for Nonprehensile Manipulation of Unseen Objects with Rapid Online Adaption

MetaPusher：基于在线快速适应，实现非抓握式操作的元学习与规划

• Authors: Donghyung Lee, Seyedali Golestaneh, Jaskrit Singh, Zhuoyun Zhong, Athanasios Kapoutsis, Constantinos Chamzas
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21122
• Pdf link: https://arxiv.org/pdf/2609.21122
• Abstract Manipulating previously unseen objects remains challenging, as their dynamics depend on latent physical properties, such as friction and mass distribution, that cannot be inferred from perception alone. Prior experience across objects can provide an initial estimate of unseen object dynamics, but this estimate remains uncertain and can degrade further during sim-to-real transfer. Adapting the dynamics through interaction can progressively refine the estimation, however, updating the model may invalidate the planned trajectory. Successful and efficient manipulation therefore requires both rapid dynamics adaptation and a planning strategy that can incorporate this evolution. In this work, we introduce MetaPusher, a meta-learning and adaptive planning framework for nonprehensile manipulation of unseen objects without prior object-specific interactions. A meta-learned dynamics model rapidly adapts from interactions during task execution, while an adaptive kinodynamic planner updates long-horizon plans by reusing and refining its existing search tree. This coupling enables manipulation and adaptation without a separate data collection phase. We evaluate MetaPusher on unseen objects in simulation and in sim-to-real scenarios, comparing against fine-tuning and active learning methods, MPPI-based control, and a reinforcement learning policy. It achieves lower prediction error and improves task success rate by up to 20%.
• 中文摘要 操作此前未见物体依然具有挑战性，因为它们的动力学依赖于潜在的物理属性，如摩擦力和质量分布，这些特性无法仅凭感知推断。跨对象的先前经验可以提供对未见物体动态的初步估计，但该估计仍不确定，且在模拟到现实的转移过程中可能进一步恶化。通过交互调整动态可以逐步精炼估计，但更新模型可能会使计划轨迹失效。因此，成功且高效的操作需要快速的动态适应和能够纳入这种演变的规划策略。本研究介绍了MetaPusher，一种元学习和自适应规划框架，用于在没有事先对象特定交互的情况下，非抓握式操作看不见的物体。元学习动力学模型在任务执行过程中迅速适应相互作用，而自适应运动动力规划器通过重用和完善现有搜索树来更新长期计划。这种耦合使操作和适应无需单独数据收集阶段即可实现。我们在模拟和模拟到现实场景中对未见对象进行评估，并与微调和主动学习方法、基于MPPI的控制以及强化学习策略进行比较。它实现了更低的预测误差，并将任务成功率提升了多达20%。

SAGE: Safety-Aligned Gradient Enforcement for Human--Robot Collaboration

SAGE：人机协作的安全对齐梯度执行

• Authors: Yisen Li, Hao Zhang, Ruize Geng, Yves Tseng, Ding Zhao, H. Eric Tseng
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21130
• Pdf link: https://arxiv.org/pdf/2609.21130
• Abstract Multi-party human-robot collaboration poses a dual challenge: robot decisions should remain interpretable and auditable, while executed actions must satisfy safety constraints during physical interaction. Combining explainable decision-tree policies with control-barrier-function (CBF) filtering provides a promising architecture but creates two learning mismatches in multi-agent reinforcement learning. Safety projection changes the action applied to the environment, while the coupled proposal graph can misalign independently optimized actor updates with a team-level update. We present safety-aligned gradient enforcement (SAGE) to address both mismatches. Its shield-annealed internalization layer (SAIL) uses a differentiable finite-penalty proposal map while retaining the exact CBF quadratic program for execution, preserving constraint-normal sensitivity to internalize repeatedly active safety constraints. Team-averaged Lyapunov policy optimization (TALO) constructs a team-aware update reference and applies a Lyapunov half-space correction to regulate independent actor updates. Physical experiments with two humanoid robots and a human partner demonstrate deployment feasibility. Across nine simulation scenarios, SAGE achieves a 71.0% success rate with 0.5 collision steps per thousand environment steps. Ablations show that direct CBF filtering reduces collision frequency by 98.5% but decreases success from 67.3% to 59.3%. SAIL reduces proposal violation by 48.8% and proposal-execution correction by 85.2%, while TALO reduces the update-consistency gap by 50.8%.
• 中文摘要 多方人机协作面临双重挑战：机器人决策应保持可解释性和可审计性，而执行的动作必须满足物理交互中的安全约束。将可解释的决策树策略与控制-障碍-函数（CBF）过滤结合，提供了有前景的架构，但也在多智能体强化学习中产生了两种学习不匹配。安全预测改变了应用于环境的动作，而耦合提案图则可能使独立优化的演员更新与团队级更新错位。我们提出了安全对齐梯度强制（SAGE）来解决这两种不匹配问题。其盾退火内部化层（SAIL）使用可微的有限惩罚提案映射，同时保留精确的CBF二次程序执行，保持约束-正态敏感性，以内化反复激活的安全约束。团队平均的Lyapunov策略优化（TALO）构建团队感知更新参考，并应用Lyapunov半空间修正来调节独立演员的更新。使用两台类人机器人和一名人类伙伴的物理实验证明了部署的可行性。在九个模拟场景中，SAGE实现了71.0%的成功率，每千环境步中碰撞步数为0.5。消融显示，直接CBF过滤将碰撞频率降低98.5%，但成功率从67.3%降至59.3%。SAIL将提案违规率降低48.8%，提案执行修正减少85.2%，而TALO则将更新一致性差距减少了50.8%。

Stability-aware Residual Reinforcement Learning Framework for Robotic Manipulator Disturbance Compensation

机器人机械臂干扰补偿的稳定性感知残差强化学习框架

• Authors: Jihong Kim, Joonhyuk Kwon, Hwa Soo Kim, TaeWon Seo, Hyung-Tae Seo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21307
• Pdf link: https://arxiv.org/pdf/2609.21307
• Abstract Although conventional controllers and disturbance observers (DOBs) are the standard for precision tracking in manipulators, they suffer from parameter uncertainty, nonlinear friction, and compound disturbances. This study proposes a residual reinforcement learning DOB framework that pairs an analytical observer with an RL policy. The deterministic baseline operates within a reliable region, whereas the RL policy explicitly targets the residuals that the model cannot capture. To make this compensation disturbance-aware, an estimator network aligns the observation history with a privileged disturbance context, organizing the latent space by disturbance regime and enabling rapid adaptation across disturbance transitions. To guarantee stability, we derived and enforced a state-dependent action bound on the RL policy from an input-to-state stability (ISS) analysis such that the closed loop provably confines the tracking error to a certified envelope for arbitrary policy outputs. Experiments on a 6-DOF manipulator demonstrated consistent improvements in disturbance estimation and tracking, including a 27.8% tracking-error reduction on real hardware under zero-shot sim-to-real transfer and a 38.0% reduction under a base-vibration disturbance that was not observed during training.
• 中文摘要 尽管传统控制器和扰动观察器（DOBs）是机械臂精密跟踪的标准，但它们存在参数不确定性、非线性摩擦和复合扰动问题。本研究提出了一种残差强化学习DOB框架，将分析观察者与强化学习策略配对。确定性基线在可靠区域内运行，而强化策略明确针对模型无法捕捉的残差。为使补偿干扰感知，估计器网络将观测历史与特权扰动上下文对齐，按扰动状态组织潜在空间，实现扰动转变间的快速适应。为保证稳定性，我们通过输入到状态稳定性（ISS）分析推导并强制执行了基于状态的动作约束，使闭环可证明将跟踪误差限制在任意策略输出的认证包络内。在6自由度操作手上的实验显示，扰动估计和跟踪有持续提升，包括在零时模拟到实传输下真实硬件跟踪误差减少27.8%，在训练中未观察到的基底振动扰动下减少38.0%。

LEMCA: LLM-Guided Synthesis of Efficient Mode-Switching Control Architectures

LEMCA：高效模式切换控制架构的LLM引导综合

• Authors: Arjun Krishna, Vincent Pacelli, Dinesh Jayaraman
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21319
• Pdf link: https://arxiv.org/pdf/2609.21319
• Abstract Physical control tasks in the natural world, such as driving or object manipulation, frequently exhibit dramatic variations in sensory and compute complexity over time. Correspondingly, a natural resource-efficient choice for robot control is to dynamically switch between control modes with varying resource allocations. However, such "mode-switching controllers" (MSCs) have historically required laborious, expert-driven design and synthesis for each new task. Driven by these design difficulties, modern robotic control architectures often fall back to a wasteful "monolithic" one-size-fits-all structure, where resource allocation is permanently anchored to the hardest, most resource-intensive task phases. To facilitate the design of performant yet efficient MSCs, we propose LLM-Guided synthesis of Efficient Mode-Switching Control Architectures (LEMCA). LEMCA represents MSC designs as interpretable programs to be iteratively refined in an evolutionary loop. To evaluate design fitness, we propose MSC-compatible extensions of automated controller synthesis approaches, such as reinforcement learning in simulation. LEMCA then leverages the semantic priors, reasoning, and coding capabilities of Large Language Models (LLMs) to iteratively edit controller modes, their corresponding sensory-compute resource allocations, and mode transitions. Our experiments across diverse control benchmarks show that LEMCA consistently discovers strategies that surpass the Pareto frontier of monolithic designs by reclaiming wasted resources during "easy" task phases. LEMCA thus presents an automated, low-effort path to synthesize resource-efficient MSC designs.
• 中文摘要 自然界中的物理控制任务，如驾驶或物体操作，常常在感官和计算复杂度随时间显著变化。相应地，机器人控制的自然资源高效选择是动态切换不同资源分配的控制模式。然而，这类“模式切换控制器”（MSC）历来需要为每个新任务进行繁琐且专家驱动的设计和综合。受这些设计难题驱动，现代机器人控制架构常常退回浪费的“单一”一刀切结构，资源分配永久固定在最困难、最耗资源的任务阶段。为促进高效且高效MSC的设计，我们提出了高效模式切换控制架构（LEMCA）的LLM引导综合。LEMCA将MSC设计表示为可解释的程序，在进化循环中迭代完善。为评估设计适应度，我们提出了MSC兼容的自动化控制器综合方法扩展，如仿真中的强化学习。LEMCA随后利用大型语言模型（LLM）的语义先验、推理和编码能力，迭代编辑控制器模式、对应的感官计算资源分配及模式转换。我们在多种控制基准测试中的实验表明，LEMCA通过回收“简单”任务阶段的浪费资源，持续发现超越单体设计帕累托边界的策略。因此，LEMCA提供了一条自动化、低成本的综合资源高效MSC设计路径。

Deep Reinforcement Learning with Buffered Quantile Objectives

带缓冲分位数目标的深度强化学习

• Authors: Mohammad Alipour-vaezi, Sajad Khodadadian
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.21327
• Pdf link: https://arxiv.org/pdf/2609.21327
• Abstract Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision-making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.
• 中文摘要 基于分位数的强化学习通过优化累计回报分布的指定分位数，提供了一种可解释的风险敏感决策方法。尽管如此，在点分位数目标下学习仍具有挑战性：分位数在返回分布的小扰动下可能发生突变，精确的分位数敏感规划需要计算量较大的分布优化。缓冲较低的分位数通过平均目标水平以下的邻近分位数，缓解了前者的难题，提供更平滑的替代，同时保持了基础的点分位数目标。然而，基于该原则的现有方法仍基于模型，依赖显式回归律规划，其适用范围仅限于小型表格问题。我们开发了Deep-BQRL，一种无模型的分布强化学习框架，将缓冲分位数学习扩展到神经功能近似。该方法直接从抽样转移学习条件返回分位数，从学习到的分位函数相关区域构建缓冲动作得分，并利用集合不一致来指导探索。增强输入表示使所学策略能够响应轨迹信息，而无需显式地重现精确规划所需的分位数-状态递归。在资产出售最优停止问题和滑溜的FrozenLake上的实验中，比较了Deep-BQRL与基于模型的UCB-BQRL以及表格PPO和TRPO实现。在资产销售中，Deep-BQRL在报告的目标水平上，平均累计点-分位数政策差距小于PPO和TRPO，而UCB-BQRL则保持最小的差距。学习到的停止决策也随目标分位数变化，提供了方法风险敏感行为的可解释性示例。

Co-Evolving Zero-Day Jamming: Adaptive Attack Synthesis and Graph Attention-Based Online Detection

共同演进的零日干扰：自适应攻击综合与基于注意力的图在线检测

• Authors: Ghilas Aissou, Rémi A. Chou, Taejoon Kim
• Subjects: Subjects: Information Theory (cs.IT); Artificial Intelligence (cs.AI); Signal Processing (eess.SP)
• Arxiv link: https://arxiv.org/abs/2609.21334
• Pdf link: https://arxiv.org/pdf/2609.21334
• Abstract Effective evaluation of zero-day jamming detectors requires robust adversarial models. However, existing attack models often assume prior knowledge of the target receiver, limiting their utility as evaluation benchmarks. On the detection side, existing detectors fail to capture the global temporal-spectral structure of jamming behavior and cannot differentiate zero-day strategies as they emerge. This paper addresses these limitations through a two-pronged framework. First, an online detection framework is introduced that combines a graph attention network (GAT) for temporal-spectral representation learning with Dirichlet process (DP)-means clustering. This framework jointly classifies known and discovers zero-day strategies within a unified learning objective. Second, an inference-driven reinforcement learning (RL) jammer is proposed as an adversarial benchmark. The jammer treats the target receiver as a black-box, infers the detector state via hypothesis testing, and optimizes the trade-off between attack impact and stealth. Simulation results show that the proposed RL jammer outperforms benchmarks, achieving 33% higher attack efficacy and 67% higher stealth. The proposed detection framework against the proposed RL jammer is shown to achieve 20% higher detection accuracy than the benchmarks.
• 中文摘要 有效评估零日干扰检测器需要稳健的对抗模型。然而，现有攻击模型通常假设对目标接收方具备先验信息，限制了其作为评估基准的效用。在检测方面，现有检测器未能捕捉干扰行为的全局时间-频谱结构，且无法在零日策略出现时区分开来。本文通过双管齐下的框架解决了这些局限性。首先，引入了一个在线检测框架，结合了用于时间频谱表示学习的图注意力网络（GAT）与狄利克雷过程（DP）均值聚类。该框架在统一的学习目标内共同分类并发现零日策略。其次，提出了一种推理驱动强化学习（RL）干扰器作为对抗基准。干扰器将目标接收器视为黑箱，通过假设检验推断检测器状态，并优化攻击影响与隐蔽性之间的权衡。模拟结果显示，所提出的强化学习干扰器表现优于基准测试，攻击效能提升33%，隐蔽性提升67%。针对拟议强化学习干扰器的检测框架显示，其检测准确率比基准测试高出20%。

ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL

ArenaFlow：从轨迹排名到开放式智能体强化学习的层级信用传播

• Authors: Qiang Zhang, Ruixue Ding, Fanrui Zhang, Xi Chen, Boli Chen, Shihang Wang, Yinfeng Huang, Yi Zheng, Pengjun Xie, Kaipeng Zhang, Jiawei Liu, Zheng-Jun Zha
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.21378
• Pdf link: https://arxiv.org/pdf/2609.21378
• Abstract Reinforcement learning has substantially improved large language model (LLM) agents in verifiable domains, but remains difficult to apply to open-ended agent tasks, where solutions are diverse and reliable scalar rewards are hard to obtain. Recent pairwise evaluation methods alleviate reward discrimination collapse by replacing pointwise scoring with relative preferences. However, they still compress rich comparative feedback into a single trajectory-level reward, obscuring decisive intermediate steps and preventing successful behaviors from being consolidated into reusable skills. We propose ArenaFlow, a hierarchical credit propagation framework for open-ended agent reinforcement learning. ArenaFlow leverages tournament-based relative ranking to derive trajectory-level reward signals. Each comparison is further equipped with structured reflective evaluation, which reveals three types of supervision: pivotal success steps, reusable strategy skills, and usage attribution of retrieved skills. At the step level, ArenaFlow propagates trajectory-level advantages to high-confidence pivotal steps according to tournament survival depth, enabling more targeted optimization of local reasoning behaviors. At the skill level, ArenaFlow estimates skill utility from group-level usage attribution and maintains a global skill memory through utility-aware updating, pruning, and retrieval. The resulting high-utility skills further serve as policy priors for future exploration. Extensive experiments validate ArenaFlow's effectiveness on open-ended agent tasks.
• 中文摘要 强化学习显著提升了可验证领域的大型语言模型（LLM）代理，但仍然难以应用于开放式代理任务，因为解决方案多样且可靠的标量奖励难以获得。近期的成对评估方法通过用相对偏好替代点数评分，缓解了奖励判别崩溃。然而，它们仍将丰富的比较反馈压缩到单一轨迹级奖励中，掩盖了决定性的中间步骤，阻止成功行为被整合为可重用技能。我们提出了ArenaFlow，一种用于开放式代理强化学习的分层学分传播框架。ArenaFlow利用基于锦标赛的相对排名来推导轨迹级奖励信号。每次比较还配备了结构化反思评估，揭示了三种监督类型：关键成功步骤、可复用策略技能和检索技能的使用归因。在步骤层面，ArenaFlow将轨迹层优势传递给高置信度的关键步骤，基于锦标赛生存深度，实现局部推理行为的更有针对性优化。在技能层面，ArenaFlow通过群体层级使用归因估算技能效用，并通过效用感知的更新、修剪和检索维持全局技能记忆。由此产生的高效用技能还作为未来探索的政策先验。大量实验验证了ArenaFlow在开放式代理任务上的有效性。

DPed-VLN: A Benchmark for Socially Compliant Vision-and-Language Navigation in Dynamic Pedestrian Environments

DPed-VLN：动态步行环境中社会合规的视觉与语言导航基准

• Authors: Haojie Dai, Xiangyi Wang, Liuyi Wang, Kai Sheng, Zongtao He, Chengju Liu, Wei Ye, Qijun Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21504
• Pdf link: https://arxiv.org/pdf/2609.21504
• Abstract Vision-and-language navigation (VLN) has advanced rapidly in static indoor environments, but robots operating in human-populated spaces must ground language while responding to moving pedestrians and social-safety constraints. We present DPed-VLN, a Habitat 3.0 benchmark for dynamic-pedestrian VLN that couples 33,093 navigation episodes with paired global and prior-augmented instructions, ORCA-controlled humanoid pedestrians, socially constrained expert paths, and metrics that jointly assess navigation efficiency and social safety. DPed-VLN separates ordinary goal-oriented route guidance from prior-augmented instructions that expose dynamic-pedestrian cues for controlled analysis. To instantiate the benchmark, we introduce DPet (Dynamic Pedestrian-aware Network), a pedestrian-aware policy network trained with reinforcement learning and imitation learning. We further adapt representative state-of-the-art VLM-based navigation models, including NaVILA and StreamVLN, to DPed-VLN through LoRA fine-tuning. Experiments show that LoRA adaptation improves zero-shot VLM baselines in several success and safety metrics, especially reducing StreamVLN's collision rate. Among the evaluated methods, DPet-RL achieves the highest SR, SPL, and STL.
• 中文摘要 视觉与语言导航（VLN）在静态室内环境中发展迅速，但机器人在人居住空间中运行时必须在应对移动行人和社会安全限制时保持语言基础。我们展示了DPed-VLN，这是Habitat 3.0动态步行VLN基准，结合了33,093个导航片段与成对的全局和事先增强指令、ORCA控制的人形行人、社会约束的专家路径，以及联合评估导航效率和社会安全的指标。DPed-VLN将普通目标导向路线指引与先前增强的指令区分开来，后者会暴露动态行人线索供受控分析。为实现基准，我们引入了DPet（动态步行感知网络），这是一个通过强化学习和模仿学习训练的行人感知政策网络。我们进一步通过LoRA微调，将代表性的基于VLM的先进VLM导航模型，包括NaVILA和StreamVLN，适配到DPed-VLN。实验显示，LoRA的适应在多个成功和安全性指标上提升了零射击VLM基线，特别是降低了StreamVLN的碰撞率。在评估的方法中，DPet-RL实现了最高的SR、SPL和STL。

IncentRL: The Trade-Off Between Preference Guidance and Task Performance

IncentRL：偏好指导与任务绩效之间的权衡

• Authors: Xuening Wu, Yanlan Kang, Shenqin Yin
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.21525
• Pdf link: https://arxiv.org/pdf/2609.21525
• Abstract Preference-based reward shaping can guide reinforcement learning, but adding preference signals to the reward may unintentionally change the task being optimized. We address this problem with IncentRL, a framework that introduces preference guidance while explicitly characterizing its effect on external-task performance. IncentRL adds a Kullback--Leibler (KL) penalty between a specified outcome distribution and a preferred distribution. For finite discounted Markov decision processes with bounded shaping costs, we derive an external-value perturbation bound, establish a sufficient strict-action-gap condition for preserving the original optimal policy, and characterize the large-weight regime through discounted cumulative preference cost. Exact examples clarify the limits of these guarantees, including tied optima and support mismatch. We study a practical implementation using a hand-designed, distance-based outcome proxy, a fixed preference distribution, and score-weighted coefficient search. On MiniGrid DoorKey-8x8, the reported three-seed mean success rate after two million training steps reaches 98\% with coefficient 0.01, compared with 90.5\% for the reported zero-coefficient baseline, while the search progressively shifts toward smaller coefficients. Together, these results provide a principled view of the central trade-off in preference-based RL: using additional guidance to improve learning without excessively distorting the original task objective. The current experiments remain descriptive and do not yet isolate KL shaping from simpler alternatives.
• 中文摘要 基于偏好的奖励塑造可以指导强化学习，但向奖励添加偏好信号可能会无意中改变被优化的任务。我们通过IncentRL解决了这一问题，该框架引入了偏好指导，同时明确描述了其对外部任务表现的影响。IncentRL在指定结果分布与偏好分布之间增加了Kullback-Leibler（KL）惩罚。对于具有有限折现的马尔可夫决策过程，具有有界的塑形成本，我们推导了一个外部值扰动界限，建立一个足够的严格行动缺口条件以保持原始最优策略，并通过贴现累计偏好成本来表征大权重区间。具体示例阐明了这些保证的局限，包括最优和支持不匹配。我们研究了使用手工设计的基于距离的结果代理、固定偏好分布和得分加权系数搜索的实际实现。在MiniGrid DoorKey-8x8上，报告的三种子平均成功率在两百万次训练步后达到98/%，系数为0.01，而零系数基线为90.5%，而搜索结果逐渐向更小系数方向转变。综合来看，这些结果为基于偏好的强化学习提供了原则性权衡：使用额外指导提升学习，同时不过度扭曲原始任务目标。当前实验仍具描述性，尚未将基层逻辑塑造与更简单的替代方案区分开来。

OneBid: A Unified Auto-Bidding Foundation Model for Diverse oCPX Advertising Scenarios

OneBid：针对多种oCPX广告场景的统一自动竞价基础模型

• Authors: Yewen Li, Peng Jiang, Yitian Li, Pengfei Lv, Xialong Liu, Peng Jiang, Qingpeng Cai
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.21550
• Pdf link: https://arxiv.org/pdf/2609.21550
• Abstract Auto-bidding is central to computational advertising, where strategies must maximize advertisers' conversion value under economic constraints. It has evolved from rule-based controllers to reinforcement learning and generative methods such as Decision Transformer (DT). Yet these methods increasingly mismatch the prevailing optimized cost-per-X (oCPX) paradigm, which spans heterogeneous scenarios (e.g., registration, purchase), each served by a separate model, leading to fragmented pipelines and underexploring cross-scenario modeling. Inspired by foundation models like LLMs, unifying these oCPX scenarios into one model raises three challenges: multi-objective control, scalable capacity under strict latency, and safe offline policy improvement. We present OneBid, a unified auto-bidding foundation model that learns a reusable backbone from heterogeneous oCPX logs and adapts it to scenario-specific deployments via offline post-training. Building on DT, OneBid extends single Return-to-Go conditioning to two atomic signals, Return-to-Go for conversion value and Cost-to-Go for cost ratio, plus value-aware regularization on next-action prediction. To absorb distributional heterogeneity, we design a sequence-level Mixture-of-Experts architecture, where shared experts encode cross-scenario knowledge and sparsely-routed experts capture scenario-specific patterns at low latency, yielding consistent scaling with model size and data. During post-training, we align the backbone with scenario preferences via Critic-guided Relative Offline Policy optimization (CROP): a learned critic scores candidate actions group-relatively, avoiding the unsafe online exploration of GRPO-style fine-tuning while constraining policy shift to reduce OOD risk. Validated via online A/B tests and fully deployed at Kuaishou, OneBid delivers an overall +2.2% ADVV gain on oCPX Ads, peaking at +13.1% in the ROAS scenario.
• 中文摘要 自动竞价是计算广告的核心，策略必须在经济约束下最大化广告主的转化价值。它已从基于规则的控制器发展到强化学习和生成方法，如决策变换器（DT）。然而，这些方法越来越与现有的优化单次成本（oCPX）范式不匹配，后者跨越异构场景（如注册、购买），每个场景由独立模型服务，导致管道碎片化，跨场景建模缺乏探索。受大型语言模型（LLM）等基础模型启发，将这些oCPX场景统一为一个模型带来了三个挑战：多目标控制、严格延迟下的可扩展容量，以及安全的离线策略改进。我们提出了OneBid，一种统一的自动竞价基础模型，它从异构oCPX日志中学习可重用骨干，并通过离线后训练将其适配为场景特定部署。基于DT，OneBid将单一回归条件扩展到两个原子信号：回归到目的地用于转化价值，成本到去成本用于成本比，以及对下一行动预测的价值感知正则化。为吸收分布异质性，我们设计了一个序列级专家混合架构，共享专家编码跨场景知识，稀疏路由专家以低延迟捕捉场景特定模式，实现模型规模和数据的一致扩展。在后期训练中，我们通过Critic引导的相对离线策略优化（CROP）将骨干与场景偏好对齐：学习过的CRITA对候选动作进行相对评分，避免了GRPO式微调的不安全在线探索，同时约束策略调整以降低OOD风险。通过在线A/B测试验证并在快手全面部署，OneBid在oCPX广告的整体ADVV增长率为+2.2%，在ROAS场景中峰值为+13.1%。

Potential-Field Action Representation for Reinforcement Learning in Contact-Rich Manipulation

在接触富丰富操作中强化学习中的势场动作表示

• Authors: Xinyu Liu, Gökhan Solak, Arash Ajoudani
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.21609
• Pdf link: https://arxiv.org/pdf/2609.21609
• Abstract Model-free reinforcement learning can acquire contact-rich robotic manipulation skills through trial-and-error interaction, but it often requires the policy to learn both task strategy and low-level motion generation. In this setting, the action representation is critical because it determines how policy outputs are converted into robot motion, shaping both exploration and physical execution. Direct Cartesian command interfaces require the policy to generate motion at every decision step, coupling task-level adaptation with continuous low-level control and increasing the learning burden. We propose PA-RL, a reinforcement-learning framework that uses artificial potential fields as the action representation. Instead of commanding motion directly, the policy adapts the parameters of an energy-like potential field, which generates a state-dependent guidance direction executed through a Cartesian impedance controller. We evaluate PA-RL on peg-in-hole insertion, a representative contact-rich task with nonlinear dynamics and discontinuous contact transitions. In simulation, PA-RL is compared with Cartesian velocity, Cartesian pose, and variable-impedance action spaces using the same RL algorithm. PA-RL is the only method to reach a 100% evaluation success rate within the allotted training time, while the best baseline reaches 92.6%. It also reduces joint-torque variation by 55.4% and Cartesian acceleration variation by 70.8% relative to the best baseline, without explicit motion-quality penalties in the reward. The simulation-trained policy further completes 9/9 real-robot insertions without fine-tuning, demonstrating the deployment feasibility of the learned potential-field interface.
• 中文摘要 无模型强化学习可以通过试错互动获得富接触的机器人操作技能，但通常需要策略同时学习任务策略和低级运动生成。在此环境中，动作表示至关重要，因为它决定策略输出如何转化为机器人运动，影响探索和物理执行。直接笛卡尔命令接口要求策略在每个决策步骤生成运动，将任务级适应与连续低级控制结合，增加学习负担。我们提出PA-RL，一种强化学习框架，使用人工势场作为动作表示。该策略不直接指挥运动，而是调整能量样势场的参数，生成通过笛卡尔阻抗控制器执行的状态依赖引导方向。我们在孔中插入（peg-in-hole insertion）上评估PA-RL，这是一项具有非线性动力学和不连续接触转移的代表性丰富接触任务。在仿真中，PA-RL与笛卡尔速度、笛卡尔姿态和可变阻抗作用空间使用相同的强化学习算法进行比较。PA-RL是唯一在规定训练时间内实现100%评估成功率的方法，最佳基线可达92.6%。它还将关节扭矩变化降低了55.4%，笛卡尔加速度变化减少了70.8%，且奖励中无明确的运动质量惩罚。模拟训练策略进一步完成了9/9个真实机器人插入且未微调，展示了学习到的势场界面的部署可行性。

SynthDemo-RL: Breaking the Zero-Reward Barrier in VLA Adaptation with LLM-Guided Synthetic Demonstrations

SynthDemo-RL：通过LLM引导的合成演示突破VLA适应中的零奖励障碍

• Authors: Hiroaki Kingetsu, Hiroaki Kurihara, Kaoru Yokoo, Kenji Fukumizu, Manohar Kaul
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.21650
• Pdf link: https://arxiv.org/pdf/2609.21650
• Abstract Fine-tuning Vision-Language-Action (VLA) models commonly relies on human teleoperation demonstrations, while reinforcement learning (RL) with sparse binary rewards faces an exploration challenge when successful trajectories are rarely sampled. We propose SynthDemo-RL, a teacher-student framework in which an automated teacher converts simulator-privileged state into successful manipulation trajectories, a VLA student is distilled from them by supervised fine-tuning (SFT), and PPO with binary task-success rewards refines the student. We study reward coverage, the fraction of tasks for which at least one success is observed under the fixed evaluation protocol, as a complement to the average success rate. On LIBERO-PRO, a public benchmark of perturbed LIBERO tasks for which no demonstrations exist, 27 of 57 scored tasks are at exactly 0% success for a pi_0.5 policy fine-tuned on the original LIBERO tasks. Direct PPO from this policy, under the same PPO recipe and the same RL compute as SynthDemo-RL's refinement stage, rescues 10 of these 27 tasks and leaves 17 at 0%. SynthDemo-RL, with 50 synthesized trajectories per task and no new human demonstrations, rescues all 27 and reaches average success rates of 97.8% and 97.1% on the Position and Task axes of LIBERO-PRO, respectively. On standard LIBERO, the same pipeline reaches 96.0% with no human demonstrations, within 1.7 points of pi_0.5 trained on 50 human demonstrations per task. We further validate the pipeline on RoboTwin 2.0 and verify that trajectories from a policy trained in a MuJoCo twin execute open-loop on a physical robot.
• 中文摘要 微调视觉-语言-行动（VLA）模型通常依赖人工远程操作演示，而带有稀疏二元奖励的强化学习（RL）则在成功轨迹很少被采样时面临探索挑战。我们提出了SynthDemo-RL，这是一种师生框架，其中教师自动将模拟器特权状态转换为成功的操作轨迹，VLA学生通过监督微调（SFT）从中提炼出学生，PPO配合二元任务成功奖励则细化学生。我们研究奖励覆盖率，即在固定评估协议下至少观察到一项任务成功的比例，作为平均成功率的补充。在LIBERO-PRO中，一个无演示的受扰动LIBERO任务公开基准，57个评分任务中有27个在原始LIBERO任务上微调后的pi_0.5策略成功率正好为0%。该策略直接PPO，在与SynthDemo-RL细化阶段相同的PPO方案和相同的PPO条件下，挽救了27个任务中的10个，剩17个任务处于0%。SynthDemo-RL为每个任务提供50条合成轨迹且无新的人工演示，挽救了全部27个任务，在LIBERO-PRO的位置轴和任务轴上分别达到97.8%和97.1%的平均成功率。在标准LIBERO上，同一流程在无人工演示的情况下达到96.0%，每个任务训练50次人工演示后，成功率在1.7 pi_0%以内。我们进一步验证了RoboTwin 2.0上的流水线，并验证了MuJoCo双胞胎训练策略的轨迹在物理机器人上实现开环运行。

DRT: Dense Reasoning Trace for Efficient and Grounded Multimodal Reasoning

DRT：密集推理迹，用于高效且扎根的多模态推理

• Authors: Wan Xu, Yuanfan Guo, Kevin Han, LaLa Chen, Wangmeng Zuo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.21675
• Pdf link: https://arxiv.org/pdf/2609.21675
• Abstract Despite the remarkable progress in Multimodal Large Language Models (MLLMs), prevailing Chain-of-Thought (CoT) paradigms remain confined to the natural-language expression space. Consequently, they inherently incur excessive linguistic overhead, leading to information dilution and weak visual grounding. To address this challenge, we propose Dense Reasoning Trace (DRT), a paradigm that departs from natural-language-centered CoT by expressing reasoning as compact structured traces, which include concise intermediate states with symbolic connectors and disentangle visual observations from logical deductions. First, we introduce the Dense Trace Initialization to internalize the DRT reasoning mode into the model, substantially improving token efficiency while preserving visual evidence. To further enable the model to faithfully capture the logical relations within traces, we propose the Trace-Grounded Reinforcement Learning framework, which builds reference traces through a tri-perspective verification pipeline and employs Trace-Grounded GRPO with structured rewards, encouraging the model to generate concise DRT-style traces with reduced hallucination and stronger logical grounding. Extensive experiments on challenging reasoning benchmarks show that DRT achieves 5.5$\times$ token efficiency improvement while improving 1.3 accuracy points over the Qwen3-VL baseline. These findings suggest that complex multimodal reasoning may not require verbose natural-language traces, opening a more efficient path for next-generation MLLMs. Our code and data are available at: this https URL
• 中文摘要 尽管多模态大型语言模型（MLLMs）取得了显著进展，主流的思维链（CoT）范式仍局限于自然语言表达空间。因此，它们本质上承担过多的语言开销，导致信息稀释和视觉基础薄弱。为应对这一挑战，我们提出了稠密推理痕迹（Dense Reasoning Trace，DRT）这一范式，它通过以紧凑的结构化痕迹表达推理，包括带符号连接的简明中间状态和与逻辑推理的视觉观察分离。首先，我们引入稠密痕迹初始化，将DRT推理模式内化到模型中，显著提升令牌效率，同时保留视觉证据。为了进一步使模型能够忠实捕捉痕迹中的逻辑关系，我们提出了Trace-Grounded Reinforcement Learning框架，该框架通过三视角验证流水线构建参考轨迹，并采用带有结构化奖励的Trace-Grounded GRPO，鼓励模型生成简洁的DRT风格轨迹，减少幻觉并增强逻辑基础。对挑战性推理基准的大量实验显示，DRT在Qwen3-VL基线上提升了5.5$\times$的令牌效率，并提高了1.3个准确点。这些发现表明，复杂的多模态推理可能不需要冗长的自然语言痕迹，为下一代MLLM开辟了更高效的路径。我们的代码和数据可在以下网站获取：此 https URL

GEM-MPC: Balancing Exploration and Exploitation through Expert-Guided Planning

GEM-MPC：通过专家指导规划平衡勘探与开发

• Authors: Alvaro Serra-Gomez, Thomas Moerland
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.21735
• Pdf link: https://arxiv.org/pdf/2609.21735
• Abstract Effective exploration in high-dimensional continuous control remains a central challenge in reinforcement learning. Planning-based methods address this by combining online planning with learned policies and value functions, but their components can become misaligned during training: learned sampling policies may diverge from planner behavior, while planning distributions stored in replay become stale as the model and value function evolve. Reanalysis can refresh these targets, but at substantial computational cost. We propose GEM-MPC, an MPPI-based reinforcement learning method that improves the interaction between planning and learning. GEM-MPC uses MPPI to combine a policy trained to clone the planner with a KL-regularized policy that explores around it, providing complementary exploitation and guided exploration within planning. We further introduce Gated Prior Distillation, which selectively learns from stored planning distributions only when they provide a better target than the current prior, reducing the impact of stale planning data without requiring full reanalysis. Across continuous-control benchmarks, GEM-MPC consistently outperforms existing planning-based baselines under lower computational budgets.
• 中文摘要 高维连续控制的有效探索仍然是强化学习中的核心挑战。基于规划的方法通过结合在线规划与学习的策略和价值函数来解决这个问题，但它们的组成部分在训练过程中可能出现错位：学习到的抽样策略可能与规划者的行为产生偏差，而存储在回放中的规划分布随着模型和价值函数的演变而变得陈旧。重新分析可以刷新这些目标，但这会带来巨大的计算成本。我们提出了GEM-MPC，一种基于MPPI的强化学习方法，能够改善规划与学习之间的互动。GEM-MPC利用MPPI将训练用于克隆规划器的策略与围绕规划器进行探索的KL正则化策略结合起来，在规划中提供互补的利用和引导探索。我们还进一步引入了门控先行蒸馏，该方法仅在存储的规划分布提供比当前先验更好的目标时，选择性地学习，从而减少陈旧规划数据的影响，而无需完全重新分析。在连续控制基准测试中，GEM-MPC在较低计算预算下持续优于现有基于规划的基线。

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

从预训练到熟练：以最小人工干预实现长期操作的现实子任务强化学习

• Authors: Sichang Su, Benjamin Yang, Zhiyun Deng, Boyuan Liang, Yip Fun Yeung, Zelin Wang, Lingfeng Sun
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.21788
• Pdf link: https://arxiv.org/pdf/2609.21788
• Abstract A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.
• 中文摘要 预训练机器人基础策略可能执行大部分长视野任务，但在少数关键子任务中反复失败。收集额外的全任务演示以进行监督微调（SFT），操作员需要重复策略已表现良好的行为。强化学习（RL）微调提供了弥合这一差距的有前景路径，但现有方法仅用稀疏奖励难以解决长视野任务。我们提出了PARTS（基于RL的策略适应目标子任务），这是一个现实世界的子任务RL框架，集中练习于这些瓶颈，同时允许训练推广以最小限度的人为干预进行。冻结的预训练策略在执行过程中提供名义动作，而代理生成的选择器和成功验证器激活残余修正并提供局部结果奖励。这些奖励支持从成功的子任务中学习，即使完成任务成功率较高。训练结合在线强化学习与成功重加权再训练，每个重新训练的残余策略都会被重新部署以积累更多经验。人类在设置过程中识别瓶颈，并在需要时执行物理重置。在双手YAM和单臂Franka任务中，PARTS分别将完成任务成功率从32%提升到61%，从50%提升到95%，平均每个任务只需数十分钟的真实RL展开。与现有的现实强化学习微调方法相比，PARTS在相同机器人推广预算下，将全任务成功率提升超过25%，同时需要更少的人力参与。

Beyond Kinematics: Benchmarking Simulation Fidelity for Muscle-Driven Imitation Learning

超越运动学：肌肉驱动模仿学习的模拟忠真度基准测试

• Authors: Ayah G. Ahmad, Claire E. Borden, Maegan Tucker
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21909
• Pdf link: https://arxiv.org/pdf/2609.21909
• Abstract In this work, we conduct a systematic comparison of two state-of-the-art motion-imitation reinforcement learning (MIRL) pipelines, one built on SCONE/HyFyDy and one built on MuJoCo/MyoSim. HyFyDy emphasizes physiological realism through detailed musculotendon modeling, while MuJoCo prioritizes computational efficiency and scalable policy learning. While recent work has demonstrated that both pipelines reproduce human kinematics with high fidelity, it remains unclear if they accurately capture the underlying neuromuscular behavior that produced the movement. This limitation is particularly important for robotic assistive-device design and control, where outcome measures such as muscle activation patterns and metabolic cost are often used as optimization targets. To conduct a systematic comparison, our work compares both pipelines using a common set of human motion-capture and electromyography (EMG) measurements. The results find that while both pipelines produce similar kinematics with relative accuracy, the muscle activations from HyFyDy are more aligned with the experimental EMG, as supported by the average pooled (RMSE, r) values for muscle activations from HyFyDy and MuJoCo: (0.164, 0.4) and (0.344, 0.11), respectively. While we conclude that the more advanced physiological realism of HyFyDy currently makes it more suitable for musculoskeletal modeling, both require further development to bring physiological realism to GPU-parallelizable simulation environments and advance robotic assistive device design.
• 中文摘要 本研究系统比较了两条最先进的运动模拟强化学习（MIRL）流程，一条基于SCONE/HyFyDy，另一条基于MuJoCo/MyoSim。HyFyDy通过详细的肌肉腱建模强调生理真实性，而MuJoCo则优先考虑计算效率和可扩展策略学习。尽管近期研究表明两条管道都能高保真地重现人体运动学，但它们是否准确捕捉产生运动的神经肌肉行为仍不明确。这一限制对机器人辅助设备设计与控制尤为重要，因为肌肉激活模式和代谢成本等结果指标常被用作优化目标。为进行系统比较，我们的工作使用一组通用的人体动作捕捉和肌电图（EMG）测量对两条流程进行比较。结果发现，虽然两条管道在运动学上产生相似且相对准确，但HyFyDy的肌肉激活更接近实验EMG，这一点也得到了HyFyDy和MuJoCo肌肉激活的平均合并值（RMSE， r）支持：分别为（0.164， 0.4）和（0.344， 0.11）。虽然我们得出结论，HyFyDy更先进的生理真实性使其更适合肌肉骨骼建模，但两者都需要进一步发展，以将生理真实性带入GPU可平行化的模拟环境，并推动机器人辅助设备的设计。

What Should We Ask Next? Retrieval-Aware Question Learning under Partial Evidence

接下来我们应该问什么？部分证据下的检索感知问题学习

• Authors: Lyucheng Qian, John Yuehan Zhang, Pingyu Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.21924
• Pdf link: https://arxiv.org/pdf/2609.21924
• Abstract Interactive retrieval under partial evidence is a sequential information-acquisition problem: an agent must decide which question will create the most useful evidence for the next retrieval update. Existing systems train this decision by imitating an offline ordering of candidate QA pairs, although question value is determined by the response it elicits and its downstream effect on retrieval. We establish that candidate discriminativeness and perceived usefulness provide weak supervision for this objective, then introduce RAVEL, a retrieval-aware online reinforcement learning framework for interactive person re-identification. RAVEL initializes from supervised question generation, observes the current Top-4 candidates directly, and optimizes the question policy with rank feedback from the full question-answer-retrieval loop. Experiments on Interactive-PEDES show that RAVEL delivers progressively stronger retrieval performance across five interaction rounds. Further analysis shows that RAVEL reallocates the questioning budget toward localized open-ended attributes, which provide more useful retrieval evidence and yield the largest gains on initially difficult queries.
• 中文摘要 部分证据下的交互式检索是一个顺序信息获取问题：代理必须决定哪个问题能为下一次检索更新创造最有用的证据。现有系统通过模拟候选QA配对的离线排序来训练该决策，尽管问题价值由其引发的反应及其对检索的后续影响决定。我们确立候选辨别力和感知效用对该目标提供了薄弱的监督，随后引入RAVEL，一种基于检索感知的在线强化学习框架，用于交互式个体再识别。RAVEL从监督问题生成初始化，直接观察当前前四候选人，并通过完整的问答-检索循环中的排名反馈优化问题策略。在交互式PEDES上的实验表明，RAVEL在五轮交互中呈现出逐步增强的检索性能。进一步分析显示，RAVEL将提问预算重新分配到局部开放式属性，这些属性提供了更有价值的检索证据，并在最初困难的查询中获得最大收益。

Learning to Move Cities: Deep Meta-Models and Reinforcement Policies for Calibration and Control in Urban Networks

学习移动城市：城市网络校准与控制的深度元模型与强化策略

• Authors: Adewumi Augustine Adepitan, Christopher J. Haruna, Oluwasegun Adegoke, Ayooluwatomiwa Ajiboye, Oluwatobi Oluwasakin
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.21945
• Pdf link: https://arxiv.org/pdf/2609.21945
• Abstract Urban transportation networks present complex optimization challenges spanning calibration of high-fidelity simulators and real-time operational control. This paper presents a shared latent-space framework that connects simulator calibration and reinforcement learning control through a common learned representation of urban traffic dynamics. First, we develop a combinatorial MLP-autoencoder architecture that learns low-dimensional manifolds linking simulator inputs (origin-destination demand, network parameters) to outputs (travel times, congestion patterns), enabling efficient Bayesian optimization for calibration. This approach demonstrates superior sample efficiency compared to traditional dimension reduction methods, achieving better fit to observational data within fixed computational budgets. Second, we implement a deep Q-learning agent with experience replay and target networks to optimize dynamic traffic assignment through scheduling and routing adjustments. In empirical evaluations on benchmark networks, our approach reduces system-wide travel times by up to 51% compared to baseline operations. The learned latent representation is not only used to reduce the dimensionality of Bayesian calibration, but is also incorporated into the reinforcement learning state representation, allowing the control policy to operate on compressed and calibrated traffic dynamics. This shared latent-space formulation provides a unified pathway from simulator calibration to adaptive operational control within intelligent transportation systems. Our results highlight the transformative potential of deep learning methods in urban mobility planning and management, particularly for large-scale networks where traditional optimization approaches face computational bottlenecks.
• 中文摘要 城市交通网络面临复杂的优化挑战，涵盖高保真模拟器的校准和实时操作控制。本文提出了一个共享潜空间框架，通过城市交通动力学的共同学习表示，连接模拟器校准与强化学习控制。首先，我们开发了一种组合MLP-自编码器架构，学习将模拟器输入（起讫需求、网络参数）与输出（行进时间、拥堵模式）连接的低维流形，实现高效的贝叶斯校准优化。该方法在样本效率上优于传统降维方法，在固定计算预算内更好地拟合观测数据。其次，我们实现了深度Q学习代理，具备经验回放和目标网络，通过调度和路由调整优化动态流量分配。在基准网络的实证评估中，我们的方法使系统范围的行进时间比基线操作减少了高达51%。学习到的潜在表示不仅用于降低贝叶斯校准的维度，还被纳入强化学习状态表示中，使控制策略能够在压缩和校准后的交通动态下运行。这种共享潜空间表述为智能交通系统中从模拟器校准到自适应操作控制提供了统一路径。我们的结果凸显了深度学习方法在城市出行规划与管理中的变革潜力，尤其是在传统优化方法面临计算瓶颈的大规模网络中。

Automata-Theoretic Verification of Interval Markov Decision Processes

自动机理论验证区间马尔可夫决策过程

• Authors: Sarvin Bahmani, Soumyajit Paul, Sven Schewe, Sadegh Soudjani, Ashutosh Trivedi
• Subjects: Subjects: Formal Languages and Automata Theory (cs.FL); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.21966
• Pdf link: https://arxiv.org/pdf/2609.21966
• Abstract Interval Markov decision processes (IMDPs) provide a natural framework for modeling stochastic systems with uncertain transition probabilities, represented by probability intervals and resolved adversarially. Such uncertainty arises naturally, for example, when the transition model is learned from finite data or obtained through model-based reinforcement learning. In this paper, we study the automata-theoretic verification of IMDPs against rich temporal specifications, including all LTL specifications, by considering the broader class of {\omega}-regular objectives. We show that classical automata-theoretic verification techniques extend to IMDPs, but with a sharp distinction determined by the structure of the transition intervals. For stable IMDPs, where either the upper bound is zero or the lower bound is strictly positive, verification reduces to ordinary MDP analysis and can be carried out using the standard automata used in that setting (good-for-MDP automata). For unstable IMDPs, where intervals may include zero while the upper bound is strictly positive, verification becomes game-like and requires automata whose nondeterminism can be resolved on the fly (good-for-games automata). Building on these insights, we develop algorithms for verifying {\omega}-regular specifications over IMDPs and derive probabilistic guarantees when the interval model is learned from sampled data. The resulting framework enables principled verification of stochastic systems under probabilistic model uncertainty, connecting automata-based verification with data-driven stochastic modeling.
• 中文摘要 区间马尔可夫决策过程（IMDPs）为建模具有不确定转移概率的随机系统提供了自然框架，这些迁移概率由概率区间表示，并通过对抗方式解决。例如，当过渡模型是从有限数据中学习或通过基于模型的强化学习获得的时，这种不确定性自然产生。本文通过考虑更广泛的{\omega}-正则目标类别，研究了IMDP在丰富的时间规范（包括所有LTL规范）下的自动机理论验证。我们展示了经典自动机理论验证技术也扩展到IMDP，但区别明显取决于转换区间的结构。对于稳定的IMDP，当上界为零或下界严格为正时，验证简化为普通MDP分析，可以使用该环境中使用的标准自动机（适合MDP的自动机）进行。对于不稳定的IMDP，区间可能为零而上界严格为正，验证过程如博弈般复杂，需要能够即时解析非确定性的自动机（适合游戏的自动机）。基于这些见解，我们开发了验证IMDP上{\omega}-正则规范的算法，并在从抽样数据中学习区间模型时推导出概率保证。该框架使得在概率模型不确定性下对随机系统进行原则性验证，将基于自动机的验证与数据驱动的随机建模连接起来。

$λ$-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource

$λ$-受控GRPO：将流量匹配比率不稳定性转化为预算资源

• Authors: Yufeng Wang, Parivesh Priye, Meeshawn Marathe, Ramit Pahwa
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.22041
• Pdf link: https://arxiv.org/pdf/2609.22041
• Abstract Reinforcement learning is increasingly used to align image generators with reward signals, and Flow-GRPO recently extended this paradigm to flow-matching models by treating the denoising sampler as a stochastic policy that can be optimized from reward feedback. Training in this setting is unstable in a way specific to multi-step denoising: the policy update changes systematically across denoising steps, with importance ratios drifting below one, becoming increasingly dispersed, clipping at different rates, and leaving fewer usable samples late in training. Prior work treats these effects as separate failure modes and addresses each with a hand-tuned stabilizer. We show instead that they arise from a single per-step quantity, which we call path variance. This quantity is determined exactly by the sampler's Gaussian transition kernel and can be estimated cheaply during training. This reframes instability as a resource that can be measured and budgeted rather than a collection of symptoms to repair. Our method, $\lambda$-Controlled GRPO, calibrates importance-ratio behavior from this predicted law rather than from noisy empirical statistics, and allocates gradient effort across denoising steps according to their predicted cost. The two scales governing the update are fixed by standard policy choices rather than introduced as free tuning parameters. On a text-to-image model under two reward settings, rendering difficult target text scored by optical character recognition and matching human preferences scored by a preference model, $\lambda$-Controlled GRPO improves both text accuracy and preference reward over the strongest empirical stabilizer. It also keeps late-step path variance within its intended budget, precisely where the baseline systematically overshoots. The result is a Flow-GRPO update calibrated by its own transition law rather than stabilized after instability appears.
• 中文摘要 强化学习越来越多地用于将图像生成器与奖励信号对齐，Flow-GRPO最近将该范式扩展到流匹配模型，将去噪采样器视为一种随机策略，可以从奖励反馈中优化。在这种环境中，训练在多步去噪中具有不稳定性：策略更新在去噪步骤中系统性变化，重要性比会漂移到1以下，变得越来越分散，削波速率不同，且在训练后期留下的可用样本减少。以往的工作将这些效应视为独立的失效模式，并用手工调优的稳定器处理每个模式。我们展示了它们源自单一的每步量，我们称之为路径方差。该量由采样器的高斯跃迁核精确决定，且可在训练过程中廉价估计。这将不稳定性重新定义为一种可测量和预算的资源，而非一组需要修复的症状集合。我们的方法$\lambda$-受控GRPO根据该预测定律校准重要性比行为，而非噪声经验统计，并根据降噪步骤的预测成本分配梯度努力。更新的两个尺度由标准策略选择固定，而非作为自由调优参数引入。在两个奖励设置下的文本对图像模型中，通过光学字符识别评分的困难目标文本，匹配偏好模型的人类偏好，$\lambda$-控制GRPO相比最强的实证稳定器提升文本准确性和偏好奖励。它还将后步路径方差控制在预期预算内，恰好在基线系统性超转时。结果是Flow-GRPO更新由自身过渡律校准，而非在不稳定出现后稳定。

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

CodeMidas：从代码本身扩展代理编码强化学习环境

• Authors: Bowen Ye, Lei Li, Shicheng Li, Zihao Yue, Linghao Zhang, Hanglong Lv, Yuanxin Liu, Wenhan Ma, Hao Tian, Rang Li, Jinhao Dong, Yikai Zhao, Xiangwei Deng, Hailin Zhang, Liang Zhao, Qi Liu, Lingpeng Kong, Tong Yang, Fuli Luo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.22068
• Pdf link: https://arxiv.org/pdf/2609.22068
• Abstract Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich source of such tasks, while existing methods typically rely on development artifacts such as issues and commits, limiting the range of tasks that can be extracted. To better scale RL environments, we present CodeMidas, an agentic pipeline that turns implemented functionality in existing codebases into executable RL environments using source code as its only task-specific input. CodeMidas allocates agentic compute to every stage of environment construction: agents explore implemented functionality to formulate behavioral specifications, construct tests grounded in execution of the original code, and validate and filter candidate tasks through execution checks and repeated solution rollouts. The resulting dataset has 5,545 training tasks from 3,185 open-source codebases spanning 23 programming languages and 15 technical domains. Training MiMo-V2.5 on these tasks with GRPO improves performance on all five diverse benchmarks, covering issue repair (DeepSWE + 11.7%), whole-program construction (ProgramBench +17%), and terminal work (Terminal-Bench v2.1 +8.5%). Ablations show that increasing the number of high-quality training tasks improves performance. Trajectory analysis shows the RL-trained agent demonstrates better behaviors like increasing codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.
• 中文摘要 通过强化学习（RL）训练具备编码能力的代理需要多样化且可靠的验证者任务。开源代码库提供了丰富的此类任务来源，而现有方法通常依赖开发中的工件，如问题和提交，限制了可提取的任务范围。为了更好地扩展强化学习环境，我们提出了CodeMidas，这是一个代理流水线，将现有代码库中实现的功能转化为可执行的强化学习环境，使用源代码作为唯一任务特定输入。CodeMidas为环境构建的每个阶段分配代理计算：代理探索已实现的功能以制定行为规范，构建基于原始代码执行的测试，并通过执行检查和反复的解决方案推广验证和过滤候选任务。最终数据集包含来自3185个开源代码库的5545个训练任务，涵盖23种编程语言和15个技术领域。用GRPO训练MiMo-V2.5完成这些任务，提升了五个多样化基准测试的性能，涵盖问题修复（DeepSWE + 11.7%）、整程序构建（ProgramBench +17%）和终端工作（Terminal-Bench v2.1 +8.5%）。消融分析显示，增加高质量训练任务数量能提升性能。轨迹分析显示，强化学习训练的代理表现出更好的行为，如增加代码库探索和更多样化的自我验证。这些结果确立了源代码作为构建强化学习环境的可扩展基础，从而提升不同软件任务中的编码代理。

MintAct: A Unified Visual Agent for Digital Environments

MintAct：数字环境统一视觉代理

• Authors: Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy, Kaixin Ma, Vishwesh Kirthivasan, Oğuzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.22083
• Pdf link: https://arxiv.org/pdf/2609.22083
• Abstract We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.
• 中文摘要 我们介绍MintAct，这是一系列视觉语言模型，统一了UI基础、移动端、桌面和网页的多步导航，以及可视化工具使用，训练于2B、4B和8B规模。通过对环境、数据和训练方案的精心设计，MintAct模型在所有这些能力上都能匹配到各领域专家的表现。为此，我们开发了一个可扩展的环境和强化学习（RL）基础设施。在环境方面，我们托管数百个跨异构的每域后端并发实例，既服务轨迹数据收集，也服务在线强化学习。为了实现高效且可扩展的强化学习训练，异步框架明确控制跨域训练分布，并在嘈杂环境反馈和非策略漂移下保持稳定。实验结果显示，MintAct在可比模型大小下，在多种基准测试中实现了最先进的性能（在OSWorld-Verified上达到48.9）。

Keyword: diffusion policy

Demonstration Synthesis from a Single Scan via Gaussian Splatting for Visuomotor Policy Learning

通过高斯喷涂单次扫描演示合成，用于视觉运动策略学习

• Authors: Beichen Wang, Yuen-Hei Yeung, V.R. Sridhar Devarakonda, Xuesu Xiao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21112
• Pdf link: https://arxiv.org/pdf/2609.21112
• Abstract Training a visuomotor policy calls for abundant demonstrations that closely match the target environment, yet collecting them anew remains expensive. Existing demonstration synthesis methods reduce this cost but remain constrained by high manual effort, limited visual fidelity, or heavy reliance on physics simulators. This paper introduces GaussianFactory, a high-fidelity data engine that mass-produces demonstrations with a single video scan as its only human input and no physics engine in the generation loop. Specifically, GaussianFactory reconstructs the scene as an editable 3D Gaussian Splatting (3DGS) replica and samples from the object-combination tasks the scene affords. For each task, it plans grasps and trajectories purely kinematically on the geometric reconstruction, rendering photorealistic demonstrations that visually match the target environment. Physical dynamics enter the pipeline only where contact force interactions dictate the outcome---during grasp formation, via a learned contact model pretrained once on an interaction dataset. To evaluate the downstream utility of the synthesized demonstrations, we implement an end-to-end scan-to-deployment workflow in two setups: a simulated scene that stands in for the real world to enable reproducibility, and a real-world workspace with a physical UR10e robot. In each setup, a standard diffusion policy trained solely on the synthesized demonstrations achieves 95.1% and 84.2% success rates, respectively.
• 中文摘要 训练视觉运动策略需要大量与目标环境高度匹配的演示，但重新收集这些演示仍然昂贵。现有的演示合成方法降低了这一成本，但仍受限于高人工劳动力、有限的视觉真实度或高度依赖物理模拟器。本文介绍了高斯工厂（GaussianFactory），这是一种高保真数据引擎，其大量演示仅靠单次视频扫描作为人工输入，生成循环中无物理引擎。具体来说，高斯工厂将场景重建为可编辑的3D高斯喷溅（3DGS）复制品，并采样场景提供的对象组合任务。对于每个任务，它仅通过几何重建纯粹运动学规划抓取和轨迹，呈现出与目标环境视觉匹配的逼真演示。物理动力学仅在接触力相互作用决定结果时进入流程---在抓握形成过程中，通过一次预训练的学习接触模型在交互数据集上进行。为评估综合演示的下游效用，我们在两种设置中实现了端到端扫描到部署的工作流程：一个模拟场景代替真实世界以实现可重复性，另一个是使用物理UR10e机器人的真实工作空间。在每种设置中，仅基于合成演示训练的标准扩散策略分别实现了95.1%和84.2%的成功率。

Robotic Multiphase Interaction: Manipulating Coupled Liquid and Solid Dynamics with a World Model

机器人多相相互作用：利用世界模型操控耦合的液体和固体动力学

• Authors: Yixuan Feng, Peng Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21448
• Pdf link: https://arxiv.org/pdf/2609.21448
• Abstract This work presents \textit{Robotic Multiphase Interaction (RMI)}, a setting in which liquid enters a porous material and interacts mechanically with its deforming solid skeleton. Manipulation can therefore change pore volume, expel or redistribute retained liquid, and alter grasp stability at the same time. Spilled liquid can also create safety risks in domestic and manufacturing settings. This differs from most manipulation of solid objects and from tasks that involve both liquid and solid while keeping the phases spatially separate. We study a sponge filled with water as the first RMI example. We use implicit incompressible porous flow with smoothed particle hydrodynamics as the dynamics engine and enable robotic manipulation by adding Coulomb contact memory, hybrid velocity and force regulation, and a stability gate for lifting. The resulting environment connects robot commands to changes in the coupled liquid and solid state. A world model conditioned on actions predicts how this state evolves under candidate commands, while a temporal UNet generates actions using either Diffusion Policy or rectified flow matching. Our world model reduces retained water prediction error by more than $60\%$ compared with the baseline. The best action sequence selected by the world model from policy proposals further reduces the predicted terminal water error by about half. These improvements show that modelling the coupled liquid and solid state helps the robot predict how its actions affect both the porous object and the liquid held inside.
• 中文摘要 该研究提出了\textit{机器人多相相互作用（RMI）}，即液体进入多孔材料并与其变形固体骨架机械相互作用的环境。因此，操作可以改变孔体积、排出或重新分配滞留液体，同时改变抓握稳定性。泄漏的液体还可能在家庭和制造环境中带来安全风险。这不同于大多数固体物体的操作，以及同时涉及液体和固体且保持空间分离的任务。我们研究了充满水的海绵作为RMI的第一个例子。我们使用隐式不可压缩多孔流动配合光滑粒子流体力学作为动力学引擎，并通过添加库仑接触存储器、混合速度与力调节以及提升稳定门实现机器人操作。由此产生的环境将机器人指令与液态和固体耦合状态的变化连接起来。基于动作的世界模型预测该状态在候选指令下的演变，而时间UNet则通过扩散策略或整流匹配生成动作。我们的世界模型将保留水的预测误差比基线降低超过60%美元。世界模型从政策提案中选出的最佳动作序列进一步将预测的终端水误差降低约一半。这些改进表明，液态和固态耦合有助于机器人预测其动作对多孔物体和液体的影响。

Towards Fine-Grained Object Manipulation: SAM3-Guided Visuomotor Policy with Persistent Memory Learning and Focused Visual Conditioning

迈向细粒度物体操作：SAM3引导的身体运动策略结合持续记忆学习和聚焦视觉条件反射

• Authors: Haolong Meng, Fangbo Qin, Mengchen Bai, Houwu Wang, Cirong Liu, Shan Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21621
• Pdf link: https://arxiv.org/pdf/2609.21621
• Abstract Fine-grained object (FO) manipulation requires robots to distinguish a specified FO from visually similar objects and execute actions reliably despite scene distractors. However, scene-level visual conditioning lacks explicit object selection, while category-level guidance cannot reliably distinguish FOs within the same category. We present a SAM3-guided visuomotor framework that addresses these challenges through persistent object memory and focused visual conditioning. First, we introduce FO Memory-driven SAM3 (FOM-SAM3), which learns reusable FO memory tokens from limited multi-view registration images while keeping SAM3 fully frozen. Through one-vs-rest learning, these tokens encode persistent memories for localizing target FOs and rejecting similar alternatives, which can be stored in a memory bank. Second, we propose Focused Spatial-Appearance Encoding (FSAE), which combines in-FO local appearance features with explicit bounding-box coordinates to condition action policies including Diffusion Policy (DP) and Action Chunking with Transformers (ACT). The effectiveness of the proposed FOM-SAM3 was validated on the FO-30 dataset comprising 30 physical objects across four coarse categories. Across three real-robot FO manipulation tasks, our FOM-SAM3-guided policies demonstrated robustness against distractors, discrimination ability among similar FOs, and extendibility to new FOs.
• 中文摘要 细粒度对象（FO）操作要求机器人区分指定的目标对象与视觉相似对象，并在场景干扰下可靠执行动作。然而，场景级视觉条件缺乏明确的对象选择，而类别级指导也无法可靠区分同一类别内的目标对象。我们提出了一个由SAM3引导的视觉运动框架，通过持续物体记忆和聚焦视觉条件反射来应对这些挑战。首先，我们介绍了FO内存驱动SAM3（FOM-SAM3），该系统在保持SAM3完全冻结的同时，从有限的多视角注册图像中学习可重复使用的FO记忆标记。通过一对静止学习，这些标记编码持久记忆以定位目标目标，并拒绝类似的替代方案，这些可存储在内存库中。其次，我们提出了聚焦空间外观编码（FSAE），该方法结合了FO内局部外观特征与显式边界框坐标，以条件作用策略，包括扩散策略（DP）和与变换器分块（ACT）。所提FOM-SAM3的有效性在FO-30数据集上得到了验证，数据集包含4个粗类别的30个物理对象。在三个真实机器人FO操作任务中，我们的FOM-SAM3引导策略展现了对干扰物的鲁棒性、类似目标间的辨别能力以及对新FO的可扩展性。

AcousticDiffusion: Semantically Conditioned Audio-Guided Diffusion Policy for Search-and-Rescue Assistance

声学扩散：用于搜救协助的语义条件音频引导扩散政策

• Authors: Iana Zhura, Didar Seyidov, Dmitrii Plotnikov, Hajira Amjad, Miguel Altamirano Cabrera, Dzmitry Tsetserukou
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.21792
• Pdf link: https://arxiv.org/pdf/2609.21792
• Abstract Navigating toward human callers is an important capability for rescue robots operating where visual contact is degraded or occluded. We present AcousticDiffusion, a semantically conditioned, audio-guided diffusion policy for human-directed navigation. A frozen pretrained audio recognizer processes 10.24 s windows, with speech gating and distress-aware prioritization converting recognition outputs into source-level navigation roles. Microphone-array direction-of-arrival measurements are recursively integrated into a robot-centric Bayesian bird's-eye-view belief field. Ego-motion compensation aligns successive observations, progressively constraining source position while preserving bearing-induced range uncertainty. The semantic belief, recent acoustic observations, audio features, and robot state condition a diffusion model that generates waypoint trajectories. On a synthetic-navigation validation set using recorded audio, AcousticDiffusion achieves a mean end-point bearing error of 11.20 degrees, with 91.78% of trajectories aligned within 30 degrees of the caller. Distractor rejection ranges from 89.20% to 98.99%, and the policy favors a HELP-designated caller over a competing speaker in 91.07% of windows. Deployed online on a ZSL-1 quadruped without additional retraining, it achieves a mean bearing error of 64.9 degrees, compared with 98.2 degrees for A* and 90.4 degrees for RRT, with a mean planner compute time of 6.07 ms. Despite imperfect acoustic localization, the reported mean final source distance is reduced from 3.96 m for the classical planners using ODAS-derived (Open embedded Audition System) guidance to 2.48 m, a 37.4% improvement. These results demonstrate the framework's ability to translate uncertain acoustic observations into closer approaches to human callers.
• 中文摘要 向人类呼叫者导航是救援机器人在视觉接触受阻或遮挡时的重要能力。我们介绍声学扩散，这是一种语义条件化、音频引导扩散策略，用于人类引导导航。一个冻结的预训练音频识别器处理10.24秒的窗口，语音门控和求救感知优先级将识别输出转换为源级导航角色。麦克风阵列的到达方向测量被递归集成到以机器人为中心的贝叶斯鸟瞰视角信念场中。自我运动补偿对齐连续观测，逐步限制源位置，同时保持方位引起的距离不确定性。语义信念、近期声学观测、音频特征和机器人状态条件形成扩散模型，生成航点轨迹。在使用录制音频的合成导航验证集上，声学扩散实现了平均端点方位误差11.20度，91.78%的弹道对齐在来电者30度以内。干扰物拒绝率范围为89.20%至98.99%，该政策在91.07%的窗口中更倾向于帮助指定的来电者而非竞争扬声器。在无额外训练的ZSL-1四足机上在线部署时，其平均方位误差为64.9度，而A*为98.2度，RRT为90.4度，规划器计算平均时间为6.07毫秒。尽管声学定位不完美，但使用ODAS导引（Open embedded Audition System）指导的传统规划者报告的平均最终声源距离从3.96米降至2.48米，提升了37.4%。这些结果展示了该框架能够将不确定的声学观测转化为更接近人类呼叫者的能力。