生成时间: 2026-09-22 21:15:02 (UTC+8); Arxiv 发布时间: 2026-09-22 20:00 EDT (2026-09-23 08:00 UTC+8)

今天共有 78 篇相关文章

Keyword: reinforcement learning

GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training

GRRR：解码器LLM后期训练中的重塑、旋转与布线几何

• Authors: Jianing Qi, Hao Tang, Zhigang Zhu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.22146
• Pdf link: https://arxiv.org/pdf/2609.22146
• Abstract We study how post-training changes the weights of Large Language Models (LLMs) relative to their pretrained weights. Across 12 post-training chains with supervised fine-tuning (SFT) and reinforcement learning (RL), we express each weight update in the pretrained matrix's singular value decomposition (SVD) frame. This decomposition separates the changes of three geometrically distinct components: diagonal values, which reshapes singular values; off-diagonal values, which rotates the coupling between pretrained input and output directions; and null-space values, which routes outside the matrix's original nonzero SVD core. On a math evaluation suite, we find that removing the diagonal component usually preserves most of the gains from post-training. These results suggest that post-training gains are carried primarily by reconfiguring and extending pretrained pathways rather than by substantially changing singular values of pre-trained models.
• 中文摘要 我们研究后训练如何改变大型语言模型（LLM）权重相对于其预训练权重的变化。在12条训练后链中，通过监督微调（SFT）和强化学习（RL），我们将每次权重更新表达为预训练矩阵的奇异值分解（SVD）框架。该分解分离了三个几何上不同的组成部分的变化：对角值，重塑奇异值;非对角值，旋转预训练输入与输出方向之间的耦合;以及空空间值，路由矩阵原始非零SVD核心之外。在数学评估套件中，我们发现去除对角成分通常保留了大部分后训练后的收益。这些结果表明，训练后收益主要通过重构和扩展预训练路径实现，而非通过大幅改变预训练模型的奇异值。

Beyond Task Completion: Training Capable and Safe Computer-Use Agents

超越任务完成：培训具备能力且安全的计算机使用代理

• Authors: Zeyu Kang, Zhenyun Yin, Yang Zhang, Shan He, Shanzhe Lei, Yanjiu Zhong, Xinquan Chen, Yuhong Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.22178
• Pdf link: https://arxiv.org/pdf/2609.22178
• Abstract Computer-use agents (CUAs) have made rapid progress in completing complex tasks through graphical user interfaces, yet post-training centered on task success alone does not induce reliable safety behavior. A reliable CUA must condition its execution on risk: it should complete ordinary benign tasks, avoid environmental hazards and continue when a safe completion path remains, and refuse when the goal is harmful or no safe path exists. To learn this conditional policy, we develop Safety and Capability Optimization for Policy Execution (SCOPE), which jointly post-trains a CUA for task-execution capability and safety-aware decision making. To provide aligned training data for this joint objective, we further introduce SCOPE-Gen, an automated pipeline that synthesizes verifiable capability tasks and converts them into paired environment-risk variants while preserving their original goals. Using the resulting tasks, we construct SATraj-OS, a trajectory dataset comprising capability demonstrations, safe continuations, and explicit refusals. SCOPE first learns from all three trajectory types through supervised fine-tuning and then further improves task completion through online reinforcement learning. Starting from Qwen3.5-9B, SCOPE-RL achieves a 54.17% task success rate on OSWorld and a 64.30% attack-avoidance rate on OS-BLIND, yielding the best aggregate capability--safety score of 58.80% among the evaluated agents. Ablations reveal asymmetric but complementary roles for the two forms of safety supervision: refusal trajectories account for most of the attack-avoidance gain, whereas risk-handling trajectories preserve greater task utility at comparable attack-avoidance levels.
• 中文摘要 计算机使用代理（CUA）通过图形用户界面快速完成复杂任务，但仅以任务成功为中心的后期培训并不能诱导可靠的安全行为。一个可靠的CUA必须以风险为条件执行：应完成普通无害任务，避免环境危害，当安全完成路径存在时继续;当目标有害或无安全路径时拒绝执行。为学习这一条件策略，我们开发了策略执行安全与能力优化（SCOPE），该系统联合后期训练任务执行能力和安全意识决策的CUA。为实现这一联合目标提供对齐的训练数据，我们进一步引入了SCOPE-Gen，这是一种自动化流水线，综合可验证能力任务并将其转换为成对的环境风险变体，同时保持其原始目标。利用所得任务，我们构建了SATraj-OS轨迹数据集，包含能力演示、安全延续和显式拒绝。SCOPE首先通过监督微调从三种轨迹类型中学习，随后通过在线强化学习进一步提升任务完成率。从Qwen3.5-9B开始，SCOPE-RL在OSWorld上实现了54.17%的任务成功率，在OS-BLIND上达到64.30%的攻击规避率，在评估代理中获得了58.80%的最佳综合能力-安全得分。消融揭示了两种安全监管形式在不对称但互补的作用：拒绝轨迹解释了大部分攻击避免的收益，而风险处理轨迹在相似的攻击-避免水平下保持了更高的任务效用。

A Pinch of SFT, A Dash of RL: When Reinforcement Learning Helps Long-Horizon Advertising Agents

一点SFT，一点强化学习：当强化学习帮助远景广告代理时

• Authors: Aakash Kolekar, Sahika Genc, Bunyamin Sisman, Shahriar Shariat, Shree Vandana Kachroo, Avishek Saha, Qianli Wu, Ari Singer, Benoit Dumoulin
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.22194
• Pdf link: https://arxiv.org/pdf/2609.22194
• Abstract Enterprise analytics agents solve long-horizon tool-use problems over distributed business data, requiring retrieval, reasoning, API calls, code execution, and adaptation to intermediate observations. Supervised fine-tuning (SFT) calibrates tool syntax and teacher-supported behavior, whereas reinforcement learning (RL) can explore reward-supported behaviors beyond demonstrations; applied uniformly, however, RL can perturb already-calibrated skills. We study how to balance SFT and RL under production-mirroring beta APIs. We observe that, in our controlled experiment, checkpoint trajectories retrospectively separated into three regimes: Imitation, where SFT captured reliable teacher behavior; Lift, where both stages helped; and Discovery, where useful reward-observable behavior lay outside reliable teacher support. We leverage this prospectively, using teacher support and reward-observable headroom to route features to SFT only, SFT then RL, increased RL allocation, or further environment development. Across 18 subsequent feature-specific experiments, the diagnostic predicted 15/18 observed trajectories. On GPT-OSS 120B, targeted SFT then RL produced positive point estimates on 7/8 advertiser skills relative to a frontier Control; five positive gains had paired 95% confidence intervals excluding zero, while one skill had a confidence-supported regression. The largest gain was non-disclosure (+11.27 points; 95% CI [+9.72, +12.82]). A separate SME audit surfaced that targeted RL reduces standard leakage from 11.8% to 2.9% and adversarial leakage from 22.9% to 6.8% relative to SFT while preserving actionability (86.2% to 85.7%). In a matched uniform-versus-targeted comparison with shared rewards and optimization, targeted RL improved the seven-skill mean delta from +1.62 to +3.57 while using 43% less incremental RL compute.
• 中文摘要 企业分析代理解决分布式业务数据上的长期工具使用问题，需要检索、推理、API调用、代码执行以及对中间观察的适应。监督微调（SFT）校准工具语法和教师支持行为，而强化学习（RL）则可以探索超越演示的奖励支持行为;但统一应用时，强化学习可能会扰动已校准的技能。我们研究如何在生产镜像测试API下平衡SFT和强化学习。我们观察到，在受控实验中，检查点轨迹被回溯性分为三个模式：模仿阶段，SFT捕捉了可靠的教师行为;提升阶段，两个阶段均有帮助;发现阶段，有用的奖励可观察行为存在于可靠教师支持之外。我们利用这一前瞻性应用，利用教师支持和可观察的奖励余量，将特征仅引导至SFT、SFT再RL、增加强化学习分配或进一步环境开发。在随后18次特征专项实验中，诊断预测了15/18的观察轨迹。在GPT-OSS 120B中，针对性SFT和RL相较前沿对照组，广告主技能中有7/8项均为正值;5项正向提升与95%置信区间（不含零）配对，1项技能采用置信支持回归。最大提升为非披露（+11.27分;95% CI [+9.72， +12.82]）。另一项SME审计显示，针对强化学习（ROL）相较于SFT将标准泄漏从11.8%降至2.9%，对抗性泄漏从22.9%降至6.8%，同时保持可操作性（86.2%至85.7%）。在匹配的均一与目标对比中，共享奖励和优化，定向强化学习将七项技能平均差从+1.62提升至+3.57，同时增量强化学习计算减少43%。

Team DArgk at the 2026 ELOQUENT lab for evaluating generative language model quality: Residuals of Humanity: AI Detection Evasion via GRPO Fine-Tuning

2026年ELOQUENT生成语言模型质量评估实验室的DArgk团队：人类残差：通过GRPO微调实现AI检测规避

• Authors: Antonela Tommasel, Juan Manuel Rodriguez
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.22221
• Pdf link: https://arxiv.org/pdf/2609.22221
• Abstract Large language models (LLMs) can generate fluent and coherent text that is increasingly difficult to distinguish from human writing, motivating the development of automatic AI-generated text detectors. However, the robustness of such detectors under adversarial generation remains uncertain. This paper presents SHADE (Stochastic Human-like generation via Adversarial Detector Evasion), a reinforcement learning framework that formulates detector evasion as a policy optimization problem. Instead of applying post-hoc perturbations or prompting-based rewriting, SHADE fine-tunes an instruction-tuned LLaMA model with Group Relative Policy Optimization (GRPO), using feedback from a surrogate detector based on the PAN 2025 mdok system. Our experiments show that full fine-tuning with a small KL regularization penalty achieves $98.5\%$ surrogate evasion, compared to $1.5\%$ for the base model, while LoRA-based adaptation is substantially less effective under regularization. Linguistic analysis reveals that successful evasion is associated with shorter, simpler, and less lexically diverse outputs, suggesting that high detector evasion does not necessarily correspond to more human-like writing. In the official Voight-Kampff competition setting, our submissions ranked sixth and seventh, indicating that optimization against a single surrogate detector only partially transfers to unseen evaluation classifiers. These results highlight both the potential and limitations of reinforcement learning for adversarial AI-text generation and motivate more robust, multi-detector evaluation protocols for AI-generated text detection.
• 中文摘要 大型语言模型（LLMs）能够生成流畅且连贯的文本，这些文本越来越难以与人类书写区分，这促使了自动AI生成文本检测器的发展。然而，此类检测器在对抗性生成下的鲁棒性仍不确定。本文提出了SHADE（通过对抗性检测器规避生成的随机类人类生成），这是一个强化学习框架，将检测器规避作为策略优化问题。SHADE不采用事后扰动或基于提示的重写，而是利用基于PAN 2025 mdok系统的代理检测器反馈，微调指令调优的LLaMA模型，采用群相对策略优化（Group Relative Policy Optimization，GRPO）。我们的实验显示，在带有小 KL 正则化惩罚的完全微调下，替代规避率达到 $98.5%$，而基础模型仅为 $1.5\%$;而基于 LoRA 的适应在正则化下效果显著降低。语言学分析显示，成功的规避与更短、更简单且词汇多样性更少的输出相关，表明高检测器规避不一定意味着更像人类的书写。在官方 Voight-Kampff 竞赛环境中，我们的提交排名分别为第六和第七，表明针对单个替代检测器的优化仅部分转移到未见的评估分类器。这些结果凸显了强化学习在对抗性 AI 文本生成中的潜力和局限性，并激励了更稳健、多检测器评估 AI 生成文本检测的方案。

CNA: An AI-Oriented Comprehensive Normalized Assessment for Healthy Status and Application to Optimize RRT Strategies by Reinforcement Learning

CNA：一种基于人工智能的综合归一化评估，旨在健康状态及通过强化学习优化RRT策略的应用

• Authors: Jiang Liu, Chan Zhou, Yujie Li, Di Wu, Yihao Xie, Peiwei Li, Xin Shu, Jiaqi Zhu, Chunyong Yang, Yuwen Chen, Bin Yi
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.22232
• Pdf link: https://arxiv.org/pdf/2609.22232
• Abstract Millions worldwide require Renal Replacement Therapy (RRT) as a treatment essential for survival. However, optimizing RRT strategies via AI is challenging due to heterogeneous patient dynamics, missing data, and the absence of an AI-oriented health assessment criterion. We propose an AI-Oriented Comprehensive Normalized Assessment (CNA) for healthy status and apply it to optimize RRT strategies by using offline reinforcement learning (RL). The key idea of CNA is transforming vital-sign distributions into a standard normal space, enabling a unified, data-driven health-status score defined by deviations from referent intervals, which also provides an AI-oriented criterion to assess strategy quality and supports RL termination. We further design a structured 23-dimensional state representation that integrates 19 indicators with 4 RRT descriptors, and employ matrix decomposition to reconstruct missing vital signs, improving data completeness for learning. These components are incorporated into multiple offline RL algorithms and validated via systematic ablation studies on RRT feature subsets. Compared with physicians' observed treatments, the best learned strategy reduces mortality from 13.2% to 5.0% (reducing 62.24%) and shortens average in-hospital stay from 308.5 to 250.1 hours (reducing 18.93%), demonstrating both methodological innovation and the potential of CNA-guided RL to improve RRT outcomes in nephrology.
• 中文摘要 全球数百万人需要肾脏替代疗法（RRT）作为生存必需的治疗。然而，由于患者动态异质、数据缺失以及缺乏基于人工智能的健康评估标准，通过人工智能优化RRT策略具有挑战性。我们提出了基于人工智能的综合归一化评估（CNA）用于健康状态，并通过离线强化学习（RL）优化RRT策略。CNA的核心思想是将生命体征分布转化为标准的常态空间，实现统一的、数据驱动的健康状态评分，该评分由偏离所指区间定义，同时为评估策略质量提供面向人工智能的标准，支持RL终止。我们还设计了一种结构化的23维状态表示，整合了19个指标和4个RRT描述符，并利用矩阵分解重建缺失的生命体征，提升学习数据的完整性。这些组成部分被纳入多种离线强化学习算法中，并通过对RRT特征子集的系统消融研究进行验证。与医生观察到的治疗相比，最佳学习策略将死亡率从13.2%降至5.0%（减少62.24%），并将平均住院时间从308.5小时缩短至250.1小时（减少18.93%），展示了方法学创新以及CNA引导RL改善肾脏学RRT结果的潜力。

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

洞察冲突：改善视觉语言模型中的指令层级对齐

• Authors: Nicholas Sansoterra, Zishuo Zheng, Sachin Kumar
• Subjects: Subjects: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.22234
• Pdf link: https://arxiv.org/pdf/2609.22234
• Abstract Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing multimodal IH alignment as a reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfers to multimodal attacks, failing when models must decode, reconstruct, or reason over instructions across modalities. Image-based training improves robustness beyond text-only supervision, while mixed-modality training performs best overall. Importantly, the benefits generalize beyond the synthetic typographic training setting to real-image and web-agent safety tasks, while largely preserving general multimodal capability, showing that lightweight, verifiable supervision can meaningfully improve VLM robustness under adversarial, cross-modal, and interactive instruction conflicts.
• 中文摘要 指令层级（IH）对齐教导语言模型在输入冲突时优先处理更高级的指令。虽然主要在纯文本环境中研究，视觉语言模型（VLMs）为IH带来了新的挑战：指令可能嵌入图像中、分散于不同模态、视觉转换，或在代理任务中遇到。我们将多模态IH对齐定位为推理问题，使用基于规则的奖励强化学习训练VLM，比较纯文本、仅图像和混合模态监督。我们发现纯文本IH训练部分可转移至多模态攻击，当模型需要对跨模态指令进行解码、重建或推理时，攻击失败。基于图像的训练提升了超越纯文本监督的稳健性，而混合模态训练整体表现最佳。重要的是，其优势不仅限于合成排版训练设置，还能推广到实像和网页代理安全任务，同时在很大程度上保留了通用多模态能力，表明轻量级、可验证的监督在对抗性、跨模态和交互式指令冲突下可以显著提升VLM的鲁棒性。

RS-Claw-Evolution: Environment-Feedback-Driven Evolution for Lightweight Remote Sensing Agents in Long-Horizon Tasks

RS-Claw-Evolution：环境反馈驱动的轻量级遥感智能体在长视距任务中的演进

• Authors: Kai Ouyang, Dongyang Hou, Liangtian Liu, Zeyuan Wang, Ziyu Li, Chengfu Liu, Zichao Tang, Xuezhi Cui, Shengwu Ouyang, Wentao Yang, Hanwen Yu, Haifeng Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.22258
• Pdf link: https://arxiv.org/pdf/2609.22258
• Abstract Large language model-driven remote sensing (RS) agents offer a promising approach to automating geospatial analysis. However, lightweight RS agents based on compact language models struggle with multi-step interactive tasks due to loss of long-horizon states, inefficient environmental feedback utilization, and sparse optimization signals. We propose RS-Claw-Evolution, an environment-feedback-driven framework that progressively improves lightweight agents through three stages. Interaction evolution uses executable code to control observations, maintain intermediate states, and reduce context redundancy. Experience evolution combines failure-aware trajectory generation with error-turn masking to learn from informative failure-recovery experiences without imitating faulty actions. Decision evolution uses reinforcement learning with multi-dimensional environment rewards and turn-level advantage protection to optimize tool-use behaviors and improve credit assignment in long sequences. On Earth-Bench, the optimized Qwen3-4B-based agent achieves 65.9% accuracy in Autonomous Planning mode, outperforming the untrained Qwen3-32B baseline (43.8%) and DeepSeek-V3.1 (60.8%), while approaching GPT-5 (71.6%). These results demonstrate that learning from environmental feedback can improve lightweight agents and narrow their performance gap with larger models in long-horizon RS tasks.
• 中文摘要 大型语言模型驱动遥感（RS）代理为自动化地理空间分析提供了有前景的方法。然而，基于紧凑语言模型的轻量级RS代理在处理多步交互任务时遇到困难，原因包括长视野状态丢失、环境反馈利用效率低下和优化信号稀疏。我们提出了RS-Claw-Evolution，这是一个环境反馈驱动的框架，通过三个阶段逐步提升轻量级代理。交互演化利用可执行代码控制观测、维护中间状态并减少上下文冗余。体验演化结合了故障感知轨迹生成与错误转向掩蔽，从信息丰富的失败恢复经验中学习，同时不模仿错误操作。决策演化利用强化学习、多维环境奖励和回合级优势保护，优化工具使用行为并改善长序列中的信用分配。在Earth-Bench上，基于Qwen3-4B的优化智能体在自主规划模式下实现65.9%的准确率，优于未训练的Qwen3-32B基线（43.8%）和DeepSeek-V3.1（60.8%），同时接近GPT-5（71.6%）。这些结果表明，从环境反馈中学习可以提升轻量级智能体，并在长期RS任务中与大型模型缩短性能差距。

CHOREO: Every Humanoid Skill as a Trajectory

编舞：每个类人生物技能作为轨迹

• Authors: Ziyi Sun, Jingwen Chen, Yuxi Wang, Xiuze Xia, Long Cheng, Zhaoxiang Zhang, Yujun Dong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22274
• Pdf link: https://arxiv.org/pdf/2609.22274
• Abstract Recent advances in humanoid robotics have produced diverse skills through reinforcement learning, motion imitation, and generative modeling. Yet these capabilities remain siloed because they are built around incompatible representations, interfaces, and controllers. We present CHOREO, a framework for training-free composition of heterogeneous humanoid skills. Our key observation is that, regardless of how a skill is learned, it can ultimately be expressed as an executable motion trajectory. Based on this observation, CHOREO converts each capability into SkillMotion, a unified representation that combines motion states, contacts, semantics, and boundary conditions. Skills are composed through direct continuation, cubic Hermite blending, or validated bridge motions, without retraining source models or updating models at test time. On Unitree G1 in MuJoCo, CHOREO organizes 2,950 admitted SkillMotion assets derived from heterogeneous sources and achieves 95.4\% sequence success across 130 multi-action tasks, including 93.8\% success on eight-action sequences. These results demonstrate that executable trajectories provide a scalable interface for accumulating and composing pretrained humanoid capabilities.
• 中文摘要 类人机器人技术的最新进展通过强化学习、动作模仿和生成建模产生了多样化的技能。然而，这些能力仍然孤立，因为它们围绕不兼容的表示、接口和控制器构建。我们提出了CHOREO，一个用于无训练的异构类人技能组合框架。我们的关键观察是，无论技能如何学习，最终都可以表达为可执行的运动轨迹。基于这一观察，CHOREO将每个能力转化为SkillMotion，这是一种统一的表示方式，结合了运动状态、接触、语义和边界条件。技能通过直接延续、立方体Hermite混合或验证桥接动作组成，无需重新训练源模型或在测试时更新模型。在MuJoCo的Unitree G1上，CHOREO组织了2950个来自异构来源的已批准SkillMotion资产，并在130个多动作任务中实现了95.4%的序列成功率，其中8个动作序列的成功率为93.8%。这些结果表明，可执行轨迹为积累和组合预训练类人生物能力提供了可扩展的接口。

Visual Graph Reasoning via Knowledge Compilation

通过知识编译进行可视化图推理

• Authors: Rongzheng Wang, Zhe Wang, Ke Qin, Rongwei Wang, Muquan Li, Yizhuo Ma, Yihong Huang, Jielei Wang, Shuang Liang
• Subjects: Subjects: Multimedia (cs.MM); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.22327
• Pdf link: https://arxiv.org/pdf/2609.22327
• Abstract Visual graph reasoning requires answering graph-theoretic questions directly from graph images, where graph topology and state are conveyed visually rather than given in symbolic form. Despite recent progress of vision-language models (VLMs), current approaches to visual graph reasoning still fail on simple visual graph problems. This reveals a fundamental limitation of existing approaches: they prioritize final-answer supervision over the intermediate recovery of an explicit graph representation that preserves graph topology and state from visual input. To address this limitation, we propose VGCompiler, a compilation-centric paradigm for visual graph reasoning via knowledge compilation. VGCompiler organizes reasoning around two compilers: a representation compiler that recovers a structure-preserving intermediate graph representation from visual input, and an operation compiler that compiles query intent under the recovered graph state into an executable graph operation. Specifically, we build VGCompiler on Qwen3-VL-8B and train it with reinforcement learning guided by a layered reward over executability, compiled graph validity, representation quality, and operation quality. VGCompiler uses a frozen observer to summarize graph and question conditions into lightweight signatures, enabling archive retrieval and code reuse across similar regimes. Experiments on three benchmarks GVLQA, VisionGraph, and VGCURE, show that Qwen-VGCompiler, built on an 8B backbone, surpasses the strongest closed-source VLM baseline by 28.9% and the strongest code-based baseline by 23.7%, while maintaining high efficiency. We further evaluate VGCompiler on three real-world domains, including metro routing, logistics delivery, and network fault assessment, where it generalizes across heterogeneous visual graphs and domain-grounded tasks.
• 中文摘要 可视化图推理需要直接从图图像回答图论问题，图拓扑和状态通过视觉形式呈现，而非符号形式。尽管视觉语言模型（VLM）近期有所进展，当前的视觉图推理方法在简单的视觉图问题上仍然失败。这揭示了现有方法的一个根本局限：它们优先于最终答案监督，而忽视了从视觉输入中恢复显式图表示的显式图表示，从而保持图拓扑和状态。为解决这一限制，我们提出了VGCompiler，一种以编译为中心的可视化图推理范式，通过知识编译实现。VGCompiler围绕两个编译器组织推理：一个表示编译器，从视觉输入恢复结构保持的中间图表示，另一个操作编译器将恢复的图状态下的查询意图编译为可执行的图操作。具体来说，我们在Qwen3-VL-8B上构建VGCompiler，并以分层奖励为导向，训练其基于可执行性、编译图效度、表示质量和操作质量的强化学习。VGCompiler使用冻结的观察器将图和问题条件总结为轻量级签名，实现在类似环境中的归档检索和代码重用。在GVLQA、VisionGraph和VGCURE三个基准测试上，基于8B骨干的Qwen-VGCompiler在最强闭源VLM基线上提升28.9%，在保持高效的情况下，领先最强的基于代码的基线23.7%。我们还进一步在三个真实领域评估VGCompiler，包括都市规划、物流交付和网络故障评估，并在异构可视化图和域基任务中实现推广。

Learning and Control Beyond Linearity: Towards a Non-asymptotic Theory for Bilinear Systems

超越线性学习与控制：迈向双线性系统的非渐近理论

• Authors: Yahya Sattar, Yassir Jedra, Robin Strässer, Frank Allgöwer, Maryam Fazel, Sarah Dean
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG); Robotics (cs.RO); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.22338
• Pdf link: https://arxiv.org/pdf/2609.22338
• Abstract This tutorial provides a unified view of the emerging area of bilinear learning and control. Using linear systems as a benchmark, it explains what fundamentally changes in the bilinear settings, how recent theory addresses finite-sample learning and control, and how these ideas connect to broader themes in nonlinear control, representation learning, and data-driven decision making. For learning, we emphasize tools that are particularly useful in the bilinear settings, such as one-sided Bernstein's inequality for dependent and heavy-tailed covariates, blocking arguments, and martingale concentration for input-dependent noise. We then apply these tools to obtain finite-sample learning guarantees for fully observed bilinear systems, partially observed bilinear systems, and linear systems with bilinear observations. For control, we discuss quadratic control from bilinear observations, where the classical separation principle fails, and review tractable approaches based on belief-space receding horizon control. We also cover stabilization of bilinear dynamics under state feedback using semi-definite programming, LMI relaxations, sum-of-squares methods, and Koopman-based lifting. We conclude by discussing connections to reinforcement learning and machine learning, and some open problems in combined learning and control of bilinear systems.
• 中文摘要 本教程为双线性学习与控制这一新兴领域提供了一个统一的视角。以线性系统为基准，解释了双线性环境的根本变化，最新理论如何处理有限样本学习与控制，以及这些思想如何与非线性控制、表示学习和数据驱动决策等更广泛的主题相联系。在学习方面，我们强调在双线性环境中特别有用的工具，如依赖和重尾协变量的单侧伯恩斯坦不等式、阻断论元以及输入依赖噪声的马丁加尔集中。随后，我们应用这些工具获得对完全观测双线性系统、部分观测双线性系统和带有双线性观测的线性系统有限样本学习保证。控制方面，我们讨论双线性观测的二次控制，其中经典分离原理失效，并回顾基于信念空间退缩视界控制的可行方法。我们还涵盖了利用半正定规划、LMI松弛、平方和方法和基于Koopman的提升，在状态反馈下稳定双线性动力学。最后，我们讨论了强化学习与机器学习的联系，以及双线性系统结合学习与控制中的一些未解决问题。

Learning Control Policies from Heterogeneous Multi-Horizon Time Series in Battery Energy Management Systems

从电池能源管理系统中的异构多视界时间序列学习控制策略

• Authors: Sheng Yin, Vivek Teja Tanjavooru, Holger Hesse, Christoph Goebel
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.22546
• Pdf link: https://arxiv.org/pdf/2609.22546
• Abstract This paper introduces Representation-to-Decision (R2D), an end-to-end imitation learning framework that maps heterogeneous multi-horizon time-series inputs directly to battery control decisions through modular Temporal Feature Extractors (TFEs) and a shared latent representation, without an explicit load and PV forecasting step. While accurate forecasting improves prediction quality, optimal control performance remains unguaranteed in prediction-then-optimization pipelines; standard Reinforcement Learning (RL) lacks the long-horizon temporal awareness due to short-window observations, even when forecast signals are available as additional inputs. R2D offers a different perspective: rather than forecasting first and deciding second, it learns to decide directly from raw temporal inputs, with control optimality anchored by an aging-aware Mixed-Integer Linear Programming (MILP) expert through Behavior Cloning (BC). Benchmarked against six controllers on a high-fidelity electro-thermal battery simulation across five industrial sites, R2D achieves 62--77\% of the global clairvoyant optimum, outperforms the tested Model Predictive Control (MPC) and RL benchmarks, and yields battery degradation nearly identical to its MILP teacher across all five sites. Comprehensive ablation studies over temporal backbone, model size, expert formulation, aging-cost weighting, and horizon configuration confirm that an LSTM encoder with a 15-minute single-step control horizon provides the most robust and deployment-ready configuration, and cross-site and single-factor out-of-distribution tests show that generalization to unseen profiles is profile-dependent, with policies trained on high-activity sites transferring most reliably.
• 中文摘要 本文介绍了表征到决策（R2D），这是一种端到端的模仿学习框架，通过模块化时间特征提取器（TFE）和共享潜在表示，将异质多视界时间序列输入直接映射到电池控制决策，无需显式负载和PV预测步骤。虽然准确预测提升了预测质量，但在预测后优化流程中，最优控制性能仍不保证;标准强化学习（RL）由于短窗口观测，即使有额外输入预测信号，也缺乏长视野时间感知。R2D提供了不同的视角：它不再先预测后决策，而是学习直接从原始时间输入中做出决策，控制最优性由行为克隆（BC）中一个具有老化意识的混合整数线性规划（MILP）专家作为锚定。通过在五个工业站点进行高保真电热电池模拟的六个控制器基准测试，R2D实现了全球千里眼最优值的62%-77%，超过了测试的模型预测控制（MPC）和强化学习基准测试，电池劣化在五个站点上几乎与其MILP老师相同。对时间骨干、模型大小、专家公式、老化成本加权和地平线配置的全面消融研究证实，15分钟单步控制视野的LSTM编码器提供了最稳健且部署准备充分的配置，跨站点和单因子非分布测试显示，对未见剖面的泛化依赖于剖面，训练于高活跃站点的策略传输最为可靠。

Augmenting PID Control with Deep Reinforcement Learning: A Hybrid Approach to the Industrial Benchmark

用深度强化学习增强PID控制：工业基准的混合方法

• Authors: Zhengyang (Cissy)Gu, Joseph E. Hernandez, John Burtenshaw, Sean Scott, Thomas Cook, Chris Couch
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.22584
• Pdf link: https://arxiv.org/pdf/2609.22584
• Abstract As industrial processes grow in complexity, traditional Proportional-Integral-Derivative (PID) controllers are often insufficient for handling their non-linear, multi-input dynamics. We propose using advanced Deep Reinforcement Learning (DRL) to prove its advantages in these complex environments. To do this, we rely on the Industrial Benchmark (IB). The IB is a realistic simulation that tests DRL algorithms against the key challenges of industrial applications: high-dimensional state spaces, delayed effects, and conflicting multi-criterial objectives. This testbed highlights DRL's core trade-off: while its final policies can often be unstable, its unique strength is the ability to autonomously discover optimal, non-obvious policies in multi-dimensional spaces where simple controllers fail. In this paper, we propose a novel hybrid PID-RL controller that leverages DRL's discovery capability while ensuring Reliability. After developing a multi-objective reward function to make DRL viable, we use a twin-delayed deep deterministic (TD3) agent as a discovery tool to find the optimal, non-obvious settings for the IB's 'Gain' and 'Shift' parameters. By feeding these discovered parameters to a simple, tuned PID controller, our hybrid model successfully combines all three characteristics: it achieves the optimal Performance and Efficiency of the best DRL agent with the Reliability of a classical controller. This work demonstrates a practical methodology for using DRL to augment, rather than replace, trusted industrial control systems.
• 中文摘要 随着工业流程复杂化，传统的比例-积分-导数（PID）控制器往往无法满足其非线性、多输入动态的处理。我们建议利用先进的深度强化学习（DRL）来证明其在这些复杂环境中的优势。为此，我们依赖工业基准（IB）。IB是一种真实模拟，测试DRL算法针对工业应用的关键挑战：高维状态空间、延迟效应和多标准目标冲突。该测试平台凸显了DRL的核心权衡：尽管其最终策略常常不稳定，但其独特优势在于能够自主发现在多维空间中最优且不显而易见的策略，而这些空间的简单控制器无法实现。本文提出了一种新颖的混合PID-RL控制器，利用DRL的发现能力，同时保证可靠性。在开发多目标奖励函数使DRL可行后，我们使用双延迟深度确定性（TD3）智能体作为发现工具，寻找IB中“增益”和“偏移”参数的最优且不显而易见的设置。通过将这些发现的参数输入一个简单且调优的PID控制器，我们的混合模型成功结合了这三种特性：它实现了最佳DRL代理的最佳性能和效率，同时具备经典控制器的可靠性。这项工作展示了一种实用方法论，用于增强而非取代可信工业控制系统。

AutoGym: Blueprint-First Generation of Verifiable Agent Gyms

AutoGym：蓝图——第一代可验证代理道馆

• Authors: Aarati Andrea Noronha, Kavya Ravikumar, Carly Xiaoyu Lin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.22592
• Pdf link: https://arxiv.org/pdf/2609.22592
• Abstract Training agents with reinforcement learning requires a gym, comprising a task, an executable environment in which the task can be attempted, and a verifier that reliably distinguishes success from failure. Constructing such gyms remains manual, expensive, and static. Task sets saturate as models improve and are increasingly exposed to contamination. Synthetic generation offers scale, but single-pass synthesis produces tasks whose difficulty is largely cosmetic. Models comparable in capability solve them despite convoluted phrasing, and correctness must be adjudicated post-hoc by unreliable LLM judges. We present AutoGym, a framework that generates complete gyms (tasks, executable environments, and verifiers) from a minimal domain seed or prior model trajectories. AutoGym introduces three mechanisms. (1) Blueprint-first generation specifies the valid solution space, environment requirements, and verification criteria before the environment is materialized, making solvability a construction prerequisite rather than a property verified after the fact. (2) Explicit generation parameters control task topology, interaction depth, capability axes, question obfuscation, and distractor composition, enabling fine-grained difficulty steering. (3) Active curriculum synthesis uses performance-informed calibration to adjust the distribution over these parameters as model capabilities evolve. Across productivity and temporal-reasoning settings, AutoGym generates gyms spanning the capability spectrum, including instances that challenge frontier models.
• 中文摘要 用强化学习训练代理需要一个健身房，包括任务、可执行的环境，以及一个可靠区分成功与失败的验证器。构建此类健身房仍然是手动、昂贵且静态的。随着模型的改进，任务集逐渐饱和，且越来越容易受到污染。合成生成提供了规模，但单遍综合产生的任务难度大多是表面上的。能力相当的模型即使措辞复杂也能解决，且正确性必须由不可靠的大型语言模型裁判事后裁定。我们介绍AutoGym，一个从最小域种子或先前模型轨迹生成完整健身房（任务、可执行环境和验证器）的框架。AutoGym引入了三种机制。（1） 蓝图第一代在环境具体化前指定有效的解空间、环境需求和验证标准，使可解性成为构建的前提条件，而非事后验证的属性。（2） 显式生成参数控制任务拓扑、交互深度、能力轴、问题混淆和干扰器组合，实现细粒度的难度引导。（3） 主动课程综合利用性能导向校准调整这些参数上的分布，以配合模型能力的演进。在生产力和时间推理环境中，AutoGym生成跨能力谱的健身房，包括挑战前沿模型的实例。

Autonomous Model Lifecycle Management for Digital Twin-Based Manufacturing Control

基于数字孪生的制造控制自主模型生命周期管理

• Authors: Zhengyang (Cissy)Gu, Thomas Cook, Fredaljohn Rohrbaugh, Joseph E. Hernandez, Chris Couch
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.22701
• Pdf link: https://arxiv.org/pdf/2609.22701
• Abstract Manufacturing AI systems must autonomously adapt to continuous distributional shift from raw-material variability, ambient changes, and equipment aging, under strict safeguard and operator-trust requirements where model failures risk physical damage. This paper presents a closed-loop Cyber-Physical System (CPS) for autonomous model lifecycle management in automotive manufacturing, deployed since 2023. The system manages product-specialized model pairs: a sequence-to-sequence physics model (LPP) serving as a digital twin, and a deep Reinforcement Learning (RL) control policy (LCP) trained against it. Per retraining cycle, multiple model variants spanning architecture families and RL algorithms compete; only the best-scoring candidate advances. A Conductor orchestrator autonomously manages plant-wide model inventories with dependency-aware retraining and Proportional-Integral-Derivative (PID) fallback. Reflecting the principle of Human-Centric Intelligence, the LCP composite score embeds an operator-trust gate penalizing policies deviating from established practice; without it, 23% of policies are rejected by operators despite passing accuracy thresholds. Across multiple facilities, LCP-controlled processes achieve process stability improvements of 28-45% over uncontrolled baselines with zero safety incidents.
• 中文摘要 制造AI系统必须自主适应从原材料变异、环境变化到设备老化的持续分布变化，在严格的保障和操作员信任要求下，模型失效可能导致物理损坏。本文介绍了自2023年起部署的汽车制造业自主模型生命周期管理闭环网络物理系统（CPS）。该系统管理产品专用模型对：作为数字孪生的序列对序列物理模型（LPP）和针对其训练的深度强化学习（RL）控制策略（LCP）。每个再训练周期中，跨架构家族和强化学习算法的多个变体竞争;只有得分最高的候选者才能进步。指挥编排器自主管理全厂模型库存，采用依赖感知再训练和比例积分微分（PID）缓冲。体现以人为本智能的原则，LCP综合评分内嵌了一个操作员-信任门，惩罚偏离既定惯例的政策;没有该门槛，23%的政策即使通过准确阈值仍被运营商拒绝。在多个设施中，LCP控制的流程比无控制基线实现了28-45%的过程稳定性提升，且无安全事故。

UBA-ORL: Unlearning-Activated Backdoor Attacks on Offline Reinforcement Learning

UBA-ORL：离线强化学习中的“去学习激活”后门攻击

• Authors: Fengyi Wang, Cong Li, Lulu Xue, Qiyu Leng, Ziqi Zhou, Peijin Guo
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.22711
• Pdf link: https://arxiv.org/pdf/2609.22711
• Abstract Offline reinforcement learning (offline RL) enables policy learning from pre-collected static datasets without online exploration, and is increasingly deployed not only in safety-critical domains such as autonomous driving and robotic control but also in data-mining applications such as recommendation and behavior analysis. While compliance-driven data removal enhances privacy, it also opens a previously unrecognized attack surface. We introduce UBA-ORL (Unlearning-activated Backdoor Attack on Offline Reinforcement Learning), the first unlearning-activated backdoor attack for offline RL: in the evaluated settings, the attack is substantially suppressed after normal training and becomes pronounced after a compliance-driven deletion (unlearning) request. UBA-ORL employs a dual-sample mechanism: alongside backdoor trajectories (BD) that link a trigger to malicious actions under inflated rewards, the attacker injects camouflage trajectories (CM) sharing the same trigger pattern but preserving benign actions with equally high rewards. During training, BD and CM provide competing supervisory signals; upon a legitimate deletion request on the CM subset, the residual BD signal can re-dominate, reactivating the backdoor on demand. Empirical results show that UBA-ORL achieves controllable activation under the evaluated offline-RL configurations, while no-trigger return changes vary by configuration, exposing a previously overlooked security risk in compliance-driven offline RL platforms. We urge the community to develop joint pre-/post-unlearning auditing mechanisms for compliant unlearning services.
• 中文摘要 离线强化学习（离线强化学习）使得从预先收集的静态数据集中进行策略学习，无需在线探索即可，且越来越多地应用于安全关键领域如自动驾驶和机器人控制，也广泛应用于推荐和行为分析等数据挖掘应用中。虽然合规驱动的数据移除提升了隐私，但也开启了此前未被识别的攻击面。我们介绍了UBA-ORL（离线强化学习中的去学习激活后门攻击），这是离线强化学习中首个被移除后门攻击：在评估环境中，该攻击在正常训练后被大幅抑制，并在合规驱动的删除（复学习）请求后变得明显。UBA-ORL采用双采样机制：除了将触发器与恶意行为关联的后门轨迹（BD）外，攻击者还注入伪装轨迹（CM），这些轨迹共享相同触发模式，但保留了同等高奖励的良性动作。在训练过程中，BD和CM会提供竞争性的监督信号;在CM子集收到合法删除请求时，残余的BD信号可以重新主导，按需重新激活后门。实证结果显示，UBA-ORL在评估的离线强化环境中实现可控激活，而无触发返回的变更因配置而异，暴露了合规驱动的离线强化学习平台此前被忽视的安全风险。我们敦促社区为合规的去学习服务开发联合的前后复学审计机制。

ProcessLight: Process Supervision for Large Language Model Based Traffic Signal Control

ProcessLight：用于大型语言模型的交通信号控制过程监督

• Authors: Huaitao Zhao, Tianlong Zhou, Weijie Wang, Jiasheng Shi, Weixiong Rao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.22746
• Pdf link: https://arxiv.org/pdf/2609.22746
• Abstract Large Language Models (LLMs) have recently been introduced into traffic signal control (TSC) as decision agents due to their strengths in human-readable reasoning generation. Yet, existing LLM TSC methods optimize only from final outcomes and fail to distinguish valid from flawed reasoning steps, causing useful or misleading steps to be jointly updated and thus impairing the model's learning of effective reasoning. To bridge this gap, we propose an LLM-based framework ProcessLight to decompose signal decisions into verifiable semantic steps. Building on ProcessLight, we further develop Step-wise Traffic Process Policy Optimization (STeP-PO), a novel reinforcement learning framework that optimizes structured reasoning processes through step-level credit assignment. Specifically, STeP-PO uses step quality scores to evaluate local reasoning quality and step importance to measure each step's influence on the final action, and then assigns step-level advantages over a semantic step tree structure. The resulting step-level advantages are propagated to reasoning tokens, enabling fine-grained policy optimization beyond outcome-only rewards. Extensive experiments over multiple real-world datasets demonstrate the superiority of our methods. Our code is available at this https URL.
• 中文摘要 大型语言模型（LLM）因其在人类可读推理生成方面的优势，最近被引入交通信号控制（TSC）作为决策代理。然而，现有的大型语言模型TSC方法仅从最终结果中优化，未能区分有效与错误的推理步骤，导致有用或误导性的步骤被联合更新，从而削弱模型对有效推理的学习。为弥合这一差距，我们提出了基于LLM的ProcessLight框架，将信号决策分解为可验证的语义步骤。基于ProcessLight，我们进一步开发了分步骤交通流程策略优化（STeP-PO），这是一种通过步骤级信用分配优化结构化推理过程的新型强化学习框架。具体来说，STeP-PO使用步骤质量评分评估局部推理质量和步骤重要性，以衡量每一步对最终动作的影响，然后赋予步骤层面优势相较于语义步骤树结构。由此产生的步骤层面优势会传递到推理代币，实现超越单纯结果奖励的细粒度策略优化。在多个真实数据集上的大量实验证明了我们方法的优越性。我们的代码可在此 https URL 访问。

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

通过对抗强化学习，霍克斯订单流与价格影响实现强有力的做市

• Authors: Hao Yang, Zhenguo Xu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.22785
• Pdf link: https://arxiv.org/pdf/2609.22785
• Abstract Market-making strategies in real limit order book markets face substantial model uncertainty and regime-shift risk. Existing adversarial reinforcement learning approaches improve robustness by formulating the Avellaneda--Stoikov market-making problem as a zero-sum game between a market maker and an environmental adversary. However, these approaches typically rely on Poisson order arrivals and neglect trade-induced price impact, limiting their ability to capture important high-frequency market microstructure effects such as clustered order flow, self-excitation, and post-trade price feedback. We extend adversarial reinforcement learning for market making to a more complex environment with Hawkes self-exciting order arrivals and trade-induced price impact. To mitigate the increased non-stationarity introduced by the expanded regime space, we incorporate an LSTM module that explicitly models the temporal structure of recent observations. We further characterize the equilibrium properties of the proposed framework through both game-theoretic analysis and numerical experiments, and introduce a robustness evaluation protocol focused on improvements in the left tail of the return distribution. Experimental results across a range of market regimes show that the proposed method achieves improved left-tail performance in most complex microstructure environments. In particular, the gains are pronounced in regimes with strong Hawkes excitation and low-to-moderate price impact. Bootstrap tests provide no evidence that these improvements are obtained through a stronger terminal directional inventory bias. These results suggest that combining adversarial training with temporal state representation can improve the robustness of reinforcement-learning-based market-making strategies under order-flow self-excitation, price impact, and regime uncertainty.
• 中文摘要 真实限价单市场中的做市策略面临显著的模型不确定性和制度转移风险。现有的对抗强化学习方法通过将Avellaneda-Stoikov做市问题表述为做市商与环境对手之间的零和博弈，从而提升了鲁棒性。然而，这些方法通常依赖泊松订单到达，忽视交易引发的价格影响，限制了它们捕捉重要高频市场微观结构效应的能力，如聚类订单流、自激和交易后价格反馈。我们将对抗强化学习扩展到更复杂的环境中，采用Hawkes自激单到达和交易诱导的价格影响。为缓解扩展的体系空间带来的非平稳性增加，我们集成了一个LSTM模块，明确建模近期观测的时间结构。我们通过博弈论分析和数值实验进一步表征了所提框架的平衡性质，并引入了一套以改进回报分布左尾为重点的鲁棒性评估方案。在多种市场体系下的实验结果表明，所提方法在大多数复杂微观结构环境中实现了更好的左尾性能。特别是在霍克斯激发强烈且价格影响低至中等的区间，这些提升尤为明显。自助式测试未提供证据表明这些改进是通过更强的终端方向性库存偏差获得的。这些结果表明，将对抗训练与时间状态表示结合，可以提升基于强化学习的造市策略在秩序流自激、价格影响和体制不确定性下的鲁棒性。

ARCGym: Benchmarking Deep Reinforcement Learning in Autonomous Robotic Colonoscopy

ARCGym：自主机器人结肠镜中深度强化学习的基准测试

• Authors: Guanglin Ji, Martina Finocchiaro, Kenny Erleben, Hang Yin
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22803
• Pdf link: https://arxiv.org/pdf/2609.22803
• Abstract Simulations for learning-based autonomous colonoscopic navigation focus mainly on fully actuated capsule robots, failing to capture the contact-rich navigation of long and flexible clinical colonoscopes. We present the Autonomous Robotic Colonoscopy Gym (ARCGym), an open-source reinforcement learning environment and benchmark for image-based navigation in clinically derived deformable colon anatomies. ARCGym supports multiple types of colonoscope robots, spanning capsule robots and flexible endoscopes, with this work focusing on flexible endoscopes including magnetic-driven tip actuation and clinically used proximally translational actuation. This work includes five CT-reconstructed colons representing typical clinical scenarios, a set of clinically meaningful navigation subtasks, and unified success metrics. We introduce a reward combining depth-based lumen alignment with a lumen-visibility score to improve learning under occlusions. Experiments across tasks, robots, and anatomies show that autonomous navigation remains challenging for both magnetic-driven and proximal-insertion flexible robots, with proximal-insertion actuation remaining an open problem.
• 中文摘要 基于学习的自主结肠镜导航模拟主要聚焦于全驱动胶囊机器人，未能捕捉长而灵活的临床结肠镜的接触丰富导航。我们介绍自主机器人结肠镜健身房（ARCGym），这是一个开源强化学习环境，也是临床衍生的可变形结肠解剖中基于图像导航的基准。ARCGym支持多种类型的结肠镜机器人，涵盖胶囊机器人和柔性内镜，该工作重点关注柔性内镜，包括磁力驱动的尖端驱动和临床使用的近端平移驱动。该工作包括五个代表典型临床场景的CT重建结肠、一组临床意义的导航子任务以及统一的成功指标。我们引入了结合深度导向腔对齐与腔能见度评分的奖励，以改善闭塞下的学习。跨任务、机器人和解剖结构的实验表明，自主导航对于磁驱动和近端插入柔性机器人来说依然具有挑战性，近端插入驱动仍是一个悬而未决的问题。

Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradients

通过模型无关元学习实现的个性化联合强化学习：精确元政策梯度与无黑森元政策梯度的收敛

• Authors: Ali Beikmohammadi, Sarit Khirirat, Sindri Magnússon
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.22833
• Pdf link: https://arxiv.org/pdf/2609.22833
• Abstract We study personalized federated reinforcement learning, in which $n$ agents, each acting in its own Markov decision process, collaborate through a server to learn a shared MAML-style policy initialization that becomes effective for an individual agent once that agent adapts it with a single local policy-gradient step. We propose Per-FedAvg-PG, in which agents take $\tau$ local stochastic meta-policy-gradient steps between communication rounds, and prove that it reaches an $\varepsilon$-approximate first-order stationary point of the personalized objective in $K=\mathcal O(\varepsilon^{-3/2})$ rounds with $\tau=\Theta(\varepsilon^{-1/2})$ local steps. The analysis rests on a structural feature of the reinforcement learning setting: under standard policy-class regularity, the per-agent objectives have uniformly bounded gradients and Hessians with explicit constants, so the bounded-gradient and bounded-heterogeneity conditions imposed by the supervised theory hold automatically and no separate heterogeneity assumption is needed. The exact meta-gradient requires the inner-loop policy Hessian, which our experiments identify as the practical bottleneck. We therefore analyze the Hessian-free variant, bound its bias, and exhibit fixed points at which the meta-gradient is nonzero and of order $\alpha$, showing that the resulting stationarity floor is a property of the method rather than of the bound. Experiments on tabular and neural navigation confirm the predicted behavior and show transfer to unseen agents at an order of magnitude lower sample cost than independent training. Together these results identify the adaptation step size as a tunable personalization knob and the curvature estimate as the quantity that governs whether exact meta-gradients are affordable.
• 中文摘要 我们研究个性化联邦强化学习，其中$n$代理各自以自己的马尔可夫决策过程行动，通过服务器协作学习共享的MAML风格策略初始化，一旦该代理通过单一局部策略梯度步骤调整该策略，该初始化对该策略生效。我们提出Per-FedAvg-PG方法，代理在通信轮之间采取$\tau$局部随机元策略梯度步，并证明在$K=\mathcal O（\varepsilon^{-3/2}）$轮中达到个性化目标的$\varepsilon$近似一阶平稳点，且该回合为$\tau=\Theta（\varepsilon^{-1/2}）$局部步。分析基于强化学习设置的一个结构特征：在标准策略类正则性下，每个代理的目标具有均匀有界梯度，且具有显式常数的黑森分布，因此监督理论施加的有界梯度和有界异质性条件自动成立，无需单独的异质性假设。精确的元梯度需要内环策略Hessian，我们的实验将其识别为实际瓶颈。因此，我们分析了无Hessian变体，界定其偏置，并展示了元梯度非零且阶为$\alpha$的固定点，表明平稳底是方法的性质，而非界限的性质。表格和神经导航的实验证实了预测行为，并显示对未见代理的迁移成本比独立训练低一个数量级。这些结果共同确定了适应步长为可调节个性化旋钮，曲率估计为决定精确元梯度是否可负担的量。

ForceRFT: Refining VLA Actions through Force-Guided Residual Reinforcement Learning

ForceRFT：通过力引导残留强化学习优化VLA动作

• Authors: Yichen Wang, Chaoyang Zhang, Xuqi Su, Jun Ma, Haiyue Zhu, Xiaocong Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22840
• Pdf link: https://arxiv.org/pdf/2609.22840
• Abstract Force-conditioned vision-language-action (VLA) policies can respond to contact, but when trained solely on demonstrations, their recovery behavior may be limited by demonstration coverage, and they do not learn from deployment outcomes. Human corrective imitation provides additional recovery examples, but its objective matches local action targets without explicitly optimizing task return. We present ForceRFT, a force-guided residual reinforcement learning framework that learns contact-dependent corrections from human supervision and autonomous task outcomes. A frozen, demonstration-trained SmolVLA-based prior generates force-conditioned action chunks, while a lightweight residual actor refines individual end-effector pose commands using wrist feedback acquired during chunk execution. The decision-time wrist wrench, its temporal change, and the selected base motion condition both residual correction and value estimation. Human corrections supervise the residual actor, while verified autonomous transitions train the twin critics and support value-guided updates to the same actor. Bootstrapping is restricted to autonomous segments, preventing TD credit from crossing human-intervention boundaries. Real-robot experiments on plug insertion, ring-on-peg assembly, and whiteboard wiping show higher autonomous success rates than the evaluated demonstration-trained and residual-imitation baselines. Comparisons with residual imitation support value-guided residual optimization, while plug-insertion ablations indicate the benefit of direct execution-time wrist feedback.
• 中文摘要 力制约视觉-语言-行动（VLA）策略可以响应接触，但仅基于演示训练时，其恢复行为可能受限于示范覆盖范围，且无法从部署结果中学习。人类纠正模拟提供了额外的恢复示例，但其目标是匹配局部行动目标，而非明确优化任务返回。我们介绍ForceRFT，一种力引导的残余强化学习框架，通过人类监督和自主任务结果学习接触依赖的修正。一个冻结的、基于演示训练的SmolVLA先验生成力制约动作块，而轻量级残差演员则利用块执行时获得的手腕反馈细化单个末端执行器姿态指令。决策时间腕扳手、其时间变化以及所选基础运动条件，同时提供残差修正和价值估计。人工修正监督残余演员，而经过验证的自主转换则训练双重批评者，并支持对同一参与者的价值引导更新。自助机制仅限于自主片段，防止TD信用跨越人工干预边界。真实机器人实验中，插头插入、环形连接组装和白板擦拭显示，自主成功率高于已评估的演示训练和残差仿制基线。与残差模仿的比较支持价值引导残差优化，插头插入消融显示直接执行时间手腕反馈的益处。

AoI-Driven Hierarchical Learning for Cooperative Resource Sharing in Multi-Operator UAV Networks

多运营商无人机网络中基于AoI驱动的协作资源共享分层学习

• Authors: Atefeh Hajijamali Arani, Mahyar Shirvanimoghaddam, Abolfazl Mehbodniya, Halim Yanikomeroglu, Fumiyuki Adachi
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.22865
• Pdf link: https://arxiv.org/pdf/2609.22865
• Abstract Uncrewed aerial vehicle (UAV)-assisted networks provide a versatile paradigm for on-demand connectivity. However, in multi-operator aerial networks (MOANs), the joint optimization of cooperative resource sharing and 3D trajectory control to maintain information freshness is a complex combinatorial problem, which can be shown to be NP-hard. To address this computational complexity, we propose an age of information (AoI)-driven hierarchical deep reinforcement learning (DRL) framework. Specifically, a Dueling Double Deep Q-Network (D3QN) architecture is deployed at both the operator and UAV decision layers to mitigate overestimation bias and enhance stability in high-dimensional state spaces. To improve system resilience, we introduce an AoI- and load-aware outage compensation mechanism that prioritizes users based on instantaneous transmission demands and temporal freshness. Furthermore, a normalized load exchange balance metric is incorporated to regulate cooperative behavior and ensure resource fairness across operators. Simulation results demonstrate that the proposed hierarchical D3QN significantly outperforms conventional DRL, non-cooperative, and cooperative benchmarks, reducing the average AoI by up to 56.1% under severe congestion while ensuring superior inter-operator fairness and outage mitigation.
• 中文摘要 无人机（UAV）辅助网络为按需连接提供了多样化的范式。然而，在多运营商空中网络（MOAN）中，协同资源共享与三维轨迹控制的联合优化以保持信息新鲜性是一个复杂的组合问题，且可证明为NP难问题。为解决这一计算复杂性，我们提出了一个信息时代（AoI）驱动的层级深度强化学习（DRL）框架。具体来说，在操作员层和无人机决策层均部署了对比双深度Q网络（D3QN）架构，以减少高估偏差并提升高维状态空间的稳定性。为提升系统韧性，我们引入了基于瞬时传输需求和时间新鲜度的AOI感知和负载感知的断电补偿机制，优先级对用户进行调整。此外，还采用了归一化负载交换平衡指标，以规范合作行为并确保运营商间的资源公平性。模拟结果显示，所提的分层D3QN显著优于传统日程、非合作式和合作式基准，在严重拥堵情况下平均AOI可降低多达56.1%，同时确保运营商间公平性和停机缓解效果优异。

Towards Full Pipeline FP8 Reinforcement Learning for LLMs

迈向大型语言模型的全流水线FP8强化学习

• Authors: Fanchao Chen, Ziheng Jiang, Ziyun Wei, Zheng Zhong, Du Li, Chi Zhang, Haibin Lin, Shivaram Venkataraman
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.22870
• Pdf link: https://arxiv.org/pdf/2609.22870
• Abstract Reinforcement learning (RL) has become a key technique for improving the reasoning and agentic abilities of large language models (LLMs). Although FP8 quantization can accelerate RL training, maintaining stability throughout an FP8 RL pipeline remains challenging. While previous works have focused on resolving train-inference mismatches using correction techniques like TIS, we reveal that full-pipeline FP8 RL still suffers from severe training instability, manifesting as anomalous mid-training entropy surges and garbled outputs. We trace this instability to a previously overlooked cause: compounded FP8 quantization noise distorts the importance ratio, disproportionately pushing negative-advantage tokens outside the trust region and erroneously zeroing out their gradients. As a result, pathological outputs are not properly penalized and accumulate over the course of training. To address this, we propose Calibrated Clipping, a dynamic method that aligns the FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound clipping quantile and rebalancing the upper bound accordingly. Extensive experiments across GRPO and DAPO algorithms, model scales from 8B to 32B, and multiple FP8 scaling granularities demonstrate that our approach successfully eliminates entropy surges and restores performance comparable to the BF16 baseline.
• 中文摘要 强化学习（RL）已成为提升大型语言模型（LLM）推理和代理能力的关键技术。尽管FP8量化可以加速强化学习，但在整个FP8强化流水线中保持稳定性仍具挑战性。虽然以往研究主要通过TIS等纠正技术解决列车与推理不匹配，但我们发现全流水线FP8 RL仍存在严重的训练不稳定性，表现为异常的训练中段熵激增和输出杂乱。我们将这种不稳定性追溯到一个此前被忽视的原因：复合FP8量化噪声扭曲了重要性比，不成比例地将负优势标记推出信任区，错误地将其梯度归零。结果是，病态输出未能得到适当惩罚，并在训练过程中不断累积。为此，我们提出了校准裁剪（Calibrated Clipping）这一动态方法，通过匹配下界裁剪分位数并相应重新平衡上界，使FP8裁剪边界与高精度BF16分布对齐。通过GRPO和DAPO算法、8B至32B的模型尺度以及多重FP8缩放粒度的广泛实验，表明我们的方法成功消除熵激增，并恢复与BF16基线相当的性能。

Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization

优先推广以实现高效的基于世界模型的愿景-语言-行动政策优化

• Authors: Yifei Sheng, Haoxiang Ren, Zhilong Zhang, Haonan Wang, Runjie Xu, Yihao Sun, Nan Tang, Zhichao Wu, Lei Yuan, Haoxin Lin, Yang Yu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.22879
• Pdf link: https://arxiv.org/pdf/2609.22879
• Abstract Vision-Language-Action (VLA) models have emerged as a powerful paradigm for embodied intelligence, but fine-tuning them with reinforcement learning (RL) remains constrained by the cost of real-world robot interaction. Model-based reinforcement learning (MBRL) reduces this cost by using a learned world model to generate rollouts for policy optimization. However, it becomes computationally expensive as VLA policies and world models scale. Existing methods typically treat states equally, overlooking substantial differences in their utility for policy improvement. In this paper, we show that policy uncertainty helps identify states with greater potential for policy improvement. The policy exhibits high uncertainty at only a small subset of states, often during decision-sensitive stages where small action differences can alter task outcomes, suggesting that policy improvements at these states could be particularly valuable. Building on these findings, we introduce U-GROW, a lightweight, plug-and-play sampling layer that directs more model rollouts to these informative states. By modifying only the branched-start distribution, U-GROW can be integrated into existing MBRL pipelines without changing the policy optimization objective. Experiments in both simulated and real-world manipulation tasks demonstrate the efficiency and effectiveness of U-GROW, supporting the use of policy uncertainty to guide experience generation.
• 中文摘要 视觉-语言-行动（VLA）模型已成为具身智能的强大范式，但通过强化学习（RL）微调它们仍受现实机器人交互成本的限制。基于模型的强化学习（MBRL）通过使用学习世界模型生成政策优化的推广，降低了这一成本。然而，随着VLA政策和世界模型的扩展，计算成本会增加。现有方法通常平等对待状态，忽视了它们在政策改进中效用上的显著差异。本文展示了政策不确定性有助于识别具有更大政策改进潜力的状态。该政策仅在少数状态表现出高不确定性，通常发生在决策敏感阶段，这些阶段的微小行动差异可能改变任务结果，表明这些状态下的政策改进可能特别有价值。基于这些发现，我们引入了U-GROW，一种轻量级的即插即用采样层，能够将更多模型的部署引导到这些信息状态。仅通过修改分支起始分布，U-GROW可以集成到现有的MBRL流水线中，而不改变策略优化目标。在模拟和现实操作任务中的实验展示了U-GROW的高效性和有效性，支持利用策略不确定性来指导体验生成。

Reproducibility Challenges in Computational Network Science: Evidence, Causes, and Recommendations

计算网络科学中的可重复性挑战：证据、原因与建议

• Authors: Akrati Saxena
• Subjects: Subjects: Social and Information Networks (cs.SI)
• Arxiv link: https://arxiv.org/abs/2609.22914
• Pdf link: https://arxiv.org/pdf/2609.22914
• Abstract Reproducibility is essential for scientific progress, enabling validation, fair comparison, and building upon prior work. In computational network science (CNS), however, reproducibility remains limited due to missing code, inaccessible datasets, and insufficient reporting of experimental details. This paper presents a taxonomy of reproducibility in CNS, structured around artifact availability, algorithmic clarity, experimental environments, and data processing and experimental pipelines. To systematically examine these challenges, we conduct four case studies spanning diverse methodological settings: topic-based influential user detection (network science and natural language processing-based methods), influence-based community detection (pure network-based methods), influence maximization (classical, heuristic, approximation, and AI-based mixed approaches), and reinforcement learning for network analysis (learning-based methods). Across these domains, we observe a consistent lack of publicly available artifacts, particularly code and datasets, hindering verification and comparison of results. We identify key causes of this reproducibility gap, including limited incentives for sharing artifacts, data access restrictions, incomplete experimental descriptions, and a complex methodological pipeline. Finally, we outline recommendations to improve reproducibility, including mandatory artifact sharing policies, standardized benchmarks, and comprehensive reporting of experimental setups. Addressing these gaps is critical to ensure transparency, comparability, and sustained progress in computational network science.
• 中文摘要 可重复性对于科学进步至关重要，能够进行验证、公平比较，并建立在既有工作的基础上。然而，在计算网络科学（CNS）中，由于缺失代码、数据集无法访问以及实验细节报告不足，可重复性仍然有限。本文提出了CNS中可重复性的分类法，围绕工件可用性、算法清晰度、实验环境以及数据处理和实验流程进行结构化。为系统地探讨这些挑战，我们进行了四个跨越多种方法论环境的案例研究：基于主题的影响力用户检测（网络科学和基于自然语言处理的方法）、基于影响力的社区检测（纯基于网络的方法）、影响力最大化（经典、启发式、近似和基于人工智能的混合方法）以及基于学习的网络分析强化学习。在这些领域中，我们观察到公开工件，尤其是代码和数据集的持续缺乏，阻碍了验证和结果比较。我们识别了这一可重复性差距的关键原因，包括有限的共享工件激励、数据访问限制、实验描述不完整以及复杂的方法论流程。最后，我们提出了提高可重复性的建议，包括强制工件共享政策、标准化基准测试以及对实验设置的全面报告。解决这些差距对于确保计算网络科学的透明度、可比性和持续进步至关重要。

Computationally efficient safe exploration in reinforcement learning

强化学习中的计算高效安全探索

• Authors: Shreeram Murali, Shankar A. Deka, Dominik Baumann
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.22919
• Pdf link: https://arxiv.org/pdf/2609.22919
• Abstract Reinforcement learning in real-life applications requires safety guarantees during exploration. Typical reinforcement learning algorithms do not provide such guarantees, and many modifications that do rely on Gaussian processes (GPs), which have a large computational cost. We propose a computationally lightweight algorithm based on the Nadaraya-Watson estimator that safely explores and optimizes constrained Markov decision processes (MDPs). Our algorithm, \textsc{CoLSafe-MDP}, uses an estimator that scales in constant-time with bounds on the estimates, a significant improvement from its GP-based counterparts that scale cubically with the number of data points. We then evaluate its performance in a grid-based environment and on observational Martian terrain data.
• 中文摘要 现实应用中的强化学习在探索过程中需要安全保障。典型的强化学习算法不提供此类保证，许多提供此类保证的修改依赖于高斯过程（GP），而高斯过程计算成本较高。我们提出了一种基于Nadaraya-Watson估计量的计算轻量级算法，能够安全地探索和优化受限马尔可夫决策过程（MDP）。我们的算法\textsc{CoLSafe-MDP}使用一个估计量以常数时间扩展且估计值有界限的估计器，这相比其基于GP的GP算法显著提升，后者随数据点数量按立方扩展。随后，我们评估其在网格环境和火星地形观测数据中的表现。

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

RewardVerse：基于评分标准的策略优化，用于视频奖励建模

• Authors: Zhenchen Tang, Yang Li, Songlin Yang, Bo Peng, Xiaotong Zhao, Shuai Li, Haotian Fan, Alan Zhao, Jing Dong
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.22947
• Pdf link: https://arxiv.org/pdf/2609.22947
• Abstract Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existing video reward models often produce unstable scalar scores because they directly map complex, subjective video quality into a single score without explicit evaluation criteria. This leads to scalar drift, where the scoring scale collapses or shifts across different prompts, making the reward unreliable for RL. Drawing inspiration from professional human annotation engineering, we address this problem with RewardVerse, a rubric-based video reward framework that introduces a dynamic rubric as an intermediate representation between the evaluation query and the scorer. Instead of unconstrained direct scoring, RewardVerse first generates explicit evaluation criteria and then performs rubric-guided scoring, providing a stable semantic anchor that mitigates scalar drift. To efficiently optimize this collaborative pipeline, we propose Rubric-Guided Policy Optimization (RGPO), a two-stage training algorithm. RGPO first warms up the scorer using self-evolving seed rubrics and then jointly optimizes the rubric generator to produce query-adaptive evaluation criteria while continuously aligning the scorer with human ratings. Extensive experiments on the 16-dimensional EvalVerse benchmark and external datasets demonstrate that RewardVerse mitigates scalar drift, achieves state-of-the-art performance on both pointwise and pairwise evaluation, and provides a robust and interpretable reward signal for RL in video generation.
• 中文摘要 强化学习（RL）对于优化视频生成模型至关重要，而稳健的奖励模型（RM）是基石。然而，现有的视频奖励模型常常产生不稳定的标量分数，因为它们将复杂、主观的视频质量直接映射到一个评分中，而没有明确的评分标准。这导致了标量漂移，即评分尺度在不同提示之间崩溃或移动，使得奖励对强化学习来说不可靠。我们借鉴了专业的人工注释工程，通过RewardVerse解决了这一问题，这是一种基于评分标准的视频奖励框架，引入动态评分标准作为评估查询与评分者之间的中间表示。RewardVerse不是直接无约束地评分，而是先生成明确的评分标准，然后执行评分标准引导评分，提供一个稳定的语义锚点，减轻标量漂移。为了高效优化这一协作流程，我们提出了两阶段训练算法——评分标准引导策略优化（RGPO）。RGPO首先通过自演化的种子评分标准预热评分器，然后联合优化评分标准生成器，生成查询自适应的评估标准，同时持续使评分器与人类评分对齐。在16维EvalVerse基准测试和外部数据集上的大量实验表明，RewardVerse能够减轻标量漂移，在点数和成对评估中都达到最先进的性能，并为视频生成中的强化学习提供了稳健且可解释的奖励信号。

General Collaborative Intelligence: Architecting Cognition for Resilient Multi-Agent Ecosystems

通用协作智能：为韧性多智能体生态系统构建认知

• Authors: Lei Zhang, Chun Ye, Le Yang, Zhaozhong Wang, Deng-Ping Fan, Hang Dai, Binglu Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22967
• Pdf link: https://arxiv.org/pdf/2609.22967
• Abstract Multi-agent unmanned systems are moving from isolated, ego-centric sensing toward collaborative intelligence, in which distributed agents exchange compact features to overcome a local observation trap that no single agent can escape: occlusions, finite sensor range, and environmental degradation. The field has matured across architectural, communication, embodied, resilience, and trust dimensions, yet existing surveys examine these dimensions in isolation and rarely expose their dependencies. This review offers a unified synthesis through two complementary lenses. The first is a five-dimensional taxonomy spanning collaboration stage, communication paradigm, fusion architecture, learning strategy, and application domain. The second is three cognitive synergy conditions, Semantic Disambiguation, Pragmatic Information Exchange, and Proactive Informational Foraging, that turn cognitive synergy into operational criteria. Across these lenses we survey collaboration architectures and topologies, neural-communication co-design that treats the channel as a differentiable pipeline component, embodied action-perception loops via multi-agent reinforcement learning, and resilience mechanisms for synchronization, uncertainty quantification, and label-efficient learning. We then map these advances onto four operational domains, V2X, unmanned aerial, industrial logistics, and smart cities, and onto the safety-privacy-utility triad. To counter benchmark saturation and evaluation fragmentation, we propose GCI-Bench, a five-pillar scoring protocol with a maturity model that makes the trade-offs of collaborative methods comparable across studies. A critical reflection on reproducibility, the sim-to-real gulf, and conditions under which collaboration degrades performance identifies open challenges and charts directions toward general collaborative intelligence under real-world uncertainty.
• 中文摘要 多智能体无人系统正从孤立、以自我为中心的感知转向协作智能，其中分布式智能体通过交换紧凑特征，克服单个智能体无法逃脱的局部观察陷阱：遮挡、有限的传感器范围和环境退化。该领域在架构、通信、具身、韧性和信任等维度上已成熟，但现有调查仅单独考察这些维度，鲜少揭示其依赖关系。本综述通过两个互补视角提供统一综合。第一是跨协作阶段、通信范式、融合架构、学习策略和应用领域的五维分类法。第二是三种认知协同条件：语义消歧、实用信息交换和主动信息采集，将认知协同转化为操作标准。在这些视角下，我们考察了协作架构和拓扑、将通道视为可微流水线组件的神经通信协同设计、通过多智能体强化学习实现的具身动作-感知循环，以及同步、不确定性量化和标签高效学习的韧性机制。随后，我们将这些进展映射到四个运营领域：V2X、无人机、工业物流和智慧城市，以及安全-隐私-公用三位一体。为应对基准饱和和评估碎片化，我们提出了GCI-Bench，这是一套五大支柱评分协议，采用成熟度模型，使协作方法在各研究间的权衡具有可比性。对可重复性、模拟与现实差距以及协作如何降低性能的条件进行了批判性反思，识别了开放挑战，并在现实世界不确定性下为通用协作智能指明了方向。

DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale

DeepSeek 弹性计算（DSec）：一个用于大规模有效代理训练的沙盒基础设施

• Authors: Jialiang Huang, Hongxuan Tang, Jingchang Chen, Yuxuan Liu, Yixiao Chen, Yuan Cheng, Yi Tao, Jingli Zhou, Yupeng Chen, Haoyu Chen, Jiarui Wang, Shengkai Lin, Chuqi Zhang, Bryan Lee Teng, Lian Guo, Zhe Fu, Wenjun Gao, Yisong Wang, Liang Zhao, Zehao Wang, Ziwei Xie, Yongqiang Guo, Peixin Cong, Ziyi Gao, Shuiping Yu, Hanwei Xu, Zuofan Wu, Zhizhou Ren, Yuyang Zhou, Bowei Zhang, Zhihuan Huang, Qihao Zhu, Lei Wang, Tianle Lin, Han Yu, Jiewen Hu, Dejian Yang, Shuo Yang, Shanghao Lu, Shaoyuan Chen, Junjie Qiu, Zhangli Sha, Yinmin Zhong, Yongtong Wu, Shiyu Wang, Wei Liu, Bingzheng Xu, Longhao Chen, Qiushi Du, Yuzhen Huang, Shirong Ma, Yaohui Wang, Mingshu Chen, Tongrui Xiong, Y.C. Yan, Haowen Luo, Haofen Liang, Xiaokang Zhang, Weihao Zeng, Runxin Xu, Peiyi Wang, Jinhua Zhu, Ruoyu Zhang, Wenkai Yang, Qi Tang, Jiping Yu, Tian Ye, Ruizhe Pan, Honghui Ding, Xiaodong Liu, Lingxiao Luo, Zhihong Shao, Yuhan Wu, Jibai Lu, Wen Liu, Haoling Zhang, Jingcheng Hu, Yaoyang Ye, Chaofan Lin, Zhaochen Zhang, Jianan Tong, Hengxu Wu, Zhihao Li, Yicheng Wang, Luyao Wang, Yuzhuo Bai, Lingyue Fu, Ruifan Xu, Y.Z. Wang, Zonglin Li, Mingqi Wei, Haiyang Shen, Chengyuan Zhang, Chao Jin, Zili Zhang, R.H. Yang, Xinbo Xu, Jian Zhou, Ruidong Zhu, Yuzhe Guo
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.22978
• Pdf link: https://arxiv.org/pdf/2609.22978
• Abstract Large-scale agentic training and evaluation with large language models (LLMs) rely on isolated, stateful execution environments in which models inspect repositories, invoke tools, execute commands, and interact with task-specific services. These workloads create sandboxes in large bursts, span heterogeneous functionality and isolation requirements, retain state across long interactions, and draw from large image corpora with limited reuse. Supporting them therefore requires an elastic execution platform rather than a single sandbox runtime. This report presents DeepSeek Elastic Compute (DSec), a production sandbox platform that exposes FnCall, container, microVM, and full-VM sandbox backends through a unified SDK. DSec coordinates placement and lifecycle management across the cluster, composes environments from independently versioned layers, combines memory sharing, reclamation, and CPU scheduling for high-density execution, and loads image data on demand from Fire-Flyer File System (3FS), a cluster-wide distributed filesystem. DSec is co-designed with the reinforcement learning (RL) framework, decouples stateful rollout execution from preemptible GPU training, coordinates sandbox lifecycle with training to preserve rollout state while reclaiming idle resources, and mitigates agent misbehavior such as reward hacking. A single production-scale unit of DSec spans around 160 nodes, serving about 3 million sandboxes per day; in production, it supports over 380,000 concurrent sandboxes and sustains over 5,000 sandbox creations per second. Our evaluation and deployment experience show that these mechanisms reduce environment setup and image-distribution overhead, improve memory efficiency, and preserve latency-sensitive performance under high-density overcommit.
• 中文摘要 大规模代理训练和评估依赖于孤立的、有状态的执行环境，模型在其中检查仓库、调用工具、执行命令并与任务特定服务交互。这些工作负载以大规模突发方式创建沙盒，跨越异构功能和隔离需求，在长时间交互中保持状态，并有限度地从大型图像语料库中提取资源。因此，支持它们需要一个弹性执行平台，而非单一沙盒运行时。本报告介绍了DeepSeek Elastic Compute（DSec），一个生产沙盒平台，通过统一SDK暴露FnCall、容器、microVM和全虚拟机沙箱后端。DSec 协调集群内的布局和生命周期管理，从独立版本化层组成环境，结合内存共享、回收和 CPU 调度以实现高密度执行，并按需从 Fire-Flyer 文件系统（3FS）加载图像数据，该文件系统是一个集群范围的分布式文件系统。DSec 与强化学习（RL）框架共同设计，将有状态的部署执行与抢占式 GPU 训练解耦，协调沙箱生命周期与训练，以保留部署状态同时回收闲置资源，并减少代理行为不当，如奖励黑客行为。单个生产规模的 DSec 设备覆盖约 160 个节点，每天服务约 300 万个沙盒;生产环境中，它支持 38 万多个并发沙盒，每秒维持超过 5,000 个沙盒创建。我们的评估和部署经验表明，这些机制降低了环境搭建和镜像分发的开销，提高了内存效率，并在高密度超额度下保持了对延迟敏感的性能。

Attributable Post-Rationalization in RAG Citations: A Controlled Reproduction and an RLVR Comparison

RAG引用中的归因性后合理化：受控复制与RLVR比较

• Authors: Mehedi Khan, Md. Shariful Islam Bhuyan
• Subjects: Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.23053
• Pdf link: https://arxiv.org/pdf/2609.23053
• Abstract A RAG system can hand you the right answer and cite a source it did not actually use. Models output these unfaithful citations via post-rationalization: they write the answer first and then attach a citation to whatever passage looks close enough. Search agents are now trained with reinforcement learning from verifiable rewards (RLVR), which pays them for getting the answer right. We asked whether that training also teaches them to cite honestly. Improving an existing methodology with a required control, we compared an instruction-tuned model against three RLVR agents trained from it, on four question-answering datasets, using only free-tier Kaggle GPUs. Post-rationalization is everywhere: on Wikipedia-based questions roughly one citation in seven is unfaithful. RLVR does not fix it. The agents post-rationalize at their base model's rate, and one lands slightly worse. Rewarding correct answers buys nothing in citation faithfulness, so faithfulness has to be trained and measured on its own terms.
• 中文摘要 RAG系统可以给你正确答案，并引用它实际上未使用的来源。模型通过后合理化输出这些不忠实引用：先写答案，然后对看起来足够接近的段落附加引用。搜索代理现在接受可验证奖励强化学习（RLVR）训练，正确回答后获得报酬。我们询问该训练是否也教会他们诚实引用。改进现有方法，我们比较了指令调优模型与三名RLVR代理，基于四个问答数据集，仅使用免费版KaggleGPU。后合理化无处不在：在维基百科问题中，大约七个引用中有一个是不忠实的。RLVR无法解决这个问题。代理按基础模型的频率进行后合理化，其中一个表现略差。奖励正确答案并不能提升引用忠实度，因此忠实度必须按照自身标准进行训练和衡量。

Verti-WM: A Physics-Aided Exteroceptive World Model for Off-Road Reinforcement Learning

Verti-WM：一种物理辅助的外感知世界模型，用于越野强化学习

• Authors: Chenhui Pan, Tong Xu, Xuesu Xiao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23118
• Pdf link: https://arxiv.org/pdf/2609.23118
• Abstract Reinforcement learning for off-road navigation requires extensive vehicle-terrain interaction data, which are costly to collect in high-fidelity simulation. World models offer a promising alternative by replacing simulator roll-outs during policy optimization. However, an off-road world model must condition state transitions on exteroceptive terrain information, which proprioception alone does not provide. This challenge is further amplified by the need to model both rigid and deformable terrain, where data-driven and physics-based approaches offer complementary strengths. We propose Verti-WM, a physics-aided exteroceptive world model that recurrently fuses a frozen Transformer for rigid terrain and a neuro-symbolic terramechanics model for deformable terrain. Elevation and semantic observations queried from a supplied map at each predicted pose condition fusion, enabling six-degree-of-freedom rollouts for policy optimization without further simulator access. Verti-WM reduces prediction error by 34.6% and 21.7% over data-driven and physics-based baselines, respectively. Policies trained entirely within Verti-WM achieve comparable task success rates while reducing computation time by 23.6X relative to direct training in the high-fidelity simulator. We further validate Verti-WM using real-world data, enabling policy optimization within learned real-world kinodynamics and achieving a 80% success rate on the Verti-4-Wheeler platform, compared with 40% for direct sim-to-real transfer.
• 中文摘要 越野导航的强化学习需要大量车辆与地形相互作用的数据，而在高保真模拟中收集这些数据成本较高。世界模型通过在策略优化期间替代模拟器展开，提供了有前景的替代方案。然而，越野世界模型必须以外感受地形信息为条件，而本体感觉本身无法提供这些信息。这一挑战因需要同时建模刚性和可变形地形而进一步加剧，数据驱动和基于物理的方法互补优势。我们提出了Verti-WM，一种物理辅助的外感知世界模型，它循环融合刚性地形的冻结变形器和可变形地形的神经符号地形力学模型。在每个预测姿态条件融合处，从提供的地图查询高程和语义观测，实现六自由度滚动以优化策略，无需进一步模拟器访问。Verti-WM分别在数据驱动和基于物理的基线上降低了34.6%和21.7%的预测误差。完全在Verti-WM中训练的策略实现了相当的任务成功率，同时计算时间比高保真模拟器直接训练减少了23.6倍。我们进一步利用真实世界数据验证Verti-WM，使得在学习到的现实世界运动动力学中实现策略优化，并在Verti-4-Wheeler平台上实现了80%的成功率，而直接模拟到现实传输的成功率为40%。

Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochs

连续时间情节MDP中具有泊松决策纪元的可证明高效强化学习

• Authors: Kenny Guo, Valentio Iverson, Sahan Wijetunga, William Chang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.23127
• Pdf link: https://arxiv.org/pdf/2609.23127
• Abstract Many real-world reinforcement learning (RL) problems evolve in continuous time, where decisions occur at irregular, event-driven intervals rather than at fixed discrete steps. We study episodic continuous-time Markov Decision Processes (MDPs) in which decision epochs are governed by a homogeneous Poisson process and the reward and transition dynamics vary smoothly over time. We consider both a fixed number of jumps per episode and a fixed time budget with a random number of Poisson decision epochs. Under a Lipschitz continuity assumption in time, we exploit local smoothness through discretization and extend both UCRL (Auer and Ortner 2006) and Q-learning (Jin et al. 2018) to this setting, proving $\widetilde{O}(T^{2/3})$ regret bounds for both model-based and model-free algorithms. Finally, we establish matching $\widetilde{\Omega}(T^{2/3})$ minimax lower bounds, showing that the rate is optimal up to logarithmic factors. These results provide the first tight regret guarantees for Lipschitz-smooth continuous-time episodic MDPs with Poisson decision epochs.
• 中文摘要 许多现实世界强化学习（RL）问题在连续时间中演变，决策发生在不规则的事件驱动区间，而非固定的离散步骤。我们研究了情节连续时间马尔可夫决策过程（MDP），其中决策历由齐次泊松过程支配，奖励和过渡动态随时间平滑变化。我们考虑每集跳跃次数固定和随机数量泊松决策历时的固定时间预算。在Lipschitz时间连续性假设下，我们利用离散化实现局部平滑性，并将UCRL（Auer和Ortner 2006）和Q-learning（Jin等，2018）扩展到此环境，证明了基于模型和无模型算法的$\widetilde{O}（T^{2/3}）$后悔界限。最后，我们建立了匹配的$\widetilde{\Omega}（T^{2/3}）$ 极小极大值下界，表明速率在对数因子下最优。这些结果为具有泊松决策纪元的Lipschitz光滑连续时间段段MDP提供了首个严格的遗憾保证。

Proximal Residual Value Functions for Consistent Planning and Real-Time Execution

用于一致规划和实时执行的近端残值函数

• Authors: Harrison Waldon, Carson Eisenach, Akhil Bagaria, Daniel Russo, Dominique Perrault-Joncas, Alisha Zachariah, Dean Foster
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.23242
• Pdf link: https://arxiv.org/pdf/2609.23242
• Abstract We study two-timescale decision systems in which a planning layer periodically supplies a continuation-value function to a real-time optimizer that allocates arriving resources, with inventory placement as our motivating application. We propose an end-to-end reinforcement learning (RL) method for learning this function using \emph{proximal residual value functions}, which combine a strictly convex potential of post-decision inventory with a learned convex residual. This general form yields a well-posed optimization layer that supports end-to-end differentiation while preserving an explicit convex objective for real-time execution. We characterize the necessary and sufficient conditions under which a smooth value function yields decisions that are consistent across the planning and execution timescales. In an offline simulation using historical inventory arrival and demand patterns from a large e-commerce retailer, learned proximal residual value functions reduce total routing and transfer cost relative to a historical-production-system proxy by 5.0%.
• 中文摘要 我们研究了两时间尺度决策系统，其中规划层定期向实时优化器提供延续价值函数，实时优化器分配到达的资源，库存配置作为我们的动机应用。我们提出了一种端到端强化学习（RL）方法，利用 \emph{近端残差价值函数}学习该函数，结合决策后库存的严格凸势和学习到的凸残差。该一般形式产生了适态优化层，支持端到端微分，同时保持显式凸目标以供实时执行。我们刻画了平滑价值函数在规划和执行时间尺度中产生一致决策的必要条件和充分条件。在利用大型电商历史库存到达和需求模式进行离线模拟时，学习到的近端残余价值函数相较于历史生产系统代理降低了5.0%的总路由和传输成本。

Latent Telepathy: Multi-Robot Communication with Self-Supervised Perceptual Latents

潜在心灵感应：多机器人与自我监督的感知潜能交流

• Authors: Howard Wang, Han Zheng, Cathy Wu
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.23269
• Pdf link: https://arxiv.org/pdf/2609.23269
• Abstract In a decentralized multi-robot team under partial observability, the fact that decides a robot's next action is often visible only to a teammate. Existing decentralized methods communicate kinematic information, such as position or planned trajectory, which cannot convey what the teammate perceives. Learned communication in multi-agent reinforcement learning (MARL) can carry perceptual content, but the resulting messages are task-coupled and opaque. We propose Latent Telepathy. Each robot broadcasts the perceptual latent vector it already computes for its own use, the output of an encoder trained with a self-supervised joint-embedding predictive objective, frozen, and shared across the team. A teammate learns to act on it from task reward alone. Because the encoder already runs for perception, the message costs no additional computation and a single compact vector of bandwidth. Because the encoder is frozen before any policy is trained, the message means the same thing to every robot, and the receiving robot is never told what it means. We evaluate Latent Telepathy with a content-controlled protocol in which bandwidth, latency, topology and receiver are held fixed and only the message content varies. Broadcasting the latent lets a navigator avoid an occluded hazard in 99.7% of episodes, matching a noiseless hand-designed message. Position and trajectory messages remain at chance, and the raw camera image, 186 times wider, is less reliable than the compressed latent. The result holds from a discrete gridworld to rendered pixels under continuous velocity control, and the encoder decodes the hazard from a physical robot's camera in 102 of 102 live decisions. We also identify a requirement for porting MARL communication results to continuous control, that the decision a message informs must remain reachable by exploration, and show how to restore it.
• 中文摘要 在一个分散的多机器人团队中，部分可观察性，决定机器人下一步行动的事实通常只有队友可见。现有的去中心化方法传递的是运动学信息，如位置或计划轨迹，无法传达队友所感知的内容。多智能体强化学习（MARL）中的学习交流可以携带感知内容，但产生的信息是任务耦合且不透明的。我们提出了潜在心灵感应。每个机器人广播它已计算好的感知潜在向量，该向量是用自监督联合嵌入预测目标训练的编码器输出，冻结并在团队间共享。队友仅凭任务奖励学会对其采取行动。由于编码器已运行感知，消息无需额外计算，且仅使用一个紧凑的带宽向量。由于编码器在任何策略训练前就被冻结，消息对每个机器人来说含义相同，接收机器人从未被告知其含义。我们用内容控制协议评估潜在心灵感应，该协议中带宽、延迟、拓扑和接收器均固定，只有消息内容变化。广播潜在信号使导航员在99.7%的剧集中避免遮蔽危险，匹配无噪声的手工设计信息。位置和轨迹信息仍处于偶然状态，原始摄像头图像宽度为186倍，可靠性低于压缩潜在图像。结果从离散网格世界到连续速度控制渲染的像素均成立，编码器在102个实时决策中102个解析了物理机器人摄像头的危险。我们还确定了将MARL通信结果移植到持续控制的必要条件，消息所告知的决策必须通过探索保持可达性，并展示如何恢复该结果。

The Evidence Ladder for Reinforcement Learning in Healthcare: From Retrospective Policies to Trusted Interventions

医疗强化学习的证据阶梯：从回顾性政策到可信干预

• Authors: Yunfan Zhao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.23374
• Pdf link: https://arxiv.org/pdf/2609.23374
• Abstract Reinforcement learning (RL) offers a natural language for healthcare decisions whose conse- quences unfold over time, yet most reported progress remains far from routine intervention. Ex- isting surveys organize the field by algorithm or clinical application. We instead review healthcare RL through an evidence ladder: problem formulation, retrospective identification, policy estima- tion, stress testing, prospective evaluation, and lifecycle monitoring. This view connects clinical treatment, patient engagement, and health-system operations while exposing a recurring gap: evi- dence that a policy scores well in a historical dataset is not evidence that it will improve care. We synthesize the assumptions and failure modes at each rung, identify what evidence can and can- not transfer across settings, and propose reporting practices for cumulative evaluation. Restless bandits are included as one special case, not as the organizing framework. The central lesson is that healthcare RL should be evaluated as an intervention embedded in a changing sociotechnical system, rather than only as an optimizer of a retrospective reward.
• 中文摘要 强化学习（RL）为医疗决策提供了一种自然语言，其后果随时间展开，但大多数报告的进展仍远非常规干预。现有调查通过算法或临床应用组织领域。我们通过证据阶梯回顾医疗强化学习：问题制定、回顾性识别、政策估计、压力测试、前瞻性评估和生命周期监测。该观点连接临床治疗、患者参与和医疗系统运营，同时揭示一个反复出现的空白：政策在历史数据集中得分良好并不代表它会改善护理。我们综合各级的假设和失败模式，识别哪些证据能跨环境转移，哪些不能，并提出用于累积评估的报告实践。不安的强盗被列为一个特例，而非组织框架。核心教训是，医疗强化学习应作为嵌入在不断变化的社会技术体系中的干预措施来评估，而不仅仅是回溯性奖励的优化器。

One to More, More to One: Category-Aware Iterative Expert Training for Software Engineering Agents

一对多，多到一：软件工程代理的类别感知迭代专家培训

• Authors: Jie Zhao, Ziyu Jiang, Suhang Zheng, Minghui Shan, Xiaoxiao Xu, Lin Qu
• Subjects: Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.23377
• Pdf link: https://arxiv.org/pdf/2609.23377
• Abstract Repository-level software engineering (SWE) comprises heterogeneous task categories, whose progress under pooled agentic reinforcement learning can be uneven: gains in some categories coincide with regressions in others, while aggregate resolution obscures these changes. Motivated by this category see-saw, we develop a category-aware expert-training and policy-integration framework. Executable task construction and SWE Labeler, an evidence-grounded multi-axis labeling system, organize the training pools. Initial category-specific RL improves average training success while leaving uneven instance-level progress, motivating explicit consolidation of successful behavior and policy-adaptive task selection. Same-origin category experts alternate long-horizon Agentic-miniRL with Refresh-Repair-Expand (RRE): the updated policy refreshes instance mastery, reuses its own verified successful trajectories for Repair SFT, and reselects tasks for further RL. Label-routed multi-teacher on-policy distillation (MOPD) consolidates the experts into one deployable student, with ReLU-gated reward extrapolation keeping only each teacher's improving direction over the reference. Expert training and policy integration require no external model to provide solution trajectories or action targets. We evaluate Pooled RL and Balanced RL, expert development, and single-model integration through aggregate and per-category resolution, the minimum category lift over each joint-RL baseline, and expert-gain recovery. The final MOPD policy achieves mean resolution of 58.04% on Pro-618 and 59.00% on SWE-bench Multilingual, improving over the base model by 5.39 and 2.78 percentage points, respectively.
• 中文摘要 仓库级软件工程（SWE）包含异构任务类别，这些任务类别在合并代理强化学习下的进展可能不均衡：某些类别的进展与其他类别的回归重合，而总体分辨率则掩盖了这些变化。受这一类别摇摆的激励，我们开发了一个类别感知的专家培训和策略集成框架。可执行任务构建和基于证据的多轴标记系统SWE标签器组织训练池。初始类别特定RL提高了平均训练成功率，同时保持实例级进展不均，促使成功行为和策略自适应任务选择的明确整合。同一来源类别专家交替使用长期视野代理迷你RL与刷新-修复-扩展（RRE）：更新策略刷新实例掌握，重用自身验证的成功轨迹用于修复SFT，并重新选择任务进行后续强化学习。标签路由多教师政策提炼（MOPD）将专家整合到一个可部署的学生中，ReLU门槛奖励外推仅保留每位教师对参考的改进方向。专家培训和策略集成无需外部模型提供解决方案轨迹或行动目标。我们评估池化RL和平衡RL、专家开发以及通过汇总和各类别分辨率、每个联合RL基线的最小类别提升和专家收益恢复的单模型集成。最终MOPD政策在Pro-618上实现了58.04%的平均分辨率，在SWE多语言平台上实现了59.00%，分别较基础模型提升了5.39%和2.78个百分点。

RLVR$^{2}$: Reinforcement Learning with Verifiable Rubric-based Ranking

RLVR$^{2}$：基于可验证评分标准的强化学习排名

• Authors: Hao Li, Zhengkun Zhang, Gangqiang Hu, Zhen Zhang, Yude Gao, Dai Dai, Jing Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.23457
• Pdf link: https://arxiv.org/pdf/2609.23457
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, toward multifaceted quality requirements specified by multi-dimensional rubrics. Since policy optimization consumes one scalar per rollout, rubric-based pipelines must map multiple criterion scores into a scalar reward. This aggregation is often treated as score scaling, but it implicitly determines how quality dimensions trade off during training. The prevailing practice, normalizing each criterion and taking a linear combination, assumes that cardinal score differences are comparable across criteria and that gains on one criterion compensate for failures on another; both assumptions are unreliable when criteria are semantically heterogeneous. We propose Reinforcement Learning with Verifiable Rubric-based Ranking (RLVR$^2$), a verifiable ranking paradigm for rubric-based RLVR. For each criterion, RLVR$^2$ converts rubric scores into criterion-specific within-group ordinal outcomes, recovers a latent utility from the resulting comparison matrix, and merges these utilities into one training signal. By retaining only within-group ordering and discarding raw score magnitudes, RLVR$^2$ avoids calibrating heterogeneous rubric scales. It further supports objective-preserving attribute adjustment: auxiliary attributes that correlate with observed rankings but are not training objectives can enter the estimation without expanding the rubric or rewarding them directly. Across three model scales and 16 benchmarks, RLVR$^2$ consistently outperforms representative rubric-based baselines, achieving the best overall performance on most benchmarks at every scale. Analysis shows it controls systematic effects tied to reasoning efficiency and response formatting while preserving the quality objective.
• 中文摘要 带可验证奖励的强化学习（RLVR）正从具有明确正确性信号的任务（如数学和代码）扩展到由多维评分标准指定的多维质量要求。由于策略优化每次部署消耗一个标量，基于评分标准的流程必须将多个标准评分映射为标量奖励。这种聚合通常被视为分数递增，但它隐含决定了训练过程中质量维度的权衡。主流做法是对每个标准进行归一化并采用线性组合，假设基数分数差异在不同标准间是可比的，且在一个标准上的提升可以补偿另一个标准的失误;当标准语义异质时，这两种假设都不可靠。我们提出了基于可验证评分标准排名的强化学习（RLVR$^2$），这是一种基于评分标准RLVR的可验证排名范式。对于每个标准，RLVR$^2$将评分转换为标准特定的组内序数结果，从所得比较矩阵中恢复潜在效用，并将这些效用合并为一个训练信号。通过仅保留组内排序并舍弃原始分数幅度，RLVR$^2$避免校准异质化的评分量表。它还支持保持目标的属性调整：与观察排名相关但非训练目标的辅助属性可以在不扩展评分或直接奖励的情况下进入估计。在三种模型量表和16个基准测试中，RLVR$^2$始终优于代表性的基于评分标准的基线，在大多数基准测试中实现最佳整体表现。分析显示，它控制了与推理效率和响应格式相关的系统效应，同时保持质量目标。

STRIDER: Stepping-Enabled Multi-Gait Hierarchical 3D Loco-Manipulation Framework for Humanoid Robots

STRIDER：用于人形机器人的步进多步态分层3D机车操作框架

• Authors: Yuanzhuo Li, Wen Zhao, Zhe Yong, Xiang Meng, Gang Han, Hengle Ren, Xiaoyang Zheng, Zhen Wang, Yijie Guo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23483
• Pdf link: https://arxiv.org/pdf/2609.23483
• Abstract Humanoid loco-manipulation faces two prominent limitations: controllers using continuous velocity commands cannot precisely regulate individual footholds, while specialized foothold-tracking modules are difficult to integrate with whole-body manipulation. Furthermore, standard action-based imitation distillation primarily transfers expert actions, without explicitly encouraging a shared representation of heterogeneous skills. This paper introduces STRIDER, a hierarchical multi-gait framework to bridge these gaps. The framework integrates terrain-aware 3D stepping logic, Adversarial Motion Priors (AMP)-based natural walking, and Cartesian upper-body control: its stepping expert selects feasible footholds in the stance-foot frame and generates clearance-aware swing trajectories. To fuse distinct walking and stepping experts into one executable student policy, we propose Latent Distillation Proximal Policy Optimization (LD-PPO), a distillation algorithm augmented with teacher-conditioned latent alignment. By jointly optimizing on-policy reinforcement learning, DAgger-based action reconstruction, and latent alignment, LD-PPO transfers expert actions while encouraging a shared skill representation across heterogeneous modes. Simulation and real-robot evaluations on the TianGong Omni humanoid show that LD-PPO outperforms vanilla distillation-PPO in foothold-tracking and posture-tracking accuracy. Deployed on hardware, STRIDER realizes multi-gait loco-manipulation with accurate foothold and end-effector tracking.
• 中文摘要 类人机车操作面临两个显著局限：使用连续速度指令的控制器无法精确调节单个脚点，而专用脚步跟踪模块难以与全身操控集成。此外，标准基于动作的模仿提炼主要传递专家动作，而未明确鼓励共享异质技能的表征。本文介绍了STRIDER，一种层级多步态框架，旨在弥合这些空白。该框架集成了地形感知的3D步进逻辑、基于对抗运动先验（AMP）的自然行走和笛卡尔上半身控制：其步步专家选择站姿-足部框架中的可行脚点，并生成空间感知的挥杆轨迹。为了将不同的步行和步调专家融合到一个可执行的学生策略中，我们提出了潜在蒸馏近端策略优化（LD-PPO），这是一种通过教师条件潜在对齐增强的提纯算法。通过联合优化策略强化学习、基于DAgger的动作重建和潜在对齐，LD-PPO在转移专家动作的同时，促进跨异构模式的共享技能表征。对天宫全人形机器人的模拟和实机评估显示，LD-PPO在脚踏跟踪和姿态跟踪精度上优于普通蒸馏-PPO。部署于硬件上，STRIDER实现了多步态的机车操作，并实现了精准的脚踏和末端执行器跟踪。

PINGU: Extending Air-Bearing Spacecraft Emulators with Open-Source Actuators and Learned Control for Contact-Rich Proximity Operations

PINGU：扩展带有开源执行器和学习控制的空气轴承航天器模拟器，实现接触密集的近距离操作

• Authors: Ricard Marsal I Castan, Akiyoshi Uchida, Aman Arora, Pedro Lima, Matteo El-Hariry, Anrej Orsula, Francesco Grella, Antoine Richard, Cedric Pradalier, Miguel A. Olivarez-Mendez
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23554
• Pdf link: https://arxiv.org/pdf/2609.23554
• Abstract Low-cost planar air-bearing testbeds have matured into a standard proxy for free-flying spacecraft GNC, but they remain largely thruster-only and are rarely equipped for contact-rich, inertia-coupled manipulation. Building on the open-source ATMOS testbed, we contribute a reaction wheel and two force/torque-sensed robotic arms (LEVION) with interchangeable end-effectors, integrated as first-class control actuators through a unified ROS 2 abstraction layer. On top of the software stack we build a reinforcement-learning training environment and digital twin, and a controller that exploits these added degrees of freedom, letting classical optimal controllers and learned policies be swapped on the same hardware without modification. We validate the integrated system, PINGU, across four benchmark tasks: point-to-pose navigation (classical LQR vs. sim-to-real PPO), dynamic disturbance rejection under arm-induced center-of-mass shifts, reaction-wheel momentum stabilization, and force-controlled docking. The results show that these additions extend an ATMOS-class emulator into the contact-rich regime and bridge classical optimal control and reinforcement learning on one reproducible platform.
• 中文摘要 低成本平面空气轴承测试平台已成熟为自由飞行航天器GNC的标准代理，但它们大多仅依赖推进器，且很少配备接触丰富、惯性耦合的操作能力。基于开源的ATMOS测试平台，我们贡献了一个反作用轮和两只力/扭矩感应机械臂（LEVION），配备可更换的末端执行器，通过统一的ROS 2抽象层集成为一流的控制执行器。在软件栈之上，我们构建了一个强化学习训练环境和数字孪生，以及一个利用这些额外自由度的控制器，使经典的最优控制器和学习策略在同一硬件上无需修改即可切换。我们在四个基准任务中验证了集成系统PINGU：点对姿态导航（经典LQR与模拟到真实PPO）、臂引发质心移动下的动态扰动拒绝、反作用轮动量稳定和力控对接。结果表明，这些新增功能将ATMOS级仿真器扩展到接触丰富区间，并在一个可复现平台上架起经典最优控制与强化学习桥梁。

Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models

先听再推理：基于感知的测试时间强化学习，适用于大型音频语言模型

• Authors: Jiaheng Dong, Xiaofeng Yu, Jean Honorio, Abhirup Ghosh, Hong Jia, Ting Dang
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2609.23589
• Pdf link: https://arxiv.org/pdf/2609.23589
• Abstract Large audio-language models (LALMs) are increasingly used for a broader range of audio reasoning tasks. These models typically incorporate audio representations into a large language model (LLM) backbone to enable multimodal reasoning. Recent test-time reinforcement learning (TTRL) methods further improve LLM reasoning capability by leveraging unlabelled test data after pre-training. However, the importance of the perceptual capability of LALMs remains underexplored, particularly how much acoustic evidence is integrated and relied upon during reasoning, and how this contributes to final task performance. This gap limits the development of effective post-training methods like TTRL for audio reasoning. In this work, we first analyse how audio information is integrated and utilised during reasoning process. We quantify layer-wise perceptual reliance and show that stronger acoustic reliance is associated with higher accuracy and a larger performance gain attributable to the audio input. Building on this, we propose Perception-Grounded TTRL (PG-TTRL), which aligns label-free test-time optimisation with perceptually grounded reasoning, encouraging the model to structure its reasoning more strongly on the audio input. Experiments across LALMs and benchmarks show that PG-TTRL consistently improves reasoning performance over both the base models and standard TTRL, showing the value of perceptual-grounding optimisation for test-time audio reasoning.
• 中文摘要 大型音频语言模型（LALMs）越来越多地被用于更广泛的音频推理任务。这些模型通常将音频表示整合到大型语言模型（LLM）骨干中，以实现多模态推理。最新的测试时强化学习（TTRL）方法通过利用预训练后的未标记测试数据，进一步提升了LLM推理能力。然而，LALM感知能力的重要性仍未被充分探讨，特别是在推理过程中整合和依赖多少声学证据，以及这些证据如何影响最终任务的表现。这一空白限制了如TTRL等有效后期训练方法的发展。本研究首先分析了音频信息在推理过程中的整合和利用。我们量化了层级感知依赖，并表明更强的声学依赖与更高的准确性和更大的性能提升相关，这些数据归因于音频输入。基于此，我们提出了感知基础TTRL（PG-TTRL），该方法将无标签测试时间优化与感知基础推理相结合，鼓励模型更强力地构建音频输入的推理。跨LALM和基准测试的实验表明，PG-TTRL在基础模型和标准TTRL上持续提升推理性能，显示了感知基础优化对测试时间音频推理的价值。

Spiking Neural Network Actor-Critic Proximal Policy Optimization Control for Autonomous UAV Navigation Through Constrained Openings in Civil Infrastructure and Buildings

Spiking Neural Network Actor-Critic 通过受限开口在民用基础设施和建筑物中实现自主无人机导航的近端策略优化控制

• Authors: Francis Noah Walugembe, Maciej Wielgosz, Tomaž Goričan, Matej Mertik
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.23643
• Pdf link: https://arxiv.org/pdf/2609.23643
• Abstract Autonomous navigation of unmanned aerial vehicles in constrained three-dimensional environments has been a challenge in the robotics domain. The application of autonomous unmanned aerial vehicles in civil infrastructure inspection involves the use of such vehicles in bridge inspection, tunnel inspection, and structural inspection. The use of deep reinforcement learning in the autonomous navigation of unmanned aerial vehicles has been successful in constrained environments. However, the computational cost of the algorithm limits the application of the algorithm in the autonomous navigation of unmanned aerial vehicles. This paper proposes the use of the spiking neural network-based Proximal Policy Optimization algorithm in the autonomous navigation of unmanned aerial vehicles in constrained sequential environments. The proposed algorithm integrates the use of spike-based actor-critic reinforcement learning with the Proximal Policy Optimization algorithm. The proposed algorithm uses the stochastic Gaussian policy in the autonomous navigation of unmanned aerial vehicles. The proposed algorithm was implemented in the autonomous navigation of unmanned aerial vehicles in constrained 3D environments. The proposed algorithm was successful in completing 1913 episodes out of more than 3000. The proposed algorithm was successful in passing an average of 2.10 windows per episode. The proposed algorithm was successful in achieving a success rate of 63.77%. The proposed algorithm was successful in achieving success rates of more than 90% in the later stages of the algorithm.
• 中文摘要 无人机在受限三维环境中的自主导航一直是机器人领域的一大挑战。自主无人机在民用基础设施检查中的应用涉及桥梁检查、隧道检查和结构检查。深度强化学习在无人机自主导航中已在受限环境中取得成功。然而，该算法的计算成本限制了该算法在无人机自主导航中的应用。本文提出在受限序列环境中，利用基于尖峰神经网络的近点策略优化算法进行无人机自主导航。该算法将基于尖峰的演员-批评者强化学习与近端策略优化算法集成。该算法在无人机自主导航中采用随机高斯策略。该算法在受限三维环境中实现了无人机的自主导航。该算法成功完成了3000多集中的1913集。该算法平均每集通过2.10个窗口。该算法成功率为63.77%。该算法在算法后期成功率超过90%。

Which Constraints Are Missing? Ask the Verifier: Graded Rewards for Constraint-Following Music Generation

缺少哪些约束？问验证者：遵循约束的音乐生成的分级奖励

• Authors: Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoyu Ma, Haoran Shou, Xiaoying Tang
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.23665
• Pdf link: https://arxiv.org/pdf/2609.23665
• Abstract Constraint-following music generation asks a score to satisfy several user-specified properties at once, each checkable programmatically (key, meter, length, range, final note, rhythm, motion and form), yet no existing benchmark isolates this capability. We construct MusicConstraintBench, 2,180 items over eight constraint families, on which current models fail once a few constraints are combined. The natural remedy is reinforcement learning with these verifiers as reward, yet we observe that a reward paid only when every property holds leaves most training groups without a learning signal: over the first 50 updates, 0.550 of rollout groups score identically and receive no gradient, even though a failing score typically misses only one requested property. Under the joint criterion, rollouts for a prompt tend to fail together, so a binary reward cannot separate a nearly correct score from a malformed one. We therefore introduce MusicRLVR, which pays graded per-property credit behind a hard validation gate that rejects malformed outputs, plus a joint-satisfaction bonus, requiring no human annotation, learned reward model, or music-domain fine-tuning. On MusicConstraintBench, MusicRLVR lifts Qwen3-4B-Instruct from 0.160 to 0.807 on mixed constraints and leads every zero-shot baseline including Llama-3.1-70B at 0.380. It also generalises to property combinations unseen in training and to out-of-range parameter values, showing that verifiable rewards need not presuppose a target output.
• 中文摘要 遵循约束的音乐生成要求分数同时满足多个用户指定的属性，每个属性可通过程序检查（键、拍子、长度、音域、终音、节奏、运动和形式），但目前没有任何基准测试将该能力隔离开来。我们构建了MusicConstraintBench，包含8个约束族中的2180条项目，当前模型在几个约束组合后会失败。自然的解决方法是用这些验证器作为奖励的强化学习，但我们观察到，只有当所有属性都成立时支付奖励，大多数训练组却没有学习信号：在前50次更新中，只有0.550个rollout组得分相同且没有梯度，尽管失败分数通常只漏掉一个请求的属性。在联合准则下，提示的rollout往往一起失败，因此二元奖励无法区分接近正确的分数和不完整的分数。因此，我们引入了MusicRLVR，它在硬验证门后按属性分级支付积分，该门拒绝了格式化的输出，并附带联合满意度奖励，无需人工注释、学习奖励模型或音乐领域微调。在MusicConstraintBench上，MusicRLVR将Qwen3-4B-Ininstruction在混合约束下从0.160提升至0.807，并在包括Llama-3.1-70B在内的所有零次基线中均领先0.380。它还推广到训练中未见的属性组合和超出范围的参数值，表明可验证的奖励不必预设目标输出。

Statistical Convergence of Transformer Encoder-Accelerated Robust Reinforcement Learning

变压器编码器加速强化强化学习的统计收敛

• Authors: Suman Banerjee, Hiroyasu Tsukamoto
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.23775
• Pdf link: https://arxiv.org/pdf/2609.23775
• Abstract Obtaining the optimal action-value function in Markov decision processes is computationally intensive in large state--action spaces. In this study, we present statistically rigorous convergence results for a robust reinforcement learning algorithm warm-started by a transformer-based action-value function prediction, where natural language prompts encode task specifications. Our framework adopts the R-contamination model to characterize uncertainty in the state transition kernel, and employs conformal prediction to certify convergence via trajectory-level nonconformity scores constructed from the contracting Bellman residual. The resulting conformal quantile bounds the gap between the running and optimal action-value functions simultaneously over all iterations, thereby yielding a pre-certified stopping rule that requires little knowledge of the true transition kernel. Numerical case studies on perturbed maze environments of varying size and contamination level confirm that the transformer-based warm start measurably reduces the initial error and accelerates convergence, while the proposed conformal bounds track the true error trajectory more tightly than existing guarantees.
• 中文摘要 在大型状态-动作空间中，获得马尔可夫决策过程中最优动作-值函数计算量极大。本研究呈现了基于变换器的动作-值函数预测热启动的稳健强化学习算法的统计严谨收敛结果，其中自然语言提示编码任务规范。我们的框架采用R污染模型来描述状态转移核的不确定性，并通过由收缩的贝尔曼残差构建的轨迹级不符合评分来验证收敛性。所得的共形分位数在所有迭代中同时界定了运行中的动作值函数与最优动作值函数之间的间隙，从而产生一个预先认证的停止规则，几乎不需要对真实过渡核的了解。对不同规模和污染程度的扰动迷宫环境的数值案例研究证实，基于变压器的暖启动可测量地减少初始误差并加速收敛，而拟定的共形界限比现有保证更紧密地追踪真实误差轨迹。

PackLab: A Comprehensive Framework for Developing, Training, and Evaluating MLLMs in Robotic Bin Packing

PackLab：一个用于开发、培训和评估机器人垃圾桶装卸MLM的综合框架

• Authors: Donghao Zhou, Jia-Hui Pan, Fan Zhang, Xingyuan Bu, Shilong Li, Xiaojie Gao, Yun-Hui Liu, Chi-Wing Fu, Pheng-Ann Heng
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23784
• Pdf link: https://arxiv.org/pdf/2609.23784
• Abstract Robotic bin packing requires long-horizon sequential decision-making, as each object placement affects the available space for subsequent packing. Existing methods primarily rely on hand-crafted geometric heuristics that optimize predefined objectives or reinforcement learning policies learned through trial and error over predefined training configurations. Despite recent advances in multimodal large language models (MLLMs) for this task, their potential for closed-loop sequential decisions across heterogeneous packing configurations remains underexplored. To address this gap, we introduce PackLab, a comprehensive framework for developing, training, and evaluating MLLMs for closed-loop robotic bin packing. PackLab-Suite provides a physics-based simulation platform for scalable generation of diverse training packing trajectories and evaluation of their physical outcomes. PackLab-VLM is a packing-specialized MLLM that understands the evolving object and container states to jointly select objects and predict placements in a closed-loop manner. PackLab-Bench provides standardized packing scenarios at multiple difficulty levels for systematic evaluation. Extensive experiments demonstrate that, on average, PackLab-VLM outperforms conventional packing heuristics, traditional reinforcement learning methods, and general-purpose MLLMs across object sets and container configurations, highlighting the potential of MLLMs for long-horizon robotic packing. The code, model, dataset, and benchmark are available at this https URL .
• 中文摘要 机器人垃圾桶装箱需要长期顺序决策，因为每个物品的放置都会影响后续装箱的可用空间。现有方法主要依赖手工设计的几何启发式，优化通过反复试验学习的预定义目标或强化学习策略。尽管近年来多模态大型语言模型（MLLM）在该任务中有所进展，但它们在异构装箱配置中实现闭环顺序决策的潜力仍未被充分开发。为弥补这一空白，我们引入了PackLab，一个用于开发、培训和评估闭环机器人装箱MLLM的综合框架。PackLab-Suite提供了一个基于物理的仿真平台，支持可扩展生成多样化训练打包轨迹及其物理结果的评估。PackLab-VLM 是一款专注于打包的 MLLM，能够理解对象和容器状态的变化，从而以闭环方式共同选择对象并预测位置。PackLab-Bench 提供多种难度级别的标准化打包场景，供系统评估使用。大量实验表明，PackLab-VLM 平均在对象集和容器配置中优于传统打包启发式、传统强化学习方法和通用 MLLM，凸显了 MLLM 在长期机器人打包中的潜力。代码、模型、数据集和基准测试可在该 https 网址获取。

FLARE: A Full-Lifecycle Dense Supervision Paradigm for Long-Horizon Coding Agents via Generative Reward Model

FLARE：通过生成奖励模型实现长视野编码代理的全生命周期密集监督范式

• Authors: Jingxuan Xu, Gang Wu, Yanan Wu, Yutao Mou, Songwei Yu, Tianzhuang He, Zhengshuo Gong, Zhao Liu, Zihang Xu, Wenqiang Zhu, Xinping Lei, Weihao Li, Yuhui Bai, Zhongqiu Wang, Yan Wu, Ariel Deng
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.23808
• Pdf link: https://arxiv.org/pdf/2609.23808
• Abstract While test-time scaling enhances Large Language Model (LLM) agents in long-horizon software engineering (SWE), sparse binary rewards (Pass/Fail) create a severe credit assignment crisis and waste failed exploratory trajectories. Current trajectory optimization and scaling methods are costly and structurally limited, relying on heuristic state reuse without causal diagnosis or delayed scalar scoring without actionable online guidance. We propose FLARE (Full-Lifecycle Alignment and Reward Engine), a novel dense supervision paradigm driven by a lightweight Generative Reward Model (GRM). First, RADAR, an offline causal-aware diagnostic framework, extracts high-fidelity, hindsight-free supervision through causal-chain backtracking to distill a GRM providing real-time, step-level risk feedback. Second, FLARE uses this GRM to continuously optimize the agent across its entire lifecycle. During inference, FLARE acts as an Active Scaffold, autonomously intercepting high-risk generation steps for localized breakpoint re-execution, drastically reducing compute overhead. During post-training, the GRM's structured signals serve as process-supervised reranking scores for Supervised Fine-Tuning (SFT) and step-level dense rewards for Reinforcement Learning (RL), mitigating policy collapse in sparse environments. Extensive evaluations show that FLARE establishes a new Pareto frontier across the agent lifecycle: FLARE (N=1) outperforms Global Rollout (N=5) with a 5x reduction in token consumption. Extending FLARE to training overcomes the sparse reward problem in long-horizon interactive tasks, delivering relative performance gains of 19.13% in SFT through process-aware data curation and a consistent 9.19% improvement in RL.
• 中文摘要 虽然测试时间缩放增强了长期软件工程（SWE）中的大型语言模型（LLM）代理，但稀疏的二元奖励（通过/失败）则导致严重的信用分配危机，浪费了失败的探索轨迹。当前的轨迹优化和缩放方法成本高且结构有限，依赖启发式状态重用而无因果诊断，或延迟标量评分而无可操作的在线指导。我们提出了FLARE（全生命周期对齐与奖励引擎），这是一种由轻量级生成奖励模型（GRM）驱动的新型密集监管范式。首先，RADAR这一离线因果感知诊断框架通过因果链回溯提取高保真、无事后视角的监督，提炼出提供实时、阶梯级风险反馈的GRM。其次，FLARE利用该GRM在整个生命周期内持续优化智能体。在推理过程中，FLARE充当主动支架，自主拦截高风险生成步骤以实现局部断点重执行，大幅降低计算开销。在训练后，GRM的结构化信号作为过程监督的重新排序分数，用于监督微调（SFT）和强化学习（RL）的步骤级密集奖励，减轻稀疏环境中的策略崩溃。广泛评估表明，FLARE在智能体生命周期中建立了新的帕累托前沿：FLARE（N=1）表现优于全局部署（N=5），令牌消耗减少了5倍。将FLARE扩展到训练中解决了长期交互任务中奖励稀疏的问题，通过流程感知数据管理，SFT表现相对提升19.13%，强化学习持续提升9.19%。

VISTA: An Attention-Based Multi-Agent Reinforcement Learning Architecture for Space Situational Awareness Sensor Tasking

VISTA：基于注意力的多智能体强化学习架构，用于空间态势感知传感器任务

• Authors: Miguel Leiva-Vélez, Adalberto Claudio Quiros, Nicolas Gaston Rozado, Hodei Urrutxua, Víctor Rodríguez-Fernández
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.23875
• Pdf link: https://arxiv.org/pdf/2609.23875
• Abstract The rapid growth of resident space objects is increasing the complexity of space situational awareness sensor tasking, challenging classical optimization methods as they allocate finite, heterogeneous, and distributed sensing resources across ever-larger catalogues. Existing deep reinforcement learning approaches show promise in reduced settings, but fixed-dimensional state and action representations limit their ability to scale to large, dynamic catalogues and distributed sensing networks. We introduce VISTA (Variable-Entity Intelligent Sensor Tasking Architecture), a scalable deep reinforcement learning architecture for persistent uncertainty-driven catalogue maintenance across variable object populations and sensor configurations. VISTA combines physics- and mission-informed top-K retrieval with entity-centric attention, recurrent memory, and pointer-based action decoding, thereby keeping each agent's observation and action spaces independent of catalogue size. We evaluate VISTA across different scenarios, from fixed-size single-sensor benchmarks to large-scale space-based tasking and heterogeneous cooperative sensing. With 30 orbiting targets, VISTA recovers the catalogue 31.2% faster than the fixed-dimensional recurrent baseline. In the large-scale regime, VISTA reduces five-hour uncertainty by 97.5% relative to the strongest classical reference and by 99.3% relative to the recurrent learner. Zero-shot tests up to 20,000 objects reveal near-linear relations between sensing capacity, catalogue size, and recovery horizon. Learned policies also exhibit sensor modality adaptation and generalization to population and initial-uncertainty shifts. Together, these results demonstrate that VISTA provides a scalable framework for adaptive space situational awareness sensor tasking across large, distributed networks of heterogeneous ground- and space-based sensors.
• 中文摘要 驻留空间物体的快速增长增加了空间态势感知传感器任务的复杂性，挑战了传统优化方法，因为它们在日益庞大的目录中分配有限、异构和分布式的传感资源。现有深度强化学习方法在更小的环境中展现出潜力，但固定维的状态和动作表示限制了其扩展到大型动态目录和分布式传感网络的能力。我们介绍了VISTA（可变实体智能传感器任务架构），这是一种可扩展的深度强化学习架构，用于在变量对象群体和传感器配置中持续维护不确定性驱动的目录。VISTA结合了物理和任务知情的顶K检索、以实体为中心的注意力、循环记忆和基于指针的动作解码，从而保持每个代理的观察和动作空间不受目录大小影响。我们在不同场景下评估VISTA，从固定尺寸单传感器基准到大规模基于空间的任务执行和异构协同感测。对于30个轨道目标，VISTA恢复目录的速度比固定维度循环基线快31.2%。在大尺度区域内，VISTA相较于最强经典参考降低了97.5%的五小时不确定性，相对于重复学习者减少了99.3%。对最多2万个物体进行零次测试揭示了传感能力、目录大小和恢复视野之间的近线性关系。学习的策略还表现出传感器模态的适应性以及对种群和初始不确定性变化的泛化。综合来看，这些结果表明VISTA为跨大型分布式异构地面和空间传感器网络的自适应空间态势感知传感器任务提供了可扩展的框架。

BarrierFormer: Transformer-Guided Predictive Barrier Enforcement for Safe Robot Control

BarrierFormer：变压器引导预测性障碍执行，实现机器人安全控制

• Authors: Anandsingh Chauhan, Kunal Garg
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.23896
• Pdf link: https://arxiv.org/pdf/2609.23896
• Abstract Control barrier functions (CBFs) have become one of the most popular tools for encoding and enforcing state constraints in safety-critical robotics. Standard CBF approaches are inherently myopic in nature as they enforce safety only at the current time step. Consequently, the system can be driven toward the boundary of the safe set where no feasible safe control exists at a future timestep. Model predictive control (MPC) based approaches address this by enforcing state constraints over a receding horizon. However, such approaches generally require the model to be known for solving a constrained optimization problem at every step, which is computationally expensive for real-time deployment. We propose BarrierFormer, a barrier-supervised transformer framework that addresses these limitations by encoding rollout-level CBF constraints in learning a model-free safe policy. A causal transformer encodes observation-action history, autoregressively generates a predictive rollout through the dynamics head to replace the model, and provides a residual correction to a nominal controller through the action head to replace the online computation. A barrier critic operating on local observations evaluates CBF constraint violations along this rollout, and a safety teacher computes barrier-consistent actions satisfying these constraints as direct supervision targets for the learned control policy. During inference, the policy maps observation-action history to control actions without any online optimization or model knowledge, enabling real-time model-free predictive safety enforcement. Evaluations across linear and nonlinear, 2D and 3D dynamical systems for safe goal-directed navigation demonstrate that BarrierFormer outperforms existing reinforcement learning (RL)-based, diffusion-based, MPC-based, and transformer-based approaches in safety rate and inference latency.
• 中文摘要 控制障碍函数（CBF）已成为安全关键机器人中编码和执行状态约束的最常用工具之一。标准的CBF方法本质上目光短浅，因为它们仅在当前时间步执行安全。因此，系统可以被驱动到安全集边界，即未来时间步不存在可行的安全控制。基于模型预测控制（MPC）的方法通过在远征视角内强制执行状态约束来解决这个问题。然而，这类方法通常要求模型在每一步都已知解决受限优化问题，这对实时部署计算成本较高。我们提出了BarrierFormer，一种障碍监督变换器框架，通过编码部署级CBF约束来解决这些限制，以学习无模型安全策略。因果变换器编码观察-动作历史，自回归地通过动力学头生成预测滚动以替代模型，并通过动作头向名义控制器提供残差修正以替代在线计算。在局部观测上操作的障碍批评者评估该部署过程中的CBF约束违规，安全教师计算满足这些约束的障碍一致动作，作为学习控制策略的直接监督目标。在推理过程中，策略将观察-动作历史映射到控制动作，无需在线优化或模型知识，实现实时无模型的预测安全执行。跨线性和非线性、二维和三维动态系统用于安全目标导向导航的评估表明，BarrierForformer在安全率和推断延迟方面优于现有基于强化学习（RL）、基于扩散、基于MPC和基于变压器的方法。

Opt2VLA: Force-Aware Vision-Language-Action for Contact-Rich Humanoid Whole-Body Manipulation

Opt2VLA：用于丰富接触的人形全身操控的原力感知视觉-语言-行动

• Authors: Fukang Liu, Yipu Chen, Jaehwi Jang, Danfei Xu, Zsolt Kira, Ye Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23968
• Pdf link: https://arxiv.org/pdf/2609.23968
• Abstract Humanoid robots are expected to perform diverse human-level tasks in daily environments, many of which require precise regulation of interaction forces. While recent vision-language-action (VLA) models have shown promise for semantic planning and visuomotor control, existing humanoid systems primarily represent actions through geometric motion goals and rely on whole-body controllers focused on motion tracking, with limited explicit reasoning or control of interaction forces. This limitation is particularly relevant in contact-rich tasks, where geometrically similar motions may require different force regimes depending on the task context and where visual observations may become unreliable after contact. In this work, we present Opt2VLA, a force-aware VLA framework that introduces explicit force commands at the VLA-to-control interface for humanoid whole-body manipulation. A single multi-task VLA policy jointly predicts both geometric motion goals and continuous contact-force references, which are tracked by task-specific reinforcement learning (RL)-based whole-body controllers. To provide scalable and physically grounded supervision, we generate dynamically feasible and contact-consistent training data via whole-body trajectory optimization (TO) with explicit force references. We evaluate Opt2VLA on three contact-rich humanoid tasks and show that explicit force conditioning enables more accurate and consistent force regulation than motion-only control, while physically grounded torque supervision from TO further improves force tracking accuracy and stability. Closed-loop evaluations further demonstrate language-conditioned force modulation with Opt2VLA in simulation and on humanoid hardware.
• 中文摘要 类人机器人预计将在日常环境中执行多种人类级别任务，其中许多任务需要精确调节相互作用力。尽管近期视觉-语言-行动（VLA）模型在语义规划和视觉运动控制方面展现出潜力，但现有的人形系统主要通过几何运动目标表示动作，依赖专注于运动追踪的全身控制器，缺乏明确的推理或交互力控制。这一限制在接触密集任务中尤为重要，因为几何相似的运动可能需要根据任务上下文不同的力系统，且接触后视觉观察可能变得不可靠。本研究介绍了Opt2VLA，一种力感知VLA框架，在VLA与控制界面引入显式的力指令，用于人形全身操控。单一多任务VLA策略联合预测几何运动目标和连续接触力参考，这些由基于任务特定的强化学习（RL）的全身控制器跟踪。为提供可扩展且物理基础的监督，我们通过全体轨迹优化（TO）生成动态可行且接触一致的训练数据，并明确引用力。我们在三个接触丰富的人形任务中评估了Opt2VLA，表明显式力条件调节比单纯运动控制更准确、更一致，而TO的物理接地扭矩监督进一步提升力追踪的准确性和稳定性。闭环评估进一步展示了Opt2VLA在仿真和人形硬件上的语言条件力调制。

Anticipatory Robot Goalkeeping via Monotone Optimal Stopping

通过单调最优停止实现的预判机器人守门

• Authors: Hao E. Zhang, Ruize Geng, Yisen Li, Yaru Niu, Yikai Wang, Raihan Haque, Khalil Zbiss, Guanyang Luo, Hui-ping Wang, H. Eric Tseng, Ding Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23976
• Pdf link: https://arxiv.org/pdf/2609.23976
• Abstract Robots engaged in fast physical interactions often need to act before the intent of another agent is fully known. Anticipatory goalkeeping illustrates this challenge. Waiting provides more reliable information about the target but reduces the physical opportunity for interception, whereas acting early preserves reachability but requires initiating motion under uncertainty. Given a fixed closed-loop save controller, we formulate the decision of when to initiate motion as a policy-conditional finite-horizon optimal stopping problem. Building on this formulation, we propose monotone optimal stopping (MOS), a structured release-timing method for dynamic robotic interception. The quadruped save policy is trained with reinforcement learning, while MOS determines when the policy should be activated from the evolving robot state and target belief. Rather than predicting a release time or relying on confidence alone, MOS learns the return advantage of acting now over waiting for one more observation. We derive a direct Bellman recursion for this act-versus-wait margin and impose monotonicity only with respect to physical urgency, reflecting the irreversible loss of interception opportunity as time elapses. This structure enables early activation for dynamically demanding saves while preserving closed-loop adaptation when later observations change the predicted target. Under a single-crossing condition, MOS admits a threshold release boundary with a bounded approximation error. Extensive simulation studies show that MOS improves the mean save rate from 67.7% to 74.4% over a parameter-matched learned gate and increases reversal saves from 52.1% to 66.5%. Real-robot experiments further demonstrate rapid interception and post-release direction correction under human shot-direction feints.
• 中文摘要 进行快速物理交互的机器人通常需要在完全了解另一个智能体意图之前采取行动。预先目标控制展示了这一挑战。等待提供更可靠的目标信息，但减少了被拦截的物理机会;而早期行动则保持了可达性，但需要在不确定性下启动动作。给定一个固定的闭环保存控制器，我们将何时发起动作的决策表述为一个策略条件有限视界最优停止问题。基于该表述，我们提出了单调最优停止（MOS），这是一种用于动态机器人拦截的结构化释放时机方法。四足救援策略通过强化学习训练，而MOS则根据机器人状态和目标信念的演变决定何时激活策略。MOS不再仅凭释放时间预测或依赖信心，而是学会了立即行动的回报优势，而非等待最后一次观察。我们对该行动与等待间距推导出直接贝尔曼递归，仅对物理紧迫性施加单调性，反映随着时间推移拦截机会的不可逆流失。该结构允许动态要求高的救援提前激活，同时在后续观测改变预测目标时保持闭环适应性。在单次交叉条件下，MOS允许阈值释放边界且近似误差有限。大量模拟研究表明，MOS在参数匹配学习门上将平均扑救率从67.7%提升至74.4%，并将反向扑救从52.1%提升至66.5%。真实机器人实验进一步展示了在人类射击方向假动作下的快速拦截和释放后方向修正。

Synthesizing Reactive Character Behaviors for Continuous Games via Programmatic Policy Search

通过程序化策略搜索综合反应性角色行为以实现连续游戏

• Authors: Maxim Gumin, Hsueh-Ti Derek Liu, Victor Zordan, Daniel Ritchie
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24025
• Pdf link: https://arxiv.org/pdf/2609.24025
• Abstract We present a method for synthesizing reactive character behaviors for continuous games as compact, human-readable programs. Game AI practice still relies heavily on manually authored behavior trees, state machines, and scripts, while academic reinforcement learning typically produces opaque neural controllers that are expensive to train and difficult to edit. Our approach bridges this gap by searching directly over a domain-specific language for continuous-space game policies. The language is designed around reactive geometric decisions and includes higher-order constructs such as direction maximization. These constructs help discretize a continuous behavior space into enumerable program structures. To make program search practical, we introduce a large set of synthesis antipatterns that remove redundant program forms while preserving behavioral coverage. We further combine bottom-up symbolic enumeration with top-down guidance from a coding agent. Our resulting method, agentic sketching, has the agent propose high-level policy structure and call an enumerator to complete local program slots. We evaluate the method on a benchmark of 14 continuous games, ranging from classic control tasks to multi-agent football. We find that pure enumeration is often more efficient than using a coding agent alone, while the combined method substantially outperforms both. Our results suggest that programmatic policy search can be a practical authoring tool for game AI: designers specify reward functions, and the system discovers editable behaviors that are effective, portable, and often surprising.
• 中文摘要 我们提出了一种将连续游戏反应性角色行为合成为紧凑、可读程序的方法。游戏AI实践仍高度依赖手动编写的行为树、状态机和脚本，而学术强化学习通常产生不透明的神经控制器，训练成本高且难以编辑。我们的方法通过直接在领域特定语言上搜索连续空间博弈策略，弥合了这一空白。该语言围绕反应几何决策设计，包含了方向最大化等高阶构造。这些构造帮助连续行为空间离散化为可枚举的程序结构。为了使程序搜索实用，我们引入了大量合成反模式，去除冗余程序形式，同时保持行为覆盖。我们还结合了自下而上的符号枚举与编码代理的自上而下指导。我们最终生成的方法——代理绘图，要求代理提出高层策略结构并调用枚举器完成本地程序槽位。我们在14场连续游戏的基准测试中评估该方法，范围涵盖经典控制任务到多智能体足球游戏。我们发现纯枚举通常比单独使用编码代理更高效，而合并方法的表现远超两者。我们的结果表明，程序化策略搜索可以成为游戏AI的实用创作工具：设计者指定奖励函数，系统发现可编辑且有效、可移植且常常令人惊讶的行为。

Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning

Imagine-RL：残余信心引导交叉注意力，用于世界模型增强的VLA强化学习

• Authors: Kejia Hu, Wentong Zhai, Bo Zhao, Shuai Liang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24033
• Pdf link: https://arxiv.org/pdf/2609.24033
• Abstract Reliable action evaluation in contact-rich manipulation requires looking beyond the current observation to future visual and contact consequences. Existing noise-space reinforcement learning efficiently steers a frozen Vision-Language-Action (VLA) policy, but its critics largely ignore these consequences. We present Imagine-RL, which augments noise-space VLA post-training with action-conditioned visual-torque imagination. For each candidate action chunk, a frozen visual-torque latent world model (VTLWM) autoregressively predicts compact future representations without pixel reconstruction. A current image-state-action query attends to observed histories and predicted futures, while previous-window prediction residuals provide token-wise confidence priors that suppress unreliable future tokens. By combining current evidence with predicted consequences, the action critic better evaluates candidate actions and supervises the actor, while the VLA and VTLWM remain frozen. Across four real-robot tasks with 50 evaluation trials per task, Imagine-RL uses only 100 RL trajectories and improves the average success rate by (23.6%) over DSRL and by (60%) over VLA baselines.
• 中文摘要 在接触丰富操作中，可靠的动作评估需要超越当前观察，关注未来的视觉和接触后果。现有的噪声空间强化学习高效地引导了冻结的视觉-语言-行动（VLA）策略，但其批评者大多忽视了这些后果。我们介绍Imagine-RL，它通过动作条件的视觉-扭矩想象力增强噪声空间VLA的训练后工作。对于每个候选动作块，一个冻结的视觉-扭矩潜世界模型（VTLWM）自回归预测紧密的未来表示，无需像素重建。当前图像-状态-动作查询关注观察到的历史和预测的未来，而前窗口预测残差则提供按代币的置信先验，抑制不可靠的未来代币。通过将现有证据与预测结果结合，动作批评者更好地评估候选动作并监督行为者，而VLA和VTLWM保持冻结。在四个真实机器人任务中，每个任务包含50个评估试验，Imagine-RL仅使用100条强化学习轨迹，平均成功率较DSRL提升23.6%，较VLA基线提升60%。

Monitorable Chart Reasoning Agents via Verifiable Process Rewards

可监控的图表推理代理通过可验证的过程奖励

• Authors: Sanchit Sinha, Oana Frunza, Kashif Rasul, Aidong Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.24071
• Pdf link: https://arxiv.org/pdf/2609.24071
• Abstract Chart reasoning agents are increasingly used to extract actionable insights in critical domains, achieving state-of-the-art performance on multiple benchmarks. Yet, high benchmark accuracy alone is insufficient for deployment, where stakeholders must be able to audit and verify how a model reaches its answer. Existing LVLM-based chart agents produce either answer-only predictions or free-form rationales that are hard to verify, obscuring whether an error arose from misreading the chart, extracting a wrong value, or miscomputing. We propose Chart-RVR, a reinforcement learning framework for training monitorable chart agents with verifiable process rewards. Chart-RVR decomposes chart reasoning into three auditable blocks: Structure, identifying the chart type; Evidence, reconstructing the underlying data table in JSON; and Derivation, exposing the stepwise trace that computes the answer. Across six in-domain and out-of-domain benchmarks, Chart-RVR attains state-of-the-art accuracy among comparable-sized LVLMs. Beyond accuracy, we assess monitorability using a triangulated protocol that combines ground-truth surrogate metrics, an oracle information-gain measure, and an LLM-as-auditor scoring Process Verifiability and Evidence Localization, showing that Chart-RVR yields rationales that are markedly more verifiable and evidence-grounded than those from CoT prompting, SFT, and existing chart-specific baselines.
• 中文摘要 图表推理代理越来越多地被用于在关键领域提取可操作的洞见，在多个基准测试上实现最先进的性能。然而，仅靠高基准准确性不足以实现部署，因为利益相关者必须能够审计和验证模型如何得出答案。现有基于LVLM的图表代理要么生成仅有答案的预测，要么是难以验证的自由形式理由，从而模糊错误是否源于误读图表、提取错误值或计算错误。我们提出了Chart-RVR，一种强化学习框架，用于训练可监控的图表代理，并获得可验证的过程奖励。Chart-RVR将图表推理分解为三个可审计模块：结构，识别图表类型;证据，用JSON重建底层数据表;以及推导，揭示计算答案的逐步追踪。在六个域内外基准测试中，Chart-RVR在同等规模的LVLM中达到了最先进的准确性。除了准确性，我们还使用一种三角协议评估可监控性，该协议结合了地面真实替代指标、预言机信息获取指标和作为审计者的LLM评分过程验证性和证据本地化，显示Chart-RVR的理据比CoT提示、SFT及现有图表特定基线的理由更具可验证性和证据基础。

Dexterous Robot Manipulation from Human Demonstrations via Contact-Anchored Retargeting and Residual Policy Learning

通过接触锚定重定向和残余策略学习，从人类演示中实现灵巧的机器人操作

• Authors: Zihao Yang, Chengyuan Liu, Yu Zhou, Runze Lv, Tianyu Cui, Sheng Yi, Haohua Zhu, Irvine Lu, JieQ Sun
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24093
• Pdf link: https://arxiv.org/pdf/2609.24093
• Abstract Learning dexterous manipulation from demonstrations is bottlenecked by data: the contact forces that determine whether a grasp succeeds are absent from every scalable source of human demonstrations. This paper builds on two observations. First, what survives the change from a human hand to a robot hand is the contact structure of a demonstration - which finger regions touch which object locations, and in what order - rather than its joint motion. Second, physical consistency need not be engineered per task: a single residual reinforcement learning (RL) policy, trained once across diverse demonstrations, can repair kinematic recordings into physically consistent, contact-annotated trajectories, and the same residual formulation restores dynamic feasibility after retargeting. These observations yield a three-stage pipeline that converts human motion-capture recordings into dexterous robot policies with no real-robot training data: physics refinement with a simulated MANO hand recovers contacts and forces, contact-anchored retargeting transfers the demonstrated contact structure through an objective independent of hand morphology, and residual policy learning adapts the result to robot actuation. The pipeline reconstructs 25,454 single-hand trajectories (success 7.3% -> 59.3%) and 25 dual-hand tasks (16.0% -> 62.4%) with one shared policy per setting, transfers one human dataset to four morphologically distinct robot hands (+62.4 pp), and executes four contact-rich bimanual tasks on physical hardware with zero real-robot training data.
• 中文摘要 从演示中学习灵巧操作被数据瓶颈：决定抓握是否成功的接触力在所有可扩展的人类演示来源中都缺失。本文基于两个观察。首先，从人手转变为机器人手后，幸存下来的是演示的接触结构——哪些手指区域接触哪些物体的位置，以及按何种顺序——而非关节运动。其次，物理一致性不必为每个任务设计：单一残余强化学习（RL）策略，在不同演示中训练一次，即可将运动学记录修复为物理一致的接触注释轨迹，且同一残余表述在重新定向后恢复动态可行性。这些观测形成了三阶段流程，将人类动作捕捉记录转换为无真实机器人训练数据的灵活机器人策略：通过模拟MANO手进行物理细化恢复接触和力，接触锚定重新定向通过无手形态的目标转移已展示的接触结构，剩余策略学习将结果调整至机器人操作。该流程重建了25,454条单手轨迹（成功率7.3% -> 59.3%）和25个双手任务（16.0% -> 62.4%），每个设置共享一个策略，将一个人类数据集传输给四个形态不同的机器人手（+62.4页），并在物理硬件上执行四个接触丰富的双手任务，且无真实机器人训练数据。

Reinforcement Learning under State and Outcome Uncertainty: A Foundational Distributional Perspective

状态与结果不确定性下的强化学习：基础分布视角

• Authors: Larry Preuett, Qiuyi Zhang, Muhammad Aurangzeb Ahmad
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24103
• Pdf link: https://arxiv.org/pdf/2609.24103
• Abstract In many real-world planning tasks, agents must tackle uncertainty about the environment's state and variability in the outcomes of any chosen policy. We address both forms of uncertainty as a first step toward safer algorithms in partially observable settings. Specifically, we extend Distributional Reinforcement Learning (DistRL)-which models the entire return distribution for fully observable domains-to Partially Observable Markov Decision Processes (POMDPs), allowing an agent to learn the distribution of returns for each conditional plan. Concretely, we introduce new distributional Bellman operators for partial observability and prove their convergence under the supremum p-Wasserstein metric. We also propose a finite representation of these return distributions via psi-vectors, generalizing the classical alpha-vectors in POMDP solvers. Building on this, we develop Distributional Point-Based Value Iteration (DPBVI), which integrates psi-vectors into a standard point-based backup procedure-bridging DistRL and POMDP planning. By tracking return distributions, DPBVI lays the foundation for future risk-sensitive control in domains where rare, high-impact events must be carefully managed. We provide source code to foster further research in robust decision-making under partial observability.
• 中文摘要 在许多现实世界的规划任务中，代理必须应对环境状态的不确定性以及任何所选策略结果的变异性。我们将这两种不确定性作为部分可观测环境中更安全算法的第一步。具体来说，我们将分布强化学习（DistRL）——对完全可观测域的整个回报分布建模——扩展到部分可观测马尔可夫决策过程（POMDP），使智能体能够学习每个条件计划的收益分布。具体来说，我们引入了新的部分可观测分布贝尔曼算子，并证明它们在上确界p-Wasserstein度量下的收敛性。我们还提出了通过psi向量对这些返回分布的有限表示，推广了POMDP求解器中的经典α向量。在此基础上，我们开发了分布点基值迭代（DPBVI），将psi向量集成到标准的基于点备份程序中，桥接DistRL和POMDP的规划。通过跟踪回报分布，DPBVI为未来在需要谨慎管理罕见且高影响力事件的领域中的风险敏感控制奠定了基础。我们提供源代码，促进在部分可观察性下稳健决策的进一步研究。

Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?

运气不是技能：配对推广何时能帮助LLM代理的群体相对强化学习？

• Authors: Nazmus Sakib
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.24144
• Pdf link: https://arxiv.org/pdf/2609.24144
• Abstract Group-relative reinforcement learning compares rollouts of the same prompt, but independent environment noise can obscure these comparisons. We study paired rollouts, which share an event-keyed noise schedule within each group while preserving each rollout's marginal distribution. Pairing removes the between-schedule component of reward-contrast variance, but need not reduce gradient variance. For one-sided grader noise, we derive an exact condition for reduction and give a counterexample in which reward contrasts improve while gradient variance increases. A controlled study trains a 2B tool-use agent under tool faults and grader flips, with three seeds per design. The protocol was registered with a disclosed, previously completed pilot. Under tool faults, pairing improves final noisy-test success by +5.1 percentage points on average, with all three seed differences positive, but misses the registered learning-curve criterion. The criterion is also missed under grader flips: the validation-AUC difference is +0.003 (95% interval [-0.029, +0.033]). A gradient probe on eight distinct checkpoints from two fault-trained trajectories finds lower mean-centered covariance traces under both noise types: 21 to 30% for grader flips and 40 to 63% for tool faults. These finite-sample measurements support the variance mechanism without establishing a general learning-speed benefit. The results distinguish improving reward comparisons, reducing estimator variance, and improving learning.
• 中文摘要 组相对强化学习比较同一提示的各次展开，但独立的环境噪声可能掩盖这些比较。我们研究了配对展开，这些部署在每个组内共享事件键控噪声计划，同时保持每个部署的边际分布。配对消除了奖励-对比度差的间期成分，但不必降低梯度方差。对于单侧评分器噪声，我们推导出一个精确的减小条件，并给出一个反例说明奖励对比度提升而梯度方差增加。一项受控研究在工具故障和平地机翻转下训练2B工具使用代理，每个设计使用三个种子。该方案已在一个已披露且已完成的试点中注册。在工具故障条件下，配对平均使最终噪声测试成功率提升+5.1个百分点，三种种子差异均为正，但未达到注册的学习曲线标准。在分级器翻转下也未发现该标准：验证-AUC差异为+0.003（95%区间[-0.029， +0.033]）。对两个故障训练轨迹的八个不同检查点进行梯度探测，发现两种噪声类型下均中心协方差迹较低：平地机翻转为21%至30%，工具故障为40%至63%。这些有限样本测量支持方差机制，但未建立普遍的学习速度优势。结果区分了奖励比较的改善、估计量方差的降低和学习的改善。

Acceptance-Aware Draft Model Training for Speculative Decoding

投机性解码的接受感知草稿模型训练

• Authors: Tianhua Xia, Mugilan Ganesan, Yifei Feng, Haiyu Wang, Maximilian Egger, Sai Qian Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.24150
• Pdf link: https://arxiv.org/pdf/2609.24150
• Abstract Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are verified by the target model in a single forward pass. Its speedup is largely determined by the acceptance length, yet existing draft-model training methods mainly optimize cross-entropy or Kullback-Leibler (KL) divergence as proxies. These objectives encourage distribution matching but do not directly optimize acceptance length, and the acceptance mechanism also differs between greedy and sampling-based decoding. In this work, we propose acceptance-length-aware training losses that directly optimize the expected number of accepted tokens within a speculative window. For greedy verification, we derive an expected accepted length (EAL) loss that explicitly maximizes expected acceptance length. For sampling-based decoding, we introduce a window total variation (WTV) loss that optimizes the overlap between temperature-scaled draft and target distributions while accounting for sequential acceptance dependencies. Both objectives can be further combined with a group-relative reinforcement learning stage (GRPO) using simulated acceptance length as the reward. Experiments across different target and draft models, tasks, and decoding settings show that our losses consistently improve acceptance length over KL-based training. WTV provides particularly strong gains under sampling-based decoding, while EAL better matches greedy verification. These results show that directly optimizing the acceptance objective, with losses tailored to the decoding mode, is more effective than conventional distribution-matching objectives.
• 中文摘要 推测解码通过使用轻量级草稿模型生成多个候选令牌，目标模型在一次前向传递中验证这些令牌，加速了大型语言模型（LLM）的推断。其加速主要由接受长度决定，但现有的草稿模型训练方法主要优化交叉熵或Kullback-Leibler（KL）发散作为代理。这些目标鼓励分布匹配，但不直接优化接受长度，且接受机制在贪婪和基于抽样的译码之间也存在差异。本研究提出接受长度感知训练损失，直接优化推测窗口内期望接受令牌数量。对于贪婪验证，我们推导出期望接受长度（EAL）损失，显式最大化期望接受长度。对于基于抽样的译码，我们引入了窗口总变异（WTV）损失，以优化温度尺度选秀与目标分布之间的重叠，同时考虑顺序接受依赖性。这两个目标还可以进一步结合使用组相对强化学习阶段（GRPO），以模拟接受长度作为奖励。跨不同目标和选秀模型、任务及译码设置的实验表明，我们的损失在基于KL的训练中持续提升接受长度。WTV在基于抽样的译码下获得特别强的增益，而EAL更能匹配贪婪验证。这些结果表明，直接优化接受目标，并针对译码模式进行损耗调整，比传统分布匹配目标更为有效。

ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation

ChartJudgeBench：评估LMM法官以生成图表到代码

• Authors: Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang, Jiahao Tang, Jiajun Wu, Alex Jinpeng Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.24210
• Pdf link: https://arxiv.org/pdf/2609.24210
• Abstract Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward signal. Large Multimodal Models (LMMs) play a natural critical role in jointly assessing chart visual appearance and task requirements. They are therefore increasingly used as visual critics and reward models, yet their reliability as judges remains largely unexplored. To this end, we introduce ChartJudgeBench, a diagnostic vision-language benchmark for assessing LMM judges in chart-to-code workflows. It includes 1,003 Chart Perception Alignment (CPA) instances for pairwise chart comparison and 650 Chart Reasoning Judgment (CRJ) instances for binary Accept/Reject verification in Chart Reproduction and Chart Editing. Together, these tasks emulate the core judging decisions required in agentic refinement and RL-based chart optimization. Our evaluation of strong LMMs reveals four systematic limitations: (i) positional bias in pairwise comparison, (ii) a strong tendency to overpredict Accept, (iii) difficulty in matching visual styles and aesthetics, and (iv) an unexpected leniency bias in RL-trained models. These findings show that current LMM judges require explicit reliability validation before being used as critics or reward models in chart-to-code optimization. The code and data are available on ChartJudgeBench.
• 中文摘要 构建强大的图表到代码系统越来越依赖强化学习，其有效性关键在于奖励信号的质量。大型多模态模型（LMM）在联合评估图表视觉外观和任务需求方面发挥着自然的关键作用。因此，它们越来越多地被用作视觉批评和奖励模型，但其作为评判的可靠性仍大多未被深入探讨。为此，我们介绍了ChartJudgeBench，这是一个用于评估图表到代码工作流程中LMM评委的诊断视觉语言基准。它包含1003个图表感知对齐（CPA）实例用于两两图表比较，以及650个用于图表重现和图表编辑中二元接受/拒绝验证的图表推理判断（CRJ）实例。这些任务共同模拟了代理细化和基于强化学习的图表优化中的核心判断决策。我们对强LMM的评估揭示了四个系统性局限：（i）成对比较中的位置偏差，（ii）强烈高估接受率，（iii）视觉风格和美学匹配困难，（iv）强化学习训练模型中意外的宽松偏差。这些发现表明，现有LMM评委在被用作图表对代码优化的批评者或奖励模型前，要求明确验证可靠性。代码和数据可在ChartJudgeBench上获取。

Audio-based UAV Localization with Adaptive Temporal Correspondence via Reinforcement Learning

基于音频的无人机定位，通过强化学习实现自适应时间对应

• Authors: Haoxiang Lei, Mingzheng Feng, Daotong Wang, Shenghai Yuan
• Subjects: Subjects: Robotics (cs.RO); Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2609.24218
• Pdf link: https://arxiv.org/pdf/2609.24218
• Abstract Audio-based localization provides a low-cost and illumination-independent sensing solution for anti-UAV early warning. However, existing methods typically rely on a predefined fixed audio segment length, which limits temporal correspondence and creates a trade-off between sufficient acoustic evidence and timely localization. To address this issue, we propose an audio-based localization framework with adaptive temporal correspondence. A probe segment is first used to extract a compact acoustic state that characterizes the reliability and consistency of the observation. Guided by the state, a reinforcement learning controller dynamically determines the required audio window size for each localization decision. The selected audio segment is then processed by a Mamba-based localization network with adaptive temporal feature modulation for 3D position estimation. Extensive experiments demonstrate that our method achieves competitive 3D localization accuracy with substantially reduced temporal correspondence latency compared to SOTA methods and exhibits strong generalization across scenarios.
• 中文摘要 基于音频的定位为反无人机预警提供了低成本且无照度无关的传感解决方案。然而，现有方法通常依赖预定义的固定音频段长度，这限制了时间对应，并在足够的声学证据与及时定位之间产生权衡。为解决这一问题，我们提出了一种基于音频的自适应时间对应定位框架。首先使用探针片段提取一个紧凑的声学状态，以表征观测的可靠性和一致性。在该状态的指导下，强化学习控制器动态确定每个定位决策所需的音频窗口大小。选定的音频片段随后由基于Mamba的定位网络处理，并具备自适应时间特征调制以实现三维位置估计。大量实验表明，我们的方法在与SOTA方法相比时效对应延迟大幅降低的情况下，实现了具有竞争力的三维定位精度，并在各场景中展现出强烈的泛化能力。

Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcement

驯服VLM中CoT混淆：从机制证据到激活强制

• Authors: Xutao Mao, Jianing Zhu, Jinman Zhao, Tongliang Liu, Xiaowen Chu, Cong Wang, Bo Han
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24243
• Pdf link: https://arxiv.org/pdf/2609.24243
• Abstract Reinforcement learning (RL) improves reasoning in vision-language models (VLMs) but can induce chain-of-thought (CoT) obfuscation: an operational, non-intentional outcome where task reward or accuracy rises while traces become less grounded and monitorable. Prior work largely documents this decay behaviorally, leaving its representation-level correlates and actionable controls unclear. We find that template- and ground-associated activations become less separable during RL; matched interventions support the contribution of selected features to monitorability degradation. Guided by this evidence, we propose Targeted Anti-obfuscation with Mechanistic Enforcement (TAME), which uses Sparse Autoencoders (SAEs) to combine behavioral feedback with targeted suppression of template-associated activations during RL. Its asymmetric constraint penalizes template activations only above their pre-RL baseline, anchoring the localized features while behavioral feedback promotes grounded refinements. Across VIRL-39k, SPA-VL, and two model families, TAME improves CoT monitorability by up to 30.9 and 16.7 percentage points over Group Relative Policy Optimization (GRPO), respectively. Blinded human evaluation finds higher human monitorability on both datasets, and two held-out monitor families reproduce the monitorability gains. Task accuracy changes are small and mixed, and general-capability benchmarks show task-specific trade-offs. These results provide a path from behavioral monitoring to representation-level oversight for more auditable RL-trained multimodal systems.
• 中文摘要 强化学习（RL）提升视觉语言模型（VLM）中的推理能力，但可能诱发思维链（CoT）混淆：这是一种操作性、非意图的结果，任务奖励或准确率上升，而痕迹变得不那么扎实和可监控。以往研究主要记录了这种行为性衰退，其表征层级相关性和可操作控制尚未明确。我们发现模板和地面相关的激活在强化学习中变得难以分离;匹配干预支持选定特征对可监测性退化的贡献。基于这些证据，我们提出了具有机制强制的定向反混淆（TAME），该方法利用稀疏自编码器（SAE）将行为反馈与针对模板相关激活的有针对性抑制结合起来。其非对称约束仅惩罚模板激活高于其强化前基线的激活，锚定局部特征，同时通过行为反馈促进基于基础的细化。在VIRL-39k、SPA-VL及两个模型家族中，TAME分别将CoT可监测性提升了高达30.9个百分点和16.7个百分点，优于组相对策略优化（GRPO）。盲测人类评估发现两数据集的人类可监测性更高，且两组未被保留的监测族还原了可监测性提升。任务准确性变化小且混合，通用能力基准显示任务特定权衡。这些结果为更易审计的强化学习训练多模态系统提供了从行为监测到表示层监督的路径。

Reinforcement Learning Inspired Black-box Adversarial Attacks for Computer Vision

强化学习启发了计算机视觉的黑箱对抗攻击

• Authors: Florian Krone, Elena Hoemann, Sven Hallerbach
• Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.24249
• Pdf link: https://arxiv.org/pdf/2609.24249
• Abstract Neural networks, both convolution or transformer based, are essential for modern computer vision systems. However, they are vulnerable to small perturbations, almost imperceptible to humans, which significantly alter the model's prediction. These adversarial attacks are often considered to be a significant threat to the implementation of neural networks in safety-critical applications. Most attacks utilize the white-box threat model and therefore require full access to the target model, making them unrealistic to use in practice. We propose a novel approach under the more realistic black-box threat model that utilizes concepts from reinforcement learning to optimize perturbations with a non-differentiable target model. Reinforcement learning algorithms have already been optimized to be query efficient, making them an ideal starting point when designing black-box adversarial attacks. We show the success of our reinforcement learning inspired black-box adversarial attack (RIBA) in generating adversarial perturbations using only a small number of queries to the target model, by comparing it to state of the art attacks on different models on the Cifar10 and ImageNet data sets. RIBA takes $25.4\%$ fewer median queries to generate attacked images against a ResNet-18 on Cifar10 and $22.5\%$ fewer median queries to fool a Vit-B/16 model on ImageNet. Additionally, we demonstrate that RIBA can match the performance of white-box attacks on an adversarially trained model.
• 中文摘要 基于卷积或变换器的神经网络对现代计算机视觉系统至关重要。然而，它们容易受到人类几乎察觉不到的小扰动，这会显著改变模型的预测。这些对抗性攻击常被视为对神经网络在安全关键应用中实现的重大威胁。大多数攻击采用白盒威胁模型，因此需要对目标模型的全面访问，使其在实际操作中不切实际。我们提出了一种基于更现实黑盒威胁模型的新方法，利用强化学习的概念来优化非微分目标模型的扰动。强化学习算法已被优化为查询效率高，是设计黑盒对抗攻击的理想起点。我们通过比较Cifar10和ImageNet数据集上不同模型的先进攻击，展示了我们受强化学习启发的黑盒对抗性攻击（RIBA）仅用少量查询生成对抗性扰动的成功。RIBA在Cifar10上生成攻击图像的中位数查询次数减少了25.4美元，欺骗ImageNet上的Vit-B/16模型则减少了22.5美元。此外，我们还证明RIBA能够匹配白盒攻击对敌对训练模型的性能。

TTSE: A Two-Track Online Self-Evolution Framework

TTSE：双轨在线自我进化框架

• Authors: Ruimin Pei, Yongkang Wu, Shangyi Zheng, Yaqing Zhang, Deyang Li, Jianjun Tao, Xinyu Zhang, Xiang Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24289
• Pdf link: https://arxiv.org/pdf/2609.24289
• Abstract As Large Language Model (LLM) agents are applied in continuously interactive environments, driving the evolution of their own capabilities becomes a core problem for achieving long-term autonomy. Currently, environmental knowledge is typically treated as an external fixed input rather than as part of the agent's ongoing evolution. Reinforcement learning methods usually optimize policies through environmental interaction but tend to adapt only to fixed task distributions or single environments. This paper proposes TTSE (Two-Track Self-Evolution), a dual-track online self-evolution framework that separates evolving knowledge into FACT (environmental facts, whose reliability is continuously verified through interaction evidence) and TIP (task-conditioned implementation procedures). From a decision-theoretic perspective, we decompose the agent's excess risk into environment-representation regret and conditional-execution regret, characterize the conditions under which environment-conditioned policies strictly outperform condition-agnostic policies, and bound the downstream risk in terms of FACT identification error and cross-condition mismatch cost. In practice, TTSE's ablation experiments on GDPevo validate the advantage of dual-track evolution. On the classic agent task benchmarks ALFWorld and ScienceWorld, TTSE further demonstrates superior task adaptation. Moreover, TTSE is broadly compatible with existing skill self-evolution methods; combined with the Bayesian-Agent algorithm, a single-track ablation validates the dual-track advantage, substantially improving the aggregate score across the five major domains of SOPBench over three independent repetitions. Finally, on the real end-to-end task benchmark PinchBench, TTSE is integrated into a general agent framework via retrieval-based injection and stably outperforms the baseline across three independent runs.
• 中文摘要 随着大型语言模型（LLM）智能体在持续交互环境中应用，推动自身能力的演进成为实现长期自主性的核心问题。目前，环境知识通常被视为外部固定输入，而非智能体持续进化的一部分。强化学习方法通常通过环境交互优化策略，但往往仅适应固定任务分布或单一环境。本文提出了TTSE（双轨自我进化），这是一种双轨在线自我进化框架，将不断演变的知识分为FACT（环境事实，其可靠性通过交互证据持续验证）和TIP（任务条件实施程序）。从决策理论角度，我们将代理的超额风险分解为环境表征遗憾和条件执行遗憾，描述环境条件政策严格优于条件无关策略的条件，并以事实识别错误和交叉条件不匹配成本界定下游风险。在实际操作中，TTSE在GDPevo上的消融实验验证了双轨演化的优势。在经典代理任务基准测试ALFWorld和ScienceWorld上，TTSE进一步展示了更优越的任务适应能力。此外，TTSE与现有技能自我演化方法大致兼容;结合贝叶斯-代理算法，单轨消融验证了双轨优势，显著提升了SOPBench五大主要领域的三次独立重复综合得分。最后，在真实端到端任务基准测试PinchBench上，TTSE通过基于检索的注入集成到通用代理框架中，并在三次独立运行中稳定地优于基线。

Performance-Preserving Online Adaptation in Social Navigation via Diffusion Steering

通过扩散引导实现社交导航中保持性能的在线适配

• Authors: Haruto Nagahisa, Kohei Matsumoto, Yuki Hyodo, Ryo Kurazume
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24317
• Pdf link: https://arxiv.org/pdf/2609.24317
• Abstract In social navigation, modeling the complex interactions between humans and robots is difficult, and deep reinforcement learning has therefore been actively studied. However, because simulation alone cannot fully reproduce diverse scenarios, robot dynamics, and the social conventions that vary across deployment environments, fine-tuning in the deployment environment is promising. In doing so, learning that preserves the base model's performance is required, so as not to compromise the primary objective of navigation, namely avoiding pedestrians and reaching the destination. In this study, we propose a method that applies diffusion steering via reinforcement learning (DSRL), which trains only the noise policy while keeping the diffusion policy fixed, thereby achieving learning that preserves performance. Furthermore, we integrate diffusion-based RL policies trained with multiple seeds to construct the base policy, improving learning performance. Our evaluation shows that, compared with other methods, the proposed method enables efficient learning while preserving performance, and we confirm flexible behavior control through adaptation to social conventions, as well as its effectiveness on a physical robot through hardware-in-the-loop simulation.
• 中文摘要 在社会导航中，建模人机复杂互动较为困难，因此深度强化学习被积极研究。然而，由于仅靠仿真无法完全再现多样化场景、机器人动态及不同部署环境中的社会惯例，部署环境的微调具有前景。在此过程中，学习必须保持基础模型的性能，以免牺牲导航的主要目标——避开行人并到达目的地。本研究提出一种通过强化学习（DSRL）应用扩散引导的方法，该方法仅训练噪声策略，保持扩散策略固定，从而实现保持性能的学习。此外，我们将训练的基于扩散的强化学习策略与多个种子整合构建基础策略，提升学习性能。我们的评估表明，与其他方法相比，该方法实现了高效学习同时保持性能，并通过适应社会惯例确认了灵活行为控制，以及通过硬件在环模拟对物理机器人的有效性。

Conduit: An Experience Data Plane for Distributed Reinforcement Learning

导管：分布式强化学习的体验数据平面

• Authors: Sitong Zhang, Tuo Shi, Mario Di Francesco, Zeke Wang, Bo Zhao
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24456
• Pdf link: https://arxiv.org/pdf/2609.24456
• Abstract Distributed reinforcement learning (RL) scales training by parallelizing actors and learners around an Experience Buffer. As RL workloads grow, however, the buffer becomes more than a replay queue: it is the storage substrate of a large-capacity, latency-critical experience path that every iteration traverses to move, transform, sample, and batch experiences before learner updates can begin. Existing RL systems embed this path inside framework control flow or expose it as a request-driven buffer service, leaving experience placement fixed and experience-path work difficult to schedule independently as a runtime-level optimization target. We present Conduit, a framework-agnostic runtime that exposes RL experience management as an explicit systems optimization problem. At its core is the Experience Data Plane (EDP), a runtime abstraction that separates RL experience-handling semantics from framework-specific execution logic by exposing experience ingestion, experience placement, and experience delivery as explicit control points. Built on EDP, Conduit introduces capacity-constrained, bandwidth-aware placement, which distributes experience state across CPU/GPU memory tiers and nodes under heterogeneous interconnect and device-memory constraints, and latency-aware scheduling, which controls when experience-path handling runs to reduce exposed experience-path latency while preserving RL semantics. Integrated with RLlib without changing its framework execution logic, Conduit reduces exposed experience-path latency by up to 97% and end-to-end iteration latency by up to 38%, scales to 1,024 GPUs, and preserves convergence.
• 中文摘要 分布式强化学习（RL）通过将演员和学习者并行化绕体验缓冲区来扩展训练。然而，随着强化学习工作负载的增长，缓冲区不仅仅是一个重放队列：它是大容量、延迟关键的体验路径的存储基底，每次迭代都会经过它，以便在学习者更新开始前移动、转换、采样和批处理体验。现有的强化学习系统将该路径嵌入框架控制流中，或将其暴露为请求驱动的缓冲区服务，使得体验放置固定，体验路径工作难以作为运行时级优化目标独立调度。我们介绍Conduit，一个框架无关的运行时，它将强化学习体验管理暴露为显式系统优化问题。其核心是体验数据平面（EDP），这是一个运行时抽象，通过将体验摄取、体验放置和体验交付作为显式控制点，将强化学习体验处理语义与框架特定执行逻辑分离。Conduit 基于 EDP，引入了容量受限、带宽感知的布局，能够在异构互联和设备内存约束下将体验状态分布到CPU/GPU内存层和节点之间;以及延迟感知调度，控制体验路径处理的运行时间，以减少暴露的体验路径延迟，同时保持RL语义。与RLlib集成且不改变其框架执行逻辑，Conduit可将暴露的体验路径延迟降低高达97%，端到端迭代延迟最高降低38%，可扩展至1,024个GPU，并保持收敛性。

Fathom-Vaidya: Advancing Medical Reasoning with Rubric-Based Rewards

法通-维迪亚：基于评分标准的奖励推进医学推理

• Authors: Kalash Shah, Kunal Singh, Snehan J, Shreyas Singh
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.24480
• Pdf link: https://arxiv.org/pdf/2609.24480
• Abstract Deploying Large Language Models (LLMs) in healthcare requires robust performance across two complementary dimensions - diagnostic reasoning: the convergent, evidence-driven task of inferring a patient's condition from clinical data to produce a diagnosis, and clinical healthcare reasoning: the broader, navigational judgment required to communicate, plan, and adapt across multi-turn clinical interactions where a single correct answer may not exist. Recent benchmarks such as HealthBench and MedXpertQA reveal persistent weaknesses in both areas, exposing failures in complex diagnostic scenarios and limitations in contextual, patient-centered dialogue. We introduce a sequential training framework that targets these facets using synthetic data and rubric-based reinforcement learning. First, we improve diagnostic reasoning using MedBullets-derived questions with rule- and rubric-guided Reinforcement Learning (RL). We then shift to clinical reasoning by generating 5.3k synthetic multi-turn scenarios, each paired with multi-dimensional rubrics to comprehensively assess the response. This approach yields over 10% improvement on MedXpertQA, and our 30B model achieves 50.1% accuracy on HealthBench-Hard, surpassing proprietary baselines including GPT-5 (thinking). Our results show that targeted synthetic datasets and rubric-based training can systematically improve both diagnostic and interactive clinical reasoning in medical LLMs.
• 中文摘要 在医疗领域部署大型语言模型（LLMs）需要在两个互补维度上表现出强有力的性能——诊断推理：通过临床数据推断患者状况以做出诊断的趋同且循证驱动的任务;以及临床医疗推理：在多回合临床互动中进行沟通、规划和适应所需的更广泛导航性判断，在这些互动中可能没有单一正确答案。近期基准测试如HealthBench和MedXpertQA揭示了这两个领域的持续弱点，暴露了复杂诊断场景中的失败以及情境化、以患者为中心的对话的局限性。我们引入了一套顺序训练框架，利用合成数据和基于评分标准的强化学习，针对这些方面进行优化。首先，我们利用MedBullets衍生的问题，结合规则和评分标准引导的强化学习（RL）来提升诊断推理能力。随后，我们通过生成5.3k个合成多回合情景，每个情景配合多维评分标准，全面评估响应。该方法在MedXpertQA上提升超过10%，我们的30B模型在HealthBench-Hard中准确率达到50.1%，超过包括GPT-5（思维）在内的专有基线。我们的结果表明，针对性的合成数据集和基于评分标准的训练能够系统提升医学大型语言模型的诊断性和交互式临床推理能力。

Lifted Bellman Linear Programming for Offline Reinforcement Learning

离线强化学习的提升贝尔曼线性规划

• Authors: Hyukjun Yang, Jongchan Park, Narim Jeong, Donghwan Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24489
• Pdf link: https://arxiv.org/pdf/2609.24489
• Abstract Offline reinforcement learning (RL) typically trains a critic by minimizing a regression loss against bootstrapped value targets stabilized by target networks with exponential moving average (EMA) updates. Multi-step targets incorporate behavior-policy actions and therefore require off-policy correction. We instead impose in-sample Bellman optimality on the critic through inequality constraints. We formulate the Lifted Bellman Linear Program (LBLP), which lifts the linear programming characterization of Bellman optimality to the joint $(Q,V)$ space so that every constraint involves only state-action pairs in the dataset. Its unique minimizer is the in-sample optimal pair, and constraints along $K$-step segments of dataset trajectories leave this minimizer unchanged for any rollout policy and horizon. Under deterministic dynamics, this minimizer lies between the best dataset return and the optimal value. Relaxing the constraints into hinge penalties recovers the same solution above a finite penalty coefficient in the tabular case. Approximate Lifted Bellman Unconstrained Minimization (ALBUM) implements this relaxation with neural networks and detaches the $K$-step rollout targets by stop gradient. Its objective contains no squared regression onto bootstrapped targets, so it can be trained without target networks or EMA updates. Under deterministic dynamics, the LBLP solution is a stationary point of the detached update under a coefficient condition independent of $\gamma$ and $K$, and the inequality constraints allow discounted returns along dataset trajectories to serve as lower bounds without off-policy correction or action chunking. On OGBench, ALBUM uses a single critic with a Gaussian policy, matches the average performance of FQL, and is comparable to recent action-chunking methods, while using the fewest parameters and the least peak GPU memory among all compared methods.
• 中文摘要 离线强化学习（RL）通常通过最小化对由指数移动平均（EMA）更新的目标网络稳定的自助值目标的回归损失来训练批评者。多步目标包含行为-策略动作，因此需要非策略纠正。我们通过不等式约束对批评者施加样本内贝尔曼最优性。我们提出了提升贝尔曼线性规划（LBLP），将贝尔曼最优性的线性规划特征提升到联合的$（Q，V）$空间，使得数据集中每个约束仅涉及状态-动作对。其唯一的最小化器是样本内最优对，数据集轨迹中$K$步段的约束使该最小化器在任何推广策略和视野中保持不变。在确定性动力学下，该最小化器位于最佳数据集返回和最优值之间。将约束放宽为铰链惩罚，在表格情况下可恢复有限惩罚系数以上的相同解。近似提升贝尔曼无约束最小化（ALBUM）通过神经网络实现了该放松，并通过停止梯度分离$K$步的滚动目标。其目标不包含对自举目标的平方回归，因此可以在不需目标网络或EMA更新的情况下进行训练。在确定性动力学下，LBLP解是分离更新在与$\gamma$和$K$无关系数条件下的平稳点，不等式约束允许沿数据集轨迹的贴现收益作为下界，而无需非策略修正或动作分块。在OGBench中，ALBUM使用单一批判者，采用高斯策略，性能与FQL的平均性能相当，且可与近期的动作分块方法媲美，同时在所有比较方法中参数最少且GPU峰值内存最少。

InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality

InsertAnything：从模拟到现实的可推广丰富接触精度插入

• Authors: Zhenghua Ma, Xinpan Meng, Zeyu Liu, Muyuan Ma, Hengdi Zhang, Houcheng Li, Long Cheng
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24511
• Pdf link: https://arxiv.org/pdf/2609.24511
• Abstract Contact-rich precision insertion is a key manipulation skill in robotic assembly. Tight clearances make insertion more sensitive to alignment errors and prone to collisions and jamming, while variations in geometry and clearance across parts further complicate policy reuse. We present a reinforcement learning framework that trains insertion policies entirely in simulation for direct deployment without real-world demonstrations or policy fine-tuning. By combining target poses with compact three-dimensional fingertip force feedback, the policy learns to search for alignment and correct its motion despite errors in the estimated hole position. A decoupled gated reward coordinates alignment and insertion. Force-signal smoothing and state-independent standard deviations stabilize the learning process. The resulting policies perform real-world insertion across multiple hole geometries with a minimum nominal clearance of 0.02 mm and improve success while reducing peak contact forces under hole-position errors. Cross-clearance and cross-geometry evaluations further confirm policy generalization. The system achieved the first perfect score of 20/20 on ManipulationNet's peg-in-hole benchmark under its Human-in-the-Loop protocol, with fully autonomous insertion motions. A single policy trained only on a simulated hexagonal insertion task achieved an overall success rate of 95.0% across eight unseen real-world insertion tasks. These results show that learning entirely in simulation can yield precision insertion skills that can be deployed directly and reused across real-world tasks. The project website (this https URL) provides open-source simulation and real-robot experiment scripts, assets, and trained checkpoints.
• 中文摘要 接触丰富精度插入是机器人组装中的关键操作技能。紧密的间隙使插入对准错误更敏感，容易发生碰撞和卡壳，而几何形状和间隙的差异进一步复杂化策略的重复使用。我们提出了一个强化学习框架，完全在模拟中训练插入策略，无需真实演示或策略微调即可直接部署。通过将目标姿态与紧凑的三维指尖力反馈结合，策略学习在估计孔位误差下寻找对齐并修正其运动。解耦的门控奖励协调对准和插入。力信号平滑和状态无关标准差稳定学习过程。由此产生的策略在多个孔几何体中实现实际插入，最低标称间隙为0.02毫米，提升成功率，同时减少孔位误差下的峰值接触力。交叉净空和交叉几何评估进一步证实了策略的泛化性。该系统在ManipulationNet的“人机环路”协议下，采用了全自主插入动作，在ManipulationNet的钉入井基准测试中取得了首个满分20/20。仅在模拟六边形插入任务上训练的单一策略在八个未见的真实插入任务中实现了95.0%的总体成功率。这些结果表明，完全在模拟中学习可以获得可直接部署并在现实任务中重复使用的精确插入技能。项目网站（此链接链接）提供开源的仿真和真实机器人实验脚本、资源和训练好的检查点。

ME-VLM:A Unified VLM for Embodied Cognition and Agent Coordination

ME-VLM：一个用于具身认知与代理协调的统一VLM

• Authors: Foundation Model, Li Auto Inc
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.24526
• Pdf link: https://arxiv.org/pdf/2609.24526
• Abstract Physical AI requires models to ground visual and linguistic understanding in real-world environments while accounting for environmental constraints and execution feedback. We introduce MachEmbodied-VLM (ME-VLM), a unified vision-language model with two variants, 4B and 35B-A3B, that brings together embodied cognition and multimodal agent capabilities. Our work emphasizes physical perception and spatiotemporal reasoning, together with planning, interaction, and outcome assessment in both digital and physical environments. We construct training data spanning embodied and multimodal agent tasks, including execution observations and feedback to support outcome assessment and decision refinement. The training pipeline comprises embodied capability injection, separate reinforcement learning of embodied and multimodal-agent experts, and multi-teacher on-policy distillation that consolidates their complementary capabilities into a single model. Experiments show competitive performance on both embodied and agent benchmarks, as well as on autonomous-driving and embodied-navigation tasks. For edge deployment, visual token compression, W4A8 quantization, and hardware--software co-optimization enable on-device inference of the 4B variant on the M100, reducing prefill latency from 400 ms to 188 ms. Project Page: this https URL Code Repository: this https URL
• 中文摘要 物理人工智能需要模型将视觉和语言理解扎根于现实环境中，同时考虑环境约束和执行反馈。我们引入了MachEmbodied-VLM（ME-VLM），这是一个统一视觉语言模型，包含4B和35B-A3B两个变体，结合了具身认知与多模态代理能力。我们的工作强调物理感知和时空推理，以及数字和物理环境中的规划、交互和结果评估。我们构建涵盖具身和多模态代理任务的训练数据，包括执行观察和反馈，以支持结果评估和决策优化。培训流程包括具身能力注入、具身与多模态代理专家的独立强化学习，以及多教师的政策提炼，将它们互补的能力整合为单一模型。实验显示，在内涵和代理基准测试，以及自动驾驶和内在导航任务上均具竞争力。对于边缘部署，可视化令牌压缩、W4A8量化以及硬件-软件协同优化使M100的4B版本能够在设备内推断，将预填充延迟从400毫秒降至188毫秒。项目页面：此 https URL 代码库：此 https URL

Smoothness as a Constraint for Stable Humanoid Locomotion

平滑性作为稳定人形运动的约束条件

• Authors: Utsav Panchal, Denis Kleyko, Unal Artan, Amy Loutfi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24552
• Pdf link: https://arxiv.org/pdf/2609.24552
• Abstract Embodied AI systems, particularly humanoid robots deployed in real world scenarios require whole-body control policies that are both task-responsive and physically smooth. However, smoothness is not uniform across the body: lower body must remain sufficiently reactive, while the upper body must be tightly regulated to preserve stability. Existing reinforcement learning approaches typically impose smoothness through auxiliary terms in the reward function, which compete with task objectives, treating the body as uniform and provide no direct control over the physical quantities responsible for smooth behavior. We introduce DeCap (Decoupled Constraint-aware policy), a constrained reinforcement learning algorithm that decouples whole-body smoothness into separate upper- and lower-body constraint groups, each formulates smoothness as explicit constraints on physical motion limits. To improve constraint satisfaction near feasibility boundaries, DeCap incorporates a bounded barrier penalty that activates proactively as limits are approached and remains bounded at the constraint limit. On real-world humanoid whole-body control task, DeCap reduces upper-body action rate by 2.50x and acceleration by 2.18x relative to reward-based smoothness policies, while also improving lower-body smoothness and reducing transient motion. We demonstrate that a fixed set of smoothness constraints transfers across diverse terrains, alleviating the need of extensive reward tuning.
• 中文摘要 具身型人工智能系统，尤其是实际场景中部署的人形机器人，需要既符合任务响应又物理平滑的全身控制策略。然而，平滑性在全身并不均匀：下半身必须保持足够的反应性，而上半身则必须严格调节以保持稳定性。现有强化学习方法通常通过奖励函数中的辅助项来强加平滑性，这些辅助项与任务目标竞争，将身体视为统一，且不直接控制导致平滑行为的物理量。我们引入了DeCap（解耦约束感知策略），这是一种受限强化学习算法，将全身光滑性解耦为上下半身的独立约束组，每个组都将平滑性明确表述为对物理运动极限的约束。为提升可行边界附近的约束满足，DeCap引入了有界障碍惩罚，该惩罚在接近极限时主动激活，并保持限制限制。在现实中的人形全身控制任务中，DeCap相较于基于奖励的平滑策略，将上半身动作率降低2.50倍，加速度降低2.18倍，同时提升下半身平滑度并减少瞬态运动。我们证明固定的平滑约束可在不同地形间转移，减轻了大量奖励调优的需求。

Offline Reinforcement Learning for Distribution-Grid Protection

配电网保护的离线强化学习

• Authors: Julian Oelhaf, Alexander Luce, Christian Bergler, Andreas Maier, Siming Bayer
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.24703
• Pdf link: https://arxiv.org/pdf/2609.24703
• Abstract Data-driven protection may complement conventional relays in distribution grids whose operating conditions vary with distributed generation, switching events, and changing short-circuit levels. We study line-selective tripping from static trajectories of a realistically simulated CIGRE medium-voltage network using offline reinforcement learning. A convolutional Q-network receives causal voltage-current phasor and apparent-impedance features, optionally together with raw waveforms, and is trained with conservative Q-learning (CQL). A controlled sensitivity study evaluates two observation windows, reward variants, and three CQL weights under a common split and training protocol; one exploratory post-hoc run additionally increases the discount factor from $\gamma$=0.95 to 0.99. On 225 held-out episodes, the best per-timestep result is obtained with combined input and CQL weight $\alpha$=0.9, reaching precision 0.9993, recall 0.9496, and F1-score 0.9738. Because dense per-timestep scores do not encode the terminal semantics of relay operation, we also evaluate the first non-wait action in each episode. The default combined-input agent selects the correct line-trip action first in 98.13% of 214 fault episodes, but trips in 72.73% of the 11 non-fault episodes. In the post-hoc run, the corresponding rates are 98.60% and 54.55%, respectively. The results show that dense predictive performance and terminal protection behavior can lead to different model rankings. Offline CQL therefore demonstrates strong faulted-line selection on the simulated fault episodes, while the static trajectories, small non-fault set, and single-seed post-hoc design preclude conclusions about practical relay security or deployment readiness.
• 中文摘要 数据驱动保护可以补充配电网中的传统继电器，其工作条件会随着分布式发电、切换事件和短路电平变化而变化。我们利用离线强化学习研究了从真实模拟CIGRE中压网络静态轨迹中的线路选择性跳闸。卷积Q网络接收因果电压-电流相量和表观阻抗特征，可选择性地结合原始波形，并通过保守Q学习（CQL）进行训练。一项受控敏感性研究评估了两条观察窗口、奖励变体和三种CQL权重，采用统一的拆分和训练协议;一次探索性事后运行还将折价因子从$\gamma$=0.95提升至0.99。在225个暂停的集数中，结合输入和CQL权重$\alpha$=0.9，获得最佳每时间步长结果，精度为0.9993，回忆0.9496，F1得分0.9738。由于密集的每时间步得分不编码中继操作的终端语义，我们还评估了每集的第一个非等待动作。默认的合并输入代理在214个故障事件中98.13%优先选择正确的线路跳行动作，但在11个非故障事件中，跳闸率为72.73%。在事后运行中，相应的跳闸率分别为98.60%和54.55%。结果表明，密集的预测性能和终端保护行为可能导致不同的模型排名。因此，离线CQL在模拟故障事件中展示了强的故障线选择，而静态轨迹、小型非故障集和单种子事后设计则排除了关于实际中继安全性或部署准备度的结论。

Steerable and Reactive Grasping Through Modular Design with a Three-Point Interface

通过模块化设计实现可转向和反应式抓取，采用三点接口

• Authors: Andrew Nguyen, Yonghyeon Lee, Sangbae Kim
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24896
• Pdf link: https://arxiv.org/pdf/2609.24896
• Abstract Dexterous grasping requires deciding where to grasp, reaching the target, and maintaining stable contact. We connect these stages through a compact three-point interface that separates global geometric reasoning from local contact control. Given object geometry and optional language commands, our framework samples contact triples from a precomputed grasp-affordance heatmap. A model-based reactive controller tracks the object, avoids collisions, and guides the hand toward the selected contacts. In the final centimeters, a Reinforcement Learning (RL) policy uses proprioceptive feedback to refine and stabilize the grasp despite reaching and perception errors. It observes only finger joint states and its recent actions, with no target points, visual observations, or object geometry, so a single policy is shared across objects and grasp configurations. In simulation, we compare grasp-and-lift success against squeeze and end-to-end baselines, characterize reaching convergence, and demonstrate grasp steering; hardware demonstrations on two training objects and one unseen object illustrate the full pipeline. Our modular framework uses geometry to guide the reach and local feedback to secure the grasp.
• 中文摘要 灵巧抓取需要决定抓握位置、达到目标并保持稳定接触。我们通过紧凑的三点接口连接这些阶段，将全局几何推理与局部接触控制分离。在物体几何和可选语言命令下，我们的框架从预先计算的抓握-供能热图中抽取接触三重。基于模型的反应式控制器追踪物体，避免碰撞，并引导手部朝向选定的接触点。在最后几厘米，强化学习（RL）策略利用本体感觉反馈来细化和稳定抓握，尽管有触及和感知误差。它仅观察手指关节状态及其近期动作，没有目标点、视觉观察或物体几何，因此单一策略在物体和抓取配置间共享。在模拟中，我们将抓取抬举成功率与挤压和端到端基线进行比较，描述到达收敛情况，并演示抓握引导;在两个训练对象和一个看不见物体上的硬件演示展示了整个流程。我们的模块化框架利用几何结构引导伸手，并通过局部反馈确保抓握。

VideoGen-Agent: Reinforcing Video Generation Agents

VideoGen-Agent：强化视频生成代理

• Authors: Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang, Zihao Lin, Kaituo Feng, Suozhi Huang, Xiangyi Li, Yu Li, Chunyuan Li, Shilong Liu, Mengdi Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.24997
• Pdf link: https://arxiv.org/pdf/2609.24997
• Abstract Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools.
• 中文摘要 视频生成模型的最新进展使得高保真、时间相干的视频生成成为可能。然而，这些模型常常难以满足需要专业知识、特定身份、物理一致性或有序事件的提示。本文介绍了VideoGen-Agent，一种通过多任务代理强化学习训练的多模态代理，能够使用外部工具进行视频生成。该代理通过多回合交互协调增强、生成和验证工具，利用提示和中间观察来指导决策。我们在涵盖六个任务的类别平衡数据集上训练共享策略。对教师生成轨迹进行监督微调，建立工具使用行为，随后通过强化学习进行细化。类别感知的混合奖励评估工具调用有效性、任务适用工具使用及生成视频质量。我们还进一步介绍了VABench，这是一个包含600个提示的基准测试，涵盖程序知识、单实体和多实体身份保护、物理一致性、场景构成以及多镜头时间结构。在VABench上，VideoGen-Agent相较其基础文本转视频生成器提升了19.1分，从56.5分提升至75.6分。升级生成工具后，得分提升至86.1分，无需额外代理训练。在84.3%的比较中，人工评分者更倾向于升级后的配置而非最强的独立基线。这些结果支持了在视频生成任务中学习工具的使用，并表明受过训练的代理可以从后续生成工具的进步中受益。

Keyword: diffusion policy

Latent Policy Steering: An Efficient and Flexible Framework for Cross-Embodiment Transfer

潜在政策引导：一个高效且灵活的跨身体转移框架

• Authors: Yiqi Wang, Mrinal Verghese, Jeff Schneider
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22521
• Pdf link: https://arxiv.org/pdf/2609.22521
• Abstract The performance of learned robot visuomotor policies depends heavily on the size and quality of their training data, yet collecting high-quality demonstrations remains costly for robots in the real world. Although large-scale robot and human datasets are increasingly available, embodiment gaps and mismatched action spaces make them difficult to leverage directly. Cross-embodiment transfer, reusing experience from other embodiments to improve learning on a target embodiment, is therefore crucial for scaling robot learning beyond per-robot data collection. In this work, we find that efficient transfer can be achieved by learning from what is shared across embodiments, the visual dynamics of how the world responds to motion, and by effectively exploiting the scarce target-embodiment data at test time. The proposed framework, called Latent Policy Steering (LPS), implements an embodiment-agnostic pretraining phase, which trains an image-based World Model (WM) with optical flow across diverse embodiments. The resulting WM is finetuned on the target embodiment with robot actions. It then steers the base policy toward better actions by searching in the WM's latent space for plans that stay close to the finetuning data. LPS is a policy-agnostic framework: it can flexibly accommodate different policies without having to retrain them. In Robomimic and real-world evaluations, LPS improves the average performance of Diffusion Policy relatively by 16% and 62%, and Pi0.5 by 8% and 14%, with only 50 demonstrations on an unseen target embodiment.
• 中文摘要 学习机器人视觉运动策略的性能高度依赖于训练数据的大小和质量，但对现实世界中机器人来说，收集高质量演示仍然成本较高。尽管大规模机器人和人类数据集日益普及，但体现差距和动作空间不匹配使得它们难以直接利用。跨身体迁移，即重复利用其他身体的经验以提升目标身体的学习，因此对于将机器人学习扩展到超越单个机器人数据收集至关重要。在这项工作中，我们发现通过学习跨身体的共享数据、世界对运动的视觉动态学习，以及在测试时有效利用稀缺的目标-身体数据，可以实现高效的迁移。所提出的框架称为潜在政策引导（Latent Policy Steaning，LPS），实现了与身体无关的预训练阶段，通过在不同身体上训练基于图像的世界模型（WM）并以光流进行训练。生成的WM通过机器人动作在目标具体化上进行微调。然后通过在WM的潜在空间中搜索与微调数据相近的计划，引导基础策略朝向更好的行动。LPS是一个政策无关的框架：它可以灵活地适应不同策略，而无需重新训练它们。在机器人模拟和现实世界评估中，LPS在未见目标实例上仅有50个演示，使扩散政策的平均表现相对提升了16%和62%，Pi0.5平均提升了8%和14%。

BEACON: Belief-Enabled Adaptive CONtrol for Imitation Learning under Uncertainty

信标：信念驱动的自适应控制系统，用于在不确定性下的模仿学习

• Authors: Moonyoung Lee, Soumojit Bhattacharya, George Kantor, Oliver Kroemer
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22730
• Pdf link: https://arxiv.org/pdf/2609.22730
• Abstract Robot manipulation tasks often involve hidden state information that cannot be directly observed and must be inferred through sequential physical interactions. In such partially observable settings, conditioning an imitation learning policy directly on the recent raw observation history leads to poor performance. This is due to state aliasing, wherein identical observations may arise from different hidden states, and the policy receives conflicting action labels for the same input. To enable history-aware disambiguation capability, we propose conditioning a diffusion policy on a structured representation of the hidden states using Bayesian belief that exposes both the current most likely state estimate and the remaining uncertainty. This representation replaces raw history with a structured, compact input, enabling the policy to implicitly modulate between exploratory and exploitative behaviors based on belief uncertainty, without explicit mode switching or reward shaping. We evaluate across two domains with qualitatively different belief representations: a continuous belief for cornstalk gripper alignment via tactile sensing, and a discrete categorical distribution for latched door opening. In both domains, the belief-conditioned policy substantially outperforms the observation-only baseline and approaches privileged ground-truth performance, with ablations illustrating that the policy adapts its exploration behavior depending on the belief uncertainty at inference time.
• 中文摘要 机器人操作任务通常涉及无法直接观察的隐藏状态信息，必须通过连续物理交互推断。在此类部分可观察的环境中，直接基于近期原始观测历史来条件模拟学习策略会导致性能不佳。这是由于状态混叠，即相同的观察可能来自不同的隐藏状态，策略对同一输入会获得冲突的动作标签。为了实现历史感知消歧功能，我们提出以贝叶斯信念为前提，将扩散策略条件为隐藏状态的结构化表示，同时揭示当前最可能状态估计和剩余不确定性。该表示用结构化、紧凑的输入替代原始历史，使策略能够基于信念不确定性隐式调节探索性和利用性行为，无需显式切换模式或奖励塑形。我们评估了两个具有定性差异信念表征的领域：通过触觉感知实现玉米茎夹具对齐的连续信念，以及锁门打开的离散类别分布。在这两个领域中，信念条件策略的表现显著优于仅观察基线，并接近特权的地面真实表现，消融表明策略会根据推断时信念不确定性调整其探索行为。

Whole-Body UMI: Transferring UMI Manipulation Skills to Humanoid Whole-Body Manipulation via Real-Time Motion Generation

全身UMI：通过实时动作生成将UMI操作技能转移到人形全身操控

• Authors: Yuxuan Nai, Leixin Chang, Liangjing Yang, Shuo Yang, Zhongyu Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.22829
• Pdf link: https://arxiv.org/pdf/2609.22829
• Abstract Collecting whole-body demonstrations for humanoid manipulation mostly relies on teleoperation, which is costly and hard to scale up. The Universal Manipulation Interface (UMI) provides a scalable data collection paradigm, but end-effector trajectories alone underdetermine humanoid whole-body coordination, which is insufficient for whole-body demonstration collection. Therefore, we introduce Whole-Body UMI (WB-UMI), a task-agnostic, real-time and end-effector conditioned motion generator that decouples whole-body coordination learning from task semantics learning through a shared end-effector interface. A diffusion policy learns from native UMI demonstrations, while WB-UMI learns independently from retargeted motion capture, requiring no body trackers or paired image--whole-body demonstrations during task-specific data collection. In real deployment, an asynchronous hierarchy integrates the diffusion policy, motion generator, and a whole-body controller with latency compensation and measured-state feedback. Real-robot experiments on G1 support real-time closed-loop transfer across four tasks, achieving 90% success in drawer closing, 80% in shelf pick-and-place, 30% in ball toss, and 40% in Loco-PnP, which shows the effectiveness of this hierarchy in transferring native UMI skills to humanoid whole-body manipulation.
• 中文摘要 用于人形操作的全身演示收集主要依赖远程操作，而远程操作成本高且难以扩展。通用操作接口（UMI）提供了可扩展的数据收集范式，但仅靠末端执行器轨迹不足以决定人形与全身的协调，无法实现全身演示收集。因此，我们引入了全身UMI（WB-UMI），这是一种任务无关的实时和终端执行器条件运动生成器，通过共享终端执行器接口将全身协调学习与任务语义学习解耦。扩散策略从原生UMI演示中学习，而WB-UMI则独立于重定向动作捕捉学习，无需身体追踪器或任务特定数据采集时的成对图像-全身演示。在实际部署中，异步层级集成了扩散策略、运动发生器和全体控制器，并实现延迟补偿和测量状态反馈。G1上的真实机器人实验支持四项任务的实时闭环传输，收拾抽屉成功率达90%，架子摆放成功率80%，投球成功率30%，Loco-PnP成功率40%，显示出该层级在将原生UMI技能转化为人形全身操作方面的有效性。

ContactDP: Contact-Guided Diffusion Policy for Tight Insertion Tasks

ContactDP：针对紧密插入任务的接触引导扩散政策

• Authors: Chengyi Xing, Shaoxiong Yao, Diego Romeres, Devesh K. Jha
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.23800
• Pdf link: https://arxiv.org/pdf/2609.23800
• Abstract High-precision connector insertion remains challenging for robotic systems due to tight mechanical tolerances, partial observability during contact, and multimodal uncertainty arising from occlusion and contact ambiguity. Successful insertion requires closed-loop contact guidance that continuously integrates global alignment cues with local contact feedback to produce stable corrective actions under interaction. In this work, we present ContactDP (Contact-Guided Diffusion Policy for Tight Insertion Tasks), a multimodal diffusion-policy framework for contact-rich insertion. ContactDP jointly integrates wrist RGB observations, fingertip tactile sensing, and wrist-mounted force-torque measurements to infer contact state and generate temporally consistent corrective motions during insertion. To ensure stable execution under contact, the learned policy operates together with a hybrid position-force controller that provides compliant low-level interaction. We evaluate our approach on a suite of industrial-grade connector insertion tasks with varying connector geometries, grasp conditions, and initial misalignment. Across all tasks, ContactDP significantly outperforms vision-only diffusion policies for performance, reliability and generalization.
• 中文摘要 由于机械公差严格、接触时部分可观测性以及多模态因遮挡和接触模糊性引起的不确定性，高精度连接器插入对机器人系统仍具挑战性。成功插入需要闭环接触引导，持续整合全局对齐线索与局部接触反馈，以在交互下实现稳定的纠正动作。本研究提出了ContactDP（紧密插入任务接触引导扩散策略），这是一个多模态扩散策略框架，用于接触富丰富插入。ContactDP整合腕部RGB观测、指尖触觉感测和腕部装载力-扭矩测量，推断接触状态并生成插入时时间一致的修正动作。为确保接触下的稳定执行，该策略与混合位置-力控制器协同工作，提供顺应的低层交互。我们评估了一系列工业级连接器插入任务，这些任务具有不同的连接器几何形状、抓握条件和初始错位。在所有任务中，ContactDP在性能、可靠性和泛化性方面均显著优于仅视觉扩散策略。

Performance-Preserving Online Adaptation in Social Navigation via Diffusion Steering

通过扩散引导实现社交导航中保持性能的在线适配

• Authors: Haruto Nagahisa, Kohei Matsumoto, Yuki Hyodo, Ryo Kurazume
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24317
• Pdf link: https://arxiv.org/pdf/2609.24317
• Abstract In social navigation, modeling the complex interactions between humans and robots is difficult, and deep reinforcement learning has therefore been actively studied. However, because simulation alone cannot fully reproduce diverse scenarios, robot dynamics, and the social conventions that vary across deployment environments, fine-tuning in the deployment environment is promising. In doing so, learning that preserves the base model's performance is required, so as not to compromise the primary objective of navigation, namely avoiding pedestrians and reaching the destination. In this study, we propose a method that applies diffusion steering via reinforcement learning (DSRL), which trains only the noise policy while keeping the diffusion policy fixed, thereby achieving learning that preserves performance. Furthermore, we integrate diffusion-based RL policies trained with multiple seeds to construct the base policy, improving learning performance. Our evaluation shows that, compared with other methods, the proposed method enables efficient learning while preserving performance, and we confirm flexible behavior control through adaptation to social conventions, as well as its effectiveness on a physical robot through hardware-in-the-loop simulation.
• 中文摘要 在社会导航中，建模人机复杂互动较为困难，因此深度强化学习被积极研究。然而，由于仅靠仿真无法完全再现多样化场景、机器人动态及不同部署环境中的社会惯例，部署环境的微调具有前景。在此过程中，学习必须保持基础模型的性能，以免牺牲导航的主要目标——避开行人并到达目的地。本研究提出一种通过强化学习（DSRL）应用扩散引导的方法，该方法仅训练噪声策略，保持扩散策略固定，从而实现保持性能的学习。此外，我们将训练的基于扩散的强化学习策略与多个种子整合构建基础策略，提升学习性能。我们的评估表明，与其他方法相比，该方法实现了高效学习同时保持性能，并通过适应社会惯例确认了灵活行为控制，以及通过硬件在环模拟对物理机器人的有效性。

TACIT: Tactile Contact Supervision for Spatial Attention in Dexterous Manipulation

默契：灵巧操作中空间注意力的触觉接触监督

• Authors: Yanhou Lai, Fucai Zhu, Ruiqiang Wang, Koichi Hashimoto
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.24507
• Pdf link: https://arxiv.org/pdf/2609.24507
• Abstract Visuomotor policies trained from a few demonstrations may reproduce demonstrated trajectories without reliably following changes in object position. Existing approaches with explicit attention typically obtain spatial priors from human annotation or visual models. We introduce TACIT (tactile contact informs attention), which uses measured tactile contacts from teleoperated demonstrations to supervise spatial attention without additional point annotation. Gaussian targets over preceding camera point clouds supervise an attention head whose pooled output conditions a visuotactile diffusion policy. Targets are used only during training; tactile observations remain inputs at inference. In the primary real-robot benchmark, with ten demonstrations per task and five demonstrated placement regions, TACIT achieves 66.7% success on ball placement and 73.3% on peg insertion, compared with 10.0% and 20.0% for input-matched 3D visuotactile fusion and 20.0% and 43.3% for vision-only DP3. TACIT enters the 150 mm palm-to-object approach region within 12 seconds in all 30 trials per task; all remaining failures occur after arrival. Across three training seeds on real ball and simulated peg, TACIT outperforms input-matched fusion and an architecture-matched control without explicit attention supervision, supporting the contribution of supervision beyond branch capacity. Pre-contact and contact-time supervision show no consistent ordering. These results demonstrate that measured tactile contact provides effective spatial supervision for approach behavior from few demonstrations within the evaluated workspace.
• 中文摘要 通过少数演示训练的视觉运动策略可能在不可靠跟踪物体位置变化的情况下重现已示范的轨迹。现有的显性注意方法通常通过人类注释或视觉模型获得空间先验。我们引入了TAKIT（触觉接触告知注意力），利用远程操作演示中的测量触觉接触来监督空间注意力，无需额外点标注。高斯目标覆盖前方摄像头点云，监督一个注意力头，其集中输出条件为视觉触觉扩散策略。目标仅在训练期间使用;触觉观察仍作为推断输入。在主要的实机器人基准测试中，每个任务有十个演示和五个展示区域，TACIT在球体放置和插销插入成功率为73.3%，而输入匹配的三维视觉触觉融合为10.0%和20.0%，仅视觉DP3为20.0%和43.3%。TACIT在所有30次试验中均在12秒内进入150毫米手掌对物体接近区;其余所有失败均发生在到达后。在真实球和模拟插销的三个训练种子中，TACIT优于输入匹配融合和架构匹配对照组，无需明确注意监督，支持监督超出分支能力的贡献。接触前和接触时间监督显示出不一致的排序。这些结果表明，测量触觉接触能通过评估工作区内少数演示，有效为接近行为提供空间监督。

Touch2Robot: Robot Touch in the Human Demonstration Loop

Touch2Robot：人体演示环路中的机器人触摸

• Authors: Shengcheng Luo, Xiaoyang Cheng, Hong Ying, Xiaoying Zhou, Jiaming Jiang, Haoran Guo, Wanlin Li, Ziyuan Jiao, Chenxi Xiao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.24660
• Pdf link: https://arxiv.org/pdf/2609.24660
• Abstract Human demonstrations offer a scalable way to collect manipulation data, but their contacts may be unstable or infeasible when transferred to a robot hand. Collecting demonstrations directly on the target robot avoids this mismatch, but substantially increases the cost of data collection. To address this trade-off, we present \textbf{Touch2Robot}, a framework that lets humans collect demonstrations while seeing how the target robot hand would contact the object. We capture human hand motion, tactile-glove measurements, and object motion during human manipulation. These recordings guide object-specific RL policies to reproduce the demonstrated object motion while favoring contacts consistent with the recorded human touch. We distill the learned behaviors into a unified real-time retargeter that maps incoming human observations and object geometry to robot hand configurations. During collection, the predicted robot configuration is synchronized with the tracked object pose in simulation to reconstruct robot-object contacts, which are visualized to help the demonstrator adapt subsequent interactions to the target hand. Across four real-world tasks, Touch2Robot improves average real-robot replay completion from 37.9\% to 72.1\% over visual-only feedback, while reducing the collection time per replay-successful demonstration from 58.6~s to 18.2~s. Reconstructed target-hand contacts achieve 44.2\% F1 against real-robot tactile measurements, and policies trained on Touch2Robot demonstrations improve downstream Diffusion Policy performance by 29.1 percentage points over visual-only feedback. These results show that bringing robot touch into the human demonstration loop improves both the quality and efficiency of scalable dexterous data collection. \textit{Project webpage: \href{this https URL}{this https URL}.}
• 中文摘要 人工演示提供了一种可扩展的操作数据收集方式，但当接触转移到机器人手时可能不稳定或不可行。直接在目标机器人上收集演示可以避免这种不匹配，但大幅增加数据收集成本。为解决这一权衡，我们提出了 \textbf{Touch2Robot} 框架，允许人类在观察目标机器人手如何接触物体的同时收集演示。我们捕捉了人类手部运动、触觉手套测量和人体操作过程中的物体运动。这些记录指导了针对对象的强化学习策略，以再现已演示的物体运动，同时优先考虑与记录到的人类触摸一致的接触。我们将学习到的行为提炼成统一的实时重定向器，将输入的人类观察和物体几何映射到机器人手的配置。采集过程中，预测的机器人配置与模拟中跟踪的物体姿势同步，重建机器人与物体的接触，并以可视化方式帮助演示者调整后续交互以适应目标手。在四个实际任务中，Touch2Robot将模拟机器人回放完成率从37.9%提升至仅视觉反馈的72.1%，同时将每次回放成功演示的收集时间从58.6~秒缩短至18.2~秒。重建的目标手接触在真实机器人触觉测量中达到44.2%的F1，基于Touch2Robot演示训练的策略比仅视觉反馈提升了下游扩散策略性能29.1个百分点。这些结果表明，将机器人触摸引入人类演示循环，提高了可扩展灵巧数据收集的质量和效率。\textit{项目网页：\href{此 https URL}{此 https URL}.}

PredActor: Predictive Action Diffusion for Steerable Onboard Humanoid Control

PredActor：可转向人形控制的预测行动扩散

• Authors: Lei Ye, Haibo Gao, Yitang Li, Peng Xu, Zetong Jing, Junhan Sun, Fanrong Dong, Ziqi Han, Xue Wang, Jianhua Sun, Cewu Lu, Hao Zhao, Liang Ding
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.24840
• Pdf link: https://arxiv.org/pdf/2609.24840
• Abstract Diffusion models offer flexible motion generation, but translating this flexibility into feedback-responsive humanoid control remains challenging. Hierarchical systems steer motion through references that may exceed a separate tracker's capabilities, leaving recovery and physical execution largely to the tracker. Action-only diffusion generates actions directly but lacks an explicit future-state trajectory for test-time motion objectives. Joint state-action diffusion provides this representation, yet representative controllers often depend on privileged full-body states, and support for learned behavior selection and test-time motion steering remains fragmented. We present PredActor, a predictive action diffusion policy that brings these complementary steering capabilities into one directly executed policy using proprioceptive observations. Conditioned on proprioceptive history and optional task context, PredActor jointly generates executable actions and an internal future-state trajectory. Classifier-free guidance strengthens text-conditioned behavior, while classifier guidance steers predicted states toward test-time objectives. Only actions are executed, without a separate motion-reference tracker or externally estimated full-body states as policy inputs. In simulation, PredActor reaches all 15 destination targets and achieves a text retrieval score of 0.580, compared with 0.373 for conditional action diffusion, with similar observed disturbance survival. To make this guided policy practical onboard, rolling denoising and computation-preserving runtime optimizations reduce the complete callback to 16.790 ms median and 19.383 ms p95 on a Jetson Orin NX, both below the 20 ms control period. We deploy PredActor on a Unitree G1; evaluations across simulation and physical hardware demonstrate text-conditioned motion, disturbance response, joystick control, and semantic interpolation.
• 中文摘要 扩散模型提供灵活的运动生成，但将这种灵活性转化为反馈响应型人形控制仍然具有挑战性。分层系统通过可能超出独立追踪器能力的引用引导运动，恢复和物理执行主要由追踪器负责。仅动作扩散直接生成动作，但缺乏测试时间运动目标的明确未来状态轨迹。联合状态-动作扩散提供这种表示，但代表性控制器通常依赖特权的全身状态，且对学习行为选择和测试时动作引导的支持仍零散。我们介绍PredActor，一种预测性动作扩散策略，将这些互补引导能力整合为一个直接执行策略，利用本体感知观察实现。基于本体感知历史和可选任务上下文，PredActor联合生成可执行动作和内部未来状态轨迹。无分类器的引导强化文本条件行为，而分类器引导引导预测状态朝向测试时间目标。仅执行动作，无需单独的运动参考跟踪器或外部估计的全身状态作为策略输入。在模拟中，PredActor 能够覆盖全部 15 个目标，文本检索得分为 0.580，而条件动作扩散为 0.373，且观察到的干扰存活率相似。为了使该引导策略在机载上实用，滚动去噪和计算保持运行时优化将 Jetson Orin NX 的中位数回调降至 16.790 毫秒，在 Jetson Orin NX 上为 19.383 毫秒的 p95，均低于 20 毫秒的控制周期。我们将 PredActor 部署在 Unitree G1 上;跨仿真和物理硬件的评估展示了文本条件运动、扰动响应、摇杆控制和语义插值。