生成时间: 2026-09-23 21:26:40 (UTC+8); Arxiv 发布时间: 2026-09-23 20:00 EDT (2026-09-24 08:00 UTC+8)

今天共有 38 篇相关文章

Keyword: reinforcement learning

Towards Adaptive Interaction Strategies for Human Companion Robot via Deep Reinforcement Learning

通过深度强化学习迈向人类伴侣机器人的自适应交互策略

• Authors: Cong-Thanh Vu, Yen-Chen Liu
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.25031
• Pdf link: https://arxiv.org/pdf/2609.25031
• Abstract In the field of Human-Robot Interaction (HRI), achieving flexibility in human-accompanying within real-world environments holds great potential for various applications but also poses significant challenges. Traditional methods typically restrict robots to fixed positions relative to humans, such as tracking from behind, in front, or side-by-side, which limits robot adaptability in dynamic workspaces. This study introduces a novel human-companioning strategy that uses Reinforcement Learning (DRL) to enable mobile robots to dynamically adjust their tracking positions according to varying conditions. An interaction space is defined to capture the relationship between the human and the robot while considering the environment, which serves as the basis for state spaces in DRL to assist the robot in adapting to environmental changes. A human-robot companion controller is developed by integrating Model Predictive Path Integral (MPPI) control with Control Barrier Functions (CBF), ensuring that the robot accurately follows the target's movement in both position and orientation while avoiding obstacles and enhancing social acceptance and safety. The proposed approach is evaluated in real-world scenarios, both indoors and outdoors, and compared with other studies. The results show that the proposed method improves the success rate and tracking accuracy by at least 24% and 47%, respectively, while enhancing human comfort. Experiments demonstrate the robot's ability to flexibly accompany a person walking at speeds of up to 1.7 m/s, dynamically adjusting its strategy without being confined to a fixed position. Additionally, the robot respects the human's intimate space to ensure safety, comfort, and effective obstacle avoidance.
• 中文摘要 在人机交互（HRI）领域，实现在真实环境中实现人伴的灵活性具有巨大潜力，但也带来了重大挑战。传统方法通常将机器人限制在相对于人类的固定位置，如从后方、前方或并排跟踪，这限制了机器人在动态工作空间中的适应性。本研究引入了一种新型人伴策略，利用强化学习（DRL）使移动机器人能够根据不同条件动态调整其跟踪位置。定义交互空间以捕捉人与机器人之间的关系，同时考虑环境，这成为DRL状态空间的基础，帮助机器人适应环境变化。人机伴随控制器通过将模型预测路径积分（MPPI）控制与控制障碍函数（CBF）集成开发，确保机器人在目标位置和方向上准确跟踪其移动，同时避开障碍物，提升社会接受度和安全性。该方法在室内外的真实场景中进行了评估，并与其他研究进行了比较。结果显示，该方法分别提升了成功率和追踪准确率至少24%和47%，同时提升了人体舒适度。实验展示了机器人能够灵活地以最高1.7米/秒的速度陪伴行走者，动态调整策略而不被限制在固定位置。此外，机器人尊重人类的亲密空间，以确保安全、舒适和有效的障碍避让。

RULER: Instance-aware Rubric Rewards for SVG Generation

RULER：SVG生成的实例感知评分奖励

• Authors: Hangyu Ran, Yuhao Zheng, Yingying Zhang, Kevin Qinghong Lin, Han Peng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.25270
• Pdf link: https://arxiv.org/pdf/2609.25270
• Abstract Generating Scalable Vector Graphics (SVG) code from natural-language instructions is an open-ended task without absolute visual ground truth, leaving both evaluation and policy optimization without a faithful signal. Scalar metrics (CLIP, Aesthetic) calibrated on natural images transfer poorly to stylized vector content, and reusing them as RL rewards triggers reward hacking. We address both limitations with rubric-based scoring. We first establish empirically that prompting a vision-language judge with a multi-axis rubric correlates with human judgments far better than scalar metrics, both across samples and within instructions. Building on this finding, we introduce RULER (Instance-aware Rubric Rewards for Reinforcement Learning), which converts each instruction into an instance-aware rubric of six items spanning semantic, visual, and stylistic axes; a judge VLM scores rendered rollouts item-by-item, and the weighted satisfactions form a fine-grained reward optimized via Group Relative Policy Optimization. Because the rubric is derived from text alone, RULER requires neither paired SVG ground truth nor human preference labels. On MMSVG-Illustration and MMSVG-Icon, RULER lifts the rubric score from 0.432/0.395 to 0.693/0.683, surpassing dedicated SVG specialists and matching the substantially larger DeepSeek-V3, with ablations identifying rubric design as the active lever for RL on open-ended SVG generation. The project page is available at this https URL.
• 中文摘要 从自然语言指令生成可扩展矢量图形（SVG）代码是一项开放式任务，缺乏绝对的视觉基础，导致评估和策略优化都缺乏忠实信号。在自然图像上校准的标量指标（CLIP、美学）在风格化矢量内容上转换效果不佳，且将其作为强化学习奖励的重复使用会触发奖励黑客行为。我们通过基于评分标准的评分解决了这两个局限性。我们首先实证地证明，使用多轴评分标准的视觉语言评判，比标量指标在样本间和指令内的判断中，与人类判断的相关性远佳。基于这一发现，我们引入了RULR（实例感知的强化奖励评分标准），它将每条指令转换为包含六项的实例感知评分标准，涵盖语义、视觉和风格轴;评委VLM逐项对渲染的推广进行评分，加权满意度形成通过小组相对策略优化的细粒度奖励。由于评分标准仅源自文本，RULER无需配对的SVG真实性或人工偏好标签。在MMSVG-Illustration和MMSVG-Icon上，RULER将评分标准分数从0.432/0.395提升至0.693/0.683，超越专门的SVG专家，匹配更大质量的DeepSeek-V3，并带有消融处理，标准设计作为开放式SVG生成中强化学习的主动优势。项目页面可在此 https 网址访问。

Learning to Plan in Human-Robot Collaboration: Multimodal Reinforcement Learning for Adaptive Interaction

人机协作中的规划学习：多模态强化学习用于自适应交互

• Authors: Afagh Mehri Shervedani, Siyu Li, Natawut Monaikul, Bahareh Abbasi, Barbara Di Eugenio, Miloš Žefran
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25274
• Pdf link: https://arxiv.org/pdf/2609.25274
• Abstract Robot assistants for older adults and people with disabilities need to perform collaborative tasks with users effectively. The core component of these systems is an interaction manager whose job is to observe and assess the task and infer the state of the human and their intent for the robot to choose the best course of action. Due to the sparseness of the data in this domain, the policy for such multimodal systems is often crafted by hand; as the complexity of interactions grows, this process is not scalable. This paper proposes a reinforcement learning (RL) approach to automatically generate the multimodal policy of the robot. Our system focuses on a realistic scenario where a robot assists a user in locating objects within a home environment, managing multimodal signals, including language and physical actions, to select the best action. In contrast to traditional dialog systems, our agent is trained with a simulator that uses human data and can deal with multiple modalities. We use a simple high-level reward function that needs no fine-tuning and enforce some preconditions to speed up the training process. A human study evaluating the system in a real-world setting demonstrates promising results, indicating high usability and effective task completion. This RL-based approach offers a scalable and interpretable alternative for designing interaction managers in multimodal human-robot collaborations.
• 中文摘要 老年人和残障人士的机器人助手需要有效地与用户协作完成任务。这些系统的核心组件是一个交互管理器，其职责是观察和评估任务，推断人类状态及其意图，促使机器人选择最佳行动方案。由于该领域数据稀少，这类多模态系统的策略通常由手工制定;随着交互复杂度的增加，这一过程无法扩展。本文提出了一种强化学习（RL）方法，以自动生成机器人的多模态策略。我们的系统聚焦于一个现实情景，机器人协助用户在家庭环境中定位物体，管理包括语言和身体动作在内的多模态信号，以选择最佳行动。与传统对话系统不同，我们的智能体使用使用人类数据的模拟器训练，能够处理多种模态。我们使用一个无需微调的简单高级奖励函数，并强制执行一些前提条件以加快训练过程。一项在真实环境中评估系统的人类研究显示出有希望的结果，表明其高度可用性和高效的任务完成。这种基于强化学习的方法为设计多模态人机协作中的交互管理器提供了可扩展且可解释的替代方案。

TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks

TelecomGPT-R1：跨异构电信任务推理的统一后期训练

• Authors: Bohao Wang, Chenwei Wu, Hang Zou, Yu Tian, Lina Bariah, Li Wei, Chongwen Huang, Yongliang Shen, Zhaoyang Zhang, Merouane Debbah
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.25356
• Pdf link: https://arxiv.org/pdf/2609.25356
• Abstract Large language models (LLMs) offer great potential to automate a broad range of telecom engineering tasks by reasoning over standards, network configurations, mathematical models, source code, and operational logs. However, existing telecom LLMs struggle to reliably reason across these diverse tasks and data types. General-purpose LLMs often lack reliable grounding in telecom-specific knowledge, while telecom-specialized models are typically developed for narrower task families and exhibit limited multi-task performance. To fill this gap, we introduce TelecomGPT-R1, a family of open source unified telecom reasoning models structured around four complementary axes: protocol, knowledge, modeling, and fault. We first develop an axis-aware data generation framework that refines coarse public telecom artifacts into verified question-answer pairs and high quality chain-of-thought (CoT) reasoning trajectories, yielding a training corpus containing 104,880 examples. Building on this corpus, supervised fine-tuning (SFT) instills telecom knowledge and evidence-grounded reasoning patterns to overcome the cold start barrier for reinforcement learning (RL). We then apply dynamic sampling policy optimization (DAPO) with task-routed rubric rewards to keep RL updates informative and stable across heterogeneous telecom reasoning tasks. These rewards decompose axis-specific CoT traces into verifiable reasoning units and combine grounded dense process credit with outcome correctness, allowing RL to learn generalizable problem solving behaviors from verifiable telecom evidence. We release the TelecomGPT-R1 models and a reproducible training recipe to support further community development. Evaluations on seven benchmarks of the GSMA Open Telco Leaderboard show that the open-source TelecomGPT-R1-27B achieves an 89.64% mean score, outperforming leading proprietary models, including GPT-5, Claude, and Gemini.
• 中文摘要 大型语言模型（LLMs）在通过标准、网络配置、数学模型、源代码和运营日志推理，具有自动化广泛电信工程任务的巨大潜力。然而，现有电信LLM在这些多样化任务和数据类型之间难以可靠推理。通用LLM通常缺乏可靠的电信专属知识基础，而电信专用模型通常为更狭窄的任务族开发，且多任务性能有限。为填补这一空白，我们引入了TelecomGPT-R1，这是一系列开源统一电信推理模型，围绕协议、知识、建模和故障四个互补轴构建。我们首先开发了一个基于轴的感知数据生成框架，将粗糙的公共电信产物精炼为经过验证的问答对和高质量的思维链（CoT）推理轨迹，生成包含104,880个示例的训练语料库。基于该语料库，监督式微调（SFT）注入电信知识和基于证据的推理模式，以克服强化学习（RL）中的冷启动障碍。随后，我们应用动态抽样策略优化（DAPO）与任务路由的评分奖励，使强化学习在异构电信推理任务中保持信息性和稳定性。这些奖励将轴特定CoT痕迹分解为可验证的推理单元，结合扎实的密集过程功劳与结果正确性，使RL能够从可验证的电信证据中学习可推广的问题解决行为。我们发布TelecomGPT-R1模型和可重复的训练方案，以支持社区进一步发展。对GSMA Open电信排行榜七项基准的评估显示，开源的TelecomGPT-R1-27B平均得分为89.64%，优于包括GPT-5、Claude和Gemini在内的领先专有模型。

HOTICE: Whole-Body Humanoid Object Transportation in Cluttered Environments

HOTICE：在杂乱环境中实现全身类人生物物品运输

• Authors: Toan Nguyen, Weiduo Yuan, Siheng Zhao, Yue Wang, Daniel Seita
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25363
• Pdf link: https://arxiv.org/pdf/2609.25363
• Abstract Object transportation is a fundamental capability for humanoid robots operating in real-world, human-centric environments, yet existing methods struggle when clutter constrains free space around both the robot and its carried payload. We present HOTICE, a whole-body humanoid learning framework for transporting objects through such cluttered environments. First, we introduce Humanoid-Object Decoupled Potential Fields, which jointly encode collision-avoidance guidance for the robot and the carried object, enabling coordinated, obstacle-aware motion for both. Second, to address the large action space inherent to whole-body loco-manipulation, we design a dual-agent reinforcement learning architecture that decouples upper- and lower-body control while preserving whole-body coordination via shared state observations and rewards. To train a policy that generalizes across diverse cluttered scenes, we further employ a specialist-to-generalist distillation strategy, in which privileged teacher policies are distilled into a single deployable student policy. We evaluate HOTICE in MuJoCo simulation and on a real Unitree G1 humanoid, demonstrating effective and robust object transportation across cluttered scenarios for objects of varying shapes. Our results show that HOTICE reliably coordinates whole-body motion and object-aware collision avoidance, generalizing effectively to previously unseen cluttered environments while achieving strong performance in sim2real deployment.
• 中文摘要 物体运输是现实世界以人为中心环境中人形机器人的基本能力，但现有方法在杂波限制机器人及其携带有效载荷周围的自由空间时难以实现。我们介绍HOTICE，一种用于在此类杂乱环境中运输物体的全身人形学习框架。首先，我们引入了人形-物体解耦势场，该系统共同编码机器人和携带物体的碰撞避免引导，实现双方协调且感知障碍的运动。其次，为了应对全身机动操作固有的巨大动作空间，我们设计了一种双代理强化学习架构，该架构在通过共享状态观察和奖励保持全身协调的同时，实现上下半身控制的解耦。为了训练能在多样杂乱场景中泛化的策略，我们进一步采用了从专家到通才的提纯策略，将特权教师策略提炼成单一可部署的学生策略。我们在MuJoCo模拟和真实的Unitree G1类人生物上评估了HOTICE，展示了在杂乱场景中对不同形状物体的有效且稳健的物体传输。我们的结果表明，HOTICE可靠地协调了全身运动和物体感知碰撞避免，能够有效泛化到此前未见的杂乱环境中，同时在sim2real部署中实现了强劲的性能。

Deep Reinforcement Learning on Item-Compatibility Graphs for One-Dimensional Bin Packing

基于一维箱装箱的物品兼容性图的深度强化学习

• Authors: M. Aslı Aydın
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.25397
• Pdf link: https://arxiv.org/pdf/2609.25397
• Abstract The one-dimensional bin packing problem (1D-BPP) is a classical NP-hard combinatorial optimization problem with applications ranging from logistics and manufacturing to cloud resource management. Although deep reinforcement learning (DRL) has become a competitive paradigm for data-driven optimization, most learned packing methods target 2D and 3D variants, and intelligent learned solvers for 1D-BPP remain scarce. In this paper, we present a novel end-to-end, size-agnostic graph reinforcement learning framework for 1D-BPP. We formulate the packing process as a Markov decision process on an item-compatibility graph, serving as a structural knowledge representation in which every action merges two partial bins that fit together. A graph neural network actor-critic policy extracts relational features from this representation and is trained through reinforcement learning and decoded by stochastic beam search, enabling a single trained model to generalize zero-shot to instances of any size. We conduct a systematic empirical study across graph encoders, DRL algorithms, reward functions, training distributions, and hyperparameters. Evaluated zero-shot on the full BPPLIB benchmark against a constructive heuristic, a grouping genetic algorithm, and recent learned methods, our data-driven policy lowers the mean optimality gap of the constructive heuristic from 2.66\% to 2.31\%, with the largest gains on structured instances. Against learned baselines evaluated on the same benchmark, it attains a lower gap on most of the nine families and is far more stable across instance distributions. On the hardest benchmark family, it outperforms a state-of-the-art learned solver that relies on column generation and integer programming, while using no solver at all. A grouping genetic algorithm remains ahead overall, and we analyze where and why the residual gap arises.
• 中文摘要 一维箱装问题（1D-BPP）是一个经典的NP难组合优化问题，应用范围涵盖物流、制造到云资源管理等领域。尽管深度强化学习（DRL）已成为数据驱动优化的竞争范式，但大多数学习的打包方法针对二维和三维变体，且一维BPP的智能学习求解器仍然稀缺。本文提出了一种新颖的端到端、大小无关的图强化学习框架，适用于一维BPP。我们将打包过程表述为在项目兼容性图上的马尔可夫决策过程，作为结构性知识表示，每个动作合并两个部分组合的组合。图神经网络actor-critic策略从该表示中提取关系特征，通过强化学习训练，并通过随机波束搜索解码，使单一训练模型能够将零样样本推广到任意规模的实例。我们对图编码器、DRL算法、奖励函数、训练分布和超参数进行了系统实证研究。在完整BPPLIB基准测试中，基于构造启发式、分组遗传算法及最新学到的方法进行零样本评估，我们的数据驱动策略将建构启发式的平均最优性差距从2.66%降至2.31%，在结构化实例中提升最大。与同一基准测试的学习基线相比，它在九个基准测试族中的大多数情况下差距更小，且在实例分布间更为稳定。在最难的基准测试族中，它优于依赖列生成和整数规划的先进学习求解器，且完全不使用求解器。总体上，分组遗传算法领先，我们分析了残差差距的出现原因和地点。

Norm2Tex: Augmenting Visuo-Tactile Simulations with Texture

Norm2Tex：用纹理增强Visuo触觉模拟

• Authors: Seongjin Bien, Débora Oliveira Makowski, Roberto Calandra, Florian Walter, Wolfram Burgard
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25398
• Pdf link: https://arxiv.org/pdf/2609.25398
• Abstract Large-scale datasets are essential for training generalist robot control policies. Collecting real-world tactile data is costly and time-consuming, motivating the use of tactile simulations. However, current tactile simulators capture only overall contact geometry and miss fine details like texture. This results in a significant domain shift between simulated and real tactile data. To address this gap, we introduce Norm2Tex, a plug-in method that augments simulations of vision-based tactile sensors with high-frequency surface details from normal map textures. By modifying the target object's depth map before a tactile simulator's rendering pipeline, Norm2Tex seamlessly integrates into different tactile simulators. We also evaluate sim-to-real transfer using material classification and a reinforcement learning task. Our results show that Norm2Tex preserves material-dependent tactile information across domains, improving texture recognition and producing material-dependent control behavior in the real world.
• 中文摘要 大规模数据集对于训练通用机器人控制策略至关重要。收集真实世界的触觉数据成本高且耗时，这促使人们使用触觉仿真。然而，当前的触觉模拟器仅捕捉整体接触几何，遗漏了纹理等细节。这导致模拟触觉数据与真实触觉数据之间存在显著的领域差异。为弥补这一空白，我们引入了Norm2Tex，一种插件方法，通过法线贴图纹理增强基于视觉的触觉传感器模拟，获取高频表面细节。通过在触觉模拟器的渲染流水线前修改目标物体的深度图，Norm2Tex无缝集成到不同的触觉模拟器中。我们还通过材质分类和强化学习任务评估模拟到现实的传输。我们的结果表明，Norm2Tex 能够在不同领域保留与材质相关的触觉信息，提升纹理识别能力，并在现实世界中产生依赖材质的控制行为。

WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learning

WeightBridge：高效的强化学习权重转移库

• Authors: Xuanlin Jiang, Samuel Hsia, Michael Kuchnik, Zachary DeVito, Minlan Yu, Carole-Jean Wu
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.25442
• Pdf link: https://arxiv.org/pdf/2609.25442
• Abstract Weight transfer - the propagation of updated parameters from trainers to rollout generators - is becoming an important performance bottleneck in reinforcement learning (RL) systems for LLMs. The central challenge is supporting the diverse trainer and rollout layouts and synchronization requirements of modern RL workloads without sacrificing efficiency. Existing solutions are efficient under some configurations but perform poorly or lack support under others. We present WeightBridge, a flexible, efficient weight-transfer library designed to deliver high performance across diverse RL configurations. WeightBridge first automatically extracts the correspondence between trainer and rollout weight layouts, then plans and executes redundancy-free and load-balanced weight transfer. It exposes a small, general API while coordinating workers across diverse synchronization modes. Across configurations spanning different models, parallelization layouts, and synchronization modes, WeightBridge reduces average GPU stall time by up to 42$\times$ over the state-of-the-art open-source RL framework and achieves high performance in all settings. A coding agent was able to integrate WeightBridge into two different RL frameworks without manual guidance, demonstrating the generality and ease of use of its APIs.
• 中文摘要 权重转移——即从训练器传播到展开生成器的更新参数——正成为大型语言模型强化学习（RL）系统中重要的性能瓶颈。核心挑战是如何在不牺牲效率的前提下，支持多样化的训练器和展开布局及同步需求。现有解决方案在某些配置下效率高，但在其他配置下表现较差或缺乏支持。我们介绍WeightBridge，一个灵活高效的权重转移库，旨在在多种强化学习配置中实现高性能。WeightBridge首先自动提取训练器与展开权重布局之间的对应关系，然后规划并执行无冗余且负载均衡的权重转移。它在协调不同同步模式下的工人时，暴露了一个小型通用API。跨跨不同模型配置、并行化布局和同步模式，WeightBridge 在最先进的开源 RL 框架上，平均 GPU 停顿时间减少了高达 42 美元/时间，并在所有环境中实现高性能。一名编码代理能够将 WeightBridge 集成到两个不同的 RL 框架中，无需手动操作，展示了其 API 的通用性和易用性。

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

强化学习推理大型语言模型推广效率：分类法与未来方向

• Authors: Niloofar Gholipour, Marcos Assuncao, Gursimran Singh, Timothy Yu, Rajkumar Buyya, Julien Gascon-Samson, Zhenan Fan, Yong Zhang, Xiaojie Xu, Yaqiang Yao, Xiaolong Bai
• Subjects: Subjects: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.25463
• Pdf link: https://arxiv.org/pdf/2609.25463
• Abstract Reasoning-oriented reinforcement learning enables large language models to solve mathematical, coding, and other multi-step tasks, but shifts a substantial portion of the training cost to rollout, where trajectories are generated for policy updates. Efficient rollout mechanisms are therefore essential to reduce this cost while maintaining the freshness, consistency, and statistical validity of training data. This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives. Based on this taxonomy, we analyze how different technique families address distinct sources of rollout inefficiency, examine opportunities and potential conflicts for combining them, identify gaps in the evaluation and reporting of efficiency gains, and discuss open challenges and future research directions.
• 中文摘要 以推理为导向的强化学习使大型语言模型能够解决数学、编码及其他多步任务，但将相当一部分训练成本转移到了推广阶段，后者生成策略更新的轨迹。因此，高效的推广机制对于降低成本、保持训练数据的新鲜性、一致性和统计效度至关重要。本调查系统地对推理导向强化学习中推广效率的最新研究进行了分类，从机制和瓶颈视角对现有方法进行了分类。基于该分类法，我们分析了不同技术家族如何应对不同推广效率来源，分析结合它们的机会和潜在冲突，识别效率提升评估与报告中的空白，并讨论未解决的挑战和未来研究方向。

Learning Defensive Policies against Diverse Inference Attacks for Smart Meter Privacy

学习针对多样化推理攻击的智能电表隐私防御策略

• Authors: Ruichang Zhang, Mustafa A. Mustafa
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.25484
• Pdf link: https://arxiv.org/pdf/2609.25484
• Abstract Smart meter (SM) data provides fine-grained visibility into household energy consumption, but also exposes users to privacy risks. Inference attacks, known as non-intrusive load monitoring (NILM), can perform appliance-level inference from aggregate signals and recover sensitive behavioral patterns. In practice, attacker models are unknown and heterogeneous, making robust defense challenging. We formulate SM privacy protection as a black-box inference defense problem, aiming to reduce the recoverability of appliance-level information while generalizing across diverse and unseen attackers. We propose a proxy-guided hierarchical reinforcement learning framework that learns battery-based load-shaping policies to inject realistic but misleading appliance-level signatures into the aggregate signal, thereby disrupting the structured patterns exploited by NILM. A self-supervised aggregate-structure privacy probe provides a reconstruction-error-based surrogate reward for disrupting recoverable load structure, while a signature library makes the perturbations appliance-relevant and physically realizable through battery control. We provide theoretical rationale showing that proxy-guided optimization improves inference robustness under attacker diversity. Experiments on real-world datasets UK-DALE and REDD demonstrate strong cross-model and cross-appliance generalization. Across six unseen NILM attackers, covering four appliances on UK-DALE and five on REDD, our proposed defense increases average appliance-level RMSE by 107% and 166%, respectively, while reducing F1 score by 79% and 80%.
• 中文摘要 智能电表（SM）数据提供了家庭能源消耗的细致可视化，但也使用户面临隐私风险。推理攻击称为非侵入性负载监控（NILM），可从聚合信号中进行设备级推断，恢复敏感的行为模式。但实际上，攻击者模型未知且异构，使得稳健的防御具有挑战性。我们将SM隐私保护制定为黑箱推理防御问题，旨在降低设备级信息的恢复性，同时推广到多样且未被察觉的攻击者。我们提出了一个代理引导的分层强化学习框架，学习基于电池的负载整形策略，向聚合信号注入真实但误导性的设备级签名，从而破坏NILM利用的结构化模式。自监督的聚合结构隐私探针为破坏可恢复负载结构提供基于重建误差的替代奖励，而签名库则使扰动与设备相关并通过电池控制实现物理实现。我们提供了理论依据，表明代理引导优化在攻击者多样性下提升了推理鲁棒性。在现实世界数据集UK-DALE和REDD上的实验显示了强大的跨模型和跨设备泛化能力。在六个未被发现的NILM攻击者中，覆盖UK-DALE上的四台设备和五台REDD，我们提出的防御分别使平均设备级RMSE提升107%和166%，同时降低F1分数79%和80%。

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

套娃归因：学习将语言模型输出归因于表示和权重

• Authors: Aryaman Arora, Kirill Acharya, Nathan Hu, Yanzhe Zhang, Noah Goodman, Dan Jurafsky, Christopher Potts
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.25518
• Pdf link: https://arxiv.org/pdf/2609.25518
• Abstract Attributing language model outputs to their internal computations is an open problem in interpretability. Existing methods, which use causal interventions, gradients, or learnable masks, either are infeasibly expensive or struggle to identify actual causally-important internal computations. We propose framing attribution as the problem of identifying nested subsets of internal components which minimise a downstream loss. To learn this task, we introduce Matryoshka Attribution (MAttr), a mask learning method that parametrises the mask with a simple differentiable sigmoid top-$k$ operator. We supervise training over all sparsities simultaneously by randomising $k$ over training, resulting in a learned ordering of components by attribution score. MAttr achieves number 1 on the official leaderboard of the Mechanistic Interpretability Benchmark (Mueller et al., 2025); our method identifies sparse and task-transferrable circuits across varying circuit bases. As a practical application, we show that MAttr can be trained with reinforcement learning to identify weight changes responsible for downstream behaviours in LLM finetuning. We train MAttr on refusal judge scores and find that restoring $1\%$ of Llama 3.1 8B Instruct's weights to their base model state is sufficient to remove refusals while maintaining capabilities. We view MAttr as a successful formulation of interpretability into a learnable objective that we can tackle with gradient descent, and encourage future work along these lines.
• 中文摘要 将语言模型输出归因于其内部计算，是可解释性中的一个开放问题。现有方法，使用因果干预、梯度或可学习掩码，要么成本过高，要么难以识别实际因果重要的内部计算。我们提出将归因框架为识别嵌套的内部组件子集的问题，以最小化下游损失。为学习这一任务，我们引入套娃归因（MAttr），这是一种用简单可微S形顶$k$算符参数化掩模的掩体学习方法。我们通过对训练随机化$k$，同时监督所有稀疏度的训练，从而学习出按归因分数排序的组件。MAttr 在机制解释性基准（Mechanistic Interpretationability Benchmark）官方排行榜上排名第一（Mueller 等，2025）;我们的方法识别了跨不同电路基底的稀疏和任务转移电路。作为实际应用，我们展示了MAttr可以通过强化学习训练，识别导致LLM微调中下游行为的权重变化。我们用拒绝法官评分训练MAttr，发现恢复Llama 3.1 8B Instruct权重1\%$的基础模型状态足以消除拒绝，同时保持能力。我们认为MAttr是将可解释性表述为可学习目标的成功方法，并鼓励未来在这方面的研究。

Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA

基于零基LoRA的后强化学习LLM的保持推理的微调

• Authors: Wenzhi Fang, Nicholas Tzou, Lazar Valkov, Srinivas Chappidi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.25618
• Pdf link: https://arxiv.org/pdf/2609.25618
• Abstract Reinforcement learning (RL)-based post-training has become an effective approach for eliciting reasoning capabilities in large language models (LLMs). However, adapting post-RL models to new knowledge domains or behaviors through subsequent supervised fine-tuning (SFT) can severely overwrite these capabilities. Existing approaches mitigate such forgetting through experience replay, specialized initialization, or constrained optimization using gradient projection, but either provide limited preservation or incur substantial training overhead. Our analysis shows that reasoning activations concentrate in low-dimensional subspaces, leaving substantial null-space capacity for adaptation, and that the corresponding approximate null spaces can be reliably estimated from a modest number of examples. Motivated by these observations, we propose Null-Basis Low-Rank Adaptation (NB-LoRA), a parameter-efficient method for adapting post-RL LLMs while preserving their acquired reasoning ability. We formulate reasoning retention as a layer-wise hidden-state preservation constraint and construct a fixed approximate null basis from reasoning activations. LoRA updates are then reparameterized through this basis, enforcing the preservation constraint throughout fine-tuning. Extensive experiments across multiple RL-trained LLMs and diverse downstream tasks show that NB-LoRA matches standard LoRA in adaptation performance, maintains reasoning accuracy near pre-fine-tuning levels, and generalizes this preservation to held-out reasoning benchmarks.
• 中文摘要 基于强化学习（RL）的后训练已成为激发大型语言模型（LLM）推理能力的有效方法。然而，通过后续监督微调（SFT）将后强化学习模型适应新知识领域或行为，可能会严重覆盖这些能力。现有方法通过经验回放、专门初始化或使用梯度投影的受限优化来缓解这种遗忘，但要么保留有限，要么带来较大的训练开销。我们的分析显示，推理激活集中在低维子空间，留下了相当的零空间适应能力，且相应的近似零空间可以从少量实例中可靠估计。基于这些观察，我们提出了零基低秩适应（NB-LoRA），这是一种参数高效的方法，用于适应后强化学习的LLMs，同时保持其获得的推理能力。我们将推理保留定义为层层隐藏状态保留约束，并从推理激活构建固定近似空基。LoRA更新随后通过该基底重新参数化，在整个微调过程中强制执行保留约束。在多个强化学习训练的LLM和多种下游任务中的大量实验表明，NB-LoRA在适应性能上与标准LoRA相当，推理准确度接近微调前水平，并将这种保存性推广到保留的推理基准。

PAKT: Physically-Aligned Kinesthetic Teaching for Reinforcement Learning

PAKT：强化学习的身体对齐动觉教学

• Authors: Lars Johannsmeier, Yashraj Narang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25630
• Pdf link: https://arxiv.org/pdf/2609.25630
• Abstract Real-world reinforcement learning (RL) systems still struggle with the demands of contact-rich industrial manipulation, including micrometer-level precision, success rates above 99%, and human-level cycle times. Although off-policy algorithms can improve performance by leveraging demonstrations and interventions, a key bottleneck is the lack of an intuitive interface for collecting such guidance while complying with constraints of the physical system and the policy. We propose PAKT, a framework for kinesthetic teaching in RL. As opposed to teleoperation approaches, PAKT relies on kinesthetic guidance, which is widely used in industry. However, a critical weakness of kinesthetic guidance is the possibility for the operator to move the robot along trajectories (e.g., velocities, accelerations, jerk) that the robot and/or policy cannot physically reproduce. Using PAKT, operators guide the robot through admittance control, which maps human-applied forces to motion. The downstream reference generator applies the same kinematic limits used during policy execution, keeping the collected trajectories within these limits. To support this teaching interface with an appropriate execution layer, PAKT adds a high-performance control stack that maps low-frequency RL actions to high-frequency torque commands. It consists of a reference generator and subsequent impedance controller, where the reference generator preserves the tracking performance of the impedance controller while improving contact handling and producing smoother policy actions. Across the reported runs on four insertion and industrial assembly benchmarks, including a data center compute tray, the end-to-end system reduces cycle time by 23%-48% and cumulative intervention count by 62%-86% relative to the HIL-SERL baseline. Project website: this https URL}{this https URL
• 中文摘要 现实世界的强化学习（RL）系统仍在应对接触丰富工业操作的需求，包括微米级精度、99%以上的成功率以及人类水平的周期时间。尽管非策略算法可以通过演示和干预提升性能，但关键瓶颈是缺乏直观的接口，以便在满足物理系统和策略约束的同时收集此类指导。我们提出了PAKT，一种用于强化学习动觉教学的框架。与远程操作方法不同，PACT依赖于动觉引导，该引导在工业中被广泛应用。然而，动觉指导的一个关键弱点是操作员可能沿机器人和/或策略无法物理复现的轨迹（如速度、加速度、猛拉）移动。操作员通过PAKT引导机器人通过导纳控制，将人为施加的力映射到运动。下游参考生成器应用策略执行时相同的运动学限制，保持收集轨迹在这些限制内。为支持该教学接口并配备合适的执行层，PACT增加了高性能控制栈，将低频强化操作映射到高频扭矩指令。该协议由参考发生器及其后续阻抗控制器组成，参考生成器保持阻抗控制器的跟踪性能，同时改善接触处理并实现更平稳的策略操作。在四个插入和工业装配基准测试（包括数据中心计算托盘）报告运行中，端到端系统相较HIL-SERL基线将周期时间缩短23%-48%，累计干预计数减少62%-86%。项目网站：此 https URL}{此 https URL

DynaForge: Planning-Guided Residual Learning for Dynamic Manipulation Demonstration Generation

DynaForge：动态操作演示生成的规划引导残差学习

• Authors: Yiyang Jin, Yu Zheng, Xiao He, Hesheng Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25631
• Pdf link: https://arxiv.org/pdf/2609.25631
• Abstract Dynamic object manipulation is essential for robots operating in real-world environments, yet methods for generating high-quality demonstrations remain limited. Methods designed for static tasks do not readily transfer to dynamic settings. Among dynamic demonstration generators, planning-based methods can fail near contact, while DOMINO-style replay simplifies dynamic interactions and may limit the experience available for policy learning. We present DynaForge, a planning-guided framework that learns residual corrections for dynamic manipulation demonstration generation. DynaForge combines low-frequency global planning with high-frequency object-centric inverse kinematics across task phases, and applies a residual policy to correct actions during dynamic interaction. An implicit curriculum groups rollouts under matched conditions and selects mixed-success groups, focusing residual reinforcement learning on the evolving competence frontier. On Can and Bottle, it uses 0.73x as many optimizer steps as vanilla GRPO at the same nominal environment-step budget, with higher observed final success rates. Across nine simulation tasks, DynaForge increases mean demonstration-generation success from 41.30% of the planning prior to 78.37%. With 800 demonstrations per task, DP3 policies trained on DynaForge data achieve 49.11% mean success, compared with 7.07% for DOMINO data. On three real-world dynamic tasks, DynaForge-trained policies achieve 30-60% success, compared with 0-10% for DOMINO-trained policies, showing the ability of DynaForge for sim-to-real transfer.
• 中文摘要 动态对象操作对于在现实环境中运行的机器人至关重要，但生成高质量演示的方法仍然有限。为静态任务设计的方法难以迁移到动态环境。在动态演示生成器中，基于规划的方法可能在近接触时失效，而多米诺式重放简化了动态交互，可能限制了策略学习的体验。我们介绍了DynaForge，一个规划引导框架，用于动态操作演示生成剩余修正。DynaForge结合了低频全局规划与高频以对象为中心的逆运动学，跨任务阶段，并在动态交互中应用残余策略纠正动作。隐式课程分组在匹配条件下展开推广，并选择混合成功组，将残余强化学习聚焦于不断发展的能力前沿。在罐装和瓶装中，在相同的名义环境步预算下，它使用的优化步骤是普通GRPO的0.73倍，且观察到的最终成功率更高。在九个模拟任务中，DynaForge将演示生成的平均成功率从规划前的41.30%提升至78.37%。每个任务进行800次演示，基于DynaForge数据训练的DP3策略平均成功率为49.11%，而DOMINO数据为7.07%。在三个真实世界动态任务中，DynaForge训练策略的成功率为30-60%，而多米诺训练策略为0-10%，显示了DynaForge在模拟到现实转移方面的能力。

Teaching Reinforcement Learning and Humanoid Robotics to High-School Students: An Expert-Validated Curriculum Design on a Low-Cost Open Platform

向高中生教授强化学习与类人机器人：一项专家验证的低成本开放平台上的课程设计

• Authors: Yuanzhe Dong, Jie Cao, Shuman Wang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.25674
• Pdf link: https://arxiv.org/pdf/2609.25674
• Abstract Lower cost open source robots and reinforcement learning (RL) simulation tools create new opportunities for precollege students to engage with contemporary robotics. However, translating a complete research workflow, spanning mechanical assembly, electrical setup, simulation, policy learning, system identification, and physical deployment, into a coherent course for novice learners remains challenging. We present an integrated robotics course framework that organizes these activities around a shared robotic artifact. The framework combines parallel disciplinary tracks, sequencing based on technical dependencies, progressive integration of simulation and hardware, layered performance checkpoints, and structures for balancing collaborative work with individual accountability. We illustrate the framework through a high school curriculum organized around a robot project in which pairs of students assemble an open source humanoid robot, train a walking policy in simulation, and deploy it on the physical platform. The framework was developed through an iterative design process that included formative review by five experts in robotics research, engineering, secondary STEM education, and curriculum design. Expert feedback highlighted three central design tensions: authenticity versus cognitive load, system integration versus timely visible progress, and team construction versus individual accountability. These tensions informed the final framework presented in this paper. This work offers a structured approach for adapting robotics research workflows into interdisciplinary precollege courses; future classroom studies are needed to examine implementation and student learning.
• 中文摘要 低成本开源机器人和强化学习（RL）模拟工具为学前学生接触当代机器人创造了新机会。然而，将涵盖机械组装、电气安装、仿真、政策学习、系统识别和物理部署的完整研究流程转化为面向初学者的连贯课程仍然具有挑战性。我们提出了一个集成的机器人课程框架，围绕共享的机器人产物组织这些活动。该框架结合了平行学科轨道、基于技术依赖的排序、模拟与硬件的渐进集成、分层性能检查点以及平衡协作工作与个人责任的结构。我们通过围绕机器人项目组织的高中课程来说明该框架，该项目中学生两人组装开源类人机器人，在模拟中训练行走策略，并将其部署到物理平台上。该框架通过迭代设计过程开发，包括五位机器人研究、工程、中学STEM教育和课程设计专家的形成性评审。专家反馈强调了三大核心设计张力：真实性与认知负担、系统集成与及时可见进展、团队建设与个人责任。这些张力塑造了本文最终框架。本研究提供了一种结构化方法，将机器人研究工作流程调整为跨学科的大学预科课程;未来课堂研究需进行，以检视实施情况和学生学习情况。

Fully Byzantine-Resilient Multi-Agent Reinforcement Learning

完全拜占庭弹性多智能体强化学习

• Authors: Haejoon Lee, Dimitra Panagou
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.25701
• Pdf link: https://arxiv.org/pdf/2609.25701
• Abstract We study distributed Byzantine-resilient actor-critic multi-agent reinforcement learning (AC-MARL), where agents collectively learn policies through local interactions. Existing methods guarantee convergence of the agents' parameters only to a neighborhood of the attack-free limit points, resulting in degraded performance. We propose Fully Resilient AC-MARL (FRAC-MARL), a decentralized method in which each agent leverages redundancy in two-hop messages to identify reliable messages. Under linear parameterizations of the value and team-reward functions and Byzantine edge attacks, where adversarial behavior is confined to the communication layer, we prove that agents' parameters converge almost surely to the same limit points as in the attack-free case over time-varying communication graphs. We introduce a novel topological condition for the convergence of our method, present a systematic method to construct such networks, and prove that this condition can be verified in polynomial time. Finally, we demonstrate our method on cooperative multi-robot formation control tasks.
• 中文摘要 我们研究分布式拜占庭韧性actor-critic多智能体强化学习（AC-MARL），其中代理通过局部交互集体学习策略。现有方法仅保证智能体参数收敛到无攻击极限点的邻域，导致性能下降。我们提出了完全弹性AC-MARL（FRAC-MARL），这是一种去中心化方法，每个智能体利用双跳消息的冗余来识别可靠消息。在价值函数和团队奖励函数的线性参数化以及对抗行为局限于通信层的拜占庭边缘攻击下，我们证明了智能体参数几乎必然收敛到与无攻击情况下相同的极限点，且通信图时间变化。我们引入了方法收敛的新拓扑条件，提出了构建此类网络的系统方法，并证明该条件可在多项式时间内验证。最后，我们展示了该方法在多机器人协同编组控制任务中的应用。

PLAT: Sparse Timed Keyframe Motion Tracking for Humanoid Control via Privileged Latent Transition Learning

PLAT：通过特权潜在过渡学习实现稀疏定时关键帧运动追踪，用于人形控制

• Authors: Zepeng Wang, Jiangxing Wang, Chao Ma, Xiaochuan Shi, Zongqing Lu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25754
• Pdf link: https://arxiv.org/pdf/2609.25754
• Abstract Humanoid motion tracking policies rely on dense frame-by-frame references, limiting their use as high-level motion controllers for planning and interactive motion generation. We study \emph{Sparse Timed Keyframe Motion Tracking}, where a policy receives only sparse future keyframes and their desired arrival times, and must execute stable whole-body motions that reach successive goals. We propose \textbf{PLAT}, a three-stage sparse timed keyframe motion tracking policy learning framework with \textbf{P}rivileged \textbf{LA}tent \textbf{T}ransition learning. PLAT bridges dense motion tracking and sparse goal-conditioned control by exploiting dense goal sequences as privileged supervision during training while requiring only sparse timed keyframe commands at deployment. A pretrained dense tracking expert first provides robust motion priors. A privileged latent prior is then learned through DAgger-style imitation, followed by latent residual reinforcement learning that refines latent transitions instead of directly optimizing actions. Extensive simulation experiments demonstrate that PLAT maintains accurate and stable sparse timed keyframe tracking across varying planning horizons, with particularly strong performance under long-horizon commands. Successful deployment on a Unitree G1 humanoid robot further demonstrates the effectiveness and practicality of PLAT for sparse humanoid motion control.
• 中文摘要 类人生物运动追踪策略依赖密集的逐帧引用，限制了其作为高级运动控制器用于规划和交互式运动生成的应用。我们研究了\emph{稀疏定时关键帧运动追踪}，其中策略只接收稀疏的未来关键帧及其期望到达时间，必须执行稳定的全身运动以实现连续目标。我们提出了\textbf{PLAT}，一个三阶段稀疏时序关键帧动作跟踪策略学习框架，配备与\textbf{LA}帐篷\textbf{T}学习相结合。PLAT通过利用密集目标序列作为训练中的特权监督，连接密集运动跟踪与稀疏目标条件控制，同时部署时仅需稀疏的定时关键帧命令。预先训练的密集跟踪专家提供稳健的运动先验。随后通过DAgger式模仿学习特权潜在先验，随后进行潜在残差强化学习，优化潜在过渡而非直接优化动作。大量仿真实验表明，PLAT能够在不同规划视野内保持准确稳定的稀疏定时关键帧跟踪，在长视野指令下表现尤为出色。在Unitree G1类人机器人上的成功部署进一步展示了PLAT在稀疏类人型运动控制中的有效性和实用性。

Model-Free Current Control of Permanent Magnet Synchronous Motors via ESO-Based Disturbance Feedforward and Data-Driven H-infinity Residual Feedback

通过基于ESO的扰动前馈和数据驱动的H-Infinity残差反馈，实现永磁同步电机的模型无电流控制

• Authors: YongBo Li, Shuang Liang, HongWei Ma
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.25759
• Pdf link: https://arxiv.org/pdf/2609.25759
• Abstract This paper proposes a model-free current control method for permanent magnet synchronous motors (PMSMs) based on disturbance feedforward and residual feedback. An ultra-local current model incorporates motor dynamics, parameter uncertainties, cross-coupling effects, and other nonideal factors into generalized lumped disturbances. An extended state observer (ESO) estimates these lumped disturbances and compensates for them through feedforward action, transforming the original PMSM current-control problem into regulation of a simplified post-compensation residual system. A state-feedback H-infinity controller for the residual system is then learned directly from operating data using off-policy integral reinforcement learning. Owing to the simplified residual dynamics, the value function and control policies are parameterized in quadratic and linear forms, reducing the learning problem to low-dimensional parameter estimation without neural-network approximation. The proposed method requires neither prior knowledge nor online identification of PMSM electrical parameters: input-gain mismatch is incorporated into the ESO-estimated lumped dynamics, while the residual-feedback policy is obtained from operating data. Comparative simulations against deadbeat predictive current control, model-based H-infinity control, and model-free predictive current control show fast current tracking, low current distortion, and strong robustness to large parameter variations. With the learned H-infinity policy fixed and without retraining or retuning, nearly unchanged control performance is maintained when stator resistance, stator inductance, and permanent-magnet flux linkage are simultaneously varied to 20% and 200% of their nominal values.
• 中文摘要 本文提出了一种基于扰动前馈和残差反馈的永磁同步电机（PMSM）模型自由电流控制方法。超本地电流模型将电机动力学、参数不确定性、交叉耦合效应及其他非理想因素纳入广义集总扰动。扩展状态观察器（ESO）估计这些集总扰动，并通过前馈作用补偿，将原有的PMSM电流控制问题转化为简化的补偿后残差系统的调控。然后通过非策略积分强化学习，直接从操作数据学习残差系统的状态反馈H-infinity控制器。由于简化的残差动力学，价值函数和控制策略以二次和线性形式参数化，将学习问题简化为无需神经网络近似的低维参数估计。所提方法无需先验知识或在线识别PMSM电气参数：输入增益不匹配被纳入ESO估计的集中动力学中，剩余反馈策略则从运行数据中获得。对无功预测电流控制、基于模型的H-infinity控制和无模型预测电流控制的比较模拟显示，电流跟踪快速、电流失真低且对大参数变化具有强鲁棒性。在已学到的H-infinity策略固定且未重新训练或重新调谐的情况下，当定子电阻、定子电感和永久磁通耦合同时变化至标称值的20%和200%时，控制性能几乎保持不变。

Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Models

视频跳链：多跳问题与视频推理模型的信心门槛探索

• Authors: Trung Nguyen Quang, Yuhao Dong, Shuo Sun, Shuai Liu, Shulin Tian, Kim-Hui Yap, Ziwei Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.25773
• Pdf link: https://arxiv.org/pdf/2609.25773
• Abstract HopChain has shown on still images that multi-hop data synthesis improves vision-language reasoning, because long chain-of-thought reasoning exposes errors that compound across steps, while most data used for reinforcement learning with verifiable rewards (RLVR) rarely demands a chain of visual evidence, so these weaknesses are likely to stay unexposed. We observe the same problem in video, where this framework has not yet been explored. We therefore build Video-HopChain, a dataset of 22,550 multi-hop video questions over 13,378 videos, together with a held-out benchmark of 1,000 questions. Each question chains three to six yes/no questions about moments in one video, and each yields one of two integers depending on its answer. The final answer is the sum of these integers, so an exact match on that sum gives the verifiable reward that RLVR needs. We first train Qwen3-VL-8B with GRPO on a standard video dataset, and a second stage on Video-HopChain then raises the mean over eight video understanding and reasoning benchmarks from 55.4 to 57.9 and improves every one of them. Training on such a dataset, however, exposes a known limitation of GRPO: its learning signal comes from the reward variance within a group, so hard questions whose rollouts are all incorrect and easy questions whose rollouts are all correct both leave the group with no gradient. To recover these groups at the same compute budget, we introduce Confidence-Gated Exploration (CGE). With 8 rollouts per question, CGE samples the first 4 as usual. If these 4 are either all correct or all incorrect, it samples the last 4 with the policy's most confident token masked inside the reasoning span, and removes the masked positions from the loss while all 8 rollouts enter the advantage. With CGE, the mean rises further to 59.3. We release the dataset, the checkpoint, and the data generation and training code.
• 中文摘要 HopChain 在静态图像上显示，多跳数据综合能提升视觉语言推理能力，因为长的思维链推理揭示了跨步叠加的错误，而大多数用于可验证奖励强化学习（RLVR）的数据很少需要视觉证据链，因此这些弱点很可能未被发现。我们在视频中也观察到同样的问题，而该框架尚未被探索。因此，我们构建了 Video-HopChain，这是一个包含 22,550 个多跳视频问题的数据集，涵盖 13,378 个视频，以及一个已保留的 1,000 个问题基准。每个问题链起三到六个关于视频中时刻的是非题，每个问题根据答案得到两个整数之一。最终答案是这些整数的总和，因此该总和的精确匹配即可获得 RLVR 所需的可验证奖励。我们首先在标准视频数据集上用GRPO训练Qwen3-VL-8B，第二阶段在视频跳链上将八个视频理解和推理基准的平均值从55.4提升到57.9，并不断改进。然而，在此类数据集上训练暴露了GRPO的一个已知局限：其学习信号来自组内的奖励方差，因此所有推展错误的难题和所有推展都正确的简单问题都使组内无梯度。为了在相同的计算预算下恢复这些组，我们引入了置信门控探索（CGE）。每题8个展开，CGE像往常一样采样前4个。如果这4个全部正确或全部错误，则采样策略中最有信心的标记被掩盖在推理区间内，并将掩蔽位置从损失中移除，同时8个推展全部进入优势。使用CGE时，平均值进一步提升至59.3。我们发布数据集、检查点以及数据生成和训练代码。

Sometimes You Gotta Run Before You Can Walk: Run-then-Walk Scheduling Strategy for VLM Autonomous Driving

有时候你得先跑，才能走路：VLM自动驾驶的先跑后走的排班策略

• Authors: Yuqi Ye, Shangkun Sun, Junhong Lin, Jiayi Zhao, Changhao Peng, Wei Zheng, Guoqing Liu, Tiesong Zhao, Wei Gao
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.25831
• Pdf link: https://arxiv.org/pdf/2609.25831
• Abstract Recent VLM-based autonomous driving planners adopt GRPO-style reinforcement learning to optimize driving performance. However, existing GRPO recipes either optimize driving efficiency, risking progress-seeking but unsafe behavior, or enforce early safety constraints, leading to overly conservative behavior; both require lengthy training. To solve these problems, we first reveal two distinct RL regimes: a progress regime (Run-GRPO) that aggressively explores high progress, and a safety regime (Walk-GRPO) that restores safety under stable progress. Based on this finding, we propose $\textit{Run-then-Walk}$, a simple yet effective two-stage reward scheduling strategy for GRPO, achieving both better performance and faster convergence. Unlike one-stage RL, which may focus on progress, safety, or a mixture of both within a single training phase, this schedule explicitly separates progress discovery from safety repair. In the $\textit{Run}$ phase, we focus on progress, allowing the policy to escape the conservative bias and discover high-progress modes. In the subsequent $\textit{Walk}$ phase, we introduce endpoint and safety strategy to repair unsafe behaviors from the Run phase. This reversed schedule overcomes the conservatism of Walk-first methods and the unsafe progress-seeking of joint optimization. We validate it with various VLM-based planners on multiple benchmarks: NAVSIMv1, NAVSIMv2, Navhard, and nuScenes. Extensive experiments demonstrate improved driving performance while requiring 40--50\% fewer RL training epochs than the baselines.
• 中文摘要 近期基于VLM的自动驾驶规划者采用GRPO式强化学习以优化驾驶性能。然而，现有GRPO配方要么优化驾驶效率，冒着追求进步但不安全行为的风险，要么强制执行早期安全约束，导致行为过于保守;这两种都需要长时间的训练。为解决这些问题，我们首先揭示了两种不同的强化学习模式：一种是进度模式（Run-GRPO），积极探索高进步;另一种是安全模式（Walk-GRPO），在稳定进展下恢复安全。基于这一发现，我们提出了$\textit{Run-then-Walk}$，一种简单但有效的两阶段奖励调度策略，用于GRPO，实现更优的性能和更快的收敛。与可能在单一训练阶段内关注进展、安全或两者混合的单阶段强化学习不同，该计划明确区分了进展发现与安全修复。在$\textit{Run}$阶段，我们关注进展，使策略摆脱保守偏见，发现高进展模式。在随后的$\textit{Walk}$阶段，我们引入端点和安全策略，修复运行阶段的不安全行为。这种反向计划克服了先行方法的保守性和联合优化中不安全的进展寻求。我们用多个基于VLM的规划器验证了该方法，基于多个基准测试：NAVSIMv1、NAVSIMv2、Navhard和nuScenes。大量实验证明，驾驶性能提升，且所需的强化学习训练时间比基线少40%-50%。

Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models

知情掩蔽：扩散大型语言模型中强化学习中的结构感知扰动

• Authors: Xiaoyi Yu, Enver Sangineto, Pei Fu, Fiorenzo Parascandolo, Wenhui Tan, Ruikang Zhang, Rita Cucchiara, Ruihua Song, Jian Luan
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.25927
• Pdf link: https://arxiv.org/pdf/2609.25927
• Abstract Diffusion Large Language Models (dLLMs) have emerged as an efficient alternative to autoregressive models, yet aligning them via Reinforcement Learning (RL) requires likelihood surrogates estimated from masked reconstruction subproblems under a small Monte Carlo budget per rollout. Existing methods construct these subproblems by uniform random masking, leaving open the question of which subproblems to prioritize. We identify a systematic upstream/downstream structure in dLLM rollouts. Some tokens, when revealed, trigger large confidence changes in nearby undecoded positions; we call them upstream. Others induce only small local changes and are therefore downstream. We find masking downstream tokens yields substantially better-posed subproblems than masking upstream tokens, a phenomenon we term subproblem difficulty asymmetry. Based on the observation, we propose Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens. IM is plug-and-play: when plugged into three state-of-the-art dLLM RL methods on LLaDA-8B-Instruct, it delivers up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks with improved training stability.
• 中文摘要 扩散大型语言模型（dLLMs）已成为自回归模型的高效替代方案，但通过强化学习（RL）对齐它们需要每次部署在小蒙特卡洛预算下，从掩蔽重建子问题中估算似然替代。现有方法通过均匀随机掩蔽构建这些子问题，留下了优先考虑哪些子问题的问题。我们在dLLM的推广中识别出系统性的上游/下游结构。有些令牌在揭示时会触发附近未解码位置的巨大置信变化;我们称之为上游。另一些词只引发小范围局部变化，因此属于下游。我们发现掩蔽下游令牌产生的子问题比掩蔽上游令牌更为合理，我们称之为子问题难度不对称。基于观察，我们提出了知情掩蔽（IM），该方法从去噪轨迹中推导出每标记的优先级分数，且对掩蔽采样偏向下游标记。IM即插即用：当插入LLaDA-8B-Instruct上的三种最先进的dLLM RL方法时，在数学和规划基准测试中获得最高2.01%、8.68%和5.77%的相对平均提升，且训练稳定性有所提升。

MATES: Learning Multi-Agent Interactions by Transforming Observations for Frozen Single-Agent Policies

MATES：通过变换观察值以实现冻结单智能体策略学习多智能体交互

• Authors: Elie Abboud, Oren Gal
• Subjects: Subjects: Multiagent Systems (cs.MA); Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.26010
• Pdf link: https://arxiv.org/pdf/2609.26010
• Abstract Multi-agent reinforcement learning (MARL) commonly trains decentralized policies from scratch, requiring agents to acquire individual task competence and coordination simultaneously. Yet many multi-agent problems admit a compatible single-agent counterpart in which the underlying task can be learned in isolation. We introduce Multi-Agent Observation Transformation for Existing Single-Agent Policies (MATES), an input-side adaptation framework for tasks whose multi-agent observations preserve the solo-task information while exposing separately identifiable neighbor information. From multi-agent experience, MATES learns a small adapter that maps this observation into the format expected by a frozen single-agent policy, inducing actions suited to the shared environment without updating the single-agent policy itself. MATES leaves the pretrained policy's internal architecture unchanged and retains the objectives and update procedures of the underlying MARL algorithm. We evaluate MATES using both on- and off-policy algorithms on lifelong pathfinding, navigation, and cooperative discovery, spanning discrete and continuous observation and action spaces. Across all evaluated settings, MATES optimizes only 3.5-7.3% as many parameters as full-policy training while consistently outperforming MARL training from scratch. It approaches the performance of full fine-tuning, remains competitive overall with demonstration-based baselines, and retains strong task performance at team sizes not encountered during training. These results provide evidence that, under this observation structure, effective multi-agent behavior can be learned without modifying the policy that encodes individual competence.
• 中文摘要 多智能体强化学习（MARL）通常从零开始训练去中心化策略，要求智能体同时获得单个任务的能力和协调能力。然而，许多多智能体问题也存在兼容的单智能体对应物，可以在其中单独学习底层任务。我们引入了针对现有单智能体策略（MATES）的多智能体观察转换（MATES），这是一个输入端适配框架，适用于多智能体观察保持单任务信息同时暴露可识别邻居信息的任务。通过多智能体经验，MATES学习了一个小型适配器，将该观察映射为冻结单智能体策略的格式，诱导适合共享环境的动作，而无需更新单智能体策略本身。MATES保持预训练策略的内部架构不变，保留底层MARL算法的目标和更新程序。我们利用开失策略算法评估MATES在终身路径查找、导航和协作发现方面，涵盖离散和连续的观察与动作空间。在所有评估的环境中，MATES仅优化了全策略训练的3.5%-7.3%的参数，同时从零开始就持续优于MARL训练。它接近全面微调的性能，整体上通过示范基线保持竞争力，并在训练期间未遇到的团队规模下保持强劲任务表现。这些结果表明，在这种观察结构下，可以在不修改编码个体能力的策略的情况下学习有效的多智能体行为。

Differentiable Policy Transport over Multi-Layer Network Feasibility Geometry

多层网络可行几何结构上的可微策略传输

• Authors: Zuyuan Zhang, Zeyu Fang, Mahdi Imani, Nathaniel D. Bastian, Tian Lan
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.26068
• Pdf link: https://arxiv.org/pdf/2609.26068
• Abstract Learning-based control is increasingly central to automating network operations. A learned policy, however, must satisfy cross-layer constraints on interference, power-rate coupling, flow conservation, service chains, capacity, latency, and reliability. Existing methods typically account for only a subset of this geometry and only indirectly, e.g., through reward penalties, Lagrange multipliers, or post-hoc repairs. This paper proposes \emph{Network Feasibility Geometry Reinforcement Learning} (NFG-RL), which models coupled constraints via transport theory and the residual inclusion $\bphi_{\mathfrak{N}}(x,a)\in\cK_{\mathfrak{N}}$, defining the executed policy as the pushforward of a proto-policy through a feasibility-transport map. NFG-RL compiles heterogeneous constraints into typed residual blocks and transports proto-actions through a differentiable variational operator, letting active constraints shape execution, exploration, and actor gradients. Our analysis shows that exact transport yields almost-sure feasible execution, while active constraints contract exploration onto the feasible tangent space. It further establishes a nonnegative first-order gain from critic-tilted transport over plain projection and recovers backpressure scheduling as the gradient of a lifted drift residual. In two public-trace-conditioned wireless-edge surrogate environments, NFG-RL improves feasible utility by \textbf{37.5--41.5\%} over the strongest non-NFG method in each environment, reduces raw-action violation by \textbf{48.5--60.8\%}, and lowers P99 delay by \textbf{57.0--75.5\%}, outperforming a range of optimization and learning baselines.
• 中文摘要 基于学习的控制在自动化网络操作中日益核心。然而，学习策略必须满足跨层约束，如干扰、功率耦合、流量守恒、服务链、容量、延迟和可靠性。现有方法通常只考虑这些几何的子集，且仅通过奖励惩罚、拉格朗日乘数或事后修复间接实现。本文提出了\emph{网络可行性几何强化学习}（NFG-RL），通过传输理论和剩余包含关系$\bphi_{\mathfrak{N}}（x，a）\in\cK_{\mathfrak{N}}}来建模耦合约束，定义执行策略为通过可行性-传输映射推送原始策略。NFG-RL将异构约束编译为类型化残差块，并通过可微变分算符传输原始动作，使主动约束塑造执行、探索和演员梯度。我们的分析表明，精确传输几乎确定可行执行，而主动约束则将探索收缩到可行切空间。它还进一步确立了批判倾斜传输相较于平投影的非负一阶增益，并将背压调度恢复为提升漂移残差的梯度。在两种公共轨迹条件的无线边缘代理环境中，NFG-RL相较于每个环境中最强的非NFG方法提高了可行效用的_37.5--41.5\%}，将原始动作违规降低了\textbf{48.5--60.8\%}，并将P99延迟降低了\textbf{57.0--75.5\%}，在一系列优化和学习基线中表现优异。

Test-time Reinforcement Learning for Anomalous Video Understanding

异常视频理解的测试时强化学习

• Authors: Huining Li, Yuxiang Duan, Jiyang Tan, Qian Li, MingCai Chen, Jian Zhang, Xingdong Sheng, Yuntao Du
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.26099
• Pdf link: https://arxiv.org/pdf/2609.26099
• Abstract Anomalous video understanding aims to identify abnormal events in videos and interpret their semantic meanings beyond simple anomaly detection. Recent video large language models (Video-LLMs) have demonstrated promising zero-shot capabilities for this task, yet their performance remains limited due to insufficient adaptation to diverse anomaly patterns and evolving environments. Test-time reinforcement learning offers a promising solution by enabling models to improve through self-generated feedback signals without requiring additional human annotations. However, applying it to anomalous video understanding remains challenging due to three issues: (1) generated pseudo-labels can be unreliable when consensus is weak; (2) binary reward designs fail to capture uncertainty in model generations, resulting in ineffective optimization signals; and (3) unanimous rollout groups receive identical rewards, causing group-relative advantages to collapse and eliminating effective policy-gradient signals. To address these challenges, we present a novel test-time reinforcement learning framework for anomalous video understanding by introducing dual-query consistency filtering, an entropy-aware consensus reward, and a virtual negative anchor mechanism. The framework retains reliable samples through consistency across semantically equivalent queries, combines answer agreement with generation uncertainty for reward estimation, and introduces a virtual negative anchor to create reward variation in unanimous rollout groups, thereby preserving effective group-relative optimization signals. Experiments on VAU-Bench show that our method outperforms the compared frozen and supervised baselines. The gains are most pronounced on the ECVA subset of VAU-Bench with thinking, where accuracy improves from 75.81% to 90.00% relative to the frozen backbone.
• 中文摘要 异常视频理解旨在识别视频中的异常事件，并解释其语义意义，超越简单的异常检测。近期视频大型语言模型（Video-LLM）展示了有前景的零射点能力，但由于对多样化异常模式和不断演变环境的适应不足，其性能仍然有限。测试时强化学习提供了有前景的解决方案，使模型能够通过自生成的反馈信号改进，而无需额外的人工注释。然而，将其应用于异常视频理解仍具挑战性，原因有三点：（1）生成的伪标签在共识薄弱时可能不可靠;（2）二元奖励设计未能捕捉模型生成中的不确定性，导致优化信号无效;（3）一致推广组获得相同的奖励，导致群体相对优势崩溃，消除有效的策略梯度信号。为应对这些挑战，我们提出了一种新的测试时强化学习框架，通过引入双查询一致性过滤、熵感知共识奖励和虚拟负锚机制，实现异常视频理解。该框架通过语义等效查询的一致性保持可靠样本，结合答案一致性与生成不确定性进行奖励估计，并引入虚拟负锚点以在一致展开组中创造奖励变异，从而保持有效的组相对优化信号。VAU-Bench上的实验显示，我们的方法优于冷冻基线和监督基线的比较。VAU-Bench的ECVA子集与思维模式的提升最为显著，相较于冻结骨干的准确率从75.81%提升至90.00%。

A Cross-Dataset based Zero-Day Intrusion Detection System by Integrating Siamese Network and Reinforcement Learning

通过整合暹罗网络和强化学习实现的跨数据集零日入侵检测系统

• Authors: Md. Meheraj Hossain, Saumik Das Turja, Sibgatullah Tasnim, Md. Fahmid-Ul-Alam Juboraj, Muhammad Iqbal Hossain
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.26115
• Pdf link: https://arxiv.org/pdf/2609.26115
• Abstract Zero-day threats are nascent for the Internet of Things (IoT) network security, which demands cognitive detec-tion mechanisms that can identify emerging malicious behavior. Conventional intrusion detection mechanisms fail to generalize dynamic zero-day exploits within sophisticated IoT environments. This paper proposes a hybrid zero-day intrusion detection system using Siamese network-based anomaly correlation and reinforcement learning-based adaptive defense. Furthermore, the paper uses unsupervised machine learning classifiers over benchmark IoT datasets with the intention of detection of known attack types compared to unknown anomalies using distance-based similarity analysis to detect possible zero-day attacks. To facilitate adaptability, a Proximal Policy Optimization (PPO) reinforcement learning-based agent dynamically adjusts the defense policy with continuous feedback and optimization. Experimental evaluations demonstrate 99.28% training accuracy, 99.07% accuracy in unknown attack detection, and 93.94% zero-day detection ratio, confirming the convergence and stability of the model on this http URL system offers a self-learning and extensible defense mechanism of IoT deployments by finding the right balance between precision, latency and false positives. This deep anomaly correlation with adaptive reinforcement learning is a firm base on which the next generation and autonomic cyber security solutions can take the reins as the zero-day threats keep changing their course.
• 中文摘要 零日威胁在物联网网络安全中还处于萌芽阶段，这需要识别新兴恶意行为的认知识别机制。传统的入侵检测机制无法在复杂的物联网环境中推广动态零日漏洞利用。本文提出了一种结合暹罗网络异常相关和强化学习自适应防御的混合零日入侵检测系统。此外，本文使用无监督机器学习分类器，基于基准物联网数据集，旨在通过基于距离的相似度分析检测已知攻击类型与未知异常进行对比，以检测可能的零日攻击。为促进适应性，基于近端策略优化（PPO）强化学习的智能体动态调整防御策略，并持续反馈和优化。实验评估显示，训练准确率为99.28%，未知攻击检测准确率为99.07%，零日检测率为93.94%，证实了该http URL系统模型的趋同与稳定性，通过在精度、延迟和误报之间找到合适的平衡，为物联网部署提供了自学习和可扩展的防御机制。这种与自适应强化学习的深度异常相关性，为下一代和自主网络安全解决方案在零日威胁不断变化时能够发挥作用奠定坚实基础。

From Risk Scoring to Risk Allocation: A Density-Driven Framework for Diverse Monitoring in Multi-Agent Systems

从风险评分到风险分配：多智能体系统中多样化监控的密度驱动框架

• Authors: Zhaohui Wang
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.26146
• Pdf link: https://arxiv.org/pdf/2609.26146
• Abstract Risk monitoring in multi-agent systems is commonly built on a per-state primitive that scores each state independently and selects the top K. Under crowding, where many agents share the same fragility, this approach picks redundant alerts whose risks are jointly correlated, a pattern we describe as ``herding in monitoring.'' We propose a paradigm shift from risk scoring to risk allocation, supported by two contributions. First, we identify the Crowding Paradox, namely that P(risk | x) $\propto$ p(x) rather than 1/p(x), so density rather than anomaly score is the operative risk signal; on financial data, density-based scoring reaches AUROC $\geq$ 0.94 at 5d/10d/20d crash horizons, while five anomaly baselines all fall below 0.80. Second, given a density-derived fragility score, we recast monitoring as combinatorial subset selection over interdependent states and map it to a QUBO objective with a $\lambda$-controlled risk--diversity tradeoff. The resulting Pareto frontier contains standard diverse-subset methods (MMR, k-DPP) as fixed operating points; the gain over greedy grows monotonically with scale, from +24% at n=15 to +66% at n=200; a learned $\lambda$ policy reaches 99.5% of an oracle grid-search objective; and the formulation transfers to traffic and multi-agent reinforcement learning. The same QUBO instances execute without modification on Rigetti superconducting QPUs (Ankaa-3 and Cepheus-1-108Q via Amazon Braket), which we report as a compatibility property of the formulation rather than a claim of quantum advantage at this scale.
• 中文摘要 多智能体系统中的风险监测通常基于一个状态基元，该原语独立对每个状态进行评分并选择顶部K。在拥挤模式下，许多智能体共享相同的脆弱性，这种方法选择了风险共同相关的冗余警报，我们称之为“监控中的群集”模式。我们提出了从风险评分转向风险分配的范式转变，并得到两个贡献的支持。首先，我们识别了拥挤悖论，即P（risk | x） $\propto$ p（x）而非1/p（x），因此操作性风险信号是密度而非异常值;在金融数据中，基于密度的评分在5d/10d/20d崩溃视野达到AUROC $\geq$ 0.94，而五个异常基线均低于0.80。其次，给定密度衍生的脆弱性评分，我们将监测重新定义为对相互依赖状态的组合子集选择，并将其映射到一个 QUBO 目标，具有 $\lambda$ 控制的风险-多样性权衡。最终的帕累托前沿包含标准的多样子集方法（MMR，k-DPP）作为固定操作点;贪婪的收益随规模单调增长，从 n=15 时的 +24% 到 n=200 时的 +66%;学习到的 $\lambda$ 策略达到 oracle 网格搜索目标的 99.5%;该表述可转化为流量和多智能体强化学习。相同的 QUBO 实例在 Rigetti 超导 QPU（Ankaa-3 和 Cepheus-1-108Q，通过亚马逊 Braket）上无修改地执行，我们将其报告为该表述的兼容性性质，而非该尺度上的量子优势主张。

MGRL-RSCC: Multi-Granularity Reward Reinforcement Learning for Fine-Grained Remote Sensing Change Captioning

MGRL-RSCC：多粒度奖励强化学习，用于细粒度遥感变更字幕

• Authors: Futian Wang, Mengqi Wang, Xiao Wang, Wentao Wu, Haowen Wang, Zhicheng Zhao, Jin Tang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.26166
• Pdf link: https://arxiv.org/pdf/2609.26166
• Abstract Remote Sensing Change Captioning (RSCC), which aims to generate accurate and detailed linguistic descriptions of ground object variations from bi-temporal remote sensing images, is a critical and challenging task in intelligent remote sensing interpretation. The mainstream autoregressive training paradigm faces severe exposure bias and train-test distribution mismatch, resulting in cumulative generation errors. They tend to produce conservative and template-fixed captions while ignoring subtle scene change details. To address these challenges, this paper proposes a novel multi-granularity reward reinforcement learning paradigm, termed MGRL-RSCC. Specifically, we first leverage a CNN and hierarchical self-attention module to extract and enhance visual features from bi-temporal remote sensing images. A Transformer decoder is then utilized to complete visual-to-linguistic translation. Different from existing methods, we design a dual-decoding strategy and a two-stage joint optimization scheme, which combines token-level supervised learning via greedy decoding and multi-granularity reward-driven self-critical reinforcement learning via sampling decoding. We further construct three complementary reward functions covering linguistic fluency, change state consistency, and structural-semantic relevance to comprehensively optimize caption quality and alleviate false and missing change descriptions. Extensive experiments on multiple public RSCC benchmark datasets demonstrate that the proposed MGRL-RSCC effectively mitigates exposure bias and conservative generation problems in traditional autoregressive methods. The source code and pre-trained models will be released on this https URL
• 中文摘要 遥感变更字幕（RSCC）旨在从双时空遥感图像中生成准确且详细的地面物体变化语言描述，是智能遥感解读中一项关键且具有挑战性的任务。主流自回归训练范式面临严重的曝光偏差和训练-测试分布不匹配，导致累积生成误差。它们倾向于生成保守且固定模板的字幕，忽视细微的场景变化细节。为应对这些挑战，本文提出了一种新的多粒度奖励强化学习范式，称为MGRL-RSCC。具体来说，我们首先利用卷积神经网络和分层自注意模块从双时空遥感图像中提取和增强视觉特征。随后使用Transformer解码器完成视觉到语言的翻译。与现有方法不同，我们设计了双重译码策略和两阶段联合优化方案，结合了通过贪婪解码实现的代币级监督学习和通过采样译码实现的多粒奖励驱动自批判强化学习。我们还进一步构建了三个互补的奖励函数，涵盖语言流畅性、变更状态一致性和结构语义相关性，以全面优化说明质量，缓解虚假和缺失的变更描述。在多个公开RSCC基准数据集上的大量实验表明，所提出的MGRL-RSCC有效缓解了传统自回归方法中的暴露偏差和保守生成问题。源代码和预训练模型将发布于此https URL

ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model

ForeDrive：前瞻性引导的端到端自动驾驶，结合规划相关潜在世界模型

• Authors: Sinuo Wang, Zichong Gu, Yuhan Huang, Wenxin Wen, Xun Yang, Yiqing Zhang, Xingyu Zhang, Ningyu Che, Jie Ling, Qiankun Yu, Wei Liu, Jing Xu, Xinggang Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.26299
• Pdf link: https://arxiv.org/pdf/2609.26299
• Abstract Existing latent world models are typically optimized for future predictability, yet the resulting representations are not necessarily useful for planning in autonomous driving. Predictions are commonly used for pretraining or auxiliary supervision rather than as direct conditioning signals for trajectory generation. We propose ForeDrive, which learns a planning-relevant latent representation and couples it asymmetrically to a Diffusion Transformer (DiT) planner. The planner consumes multi-horizon latent future representations learned with a JEPA-style world model; planning gradients update the shared online encoder, while stop-gradient routing trains the latent predictor with forecasting losses only. Because predicted futures have varying reliability across horizons and BEV trajectories are misaligned with image tokens, we use gated visual fusion, future-status injection, and Trajectory-Adaptive Bias (TAB) to inject future latents as guidance without overriding the current observation. Trained with pure imitation learning and using only the current front-view image as visual input at inference, ForeDrive attains 89.9 PDMS on NAVSIM v1 and 90.0 one-stage EPDMS on NAVSIM v2, without reinforcement learning or an external trajectory scorer.
• 中文摘要 现有的潜在世界模型通常针对未来可预测性进行了优化，但所得的表示不一定对自动驾驶的规划有用。预测通常用于预训练或辅助监督，而非作为轨迹生成的直接条件信号。我们提出ForeDrive，它学习与规划相关的潜在表示，并将其非对称地耦合到扩散变换器（DiT）规划器。规划器消耗通过JEPA风格世界模型学习的多视野潜在未来表示;规划梯度更新共享的在线编码器，而停止梯度路由仅用预测损失训练潜在预测器。由于预测未来在不同视野间的可靠性各异，且BEV轨迹与图像符号不匹配，我们采用门控视觉融合、未来状态注入和轨迹自适应偏置（TAB）来注入未来潜在因素作为指导，而不覆盖当前观测。通过纯模拟学习训练，并仅以当前正面图像作为推断视觉输入，ForeDrive在NAVSIM v1上实现89.9 PDMS，在NAVSIM v2上实现90.0 PDMS，无需强化学习或外部轨迹评分器。

PACT: From Credit Assignment to Critic Alignment

PACT：从署名分配到批评对齐

• Authors: Jiayan Fu, Hang Xu, Yong Zhang, Zhaokai Luo, Yao Hu, Dongyan Zhao, Mu Chuan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.26355
• Pdf link: https://arxiv.org/pdf/2609.26355
• Abstract Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.
• 中文摘要 强化学习已成为大型语言模型（LLM）训练后的核心组成部分，但代币级信用缺乏普遍接受的数学定义，使其与常用训练信号的关系不明确。我们制定了三个正则性条件，即完备性、前缀一致性和中性性，并证明它们唯一决定了代币级信用。这一特征描述为解释现有算法间现象提供了统一基础，并指导改进的演员-批评者训练程序的发展。从这个角度来看，理想中的策略提炼（OPD）教师扮演隐性批评者的角色，产生与代币级信用诱导的预期政策梯度成正比。响应级强化留一除外（RLOO）信号尽管颗粒度较粗糙，仍符合代币级信用的预期策略梯度贡献。我们进一步建立了有界结果奖励下的信用稀疏度，并展示了广义优势估计（GAE）中中间批评者误差如何与基础信用相当。这些激励了政策对齐批评者培训（PACT），该训练采用“行为者-后批评者”更新顺序，将重要抽样修正应用于批评者培训，更好地使批评者与更新策略对齐。在代理数学推理中，PACT在四个基准测试中平均准确率为72.87%，分别比GRPO和PPO高出8.80个百分点和13.16个百分点。在SWE-bench Verified测试中，PACT的通过率为67.4%，分别比PPO、GRPO和SAO高出2.4、2.0和3.8个百分点。

KwaiMind Technical Report

KwaiMind技术报告

• Authors: Junlong Wu, Zijun Li, Yuting Hu, Jia Sun, Pengcheng Wei, Yimin Zhou, Honglie Wang, Huaiqing Wang, Dewen Fan, Fei Zuo, Haixuan Gao, Lihui Peng, Tingxuan She, Yuqing Li, Boheng Zhang, Fan Yang, Wenwu Ou
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.26375
• Pdf link: https://arxiv.org/pdf/2609.26375
• Abstract Commercial image editing requires product identity preservation, accurate text rendering, and user appeal alongside general editing quality. We present KwaiMind, an image editing system combining general capabilities with e-commerce specialization. An agent-based data engine maintains approximately 1.8 million high-quality editing pairs. Built on a multimodal diffusion transformer, KwaiMind undergoes continued pre-training and supervised fine-tuning, followed by preference optimization and online reinforcement learning. A general-purpose vision-language judge and specialized rewards for click-through rate (CTR), text rendering, and product consistency guide specialized policies, which are consolidated through on-policy distillation. We introduce Ecom-Bench, covering 11 commercial editing tasks with task-specific visual evaluation and CTR-based ranking. KwaiMind achieves the strongest overall scores among evaluated open-source editors on ImgEdit, GEdit, both language splits of REDEdit, and Ecom-Bench visual quality, and the highest aggregate CTR ranking score among compared systems. Offline, CTR-guided optimization increases the proportion of generated images whose predicted CTR exceeds that of the original product image from 12.16% to 37.41%. In an online A/B experiment, CTR-based selection of product main images yields an approximately 2.44% relative increase in actual CTR. These results demonstrate the value of domain-specific data and reward-driven alignment for commercial image editing.
• 中文摘要 商业图像编辑不仅需要产品身份保护、准确的文本渲染和用户吸引力，还需要整体编辑质量。我们介绍KwaiMind，一款结合通用功能与电子商务专业化的图像编辑系统。基于代理的数据引擎维护约180万对高质量编辑对。基于多模态扩散变换器，KwaiMind持续进行预训练和监督微调，随后进行偏好优化和在线强化学习。一个通用视觉语言评判器，以及针对点击率（CTR）、文本渲染和产品一致性指南的专业奖励，这些策略通过策略内的提炼整合而成。我们引入Ecom-Bench，涵盖11项商业编辑任务，包含任务特定的视觉评估和基于CTR的排名。KwaiMind 在 ImgEdit、GEdit、REDEdit 的两种语言分割和 Ecom-Bench 视觉质量方面，在受评的开源编辑器中获得了最强的整体得分，并且在比较系统中获得了最高的综合点击率排名。离线时，CTR 引导优化将生成图像中预测点击率高于原始产品图片的比例从12.16%提高到37.41%。在一项在线A/B实验中，基于CTR选择的产品主图像产生了约2.44%的实际CTR相对增长。这些结果展示了领域特定数据和奖励驱动对齐对商业图像编辑的价值。

RouteRLT: Learning When and Which RL Specialist Should Control a Vision-Language-Action Policy

RouteRLT：了解何时以及由哪位强化学习专家来控制愿景-语言-行动策略

• Authors: Chongyu Zhu, Jaden Hinds, Hyegang Kim, Juan Sebastian Rojas, Ramy Elmallah, Chi-Guhn Lee
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.26467
• Pdf link: https://arxiv.org/pdf/2609.26467
• Abstract Vision-language-action (VLA) models provide broad manipulation competence, but often struggle during the precision-critical stages that dominate contact-rich industrial tasks such as connector insertion and cable management. A common remedy is to refine a pretrained VLA with reinforcement learning (RL), enabling task-specific improvement beyond behavior cloning. However, how to preserve its generalist behavior while deciding when RL refinement is needed and which specialized policy should act remains an open question. In this work, we present RouteRLT, a routing framework that learns when and which RL specialist, an RL policy trained for a single precision-critical phase, should take control from a generalist VLA. A phase selector identifies the active controller, a stabilizer suppresses transient switches, and an action-boundary manager handles transitions between chunked policy outputs. We evaluate RouteRLT on multi-object pick-and-place tasks in LIBERO, as well as on a real-world cable pickup and port-insertion task with multiple precision-critical stages. In simulation, the learned routing improves over the base VLA and matches routing with privileged phase boundaries, without accessing those boundaries at deployment. The real-robot evaluation validates automatic routing to both the pickup and insertion specialists under an operator-aligned handoff protocol. Altogether, these results show that learned routing applies RL specialist control where precise adaptation is most valuable while preserving generalist VLA behavior, including recovery from failed execution attempts.
• 中文摘要 视觉-语言-动作（VLA）模型具备广泛的操作能力，但在主导接触丰富工业任务（如连接器插入和电缆管理）的高精度关键阶段常常遇到困难。常见的解决方法是通过强化学习（RL）完善预训练VLA，实现超越行为克隆的任务具体改进。然而，如何在决定何时需要强化学习以及应采取哪种专业策略时保持其通用行为仍是一个开放问题。本研究提出了RouteRLT路由框架，该框架学习何时以及由哪位RL专家——为单一精度关键阶段训练的RL策略——应接管通用VLA的控制权。相位选择器识别主动控制器，稳定器抑制瞬态开关，动作边界管理器处理分块策略输出之间的转换。我们在LIBERO中多对象选址任务以及具有多个精度关键阶段的线缆拾取和端口插入任务中评估了RouteRLT。在仿真中，学习到的路由相较于基础VLA有所提升，并匹配与特权相位边界的路由，而无需在部署时访问这些边界。真实机器人评估验证了在运营商对齐的切换协议下，向拾取和插入专家验证自动路由。总体来看，这些结果表明，学习到的路由在保持通用VLA行为（包括从失败执行尝试中恢复）最为重要的地方应用了强化语言专家的控制。

Learning Air-Ground Motion Control with Temporal Mode Switching and Cross-Terrain Tracking

学习利用时间模式切换和跨地形跟踪的空地运动控制

• Authors: Ruitian Pang, Mingrui Li, Xuanting Liu, Tiancheng Lai, Juncheng Chen, Xiangyu Li, Ruibin Zhang, Qishao Wang, Jin Yu, Haiyin Piao, Fei Gao, Chao Xu, Yanjun Cao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.26564
• Pdf link: https://arxiv.org/pdf/2609.26564
• Abstract Passive-wheeled terrestrial-aerial bimodal vehicles (TABVs) combine aerial mobility with energy-efficient ground locomotion. However, reliable air-ground mode switching under limited onboard perception and robust ground trajectory tracking across diverse terrains remain challenging when targeting real-world applications. In this work, we propose a learning-based air-ground motion control framework for passive-wheeled TABVs: 1) a learned mode selector for autonomous air-ground motion mode switching. The selector uses historical single-point time-of-flight (ToF) measurements and robot states together with future reference information to determine the active locomotion mode. 2) a reinforcement learning control policy for trajectory tracking. The policy combines proprioceptive observations with future reference information to anticipate trajectory changes. For ground locomotion, multi-terrain training and dynamics randomization enable robust tracking across different terrains. Simulation and real-world experiments demonstrate reliable air-ground switching under limited perception and accurate ground tracking across diverse terrain conditions. The learned selector outperforms a rule-based mode selector in challenging transitions, while the ground controller achieves lower position RMSE than PID across all tested conditions and maintains decent tracking where NMPC fails. With these capabilities integrated, the system tracks a 101m air-ground trajectory through multiple autonomous mode transitions with a position RMSE of 0.08m.
• 中文摘要 被动轮式陆地-空中双峰车（TABVs）结合了空中机动性和节能地面移动。然而，在有限的机载感知和跨多样地形的稳健地面轨迹跟踪下，可靠的空地模式切换仍具挑战性，尤其适用于实际应用。本研究提出基于学习的被动轮式空地运动控制框架：1）用于自主空地运动模式切换的学习模式选择器。选择器利用历史单点飞行时间（ToF）测量数据和机器人状态，结合未来参考信息确定主动移动模式。2）用于轨迹跟踪的强化学习控制策略。该策略结合本体感觉观察与未来参考信息，预测轨迹变化。地面移动方面，多地形训练和动态随机化实现不同地形间的稳健跟踪。仿真和实际实验展示了在有限感知下可靠的空地切换，并在不同地形条件下实现准确的地面跟踪。学习后的选择器在具有挑战性的转换中优于基于规则的模式选择器，而地面控制器在所有测试条件下的位置RMSE低于PID，并在NMPC失效时保持较好的跟踪。整合这些能力后，系统通过多次自主模式转换跟踪101米空地轨迹，位置RMSE为0.08米。

GeoComposer: Geometry-Grounded Photographic Composition Instruction

GeoComposer：几何基础摄影构图教学

• Authors: Shuangzhi Li, Qiaoqiao Jia, Xingxin Chen, Guile Wu, Dongfeng Bai
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.26620
• Pdf link: https://arxiv.org/pdf/2609.26620
• Abstract Photographic composition aims to provide visual guidance for improving the framing, viewpoint, and spatial arrangement of an image. Early methods primarily rely on image cropping to enhance composition, which is restricted to the viewpoint and spatial arrangement of the input image. Recent methods have explored image understanding and editing to improve composition, but they mainly focus on instruction following and aesthetic quality, overlooking the importance of 3D scene geometry consistency for photographic composition. In this work, we propose GeoComposer, a novel geometry-grounded photographic composition framework that analyzes the composition of a given image to generate textual guidance and synthesizes a visual exemplar that enhances the composition of the given image. To promote geometry-grounded composition, we propose a geometry-aware representation learning mechanism that leverages geometric priors from a visual geometry foundation model to shape the intermediate representations of the composition editing model. This mechanism preserves both global structural relationships and local fine-grained correspondences for geometry-grounded composition. Furthermore, we propose a reinforcement learning strategy guided by a hybrid reward that jointly optimizes instruction following, aesthetic quality, and geometric consistency. This enables the model to generate visual exemplars that faithfully follow the composition instructions while remaining visually appealing and geometrically consistent. Extensive experiments show the superiority of our approach over state-of-the-art methods, highlighting its effectiveness in generating visually appealing and geometrically consistent composition.
• 中文摘要 摄影构图旨在为改善图像的构图、视点和空间布局提供视觉指导。早期方法主要依赖图像裁剪来增强构图，而该构图仅限于输入图像的视点和空间布局。近期方法探索图像理解和编辑以提升构图，但主要关注指令遵循和美学质量，忽视了三维场景几何一致性对摄影构图的重要性。本研究提出GeoComposer，一种基于几何的新型摄影构图框架，通过分析给定图像的构图生成文本指导，并综合视觉范例以增强图像构图。为促进几何基础构图，我们提出了一种几何感知表示学习机制，利用视觉几何基础模型中的几何先验来塑造构图编辑模型的中间表示。该机制既保留了全局结构关系，也保留了基于几何的构图的局部细粒度对应关系。此外，我们提出了一种由混合奖励引导的强化学习策略，能够共同优化指令遵循、美观质量和几何一致性。这使得模型能够生成忠实遵循合成指令的视觉范例，同时保持视觉吸引力和几何一致性。大量实验显示了我们方法相较于最先进方法的优越性，凸显了其在生成视觉上吸引人且几何上一致的构图的有效性。

MAGIC: Mixed-Granularity Agent Graphs via Incremental Construction with Dense-Reward Reinforcement Learning

MAGIC：通过增量构建结合密集奖励强化学习实现混合粒度代理图

• Authors: Kairui Yang, Ziheng Yi, Xunkai Li, Minghao An, Zhanke Liu, Zekai Chen, Rong-Hua Li
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.26667
• Pdf link: https://arxiv.org/pdf/2609.26667
• Abstract Collaboration topology shapes both the performance and execution cost of LLM-based multi-agent systems. Because tasks differ in complexity and required capabilities, recent approaches generate task-specific collaboration graphs that specify agent participation and information flow. However, representative topology generators use either individual agents or predefined groups throughout an organization, overlooking differing collaboration needs across subtasks. Our key insight is to select granularity locally for each functional role, combining fine-grained control with reusable collaboration patterns within one organization. Learning such organizations requires exploring a combinatorial construction space with limited intermediate feedback from final-answer rewards. Therefore, we propose MAGIC, a dense-reward reinforcement learning framework for mixed-granularity graph generation. Specifically, MAGIC constructs a mixed-granularity agent graph by sequentially selecting a functional role, instantiating it as a single agent or reusable group, and connecting it to existing units. We directly optimize the construction policy using returns from trajectories sampled under the current policy and use potential-based reward shaping to provide intermediate feedback from probe-based utility and structural signals while preserving the cumulative task reward. MAGIC outperforms state-of-the-art baselines across eight benchmarks and demonstrates strong inference efficiency in our efficiency study.
• 中文摘要 协作拓扑决定基于LLM的多智能体系统的性能和执行成本。由于任务复杂度和所需能力不同，最新方法生成任务专属协作图，指定代理参与和信息流。然而，代表性拓扑生成器在组织内使用单个代理或预定义的组，忽略了各子任务间的协作需求差异。我们的关键见解是为每个职能角色局部选择粒度，结合细粒度控制与组织内可复用的协作模式。学习此类组织需要探索一个组合构建空间，同时对最终答案奖励的中间反馈有限。因此，我们提出MAGIC，一种用于生成混合粒度图的密集奖励强化学习框架。具体来说，MAGIC通过顺序选择功能角色，将其实例化为单一代理或可复用组，并与现有单元连接，构建混合粒度代理图。我们直接利用当前策略下抽样轨迹的收益来优化构建策略，并利用基于潜力的奖励塑造，在保持累计任务奖励的前提下，从探针型效用和结构信号中提供中间反馈。MAGIC在八个基准测试中优于最先进基线，并在效率研究中展现出强劲的推断效率。

Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning

超越重复抽样：学习大型语言模型推理的搜索策略

• Authors: Ismail Labiad, Matthieu Kowalski, Marc Schoenauer, Rémi Munos, Julia Kempe
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.26704
• Pdf link: https://arxiv.org/pdf/2609.26704
• Abstract Large language models increasingly tackle hard reasoning problems by spending more test-time compute, yet the dominant strategy remains naive repeated sampling: draw many independent solutions and hope one is correct. Because such sampling explores only through local decoding noise, it tends to produce many near duplicate attempts rather than genuinely different ideas. We ask whether exploration can instead be steered at a semantic level, by first sampling problem specific concepts, hints, or strategies and then conditioning answer generation on them. We refine this into a simple, more exploratory procedure that emits many diverse concepts in a single trajectory, and evaluate it on hard problems where repeated sampling struggles. We then go a step further and make concept generation trainable: a small concept generator is optimized with reinforcement learning so that its concepts maximize the downstream success of a larger, frozen answer generator. On hard mathematical reasoning problems, the trained concept generator substantially improves the answer generator's pass@k over naive repeated sampling at the same answer generation allocation, surpasses concepts drawn from much larger untuned models, and transfers to answer generators it was never trained against, including a model from a different family. A small model can thus be trained into an effective, reusable search policy for a much larger one.
• 中文摘要 大型语言模型越来越多地通过投入更多测试时间计算来解决硬推理问题，但主流策略仍然是天真重复抽样：抽取多个独立解，并希望其中一个是正确的。由于这种抽样仅通过局部解码噪声进行探索，往往产生许多近似重复的尝试，而非真正不同的观点。我们探讨是否可以在语义层面引导探索，先采样问题特定的概念、提示或策略，然后以此为条件条件生成答案。我们将此优化为一种简单、更具探索性的过程，在单一轨迹中输出许多不同的概念，并在重复抽样困难的难题上进行评估。然后我们更进一步，使概念生成可训练：一个小型概念生成器通过强化学习优化，使其概念最大化更大、冻结答案生成器的后续成功率。在困难的数学推理问题上，训练有素的概念生成器相比于同一答案分配下的天真重复抽样，显著提升了答案生成器的 pass@k，超越了来自更大未调优模型的概念，并转移到了它从未训练过的答案生成器，包括来自不同模型族的模型。因此，一个小模型可以训练成一个有效且可复用的搜索策略，用于更大的模型。

Keyword: diffusion policy

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation

JAMB：用于双手操作的联合动作-运动扩散

• Authors: Chuyang Xiao, Peilin Meng, David Held
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25322
• Pdf link: https://arxiv.org/pdf/2609.25322
• Abstract Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at this https URL
• 中文摘要 协调双手操作具有挑战性，因为任一机械臂的运动都可能改变共享的三维场景，从而影响另一机械臂。然而，大多数扩散策略生成动作时并未明确建模这些未来几何后果，而预测策略通常仅将未来状态作为辅助监督或固定条件使用。我们通过提出JAMB（一种扩散策略）来解决这一限制，该策略共同去噪双手动作和未来三维点轨迹。通过允许作用假设和轨道假设在共享变换器内共同演变，双方可以在去噪过程中相互信息和细化。我们还进一步将多模态表示建立在共享的时空坐标系统中，以促进关节去噪时的几何感知交互。我们在RoboTwin 2.0和现实世界机器人中的多种双手操作任务中评估JAMB的应用，并将其与仅动作策略及跨越不同状态表示和学习目标的替代未来预测方法进行比较。在16个模拟任务中，JAMB的平均成功率为83.4%，比最强基线高出23.9个百分点。在三个真实任务中，JAMB分别比仅动作和辅助几何预测方法高出50.0和21.2个百分点。除了这些性能提升外，JAMB在拥挤场景和分布外背景的泛化能力也比评估基线更强。这些结果共同展示了我们联合动作-运动建模框架在协调双手操作中的有效性。我们的项目网站可访问此 https URL

RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation

RoboMP-DINOv2：用于强健机器人操作的提示，而非过滤器

• Authors: Han Qi, Heng Yang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25506
• Pdf link: https://arxiv.org/pdf/2609.25506
• Abstract Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action. General-purpose vision encoders are not tailored to visuomotor control, while object-centric approaches often use segmentation masks as hard filters that discard potentially useful context. We propose RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2), a full-scene vision encoder that treats masks as spatial prompts rather than visibility filters. It extracts dense DINOv2 features from the full observation, injects learned region-specific embeddings at masked locations, and jointly contextualizes prompted and unprompted tokens for action prediction. We further introduce masked-region color randomization (MCR) to improve appearance robustness, yielding RoboMP-DINOv2-MCR. Across seven simulated manipulation settings, RoboMP-DINOv2 achieves 60.7% success under spatial shifts and 59.7% under scene clutter, compared with 50.7% and 41.0% for a DINOv2-based Diffusion Policy. Under unseen object colors, RoboMP-DINOv2-MCR achieves 72.5% success versus 35.1% for the strongest color-randomized baseline. Additional experiments and representation analyses show improved robustness while preserving behaviorally relevant scene information. Code is available at this https URL.
• 中文摘要 机器人操作策略必须在视觉变化间泛化，同时保留与动作相关的场景上下文。通用视觉编码器并非专为视觉运动控制量身定制，而以对象为中心的方法常将分割遮罩作为硬滤波器，丢弃潜在有用的上下文。我们提出了RoboMP-DINOv2（机器人掩蔽提示DINOv2），这是一种全场景视觉编码器，将遮罩视为空间提示而非可见性滤镜。它从完整观察中提取密集的DINOv2特征，在遮蔽位置注入学习到的区域特定嵌入，并联合上下文化提示和非提示令牌以预测动作。我们进一步引入遮蔽区域颜色随机化（MCR）以提升外观鲁棒性，生成RoboMP-DINOv2-MCR。在七种模拟操作设置中，RoboMP-DINOv2在空间变化下成功率为60.7%，场景杂波下为59.7%，而基于DINOv2的扩散策略分别为50.7%和41.0%。在未可见物体颜色下，RoboMP-DINOv2-MCR成功率为72.5%，而最强颜色随机基线为35.1%。更多实验和表示分析显示，在保留行为相关场景信息的同时，鲁棒性有所提升。代码可在此 https URL 获取。

From Instrument-Mounted Demonstrations to In-Vivo Execution: Learning Bimanual Laparoscopic Appendectomy Without Robot-Collected Demonstrations

从器械演示到体内执行：在不依赖机器人收集演示的情况下学习双手腹腔镜阑尾切除术

• Authors: Dongho Yee, Juahn Oh, Jinseok Lee, Jiyul Lee, Yechan Seo, Seong Jeong, Minsung Kim, Seonho Shim, Younghoon Noh, Hyuk Choi, Youngbin Kong, Kyu Eun Lee, Hyoun-Joong Kong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.25625
• Pdf link: https://arxiv.org/pdf/2609.25625
• Abstract Most minimally invasive surgery is still performed with hand-held laparoscopic instruments, and the surgeon's instrument kinematics are lost when the operation ends; only the endoscope video is kept. This paper presents an end-to-end pipeline that captures this motion in the operating room and uses it to train a surgical robot policy, validated on live animals. We introduce a surgical instrument-state logger that mounts on the shaft of a standard laparoscopic instrument and recovers its pose and jaw state from an inertial sensor, a time-of-flight sensor and a Hall sensor, with no external camera or tracker. A data pipeline measures the latency of every sensor channel against a robot ground truth and aligns the channels before forming observation-action pairs. On these demonstrations we train a diffusion policy with a fine-tuned DINOv3 backbone, selecting its design by closed-loop rollouts in a physics simulator reconstructed from depth maps of an ex-vivo rabbit appendix. The policy is then retrained on 849 in-vivo demonstrations from four live rabbits and deployed on four additional live rabbits with electrosurgery armed. With the surgeon selecting the surgical phase, the policy completed the appendectomy in three of the four animals. The results show that demonstrations recorded from a surgeon's own instruments are sufficient to train, select and deploy a bimanual surgical policy in vivo. The robot serves only as the timing reference for sensor calibration and as the executor, and collects no demonstrations. Both demonstration corpora are released to support future surgical robot learning research.
• 中文摘要 大多数微创手术仍使用手持腹腔镜器械进行，手术结束时外科医生的器械运动学数据丢失;仅保留内窥镜视频。本文提出了一个端到端的流程，捕捉手术室中的这一动作，并用以训练手术机器人策略，并在活体动物上验证。我们引入了一种外科器械状态记录仪，安装在标准腹腔镜器械的轴上，通过惯性传感器、飞行时间传感器和霍尔传感器恢复其姿态和颌骨状态，无需外部摄像头或跟踪器。数据流水线测量每个传感器通道与机器人地面真实数据的延迟，并在形成观察-行动对前对齐通道。在这些演示中，我们训练一个扩散策略，使用微调的DINOv3骨干，通过物理模拟器中从体外兔虫尾的深度图重建的闭环展开来选择其设计。随后，该策略在4只活兔的849只体内演示上重新训练，并部署在另外4只配备电手术设备的活兔身上。外科医生选择手术阶段后，政策完成了四只动物中的三只的阑尾切除手术。结果显示，外科医生自身器械录制的演示足以训练、选择并部署体内双手手术策略。机器人仅作为传感器校准的时序参考和执行者，不收集演示。这两份演示语料库均发布以支持未来的外科机器人学习研究。