生成时间: 2026-11-20 01:08:56 (UTC+8); Arxiv 发布时间: 2026-11-19 20:00 EST (2026-11-20 09:00 UTC+8)
今天共有 36 篇相关文章
Keyword: reinforcement learning
Error Analysis of Deep Learning
深度学习的错误分析
- Authors: Yuling Jiao, Peiying Wu, Jerry Zhijian Yang, Pingwen Zhang
- Subjects: Subjects:
Numerical Analysis (math.NA); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.23321
- Pdf link: https://arxiv.org/pdf/2611.23321
- Abstract
This review examines deep learning theory through the lens of error decomposition, systematically addressing approximation, statistical, and optimization errors. We trace the development of error analyses from bounds on individual error components to frameworks that account for all three. Using nonparametric regression and the Deep Ritz Method as dual running examples, we demonstrate how these analyses apply across fundamentally different loss structures. We then review representative analyses in scientific machine learning, generative learning, reinforcement learning, representation learning, and large language models, showing how learning errors affect solutions, distributions, and predictions through problem-specific arguments. Throughout, we clarify which error components are controlled, how they are combined, and what assumptions are needed to obtain guarantees for trained networks.
- 中文摘要
本综述通过误差分解的视角审视深度学习理论,系统地处理近似、统计和优化误差。我们追溯了误差分析的发展历程,从单个误差成分的界限到涵盖这三者的框架。以非参数回归和 Deep Ritz 方法作为双重示例,我们展示了这些分析如何适用于根本不同的损失结构。随后,我们回顾了科学机器学习、生成学习、强化学习、表示学习和大型语言模型中的代表性分析,展示学习错误如何通过问题特定论证影响解决方案、分布和预测。全文中,我们澄清了哪些误差成分受控、如何组合以及为训练网络获得保证所需的假设。
A Survey on RLOps for Large Language Models
大型语言模型RLOps调查
- Authors: Lingzhe Zhang, Tong Jia, Chiming Duan, Minghua He, Leyi Pan, Kening Zheng, Zhaoyang Liu, Bolin Ding, Philip S. Yu, Ying Li
- Subjects: Subjects:
Software Engineering (cs.SE); Computation and Language (cs.CL); Cryptography and Security (cs.CR)
- Arxiv link: https://arxiv.org/abs/2611.23357
- Pdf link: https://arxiv.org/pdf/2611.23357
- Abstract
Reinforcement learning (RL) has become increasingly important for improving the reasoning, instruction-following, and task-solving capabilities of large language models (LLMs). However, scaling RL training exposes failures in distributed execution, feedback computation, and learning dynamics that can waste computational resources, slow training, and undermine model improvements. These challenges have motivated a growing body of techniques for LLM RLOps, which manages the efficiency and reliability of operations throughout the LLM reinforcement learning training lifecycle. Despite growing interest, these techniques remain scattered across research areas, with limited synthesis of their operational roles and interactions. To bridge this gap, we present a systematic survey of LLM RLOps methods. We organize existing approaches into five stages: Training Setup, Experience Collection, Reward Computation, Policy Optimization, and Model Evaluation. Within each stage, we examine operational concerns, management techniques, and evaluation evidence. Building on this taxonomy, we identify open challenges and future directions in training process observability, cross-stage coordination, generalizable RLOps, multi-run management, and RLOps for recursive self-improvement. These directions extend operational management from individual training runs to diverse training settings, shared infrastructure, and successive improvement cycles. We maintain a GitHub repository indexing relevant papers and open resources at this https URL.
- 中文摘要
强化学习(RL)在提升大型语言模型(LLM)的推理能力、指令跟随和任务解决能力方面变得越来越重要。然而,扩展 RL 训练暴露了分布式执行、反馈计算和学习动态中的失败,这些问题可能浪费计算资源、拖慢训练速度,并削弱模型改进。这些挑战促使 LLM RLOps 技术日益丰富,用于管理整个 LLM 强化学习训练生命周期内操作的效率和可靠性。尽管兴趣日益增长,这些技术仍分散于各研究领域,其操作角色和交互的综合有限。为弥合这一空白,我们对 LLM RLOps 方法进行了系统综述。我们将现有方法划分为五个阶段:训练设置、体验收集、奖励计算、策略优化和模型评估。在每个阶段中,我们审视运营问题、管理技术和评估证据。基于该分类体系,我们识别了培训过程可观察性、跨阶段协调、可通用RLOps、多运行管理以及递归自我提升RLOps等领域的开放挑战和未来方向。这些方向将运营管理从单个培训运行延伸到多样化的培训环境、共享基础设施以及连续的改进周期。我们维护一个GitHub仓库,在此HTTPS网址索引相关论文和开放资源。
Adaptive Line Switching for Cascade Mitigation in Power Systems via Graph Reinforcement Learning
通过图强化学习实现电力系统级联缓解的自适应线路切换
- Authors: Naeem Md Sami, Soumav Biswas, Mia Naeini
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.23423
- Pdf link: https://arxiv.org/pdf/2611.23423
- Abstract
Although cascading failures are relatively rare events in power systems, they pose significant risks and can lead to severe economic and societal consequences within a short period. Therefore, developing effective mitigation strategies to prevent cascade propagation and to minimize their impacts is essential. This paper presents a deep reinforcement learning (DRL)-based framework for adaptive transmission line switching to mitigate cascading failures. Given the important role of network topology and component interactions in cascading failure propagation in power systems, graph-based DRL approaches, including Graph-DQN, Graph-A2C, and Graph-PPO frameworks, are developed to capture and utilize the topological information of power systems. The proposed frameworks are evaluated on the IEEE 39, 57, and 118 bus systems and benchmarked against the unmitigated case, random and heuristic-based switching strategies as baselines, as well as their corresponding graph-agnostic DRL counterparts. The results demonstrate the importance of incorporating graph information into these mitigation frameworks, leading to reduced cascade size, load loss, and network clustering.
- 中文摘要
尽管级联故障在电力系统中相对罕见,但它们带来重大风险,并在短时间内可能引发严重的经济和社会后果。因此,开发有效的缓解策略以防止级联传播并最大限度减少其影响至关重要。本文提出了基于深度强化学习(DRL)的自适应输电线切换框架,以减轻级联故障。鉴于网络拓扑和组件交互在电力系统级联故障传播中的重要作用,开发了基于图的 DRL 方法,包括 Graph-DQN、Graph-A2C 和 Graph-PPO 框架,以捕获和利用电力系统的拓扑信息。所提框架在 IEEE 39、57 和 118 总线系统上进行评估,并以无缓解情况、随机和启发式切换策略作为基线,以及对应的图形无关 DRL 对应方案进行基准测试。结果表明,将图信息纳入这些缓解框架的重要性,从而降低级联规模、负载损失和网络聚类。
Learning and Control in Physically Coupled Dynamical Systems
物理耦合动力系统中的学习与控制
- Authors: Bao Trung Truong, Dat Thanh Nguyen, Thinh T. Doan
- Subjects: Subjects:
Robotics (cs.RO); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.23502
- Pdf link: https://arxiv.org/pdf/2611.23502
- Abstract
We introduce TAG (Testbed for Autonomous Games), a compact, low-cost robotic platform for studying learning and control in physically coupled dynamical systems. TAG combines an actively tilted and steered ball-in-maze board with a top-mounted rotary inverted pendulum. The pendulum's reaction torque and vibration disturb the board and marble, while the board's tilt, driven by the maze-solving policy, simultaneously disturbs the pendulum's balance. The main challenge in learning control policies for each subsystem is accounting for the impact of one subsystem's dynamics on the other's learning process. To address this challenge, we present complementary reinforcement learning paradigms, leveraging model-based and model-free approaches to learn the control policies for the maze and pendulum, respectively. Our key finding is that introducing the pendulum causes an immediate, sharp drop in the maze policy's success rate, but the policy adapts online and recovers to its pre-coupling performance within a small number of learning episodes. Experimental results show that both policies, trained independently, successfully control the maze board and the pendulum despite their physical coupling.
- 中文摘要
我们介绍了 TAG(自主游戏测试平台),这是一个紧凑、低成本的机器人平台,用于研究物理耦合动力系统中的学习与控制。TAG将主动倾斜和转向的迷宫球板与顶部旋转倒摆结合在一起。摆的反作用力矩和振动扰动了板和弹珠,而由迷宫解决策略驱动的板子倾斜,同时扰乱了摆的平衡。学习每个子系统的控制策略时,主要挑战是如何考虑一个子系统的动态对另一个子系统的学习过程的影响。为应对这一挑战,我们提出了互补的强化学习范式,采用基于模型的方法和无模型的方法,分别学习迷宫和摆的控制策略。我们的关键发现是,引入摆锤会导致迷宫策略的成功率立即急剧下降,但策略会在线调整,并在少数学习阶段内恢复到耦合前的性能。实验结果表明,这两种策略在独立训练后,尽管物理耦合,仍能成功控制迷宫板和摆。
Sample-Efficient Model Predictive Control via Proximal Policy Optimization
通过近端策略优化实现样本高效的模型预测控制
- Authors: Jiarui Wang, Sina Sharifi, Mahyar Fazlyab
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.23544
- Pdf link: https://arxiv.org/pdf/2611.23544
- Abstract
Sampling-based model predictive control (MPC) enables real-time control of complex nonlinear systems but often incurs substantial computational costs due to repeated trajectory simulations. In this paper, we bridge policy optimization in reinforcement learning and sampling-based MPC to improve computational efficiency through sample reuse. Starting from a KL-regularized distributional formulation of trajectory optimization, we derive an exact performance-difference identity based on a signed exponential advantage function. This identity motivates two algorithms, PPO-MPPI and SPO-MPPI, which adapt proximal policy optimization techniques to sampling-based MPC, enabling multiple optimization updates using previously evaluated trajectories without additional forward simulations. We establish population-level monotonic improvement guarantees for both methods and prove convergence to first-order stationarity for PPO-MPPI under fixed-covariance Gaussian sampling distributions. Experiments on quadruped velocity tracking and quadrotor path tracking demonstrate that the proposed methods achieve competitive control performance at substantially lower computational cost than multi-stage sampling approaches. Our results establish a principled connection between policy optimization and sampling-based MPC and highlight the potential of trajectory reuse for sample-efficient predictive control.
- 中文摘要
基于抽样的模型预测控制(MPC)使复杂非线性系统的实时控制成为可能,但由于重复轨迹模拟,常常会带来相当高的计算成本。本文将强化学习中的策略优化与基于抽样的MPC结合起来,通过样本重用提升计算效率。基于KL正则化分布的轨迹优化表述,我们基于带符号指数优势函数推导出精确的性能差恒等式。该恒等式驱动了两种算法PPO-MPPI和SPO-MPPI,它们将近端策略优化技术适配到基于采样的MPC,使得利用先前评估的轨迹实现多次优化更新,无需额外前向仿真。我们为两种方法建立了总体层面的单调改进保证,并证明了PPO-MPPI在固定协方差高斯抽样分布下收敛至一阶平稳性。四足速度跟踪和四旋翼路径跟踪的实验表明,所提方法在显著低于多级采样方法的计算成本下实现了具有竞争力的控制性能。我们的结果在策略优化与基于采样的MPC之间建立了原则性联系,并凸显了轨迹重用在样本高效预测控制方面的潜力。
Efficient Variance-Aware Reinforcement Learning in Time-Inhomogeneous Linear MDPs
在时间非齐次线性 MDP 中实现高效方差感知强化学习
- Authors: Qingyue Zhao, Qiwei Di, Jiafan He, Heyang Zhao, Kaixuan Ji, Quanquan Gu
- Subjects: Subjects:
Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
- Arxiv link: https://arxiv.org/abs/2611.23596
- Pdf link: https://arxiv.org/pdf/2611.23596
- Abstract
We study variance-dependent reinforcement learning in episodic time-inhomogeneous linear Markov decision processes (MDPs) with feature dimension $d$, horizon $H$, and $K$ episodes. While recent works have established variance-dependent regret with $\tilde{O}(1)$ deterministic-regime overhead for linear mixture MDPs (Zhao et al., 2023) and worst-case nearly minimax optimal regret $\tilde{O}(d\sqrt{H^3 K} + d^7 H^8)$ for linear MDPs (He et al., 2023),, existing variance-aware analyses for linear MDPs (Li and Sun, 2023; Huang et al., 2023) rely on a coarse concentration step whose resulting regret guarantees retain a polynomial $\tilde{O}(K^{1/4})$ additive term even in deterministic environments. We propose VA-LSVI-UCB++, a computationally efficient algorithm that achieves the variance-dependent regret bound $\tilde{O}(d\sqrt{H\min{\mathrm{Var}_K^,\mathrm{Var}_K^\pi}}+d^5H^4)$, where $\mathrm{Var}_K^$ and $\mathrm{Var}_K^\pi$ denote the cumulative conditional variances of the optimal value function and the played policies' value functions along the realized trajectory. Our result adapts to the minimum of the two realized trajectory variances, recovers the worst-case minimax optimal leading term $\tilde{O}(d\sqrt{H^3 K})$, and collapses to $\tilde{O}(d^5 H^4)$ in deterministic environments, free of polynomial-in-$K$ dependency. At the core of our analysis is a simultaneous three-way reference-advantage decomposition for the optimistic, pessimistic, and squared value function estimators coupled with variance self-bounding and Freedman-type value-difference arguments, which tightens the pessimistic and second-moment confidence radii by a factor of $\tilde{O}(d H)$ and removes the inverse-variance-floor blowup.
- 中文摘要
我们研究了在情节性时间不均匀线性马尔可夫决策过程(MDP)中,特征维度为$d$、视界$H$和$K$的方差依赖强化学习。尽管近期研究已在线性混合MDP中确立了$\tilde{O}(1)$确定性方案开销的方差依赖遗憾(Zhao 等,2023),以及线性MDP中最近极小最优遗憾$\tilde{O}(d\sqrt{H^3 K} + d^7 H^8)$(He等,2023),现有线性MDP的方差感知分析(Li和Sun,2023;Huang 等,2023)依赖粗略集中步骤,其所得遗憾保证即使在确定性环境中仍保留多项式 $\tilde{O}(K^{1/4})$ 加法项。我们提出 VA-LSVI-UCB++,这是一种计算效率高的算法,实现方差依赖的遗憾界限 $\tilde{O}(d\sqrt{H\min{\mathrm{Var}_K^,\mathrm{Var}_K^\pi}}+d^5H^4)$,其中 $\mathrm{Var}_K^$ 和 $\mathrm{Var}_K^\pi$ 表示最优值函数与所选策略价值函数沿实际轨迹的累积条件方差。我们的结果适应了两种实现轨迹方差中的最小值,恢复了最极小极大最优首项项$\tilde{O}(d\sqrt{H^3 K}))$,并在确定性环境中无$K$多项式依赖时坍缩为$\tilde{O}(d^5 H^4)$。我们分析的核心是对乐观、悲观和平方值函数估计量同时进行三因子参考优势分解,结合方差自界和Freedman型值差参数,使悲观和二矩置信半径收紧一个倍数,消除了反方差底层膨胀。
LoRN: Low-Rank Noise Steering for Generative Robot Policies
LoRN:生成式机器人政策中的低级噪声引导
- Authors: Wei Liu, Abrar Anwar, Huihua Zhao, Soha Pouya, Yan Chang
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.23628
- Pdf link: https://arxiv.org/pdf/2611.23628
- Abstract
Noise steering improves pretrained generative robot policies by learning to select their initial noise without updating model weights. We find that actions in the evaluated world-action models are more sensitive to video noise than action noise, making video noise an expressive input for steering. However, the noise-steering methods we evaluate, in their published forms, struggle to scale to the high-dimensional noise inputs of large policy classes such as world-action models. We introduce Low-Rank Noise Steering (LoRN), a reinforcement learning method that scales noise steering to world-action models. A small steering actor selects a direction through a fixed low-rank random projection, while our norm-controlled construction keeps the resulting noise near the typical scale of a Gaussian sample. This separates the actor's control dimension from the full noise dimension, enabling steering of both video and action inputs. Across three world-action models and ten model-task pairs, video steering improves average success by 26.2 percentage points over the base policy, compared with 7.1 points for DSRL action steering. For smaller action policies, the same construction achieves comparable final success while generally improving early-training success. Steering actors trained in simulation also transfer to three physical bimanual manipulation tasks across two base policies, without additional real-world training.
- 中文摘要
噪声引导通过学习在不更新模型权重的情况下选择初始噪声,从而改进了预训练的生成式机器人策略。我们发现,在评估过的世界动作模型中,动作对视频噪声比动作噪声更为敏感,使视频噪声成为引导时具有表现力的输入。然而,我们评估的噪声引导方法(在已发表的形式中)在适应大型策略类别(如世界动作模型)的高维噪声输入方面存在困难。我们介绍了低秩噪声引导(LoRN),这是一种强化学习方法,将噪声引导扩展到世界动作模型。一个小型引导演员通过固定的低秩随机投影选择方向,而我们的范数控制构造则将所得噪声保持在高斯样本的典型尺度附近。这将演员的控制维度与完整噪声维度分离,从而实现视频和动作输入的引导。在三种世界动作模型和十对模型任务中,视频引导的平均成功率比基础策略提升26.2个百分点,而DSRL动作引导则提升了7.1个百分点。对于较小的动作策略,同样的构造也能实现相当的最终成功率,同时总体上提升早期训练的成功率。在模拟中训练的引导演员还可在两个基础策略中转换为三个物理双手操作任务,无需额外真实世界训练。
Option-GRPO: Hierarchical RL Against AI Slop in Code Generation for Unseen SDKs
选项-GRPO:针对隐形SDK代码生成中的分层强化学习与AI偏差
- Authors: Prashank Kadam
- Subjects: Subjects:
Software Engineering (cs.SE); Computation and Language (cs.CL); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.23633
- Pdf link: https://arxiv.org/pdf/2611.23633
- Abstract
Large language models increasingly produce AI slop in generated code: verbose, redundant, non-idiomatic implementations that pass functional checks but bloat codebases. The problem compounds on SDKs released after the model's training cutoff, where the model has neither idiomatic priors nor compositional understanding. We present Option-GRPO, a hierarchical reinforcement learning framework that adapts a code LLM to an unseen SDK from its source code alone, without human-written usage examples or test cases, and that explicitly optimizes for both functional correctness and code elegance. Our key insight is that SDK source code contains compositional structure that defines a hierarchy of reusable code patterns (code options) which simultaneously serve as the building blocks of a curriculum and the action space for efficient policy optimization. We show that code options induce a valid semi-Markov decision process, give conditional bounds under which shorter option description length implies more concise and, in expectation, more API-efficient programs, and analyze under stated assumptions how the option hierarchy improves credit assignment by a factor of $\sqrt{T/K}$. Using Group Relative Policy Optimization (GRPO) with execution-based rewards at each curriculum level, Option-GRPO discovers SDK idioms, composes them into patterns, and assembles patterns into complete programs. Experiments on 12 unseen SDKs with a 32B-parameter base model demonstrate 61.4% pass@1, outperforming flat GRPO by 8.7 points, ExploraCoder by 11.2 points, and ReCode by 6.3 points. Option-GRPO produces solutions that use 34.7% fewer API calls and are 27% shorter than flat-GRPO outputs while maintaining higher correctness, directly attacking the slop problem.
- 中文摘要
大型语言模型越来越多地在生成代码中产生 AI 的粗糙:冗长、冗余、非惯用的实现通过了功能检查,却使代码库膨胀。问题在模型训练截止期之后发布的 SDK 上更加严重,因为这些模型既没有惯用先验,也缺乏组合理解。我们介绍 Option-GRPO,这是一个层级强化学习框架,它仅凭源代码将代码 LLM 适配到看不见的 SDK,无需人类编写的使用示例或测试用例,并且明确优化功能正确性和代码优雅性。我们的关键见解是,SDK 源代码包含组合结构,定义了可重用代码模式(代码选项)的层级结构,这些模式既是课程的基石,也是高效策略优化的行动空间。我们展示了代码选项诱导一个有效的半马尔可夫决策过程,给出条件界限,在该条件下,较短的选项描述长度意味着更简洁且预期更高效的程序,并分析在未明言假设下,选项层级如何以 $\sqrt{T/K}$ 的倍数提升学分分配。利用基于执行的奖励的群体相对策略优化(GRPO)在每个课程层级,Option-GRPO 发现 SDK 惯用语,将其组合成模式,并组合成完整程序。在 12 个未见过的 SDK 上,采用 32B 参数基础模型的实验显示出 61.4% 的 pass@1,比平坦 GRPO 高 8.7 分,高于 ExploraCoder 11.2 分,高 ReCode 6.3 分。Option-GRPO 的解决方案比平坦 GRPO 输出少 34.7%,且输出时间短 27%,同时保持更高的正确性,直接解决了 slop 问题。
Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks
沟通式政策提炼:通过沟通提炼黑箱教师能力,实现长期任务
- Authors: Lirui Luo, Kelong Mao, Xinwei Yang, Rongqing Li, Luyu Chen, Kieran Wong, Yudong Guo, Xinrui Wang, Jiayin Zhu, Gu Simiu, Sulong Xu, Cong Fang
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.23673
- Pdf link: https://arxiv.org/pdf/2611.23673
- Abstract
On-policy distillation (OPD) is a powerful paradigm for improving language-model agents on long-horizon tasks, yet its conventional formulation relies on teacher logits that text-only black- box models do not expose. To address this limitation, we introduce Communicative On-Policy Distillation (COPD), which uses natural-language communication as a medium for distilling a stronger model's capabilities into a smaller student. During task execution, the student au- tonomously decides when to consult a black-box teacher and formulates open-ended natural- language requests without predefined question templates or intervention schedules. Successful student-teacher interactions are retained as privileged references to supervise fresh adviser-free student rollouts through on-policy self-distillation, jointly with task-reward reinforcement learning. On CAMG-II, COPD achieves 2.46 times the gain of the strongest black-box OPD baseline. The capabilities it learns through communication generalize to Terminal-Bench 2.1 and SWE-Bench Pro V2, improving task success without adviser access.
- 中文摘要
策略上提炼(OPD)是一种强有力的范式,用于提升长期任务中的语言模型代理,但其传统表述依赖于教师日志,而纯文本黑箱模型无法披露这些信息。为解决这一局限,我们引入了沟通式策略内提炼(COPD),它利用自然语言交流作为媒介,将更强模型的能力提炼到更小的学生中。在任务执行过程中,学生自主决定何时咨询黑箱教师,并制定开放式自然语言请求,无需预设问题模板或干预计划。成功的师生互动被保留为特权参考,通过策略自我提炼,结合任务奖励强化学习,监督无指导教师的新部署。在 CAMG-II 中,COPD 获得的收益是最强黑箱 OPD 基线的 2.46 倍。通过通信学习的能力可推广至 Terminal-Bench 2.1 和 SWE-Bench Pro V2,无需顾问访问即可提升任务成功率。
Finetuning VLAs for Dexterous Manipulation: The Right Checkpoint Is All You Need
精细调校 VLA 以实现灵巧操控:正确的检查点就是你所需要的全部
- Authors: Artur Kuramshin, Charlotte Cloutier, Michael Przystupa, Kirsty Ellis, Mohammad S. Akhter, Glen Berseth
- Subjects: Subjects:
Robotics (cs.RO); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.23676
- Pdf link: https://arxiv.org/pdf/2611.23676
- Abstract
Vision-Language-Action models have demonstrated strong manipulation capabilities. A limitation of VLA models is that most manipulation datasets focus on parallel-jaw gripper systems, making transfer to multi-finger hands challenging, particularly as teleoperation datasets are harder to collect with multi-fingered robots. In this work, we study how to best adapt VLAs to dexterous hands by pairing supervised fine-tuning (SFT) with diffusion-steering reinforcement learning (DSRL). Using $\pi_{0.5}$ and a LEAP hand, we evaluate five fine-tuning recipes across two simulation tasks using only a handful of demonstrations. Our empirical analysis yields three key insights: (1) SFT success is poorly correlated with post-DSRL success, meaning the best SFT checkpoint is rarely the most effective to steer; (2) freezing the vision-language model (VLM) and tuning only the action expert produces the most steerable policy for both training and out-of-distribution placements; and (3) low-rank adaptation (LoRA) is the least steerable approach. Consequently, we recommend an adaptation recipe that freezes the VLM, fully trains the action expert, and eschews SFT rollout success as a checkpoint selection metric.
- 中文摘要
视觉-语言-动作模型已展现出强大的操作能力。VLA模型的一个局限在于,大多数操作数据集都集中在平行颚握持系统上,这使得转移到多指手部变得困难,尤其是由于多指机器人更难收集远程操作数据集。本研究中,我们研究如何通过将监督式微调(SFT)与扩散引导强化学习(DSRL)结合,最佳地将VLA适配到灵活的双手。我们利用$\pi_{0.5}$和一只LEAP手,仅用少量演示,评估了两项模拟任务中的五个微调配方。我们的实证分析得出了三个关键见解:(1)SFT成功率与DSRL后成功率相关性不佳,这意味着最佳的SFT检查点往往不是最有效的引导;(2)冻结视觉-语言模型(VLM)并仅对行动专家进行调优,能够为训练和非分发部署的部署带来最可导向的策略;(3)低秩适应(LoRA)是最难引导的方法。因此,我们建议采用一种适应配方,冻结VLM,充分训练动作专家,并避免将SFT推广成功作为检查点选择指标。
A Comparison of Heuristic, Human-Sensitive, and Self-Play DRL Policies in Human-Agent Teaming
人-客服组队中启发式、人机敏感和自玩 DRL 策略的比较
- Authors: Ayman bin Kamruddin, Christopher Lam, Gaurav Patil, Mirco Musolesi, Mario di Bernardo, Michael J. Richardson
- Subjects: Subjects:
Multiagent Systems (cs.MA); Human-Computer Interaction (cs.HC)
- Arxiv link: https://arxiv.org/abs/2611.23747
- Pdf link: https://arxiv.org/pdf/2611.23747
- Abstract
Dynamical Perceptual-Motor Primitives (DPMPs) offer a tractable framework for generating human-like movement dynamics in artificial agents (AAs) and have been used to support coordination in human-AI teaming tasks. Traditionally, DPMP controllers rely on heuristic (rule-based) decision policies derived from human behavior to guide goal selection and action sequencing. Although effective in structured tasks, such heuristics can be difficult to generalize to more complex settings, motivating interest in deep reinforcement learning (DRL) approaches for learning decision policies for human-AI interaction. A key challenge, however, is ensuring that DRL policies remain aligned with human strategies, particularly when agents are trained through self-play. To examine this issue, we compared three target-selection policies governing DPMP-based AAs in a first-person herding task: (1) a human-derived heuristic policy, (2) a self-play DRL policy (SP-DRL), and (3) a human-sensitive DRL policy (HS-DRL) trained alongside a heuristic agent. Results showed that heuristic agents most closely matched typical human-human coordination patterns, followed by HS-DRL agents, whereas SP-DRL agents deviated most. These findings highlight the importance of human-informed training when developing DRL-based teammates and suggest that simple heuristic policies may remain an effective approach for designing human-aligned AAs capable of human-AI collaboration.
- 中文摘要
动态感知-运动原语(DPMP)为生成人工代理(AA)中类似人类的运动动态提供了可操作的框架,并被用于支持人机协作任务中的协调。传统上,DPMP控制器依赖源自人类行为的启发式(基于规则)决策策略来指导目标选择和行动序列。虽然在结构化任务中有效,但此类启发式难以推广到更复杂的场景,这也激发了人们对深度强化学习(DRL)方法在人机交互决策策略学习方面的兴趣。然而,一个关键挑战是确保DRL策略始终与人类策略保持一致,尤其是在代理通过自我游戏进行训练时。为探讨这一问题,我们比较了在第一人称牧羊任务中,基于DPMP的AA的三种目标选择策略:(1)人类源自启发式策略,(2)自我游玩的DRL策略(SP-DRL),以及(3)与启发式代理共同训练的人类敏感DRL策略(HS-DRL)。结果显示,启发式智能体最接近典型的人际协调模式,其次是HS-DRL智能体,而SP-DRL智能体偏差最大。这些发现凸显了在开发基于日行车(DRL)团队成员时,以人为本的培训重要性,并表明简单的启发式策略可能仍然是设计具备人机协作能力的人类对齐AA的有效方法。
PReMind: When Personal Memory Meets Visual Hazards
PReMind:当个人记忆遇上视觉危险时
- Authors: Yeongtak Oh, Donghyeok Lee, Joonhee Oh, Jaewon Jeong, Sungroh Yoon
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.23808
- Pdf link: https://arxiv.org/pdf/2611.23808
- Abstract
Personalized vision-language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
- 中文摘要
个性化视觉语言助手利用视觉观察和个人记忆,为用户提供量身定制的建议。当观察到的场景存在物理危害时,个人情境(如过敏、医疗状况或行动限制)可能会改变适合的帮助方式。因此,要做出适当反应,就需要识别可见的危险并应用相关人员的信息。我们引入了PReMind-Bench,这是一个诊断基准测试,用于评估这种组合在不同危害和个人记忆条件下的表现。它单独测量记忆访问、接地、回避和危害响应,并评估相互依赖案例中的联合成功。现有模型往往满足其中一项要求,却未能满足另一项,而仅针对安全或个性化的训练会在联合成功上留下空白。因此,我们提出了PReMind,一种两阶段强化学习框架,结合按需和强制调用交互(记忆始终被检索),共同训练接地记忆使用和危害响应。在4B阶段,PReMind将联合成功率从12.9%提升至28.6%,超过其基于相互依赖的基础,并提升了主要测试中的危害充分性和正确弃权率。在强制呼叫评估以及第三位LLM评审下,三个二期种子的联合成功率持续存在;在这些种子上平均,联合成功率也提升了隐形危害组。这些发现凸显了评估和培训个人助理将视觉证据与相关人员情境相结合的必要性。
Evidence-Based Credit Assignment for Search-Augmented Language Models
基于证据的增量搜索语言模型学分分配
- Authors: Wen Wang, Jindong Wang, Jiahua Bao, Tu Yongsiqi, Yihao Liu, Junwei He, Wenkui Fan
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.23824
- Pdf link: https://arxiv.org/pdf/2611.23824
- Abstract
Reinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every search in a trajectory regardless of whether it retrieved useful information. This limitation is especially severe when all sampled rollouts for a question fail, collapsing group-centered outcome advantages to zero even when trajectories gather markedly different evidence. We present Evidence-based Credit Assignment (ECA), which augments terminal outcome rewards with a training-only auxiliary signal measuring how retrieved documents improve the likelihood of a reference answer. Rather than scoring the full dialogue prefix, ECA queries a frozen snapshot of the current policy under an evidence-only prompt containing solely the question and the deduplicated documents observed so far, isolating the utility of retrieved passages from surface variations in generated thoughts or search queries. Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO. Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.
- 中文摘要
通过终端答案奖励进行强化学习,使搜索增强语言模型能够在没有步骤级注释的情况下交错推理和检索,但结果监督则会为轨迹中的每一次搜索赋予相同的功劳,无论其是否检索到有用信息。当所有对某一问题进行抽样的展开都失败时,这一限制尤为严重,即便各条路径收集到的证据明显不同,也使得以群体为中心的结果优势被压缩为零。我们提出了基于证据的学分分配(ECA),它通过仅训练的辅助信号来增强终端结果奖励,衡量检索到的文档如何提高引用答案的概率。ECA不对完整的对话前缀评分,而是在仅包含问题和迄今观察到的去重文档的证据提示下查询当前策略的冻结快照,从而将检索到的文章的效用与生成的思维或搜索查询的表面差异隔离开来。在此框架下,我们比较了两种返回公式:ECA-Step(累积步进截断势能增量)和ECA-Horizon(保持伸缩视界势增益),并结合交叉折叠组归一化、新颖门槛和搜索局部令牌赋值。在七个搜索质量保证基准测试中,ECA-Horizon在仅结果GRPO基线上,严格宏量精确匹配在Qwen2.5-3B-和7B-Instruct上分别提升了1.03个百分点和4.80个百分点,优于RLOO和GiGPO。MultiHop-RAG上的零样本评估进一步显示出对多跳推理查询的强转移能力。
Learning Martial Arts Motion Skills for Animating Opponents in VR Sparring Games
学习为 VR 对练游戏中对手动画制作的武术动作技巧
- Authors: Jiqing Wen, Qixuan Shi, Robert LiKamWa, Pavan Turaga
- Subjects: Subjects:
Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Multimedia (cs.MM); Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.23825
- Pdf link: https://arxiv.org/pdf/2611.23825
- Abstract
Animating virtual opponents is an important yet challenging task in game development. In martial arts games, opponent motions must not only be physically plausible but also exhibit appropriate technique, timing, transitions, and target-directed intent. Traditional game animations often rely on handcrafted clips, state machine transitions, and procedural adjustments, which can be labor-intensive and may limit the behavioral diversity of non-player characters. This paper presents a martial arts motion skills learning framework for character animation in virtual sparring games. Inspired by Tang Soo Do, we collected a martial arts motion capture dataset containing isolated attacking and defending techniques, technique combinations and transitions, and contextual sparring variations. We first pretrained a reusable repertoire of martial arts motion skills from the captured data using reinforcement learning (RL) with adversarial style rewards, and then adapted these skills with task-specific rewards for target-directed striking and threat-aware evasion. We evaluated the motions generated by the learned policy through a user study, in which participants assessed their physical plausibility, smoothness, and martial arts form. To demonstrate their applicability to interactive gameplay, we developed a VR sparring prototype for Meta Quest headsets. The prototype combines policy-generated motion clips with lightweight state-based control to support interactive opponent behavior. Together, these findings support the feasibility of using a learned martial arts skill repertoire as a motion foundation for animating opponents in VR sparring games.
- 中文摘要
对虚拟对手进行动画制作是游戏开发中一项重要但充满挑战的任务。在武术游戏中,对手动作不仅要在物理上合理,还要展现出适当的技巧、时机、转换以及针对目标的意图。传统游戏动画往往依赖手工剪辑、状态机过渡和程序调整,这些过程劳动密集,可能限制非玩家角色的行为多样性。本文提出了一个用于虚拟对练游戏中角色动画的武术动作技能学习框架。受唐昭都启发,我们收集了一套武术动作捕捉数据集,包含孤立的攻防技术、技术组合与转换,以及情境对练变化。我们首先利用强化学习(RL)预训练了一组可重复使用的武术动作技能库,并以对抗式奖励进行训练,随后将这些技能与针对目标的打击和威胁意识回避的任务特定奖励进行了调整。我们通过用户研究评估了所学策略生成的动作,参与者评估了其身体合理性、流畅度和武术形式。为展示其在互动游戏中的适用性,我们开发了一个适用于Meta Quest头显的VR对练原型。该原型结合了策略生成的动作片段与轻量级基于状态的控制,以支持互动对手行为。综合来看,这些发现支持了将学到的武术技能库作为动作基础,在VR对抗游戏中动画对手的可行性。
SWE-growth: Teacher-Guided Adaptive Self-Play for Software Engineering Agents
SWE-growth:教师引导的软件工程代理自适应自我游玩
- Authors: Songshi Liang, Zixuan Huang, Mingxuan Song, Haoyu Tang, Shusheng Xu, Tongkai Yang, Zihe Wang, Rui Yan
- Subjects: Subjects:
Software Engineering (cs.SE)
- Arxiv link: https://arxiv.org/abs/2611.23876
- Pdf link: https://arxiv.org/pdf/2611.23876
- Abstract
Self-play offers a scalable way to train software engineering agents by generating repository-level repair tasks. However, with binary repair rewards and finite rollout budgets, all-fail groups provide no reward-based group-relative advantage. Such tasks may still offer learning opportunities when a Teacher can provide verified repairs absent from the Student's own attempts. We introduce SWE-Growth, a teacher-guided adaptive self-play framework that uses verified Teacher outcomes to identify learning opportunities for the current Student. Tasks with all students failure but reliable Teacher success provide demonstrations for supervised fine-tuning (SFT), and post-SFT Student rollouts identify mixed-outcome tasks with a positive Teacher--Student gap for reinforcement learning. Task eligibility also guides Injector updates, connecting Student training with subsequent task generation. Starting from 101 seed issues, \textsc{SWE-Growth} improves Qwen3-Coder-30B-A3B-Instruct from 27.7% to 31.7% on SWE-bench Pro and from 11.0% to 17.2% on held-out TerminalBench 2.1 by Round~2. It exceeds Student-only self-play by 2.6 and 4.1 percentage points, respectively, and Teacher-augmented self-play by 1.8 and 2.5 points under approximately matched budgets. These results support adapting both supervision and task selection to the Student's evolving repair capability.
- 中文摘要
自我游戏通过生成仓库级修复任务,为软件工程代理提供了一种可扩展的方式。然而,由于二元修复奖励和有限的部署预算,全失败小组无法提供基于奖励的群体相对优势。当教师能够提供学生自身尝试中缺乏的经过验证的修复时,这类任务仍可能提供学习机会。我们介绍了SWE-Growth,这是一个由教师指导的自适应自我游戏框架,利用经过验证的教师成果为当前学生识别学习机会。所有学生都失败但教师成功可靠的任务提供了监督微调(SFT)的演示,SFT后学生推广则识别出教师与学生之间存在正向差距的混合结果任务,以促进强化学习。任务资格还指导注入器更新,将学生培训与后续任务生成连接起来。从 101 个种子期开始,\textsc{SWE-Growth} 在 SWE-bench Pro 上将 Qwen3-Coder-30B-A3B-Instruct 从 27.7% 提升到 31.7%,在 Round~2 上从 11.0% 提升到 17.2%。它分别比仅学生自玩高出 2.6 个百分点和 4.1 个百分点,在与大致匹配预算下,比教师增强自玩高出 1.8 个百分点和 2.5 个百分点。这些结果支持将监督和任务选择调整到学生不断演进的修复能力。
CoSlice: Coordinated Multi-Agent Reinforcement Learning for 5G Network Slicing Allocation
CoSlice:用于5G网络切片分配的协调多智能体强化学习
- Authors: Pengxin Wang, Nakjung Choi, Jingdi Chen
- Subjects: Subjects:
Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
- Arxiv link: https://arxiv.org/abs/2611.23909
- Pdf link: https://arxiv.org/pdf/2611.23909
- Abstract
Radio Access Network (RAN) slicing is a shared resource allocation problem: each slice seeks to satisfy its Service Level Agreement (SLA) with minimal radio-resource consumption, while all slices compete for a limited resource budget. This creates both an intra-slice trade-off between SLA assurance and resource efficiency and an inter-slice conflict over shared resources, calling for specialized yet coordinated control. In this work, we present CoSlice, a coordinated multi-agent reinforcement learning framework for inter-slice physical resource block allocation in Open Radio Access Network (O-RAN). CoSlice uses preference-conditioned slice agents for fast resource allocation and an LLM-based coordinator to adapt their objectives using global network observations at a slower timescale. We formulate the problem as a multi-objective partially observable stochastic game (mo-POSG), establish the existence of a Preference Correlated Equilibrium, and derive a sublinear regret bound for the learning procedure. We implement the algorithm by using a two-stage training procedure that consists of multi-objective pretraining and joint optimization. We build a network slicing evaluation environment using ORANSlice and its radio-frequency simulator (RFsim), supporting realistic protocol-stack execution and application traffic across multiple slices. Experimental results show that CoSlice reduces both average and tail SLA violations compared with other centralized and multi-agent baselines under diverse slice workloads.
- 中文摘要
无线接入网络(RAN)切片是一种共享资源分配问题:每个切片都试图以最小的无线资源消耗满足其服务水平协议(SLA),而所有切片都在争夺有限的资源预算。这既造成了切片内部SLA保证与资源效率的权衡,也在共享资源方面存在切片间冲突,需要专门化但协调的控制。本研究介绍了CoSlice,一个用于开放无线接入网(O-RAN)中切片间物理资源块分配的协调多代理强化学习框架。CoSlice使用偏好条件切片代理实现快速资源分配,并利用基于LLM的协调器在较慢时间尺度上利用全局网络观测调整目标。我们将问题表述为一个多目标部分可观测随机博弈(mo-POSG),确定偏好相关均衡的存在,并为学习过程推导出一个亚线性遗憾界限。我们采用一个由多目标预训练和联合优化组成的两阶段训练过程来实现该算法。我们利用 ORANSlice 及其射频模拟器(RFsim)构建了一个网络切片评估环境,支持多个切片间的真实协议栈执行和应用流量。实验结果表明,与其他集中式和多智能体基线相比,CoSlice 在不同切片工作负载下,降低了平均和尾部 SLA 违规率。
SimplePro: A Simple Baseline for Proactive Video Understanding
SimplePro:为主动视频理解提供简易基线
- Authors: Yilin Wang, Xiangxi Zheng, Dongxing Mao, Linjie Li, Zhengyuan Yang, Alex Jinpeng Wang
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.23954
- Pdf link: https://arxiv.org/pdf/2611.23954
- Abstract
Proactive video understanding requires vision-language models (VLMs) to decide not only what to answer as a video unfolds, but also when newly arrived evidence warrants a response and when to remain silent. Although modern VLMs excel at offline long-video understanding, they struggle in proactive streaming settings, leading prior work to introduce task-specific reinforcement learning or auxiliary triggering modules. We find that explicit reporting rules and structured streaming context substantially improve a frozen VLM's proactive response performance. We exploit this observation to build SIMPLEPRO, a training-free baseline for proactive video understanding. At each decision step, the model combines recent high-resolution frames, bounded historical observations, and prior answer records to determine whether new information warrants a response and, when needed, generate a concise answer. The system then filters consecutive duplicate answers while retaining the raw outputs as context for subsequent decisions. Without any parameter updates or auxiliary triggers, SIMPLEPRO with a 4B backbone consistently outperforms specialized proactive systems across OVO-Bench, StreamingBench-PO, and ProactiveVideoQA. SIMPLEPRO shows overall performance gains as backbone size increases from 2B to 32B. We hope this simple yet effective method will serve as a strong baseline for proactive video understanding, and we will release our code and inference pipeline to support future research.
- 中文摘要
主动视频理解不仅需要视觉语言模型(VLM)在视频展开时决定应答什么,还要判断何时新出现的证据值得回应,何时保持沉默。尽管现代VLM在离线长视频理解方面表现优异,但在主动流式流环境中表现不佳,促使此前研究引入任务特定强化学习或辅助触发模块。我们发现,明确的报告规则和结构化流式上下文显著提升冻结VLM的主动响应性能。我们利用这一观察构建了SIMPLEPRO,这是一个无需训练的主动视频理解基线。在每个决策步骤中,模型结合近期高分辨率帧、有界历史观察和先前答案记录,判断新信息是否值得响应,并在需要时生成简明的答案。系统随后过滤连续重复的答案,同时保留原始输出作为后续决策的上下文。在没有任何参数更新或辅助触发的情况下,采用4B骨干的SIMPLEPRO在OVO-Bench、StreamingBench-PO和ProactiveVideoQA等专业主动系统中持续优异表现。随着骨干网大小从 2B 增加到 32B,SIMPLEPRO 展现出整体性能提升。我们希望这一简单却有效的方法能成为主动视频理解的坚实基础,并将发布代码和推理流程,以支持未来的研究。
WAM-Policy: World Action Models with On-Policy Distillation for Quadrupedal Loco-Manipulation
WAM-Policy:带有政策上提炼的世界行动模型用于四足运动操控
- Authors: Junlin Long, Xu Deng, Yiran Wang, Yue Yang, Yang Zhao, Zeyu Zhang
- Subjects: Subjects:
Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.24011
- Pdf link: https://arxiv.org/pdf/2611.24011
- Abstract
Arm-free quadrupeds can perform loco-manipulation, walking on three legs while manipulating with a front leg, yet no existing policy for it follows language instructions. World action models (WAMs) have emerged as effective language-conditioned manipulation policies, but are pretrained for robot arms and sample slowly. WAM-Policy turns one WAM into a quadrupedal loco-manipulation policy via two stages. The first, Cross-Morphology Policy Adaptation and Acceleration, adapts the arm-centric WAM to quadrupeds and distills the transferred policy into a student with one video and two action steps by on-policy self-distillation (OPSD). This student, however, is bounded by its teacher, which only imitates demonstrations. The second stage, Task-Specific Improvement and Merging, therefore tunes the teacher by reinforcement learning (RL) and merges the slow specialists into one few-step student by multi-teacher on-policy distillation (M-OPD). On QLM-Bench, ten simulated tasks with 1,000 demonstrations synthesized without teleoperation, the transferred policy reaches 77.0%, and the OPSD student keeps 76.7% while sampling 14 times faster. Task-specific RL and M-OPD lift it to 85.0%. We also deploy the OPSD student on a real Unitree Go2 Air. WAM-Policy thus achieves a stronger quadrupedal policy at a fraction of the inference cost.
- 中文摘要
无臂四足动物可以执行三条腿行走、前腿操作的操作操作,但目前尚无遵循语言指令的相关策略。世界动作模型(WAMs)已经作为有效的语言条件操作策略出现,但它们已预先训练用于机器人手臂,且采样速度较慢。WAM-Policy通过两个阶段将一个WAM转化为四足操作策略。第一阶段,跨形态策略适应与加速,将以手臂为中心的WAM适配为四足动物,并通过政策内自我提炼(OPSD)将转移的策略提炼成一个学生,包含一个视频和两个行动步骤。然而,该学生受限于其教师,而教师仅模仿演示。第二阶段——任务专项改进与融合——通过强化学习(RL)调校教师,并通过多教师的政策精炼(M-OPD)将慢的专家合并为一个“少步学生”。在 QLM-Bench 上,十个模拟任务、1,000 个演示,在无远程操作的情况下合成,转移的策略达到 77.0%,OPSD 学生保持 76.7%,同时采样速度提升 14 倍。任务专属 RL 和 M-OPD 则将该比例提升至 85.0%。我们还将 OPSD 学生部署在真实的 Unitree Go2 Air 上。因此,WAM-Policy 实现了更强的四足策略,且推理成本极低。
Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts
重播,不要重采样:长期远程学习专家的非策略整合
- Authors: Zhenrui Yue, Gongxiaohui Chen, Huimin Zeng, Honglei Zhuang, Jiashuo Sun, Zhen Han, Dong Wang
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.24059
- Pdf link: https://arxiv.org/pdf/2611.24059
- Abstract
Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD). Such consolidation is expensive: the student samples in every environment again, an expert scores every sampled token, and the student inherits the drift of the final experts. Yet we argue that the experts' RL runs have already produced what consolidation needs: rollouts with their sampling probabilities and rewards, in the order the experts learned from them. We propose multi-expert replay policy distillation (MRPD), which replays these training logs through the student in the experts' order. At each step, the policy gradient is recomputed on these scored trajectories, and the final experts' distributions are distilled at the logged prefixes. Following the experts' own trajectory reduces the training-inference mismatch and keeps the student within the clip range, so the replay is off-policy in origin but quasi-on-policy throughout consolidation. MRPD samples nothing and calls no environment or reward model: the trainer is the sole process, and consolidating the experts costs only about a seventh of the compute used to train them. On three long-horizon agentic domains (reasoning, search and SQL), MRPD yields the best student model, substantially outperforming online RL baselines, and surpassing MOPD from the same experts at a fraction of MOPD's cost. Our analysis on the same logs shows that the replayed trajectories carry most of the gain, and that both the policy gradient and the distillation term add to it. Most interestingly, the order of the logs matters more than their amount or rewards, suggesting that expert training logs are a curriculum, not a data mixture.
- 中文摘要
Frontier 的后期培训遵循一个常见的配方:每个领域先用强化学习(RL)培训一名专家,然后通过多教师策略提炼(MOPD)将专家整合到一个模型中。这种整合成本高昂:学生在每个环境中重新采样,专家对每个抽样标记进行评分,学生继承最终专家的漂移。然而我们认为,专家们的强化学习运行已经产生了整合所需的内容:按照专家从中学习的顺序,推出带有抽样概率和奖励的推广。我们提出多专家回放策略提炼(MRPD),即按照专家顺序通过学生重放这些培训日志。每一步,策略梯度都会在这些评分轨迹上重新计算,最终专家的分布则按记录的前缀提炼。遵循专家自身轨迹可以减少训练与推断之间的不匹配,并让学生保持在剪辑范围之内,因此回放在非策略性起点,但在整合过程中几乎是策略性的。MRPD不采样,也不调用环境或奖励模型:培训器是唯一的过程,整合专家的成本仅为训练他们所用计算量的约七分之一。在三个长期视野的代理领域(推理、搜索和SQL)上,MRPD提供了最佳的学生模型,显著优于在线RL基线,并以MOPD的一小部分成本超越了同一专家的MOPD。我们对同一日志的分析显示,重放轨迹承担了大部分收益,且策略梯度和提炼项对此都有额外贡献。最有趣的是,日志的顺序比数量或奖励更为重要,这表明专家培训日志是一种课程体系,而非数据混合体。
VESPA: Verifiable Symbolic Policy Adaptation for Safe Reinforcement Learning
VESPA:安全强化学习所需的可验证符号策略适应
- Authors: Paul König, Sascha Marton, Stefan Lüdtke, Christian Bartelt
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.24083
- Pdf link: https://arxiv.org/pdf/2611.24083
- Abstract
Safe reinforcement learning (RL) aims to prevent agents from taking actions that are dangerous or destructive to their environment. This is especially important for safety-critical domains, such as robotics or autonomous driving. Formal verification can be used to prove agent safety through safety constraints. However, in practice, many policies do violate safety constraints, and thus are verified to be unsafe. Since the goal is to obtain a safe policy, we would need to retrain a new policy from scratch and then verify it again until a candidate happens to pass. Both training and verification are usually not cheap, which makes this process inefficient. We introduce VESPA ($\textbf{VE}$rifiable $\textbf{S}$ymbolic $\textbf{P}$olicy $\textbf{A}$daptation), a framework for targeted fine-tuning of decision tree policies. We show that using counterexamples for policy adaptation can make existing unsafe policies safe, or at least make considerable progress towards safety. Targeted fine-tuning of the existing policy on those counterexamples enables it to learn how to avoid violating safety constraints. We thus use the result of a formal verification as a training signal for fine-tuning. On CartPole, VESPA raises the mean number of verified safe steps from 63.8 to at least 92.7, reaching after two iterations what the same loop without counterexamples reaches after ten. On MountainCarContinuous, VESPA turns an unsafe policy into a safe policy in 53 of 60 runs. We conclude that counterexamples from a verifier are an effective training signal, which can be utilized to systematically improve the safety of policies. Our implementation is available at: this https URL
- 中文摘要
安全强化学习(RL)旨在防止代理采取对其环境有害或破坏性的行为。这对于安全关键领域尤为重要,如机器人或自动驾驶。形式验证可以通过安全约束来证明代理的安全。然而,在实践中,许多策略确实违反了安全约束,因此被验证为不安全。由于目标是获得一个安全的策略,我们需要从零重新训练一个新策略,然后再次验证,直到候选策略恰好通过为止。训练和验证通常都不便宜,这使得这一过程效率低下。我们介绍了 VESPA($\textbf{VE}$rifiable $\textbf{S}$ymbolic $\textbf{P}$olicy $\textbf{A}$daptation),这是一个用于决策树策略精细调的框架。我们证明,利用对策调整进行反例可以使现有不安全政策变得安全,或者至少在安全方面取得显著进展。针对这些反例对现有政策进行有针对性的微调,使其能够学习如何避免违反安全约束。因此,我们将形式验证的结果作为微调训练信号。在 CartPole 上,VESPA 将已验证安全步的平均数从 63.8 提高到 92.7,经过两次迭代后达到的,相当于无反例的同一循环在十次后达到的水平。在 MountainCarContinuous 上,VESPA 在 60 次运行中的 53 次将不安全策略转化为安全策略。我们得出结论,验证器提供的反例是一种有效的训练信号,可用于系统性提升策略安全性。我们的实现可在以下网址获取:https URL
From Restoration to Redrawing: Revitalizing Ancient Chinese Characters by SVG Code Generation and Editing
从修复到重绘:通过SVG代码生成与编辑复兴古代汉字
- Authors: Weiguang Zhang, Linhua Dong, Bin Dong, Shanshan Jiang, Maizhen Ning, Xiaoqiang Kang, Kun Zhang, Hao Xu, Xiaowei Huang, Wei Wang, Kaizhu Huang, Qiufeng Wang
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.24084
- Pdf link: https://arxiv.org/pdf/2611.24084
- Abstract
Hanjian manuscripts written on Han Dynasty bamboo and wooden slips are invaluable heritage of ancient Chinese writing but suffer from damaged ink and incomplete strokes. Existing image-to-image restoration methods may alter observed strokes and offer limited control over stroke geometry. Inspired by experts' manual redrawing practices, we address these issues by introducing \textbf{ReDraw}, a two-stage vision-language model(VLM) that reformulates character restoration as Scalable Vector Graphics(SVG) code generation and editing. Stage one targets faithful vectorization of observed strokes geometry or conservative tracing of mildly degraded strokes. Guided by skeleton chain-of-thought reasoning and reinforcement learning with structure-aware rewards, stage two predicts skeletons before contours to redraw severely incomplete strokes by appending paths while keeping existing SVG paths unchanged. To support training, we develop a multi-layer data synthesis pipeline that disentangles ink, material, and degradation to generate 1.2M samples, and derive up to 6,400 adaptation pairs from 320 annotated real glyphs. Experiments demonstrate synthetic restoration performance and adaptability to real Hanjian glyphs. Code and demo will be available upon acceptance.
- 中文摘要
汉代竹编和木片上的汉字手稿是中国古代文字的宝贵遗产,但其墨迹受损且笔触不完整。现有的图像对图像修复方法可能改变观察到的笔画,并对笔画几何体的控制有限。受专家手工重绘实践的启发,我们通过引入 \textbf{ReDraw}——一种两阶段视觉语言模型(VLM)来解决这些问题,它将字符恢复重新表述为可扩展矢量图形(SVG)代码生成与编辑。第一阶段目标是忠实地向量化已观察到的笔画几何,或对轻度退化的笔画进行保守描摹。第二阶段通过骨架思维链推理和结构感知奖励的强化学习,预测骨骼在轮廓前通过附加路径重新绘制严重不完整的笔画,同时保持现有SVG路径不变。为支持训练,我们开发了多层数据合成流程,将墨水、材料和劣化过程解开纠缠,生成120万个样本,并从320个带注释的真实字形中推导出多达6400对适配对。实验展示了合成修复性能以及对真实汉字字的适应性。代码和演示将在接受后提供。
ViStep-RL: Stepwise Reinforcement Learning for Long-Horizon Visual Agents
ViStep-RL:面向长期视野视觉代理的分步强化学习
- Authors: Junyang Chen, Yao Qin, Xiao Li, Ningchao Wang, Yiguo Qiao, Wenbin Li, Xin Geng
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.24170
- Pdf link: https://arxiv.org/pdf/2611.24170
- Abstract
Long-horizon visual agents must do more than inspect images: they must turn observations into evidence that remains usable as a task unfolds. Existing active-perception methods teach agents where and how to look, while a final-answer reward gives little structure for deciding which observations satisfy dependent subgoals or should be retained for later use. We present ViStep-RL (Visual Stepwise Reinforcement Learning), which models execution as a dependency-linked graph of evidence-bearing subgoals. A verifier tracks which subgoals are supported by valid evidence, and changes in graph potential provide a net-progress learning signal; an active memory protocol preserves evidence across turns and links retrieval to downstream use. On ViInteract, ViStep-RL reaches 60.0% strict completion with Qwen2.5-VL-7B, compared with 10.0% for the base model and 46.5% after supervised fine-tuning. Controlled studies show that incremental task-state rewards outperform cumulative rewards by 3.2 points, while active evidence writing improves strict completion by 7.75 points over automatic writing. Under a common interaction framework, the task-trained ViStep-RL policy achieves 59.5% answer accuracy, 24.0 points above the strongest of six general-purpose reference models. Held-out episodes confirm the gain at both Qwen scales, and gains extend to three external visual reasoning benchmarks.
- 中文摘要
长视野视觉代理必须做的不仅仅是检查图像:它们必须将观察转化为在任务展开时仍可使用的证据。现有的主动感知方法教会代理在哪里以及如何观察,而最终答案奖励则难以为决定哪些观察满足依赖子目标或应保留以供后续使用提供结构。我们介绍了 ViStep-RL(视觉分步强化学习),它将执行建模为一个由证据子目标组成的依赖关联图。验证器跟踪哪些子目标得到了有效证据支持,图势变化则提供净进展学习信号;主动记忆协议在各个回合间保存证据,并将检索与后续应用联系起来。在 ViInteract 上,ViStep-RL 在 Qwen2.5-VL-7B 上可达到 60.0% 的严格完成率,而基础模型为 10.0%,监督微调后为 46.5%。受控研究显示,增量任务状态奖励优于累积奖励 3.2 分,而主动证据写作则使严格完成率比自动写作高出 7.75 分。在通用交互框架下,任务训练的 ViStep-RL 策略实现了 59.5% 的答案准确率,比六个通用参考模型中最强的高出 24.0 分。持续进行的集数在 Qwen 两个量表上都证实了这一提升,且提升幅度延伸到三个外部视觉推理基准。
CommDreamer: Radio-Geometric World Models for Mapless Cellular-Connected UAV Trajectory Planning
CommDreamer:无地图蜂窝连接无人机轨迹规划的无线电几何世界模型
- Authors: Yuqi Ping, Tianhao Liang, Tingting Zhang
- Subjects: Subjects:
Robotics (cs.RO); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.24244
- Pdf link: https://arxiv.org/pdf/2611.24244
- Abstract
Reliable cellular connectivity and collision-free flight are essential for unmanned aerial vehicle (UAV) operations in low-altitude urban canyons, where dense 3D buildings obstruct flight clearance, shadow serving base stations, and create strong co-channel interference at street intersections. Existing planners either require prior 3D city meshes and channel knowledge maps unavailable onboard, or rely on model-free reinforcement learning with low sample efficiency and poor cross-city generalization. We propose CommDreamer, a mapless radio-geometric world model that fuses raw onboard 3D LiDAR scans, multicell cellular reports, and shift-invariant relative kinematics to jointly infer local obstacle clearance and directional signal-to-interference-plus-noise ratio (SINR) dynamics. Using a categorical recurrent state-space model, CommDreamer compresses multimodal measurement histories into a coordinate-invariant latent belief, enabling the UAV to foresee deep shadow zones and train a lightweight online policy purely via multi-step latent imagination without prior maps or antenna patterns. Zero-shot evaluations across unseen urban morphologies in our open-source ray-tracing simulator, AeroCity-RT, show that CommDreamer converges within 150 episodes where model-free reinforcement learning fails to complete flights, achieving 100.0% mission success with zero collisions while lifting median route SINR to 8.02 dB and cutting effective SINR outage by over 50% relative to radio-blind world-model and reactive LiDAR baselines.
- 中文摘要
可靠的蜂窝连接和无碰撞飞行对于低海拔城市峡谷中的无人机(UAV)操作至关重要,因为密集的三维建筑阻碍飞行间隙、为基站遮蔽阴影,并在街道交叉口产生强烈的同频干扰。现有规划器要么需要机载无法获得的三维城市网格和信道知识图,要么依赖无模型强化学习,但采样效率低且跨城泛化能力差。我们提出了CommDreamer,这是一种无地图的无线电几何世界模型,融合了原始机载3D LiDAR扫描、多单元细胞报告和不变相对运动学,共同推断局部障碍物清除和方向性信干扰加噪比(SINR)动力学。通过类别递归状态-空间模型,CommDreamer 将多模态测量历史压缩为坐标不变的潜在信念,使无人机能够预见深阴影区,并仅凭多步潜在想象,无需先验地图或天线模式,就能训练出轻量级的在线策略。在我们的开源光线追踪模拟器 AeroCity-RT 中对未见城市形态进行零射击评估显示,CommDreamer 在 150 个无模型强化学习未能完成飞行的情节内收敛,实现 100.0% 任务成功率且零碰撞,同时将中位航线 SINR 提升至 8.02 dB,并将有效 SINR 中断频率相较于无线电盲世界模型和反应式 LiDAR 基线降低超过 50%。
Do Thinking VLMs Need to Keep Looking at Long Videos? Write It Down, Then Look Away
思考型 VLM 需要继续看长视频吗?把它写下来,然后移开视线。
- Authors: Pengyu Li, Zhitao Gao, Lingling Zhang, Heng Wang, Chengqian Ma, WenHao Wang, Wenxuan Huang, Chen Zheng, Jun Liu
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.24259
- Pdf link: https://arxiv.org/pdf/2611.24259
- Abstract
Vision-language models (VLMs) reason about long videos over hundreds to thousands of tokens. Every decoding step re-reads the key-value (KV) cache of the entire video. Must the model keep looking while it reasons? In LookAway, the model first writes timestamped notes on what the question needs and then puts the video away. Once the notes close, the inference engine hides the video's KV cache for the rest of the response. Across both hybrid and pure-attention VLMs, dependence on the video is concentrated early in the response. On Qwen3.6-27B, writing notes preserves accuracy better than free reasoning at the same withdrawal step. Replacing them with another video's notes harms accuracy, whereas keeping the video visible alongside the model's own notes adds no significant benefit. Without additional training, LookAway nearly matches full-video thinking accuracy at 0.12-0.13 times its modeled decode cost. To shorten this initial look, we fine-tune the model on its own correct LookAway responses. Reinforcement learning then rewards correct answers while penalizing KV reads. The trained model puts the video away after a median of 66-68 tokens. Across three held-out benchmarks, it writes more tokens than an identically trained full-video model yet reads only 27-35% as many KV positions at matched accuracy.
- 中文摘要
视觉语言模型(VLMs)会对数百到数千个令符的长视频进行推理。每一步解码都会重新读取整个视频的键值(KV)缓存。模型在推理时必须继续查看吗?在 LookAway 中,模型首先写下题目所需内容的时间戳笔记,然后将视频收起来。笔记关闭后,推理引擎会隐藏视频的 KV 缓存,用于其余的回答过程。无论是混合型还是纯注意力型 VLM,对视频的依赖都集中在回答的早期阶段。在Qwen3.6-27B中,写笔记相比自由推理在同一撤退步骤中更能保持准确性。用另一段视频的笔记替代笔记会损害准确性,而让视频与模型自身笔记并列可见则无显著益处。在没有额外训练的情况下,LookAway 几乎可以达到全视频思维的准确率,其解码成本是 0.12-0.13 倍。为了缩短初始观察时间,我们根据模型自身正确的 LookAway 响应微调了模型。强化学习随后奖励正确答案,同时惩罚 KV 读取。训练好的模型在中位数 66-68 个令牌后将视频收起来。在三个测试中,它写入的标记数量超过完全训练的全视频模型,但在匹配准确率下读取的 KV 位置仅为 27-35%。
Beyond Corpus-Driven Imitation: Evolving Musical Mindsets in the Agentic Orchestra via Quantum Reinforcement Learning
超越语料库驱动的模仿:通过量子强化学习在能动管弦乐中演变的音乐思维方式
- Authors: Eduardo Reck Miranda, Moein Fahmideh Vatandoost
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Quantum Physics (quant-ph)
- Arxiv link: https://arxiv.org/abs/2611.24290
- Pdf link: https://arxiv.org/pdf/2611.24290
- Abstract
This paper presents a framework for computer-aided music composition and performance based on quantum reinforcement learning and multi-agent interaction. We introduce the agentic orchestra, where musical agents evolve distinct mindsets through trial-and-error engagement with a musical environment, rather than through training on large datasets. The agents are implemented as parameterised quantum circuits. They evolve distinct compositional behaviours, referred to as mindsets. This enables the exploration of musical decision spaces using quantum superposition and entanglement within a hybrid quantum-classical loop. We present two complementary agent architectures: atomistic agents, which process pitch and rhythm independently, and holistic agents, which handle pitch and duration as unified note events. Mindsets are shaped through musically grounded reward functions, which can be combined and weighted to develop different mindsets across agents. The resulting system positions quantum computing agents less as autonomous composers and more as adaptive musical partners, each developing its own musical personality through interaction rather than imitation. Demonstrations show how agents generalise their learned behaviour to previously unseen material, operating in interactive and offline batch modes using MIDI-encoded music. While quantum computers are still in development, experiments conducted on IBM quantum hardware confirmed that musically meaningful learning is achievable with existing quantum computers. For composers and performers, the agentic orchestra opens new possibilities for emergent, process-oriented musical creativity, offering a scalable foundation for quantum computer-aided music composition as quantum computing technology continues to advance.
- 中文摘要
本文提出了基于量子强化学习和多智能体交互的计算机辅助音乐作曲与演奏框架。我们介绍了代理管弦乐团,音乐代理通过与音乐环境的反复试验演化出不同的思维模式,而非通过大规模数据集训练。这些代理被实现为参数化的量子电路。它们演化出不同的作曲行为,称为心态。这使得在混合量子-经典循环中利用量子叠加态和纠缠探索音乐决策空间成为可能。我们介绍了两种互补的智能体架构:原子智能体,独立处理音高和节奏;整体智能体,将音高和时值作为统一的音符事件处理。心态通过音乐基础的奖励函数塑造,这些函数可以组合并加权,以在不同智能体之间发展不同的思维模式。由此形成的系统将量子计算代理定位为适应性音乐伙伴,而非自主作曲家,每个代理通过互动而非模仿发展出独特的音乐个性。演示展示了智能体如何将所学行为推广至此前未见过的素材,通过MIDI编码音乐以互动和离线批处理模式运行。虽然量子计算机仍在开发中,但在IBM量子硬件上进行的实验证实,现有量子计算机完全可以实现具有音乐意义的学习。对于作曲家和演奏者来说,代理管弦乐为涌现的、以过程为导向的音乐创造打开了新可能,为量子计算机辅助音乐创作提供了可扩展的基础,随着量子计算技术的不断进步。
PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
PIPETTE:为强大的LLM代理提供快速注入紫色组队
- Authors: Lipeng He, Yihan Wang, Jiawen Zhang, N. Asokan
- Subjects: Subjects:
Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.24362
- Pdf link: https://arxiv.org/pdf/2611.24362
- Abstract
Large language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR). Model developers use the attacks discovered during internal red-teaming to harden agents through adversarial training before deployment, a collaborative offensive-defensive process known in cybersecurity as purple-teaming. However, we show that achieving high ASR during red-teaming does not guarantee robustness gains from subsequent adversarial training. Specifically, we find that two state-of-the-art RL-based red-teaming methods, RL-Hammer and PISmith, suffer from mode collapse, repeatedly discovering similar attack strategies whose additional successes add no additional robustness improvements to adversarial training. To address this gap, we introduce PIPETTE, a unified prompt injection purple-teaming framework that effectively converts red-teaming discoveries into defender robustness improvements. PIPETTE employs a dictionary learning-based reward that encourages broader coverage of attack strategies during red-teaming. It also integrates an on-policy adversarial training pipeline tailored to IPI defense, enabling agents to resist attacks while completing legitimate user tasks. Extensive evaluations across agentic benchmarks (AgentDojo and ASB) demonstrate that PIPETTE produces defenders that achieve near-zero attack success rates on static attacks and can withstand strong adaptive attacks, while maintaining benign task utility close to that of the undefended baseline.
- 中文摘要
大型语言模型(LLM)代理经常与不受信任的外部数据交互,使其暴露在间接提示注入(IPI)攻击之下,这些攻击会劫持代理执行攻击者选择的任务。为了大规模发现这些漏洞,自动化红队方法通常利用强化学习(RL)来生成实现高攻击成功率(ASR)的攻击。模型开发者利用内部红队攻击通过部署前的对抗训练强化代理,这一在网络安全领域称为紫队的协同攻防过程。然而,我们表明,在红队中实现高ASR并不保证后续对抗训练带来的稳健性提升。具体来说,我们发现两种最先进的基于RL的红队方法——RL-Hammer和PISmith——存在模式崩溃的问题,反复发现相似的攻击策略,而这些策略的额外成功并未为对抗训练带来额外的稳健性提升。为弥补这一空白,我们引入了PIPETTE,一套统一的提示注入紫色团队框架,有效将红队发现转化为防御方的稳健性提升。PIPETTE 采用基于词典学习的奖励机制,鼓励在红队合作期间更广泛地涵盖攻击策略。它还集成了针对IPI防御量身定制的策略内对抗训练流程,使代理能够在完成合法用户任务的同时抵御攻击。对代理基准测试(AgentDojo 和 ASB)的广泛评估表明,PIPETTE 能够在静态攻击中实现近乎零的攻击成功率,并能抵御强有力的自适应攻击,同时保持接近未防御基线的无害任务效用。
Cooperative UAV and Ground Station for Robust MEC of Dynamic Aquatic IoT: A Generative AI-Enhanced Approach
协作式无人机与地面站,实现动态水域物联网的强大MEC:一种生成式AI增强方法
- Authors: Jiahao You, Ziye Jia, Chao Dong, Qihui Wu, Zhu Han
- Subjects: Subjects:
Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.24466
- Pdf link: https://arxiv.org/pdf/2611.24466
- Abstract
The increasing aquatic Internet of Things (AIoTs) require computational support and coverage in applications such as maritime search and rescue. Unmanned aerial vehicles (UAVs) can offer low-cost, flexible aerial services, and ground stations (GSs) can provide powerful supports, which can cooperate to help the AIoTs in complex scenarios. However, the collaboration between UAVs and GSs for AIoTs faces challenges of task uncertainties, AIoTs trajectory uncertainties, heterogeneities, and limited computational resources. To address these issues, we propose a cooperative UAV and GS based robust multi-access edge computing (MEC) framework to assist AIoTs in completing computational tasks. Specifically, we formulate the optimization problem of joint task offloading and UAV trajectory to minimize the total execution time, which is in the form of mixed integer nonlinear programming and NP-hard to tackle. Therefore, we propose the algorithm of generative artificial intelligence-enhanced heterogeneous agent proximal policy optimization (GAI-HAPPO). The proposed algorithm integrates GAI models to enhance the actor network ability to model complex environments and extract high-level features, thereby allowing the algorithm to predict uncertainties and adapt to dynamic conditions. Additionally, GAI stabilizes the critic network, addressing the instability of multi-agent reinforcement learning approaches. Finally, extensive simulations demonstrate that the proposed algorithm outperforms the existing benchmark methods, thus highlighting the potentials in tackling intricate, cross-domain issues in the considered scenarios.
- 中文摘要
日益增长的水下物联网(AIoT)在海上搜救等应用中需要计算支持和覆盖。无人机(UAV)可以提供低成本、灵活的空中服务,地面站(GS)则能提供强大的支持,这些设备可以协同帮助AIoT在复杂场景中发挥作用。然而,UAV与GS在AIoT上的协作面临任务不确定性、AIoT轨迹不确定性、异构性以及有限计算资源等挑战。为解决这些问题,我们提出了一个基于无人机与GS协作的稳健多访问边缘计算(MEC)框架,以协助AIoT完成计算任务。具体而言,我们提出了联合任务卸载和无人机轨迹的优化问题,以最小化总执行时间,这一问题以混合整数非线性规划和NP难度形式出现。因此,我们提出了生成式人工智能增强的异构代理近端策略优化算法(GAI-HAPPO)。该算法集成了GAI模型,增强了演员网络建模复杂环境和提取高层特征的能力,从而使算法能够预测不确定性并适应动态条件。此外,GAI稳定了批评网络,解决了多智能体强化学习方法的不稳定性问题。最后,大量模拟表明,所提算法优于现有基准方法,凸显了在处理复杂跨领域问题的潜力。
Self-Rewarded Reinforcement Learning for Better-Structured Medical Explanations in Small Language Models
自我奖励强化学习,在小型语言模型中实现更结构化的医学解释
- Authors: Prakhar Mishra, Bhavya Yajush Awasthi, Tanuja, Nishant
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.24650
- Pdf link: https://arxiv.org/pdf/2611.24650
- Abstract
Small language models (SLMs) can run on commodity hardware and support privacy-preserving, on-premises clinical deployment, but their medical explanations are often short and poorly structured compared with large language models. Recent supervised approaches such as Meerkat transfer chain-of-thought reasoning into 7B-8B backbones, yet their training objective does not directly target the style or organization of the generated rationales. Building on the instruction-tuned Mistral-7B backbone, we investigate whether a lightweight reinforcement learning (RL) step can improve explanation structure without human preference this http URL apply Proximal Policy Optimization (PPO) with LoRA adaptation to 2,000 Meerkat-derived medical prompts, guided by a simple self-supervised reward that favors longer, lexically diverse, and well-punctuated responses while penalizing very short ones. Training for about 170 PPO steps on a single NVIDIA A100 GPU yields a stable increase in the average reward. Quantitative and qualitative comparisons show that the RL-tuned model produces longer, more organized, multi-point explanations while preserving the underlying medical this http URL discuss the promise and the limitations of self-rewarded RL for shaping the style of medical explanations.
- 中文摘要
小型语言模型(SLM)可以在通用硬件上运行,并支持保护隐私的本地临床部署,但与大型语言模型相比,它们的医疗解释往往较短且结构较差。近期的监督式方法如猫鼬将思维链推理转移到7B-8B骨干网,但其训练目标并未直接针对生成逻辑的风格或组织。基于指令调优的 Mistral-7B 骨干,我们研究轻量级强化学习(RL)步骤是否能在不依赖人类偏好的情况下改进解释结构。此 http URL 应用近端策略优化(PPO)并配合 LoRA 适应 2000 个狐獴衍生医疗提示,其指导方式为简单的自我监督奖励,该奖励有利于较长、词汇多样且标点清晰的回答,而惩罚非常短的回答。在单个NVIDIA A100显卡上训练约170个PPO步骤,平均奖励能稳定提升。定量和定性比较表明,RL调优模型能在保持医学基础的同时,产生更长、更有条理、多点的解释。http URL讨论了自我奖励RL在塑造医学解释风格方面的潜力与局限。
Can Bandits Learn to Share a Channel? Distributed Multi-Armed Bandits for Slotted ALOHA
强盗能学会共享频道吗?分布式多臂强盗用于槽式ALOHA
- Authors: Kamil Szczech, Maksymilian Wojnar, Krzysztof Rusek, Katarzyna Kosek-Szott, Szymon Szott
- Subjects: Subjects:
Networking and Internet Architecture (cs.NI)
- Arxiv link: https://arxiv.org/abs/2611.24676
- Pdf link: https://arxiv.org/pdf/2611.24676
- Abstract
Slotted ALOHA networks constitute a playground for research on machine learning (ML)-based channel access schemes. We consider ML agents deployed in such networks and operating under severe constraints: fully distributed (independent operation), fully online (no pre-training), aperiodic (no fixed schedule), and with no explicit coordination (no signaling). We ask whether a multi-armed bandit (MAB), a minimal reinforcement learning agent, can learn the optimal policy for a capture-free slotted ALOHA system. We evaluate several MAB algorithms on aggregate network throughput and per-node fairness. Our best-performing agent is based on a Softmax bandit, which we call SALSA (Softmax ALOHA Learning for Shared Access). SALSA learns a per-node transmission policy close to optimal, with throughput near the theoretical maximum for fair random access. We achieve performance comparable to a competing state-of-the-art solution, but without requiring a priori knowledge of the network size.
- 中文摘要
槽位 ALOHA 网络是基于机器学习(ML)的信道接入方案研究的乐园。我们考虑部署在此类网络中并处于严苛约束下的机器学习代理:完全分布式(独立运行)、完全在线(无预训练)、非周期性(无固定时间表)以及无明确协调(无信令)。我们探讨多臂盗贼(MAB)——一种最小强化学习代理——是否能够学习无捕获的槽隙 ALOHA 系统的最佳策略。我们评估了多种 MAB 算法在总体网络吞吐量和每节点公平性方面的表现。我们表现最好的代理基于 Softmax 盗贼,我们称之为 SALSA(Softmax ALOHA Learning for Shared Access)。SALSA 学习的每节点传输策略接近最优,吞吐量接近公平随机访问的理论最大值。我们实现的性能可与竞争对手的尖端解决方案媲美,但无需事先了解网络规模。
DrawScore: Evaluating Generated CAD Models against Engineering Drawings
DrawScore:将生成的 CAD 模型与工程图纸进行评估
- Authors: Haorui Chen, Shiteng Lu, Zhanyu Guo, Zaixi Li, Yunhe Ni, Yaosong Du, Yichen Xiang, Xiande Huang, Rui Huang
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Software Engineering (cs.SE)
- Arxiv link: https://arxiv.org/abs/2611.24681
- Pdf link: https://arxiv.org/pdf/2611.24681
- Abstract
Automatic CAD generation is commonly evaluated with volumetric IoU, Chamfer Distance, and surface F-score, which barely penalize local errors that occupy only a small fraction of a part's volume or surface area, such as a misplaced hole or a slot on the wrong face, even though such errors are critical to manufacturing. We formalize these metrics in a unified integral form and show that the weight of a local error decays as the $(k+q)$-th power of its size, where $k$ is the dimension of the integration domain and $q$ is determined by the penalty function. Guided by this analysis, we propose DrawScore, which projects both models into three orthographic views with visible and hidden lines, as in engineering drawings, and applies a saturating penalty along the projected curves, so that the weight of a local error decays only linearly. Controlled experiments confirm the predicted decay orders, and on outputs of existing CAD generation agents, DrawScore exposes errors in outputs that IoU rates as nearly perfect. DrawScore further improves generation as a reinforcement learning reward and, for CAD generation from engineering drawings, serves as a test-time verifier without 3D ground truth or an additional reward model. Our code is available at this https URL.
- 中文摘要
自动CAD生成通常通过体积IoU、倒角距离和表面F分数进行评估,这些分数对仅占零件体积或表面积一小部分的局部误差几乎不予惩罚,例如错位孔或错误面上的槽,尽管这些误差对制造至关重要。我们将这些指标形式化为统一积分形式,并证明局部误差的权重随其大小的$(k+q)$次幂衰减,其中$k$为积分域维数,$q$由惩罚函数决定。基于这一分析,我们提出了DrawScore,该方法将两个模型投影为三个正交视图,带有可见线和隐线,如工程图纸中所示,并沿投影曲线施加饱和惩罚,使局部误差权重仅线性衰减。受控实验验证了预测的衰减顺序,在现有 CAD 生成代理的输出中,DrawScore 会揭示 IoU 评为近乎完美的输出错误。DrawScore 进一步提升生成效果,作为强化学习奖励,并且在工程图纸生成 CAD 时,可作为测试时验证器,无需 3D 真实数据或额外奖励模型。我们的代码可在 https 网址获取。
Depth Before Breadth: Reverse-to-Forward Curriculum with Advantage Decomposition for LLM Agent
深度重于广度:逆向向课程体系与优势分解适用于LLM代理
- Authors: Siyuan Du, Yuxin Liu, Meng Tang, Haoran Li, Yechao Bai, Feiyang Pan, Jia He
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.24697
- Pdf link: https://arxiv.org/pdf/2611.24697
- Abstract
Reinforcement learning (RL) has become a common approach for training large language model (LLM) agents on long-horizon tasks. However, learning from sparse terminal rewards makes credit assignment and exploration difficult. We approach these challenges through a policy-induced capability region, the set of states from which the current policy can reliably complete a task. Near its boundary, an agent may need only a few actions to reach a more solvable state, even when the goal remains distant. This reduces long-horizon learning to two coupled subproblems of reaching that state and completing the task from there. We address them with paired advantages and a progressive reverse-to-forward curriculum. The advantages provide separate signals for reaching more solvable states and completing the task from them, while the curriculum expands the capability region backward along successful reference trajectories and then into off-reference states through on-policy exploration. On ALFWorld and WebShop, our method outperforms competitive agentic RL methods at both model scales, achieving success rates of 90.1%/96.1% and 80.2%/82.6% with Qwen2.5-1.5B/7B-Instruct, respectively.
- 中文摘要
强化学习(RL)已成为训练大型语言模型(LLM)代理执行长期任务的常用方法。然而,从稀疏的终端奖励中学习使得学分分配和探索变得困难。我们通过策略诱导的能力区域来应对这些挑战,即当前策略能够可靠完成任务的一组状态。在其边界附近,代理可能只需少量操作即可达到更可解的状态,即使目标仍然遥远。这将长期视野学习简化为两个耦合的子问题:达到该状态并从那里完成任务。我们通过配对优势和渐进的逆向前向课程来应对这些问题。优势为达到更多可解状态并完成任务提供了独立信号,而课程则沿成功的参考轨迹向后扩展能力区域,然后通过策略探索扩展到非参照状态。在 ALFWorld 和 WebShop 上,我们的方法在两个模型尺度上都优于竞争性的主体强化学习方法,在 Qwen2.5-1.5B/7B-Instructions 上分别取得了 90.1%/96.1% 和 80.2%/82.6% 的成功率。
HINT-WAM: Hindsight-Informed Training for World Action Models
HINT-WAM:面向世界行动模式的事后诸葛明培训
- Authors: Phuong Tuan Dat, Qi Li, Xinchao Wang
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.24789
- Pdf link: https://arxiv.org/pdf/2611.24789
- Abstract
World action models (WAMs) have shown promising generalization and data efficiency in robot manipulation by coupling action generation with predictions of future visual dynamics. Translating these capabilities into reliable task execution, however, remains a challenge for existing post-training methods. Offline supervised fine-tuning provides dense supervision on expert trajectories, but its flow-matching action expert is trained only on noised demonstration actions and never on the intermediate states produced by its own sampler; task-specific adaptation can also compromise pretrained capabilities. On-policy reinforcement learning trains on the policy's own behavior, yet sparse task rewards make exploration and credit assignment difficult, while dense reward shaping still provides scalar evaluations rather than direct action targets. We introduce \textbf{HINT-WAM}, the first on-policy self-distillation framework that supervises the action expert on its own sampling trajectories using hindsight, the future video of the demonstration, as privileged information available to the WAM itself. For each demonstration observation, the student generates its action through its own denoising trajectory, and at the states along this trajectory the same model acts as a teacher conditioned additionally on this hindsight, providing a conditional action distribution as a dense training target, while the student sees only deployment-time inputs and needs no environmental rewards or simulator interaction. To transfer this supervision to a flow-matching action expert, HINT-WAM distills the teacher into the student with an energy score, a distribution-level objective that matches the teacher's action distribution from the student's own samples rather than regressing its velocity field. The project web is available in this https URL.
- 中文摘要
世界动作模型(WAMs)通过将动作生成与未来视觉动态预测结合,展示了机器人操作中的有前景的泛化性和数据效率。然而,将这些能力转化为可靠的任务执行,仍是现有后期训练方法的挑战。离线监督微调为专家轨迹提供了密集的监督,但其流程匹配动作专家仅对带噪声的演示动作进行训练,从不针对自身采样器产生的中间状态;任务特定适应也可能损害预训练能力。策略内强化学习基于策略自身行为进行训练,但稀疏的任务奖励使得探索和积分分配变得困难,而密集的奖励塑造仍提供标量评估而非直接行动目标。我们引入了 \textbf{HINT-WAM},这是首个在策略上的自我提炼框架,利用事后诸葛(未来演示视频)来监督行动专家自身的抽样轨迹,这些信息是 WAM 自身可获得的特权信息。对于每一次示范观察,学生通过自身的去噪轨迹生成行动,而在沿该轨迹的状态下,同一模型作为基于此事后诸葛亮的教师,提供条件性行动分布作为密集的训练目标,而学生则只看到部署时间的输入,无需环境奖励或模拟器交互。为了将这种督导传递给流程匹配动作专家,HINT-WAM 将教师与一个能量评分提炼成学生,这是一个与教师从自身样本中获得的行动分布匹配的分布级目标,而非回归其速度场。项目网站可在此 https 网址获取。
Data-driven simulation of the brain's response to stimulation: a system identification approach for closed-loop neuromodulation, piloted in adaptive deep brain stimulation
基于数据驱动的大脑对刺激反应的模拟:一种用于闭环神经调控的系统识别方法,在自适应深脑刺激中试点
- Authors: Matthias Dold, Joana Pereira, Marcus L.F. Janssen, Michael Tangermann
- Subjects: Subjects:
Human-Computer Interaction (cs.HC)
- Arxiv link: https://arxiv.org/abs/2611.24818
- Pdf link: https://arxiv.org/pdf/2611.24818
- Abstract
Closed-loop neuromodulation uses a physiomarker to control stimulation. Developing such controllers requires exploring a large space of strategies and parameters, which is often not feasible with the human-in-the-loop. Simulation environments offer an alternative. Adaptive deep brain stimulation (aDBS) for Parkinson's disease is a stereotypical application: while physiomarker identification progresses rapidly, clinically available control is limited to threshold and proportional strategies. We propose to identify dynamical models of the brain with and without deep brain stimulation (DBS) using the sparse identification of nonlinear dynamics (SINDy) framework. Identified ex post from existing recordings, such models are a middle ground between abstract bottom-up models and the realistic but difficult-to-access human-in-the-loop. Forecasting neural activity from real initial conditions, the models create a simulation environment for "what if" analyses of DBS changes. In a proof-of-concept with EEG data from a single subject, we train and evaluate threshold, proportional, proportional-integral-derivative (PID), and proximal policy optimization (PPO) control, aiming to have the simulated data classified as DBS ON at minimal energy. SINDy models fitted on spatially filtered EEG maintained DBS condition-specific characteristics. All aDBS controllers delivered DBS more efficiently than continuous DBS (cDBS), with up to 20.2% energy reduction. Proportional control yielded a small but significant gain over threshold control. PID and PPO achieved a comparable reward advantage over cDBS, with PPO stimulating more conservatively and PID saving the most energy. The approach thus proved feasible, and within this simulation environment, PID control is a strong a-priori choice, while the benefit of deep reinforcement learning may only emerge with richer physiomarker signals.
- 中文摘要
闭环神经调节利用物理标志物来控制刺激。开发此类控制器需要探索广泛的策略和参数领域,而这往往无法在人为“环中”的人力中实现。模拟环境提供了一种替代方案。适应性深脑刺激(aDBS)用于帕金森病是一种典型应用:虽然物理标志物识别进展迅速,但临床上可用的控制仅限于阈值和比例策略。我们提出利用稀疏非线性动力学识别(SINDy)框架,识别有深脑刺激和不使用深脑刺激的大脑动力学模型。这些模型是从现有记录中事后确定的,是介于抽象自下而上模型与现实但难以访问的人机循环之间的中间地带。这些模型通过从真实初始条件预测神经活动,为DBS变化的“假设”分析构建了模拟环境。在一次概念验证中,我们利用单个受试者的脑电图数据进行训练和评估阈值、比例、比例积分导数(PID)和近端策略优化(PPO)控制,旨在使模拟数据在最低能量下被归类为DBS ON。安装在空间滤波EEG上的SINDy模型保持了DBS条件特异性特征。所有aDBS控制器均比连续DBS(cDBS)更高效地实现DBS,能量减少率最高达20.2%。比例控制相较阈值控制带来了小幅但显著的提升。PID和PPO在奖励优势上相较于cDBS,PPO刺激更保守,PID节能最多。因此,这一方法被证明是可行的,在这一模拟环境中,PID控制是一个强有力的先验选择,而深度强化学习的益处可能只有在更丰富的物理标志物信号时才会显现。
DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play
DiWALI:通过人与机器人游戏中的潜在行动推断,在世界模型中实现扩散政策学习
- Authors: Akshay L Chandra, Iman Nematollahi, Lingyue Zhao, Daria Kulikova, Angela P. Schoellig, Fabien Despinoy, Abhinav Valada
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.24836
- Pdf link: https://arxiv.org/pdf/2611.24836
- Abstract
Learning robot manipulation is constrained by the availability of action-labeled robot data, for both imitation and world models. Humans can demonstrate skills in seconds, but their demonstrations lack the required robot actions. We introduce DiWALI, a framework that learns and improves diffusion policies within a world model trained on human and robot play. DiWALI infers latent actions from consecutive frames using an inverse dynamics model, enabling world model training without action labels, and grounds them in robot commands through a decoder trained separately on robot play. Skill policies are trained over latent actions from action-free human demonstrations and improved with reinforcement learning in imagination. On the CALVIN benchmark, DiWALI policies trained solely from human demonstrations substantially outperform baselines trained with robot demonstrations. Incorporating human play into the world model training also yields stronger policies after fine-tuning than using the same amount of robot play alone. Experiments on a physical robot show that skills learned primarily from human demonstrations are further improved in imagination with a world model trained on human and robot play. Together, these results demonstrate that human experience can support both manipulation skill acquisition and subsequent policy improvement, reducing reliance on action-labeled robot data. Videos, dataset and code are available at this https URL.
- 中文摘要
学习机器人操作受制于带有动作标记的机器人数据的可用性,无论是模拟模型还是世界模型。人类可以在几秒内展示技能,但他们的演示缺乏所需的机器人动作。我们介绍了 DiWALI,这是一个在基于人机游戏训练的世界模型中学习并改进扩散策略的框架。DiWALI 利用逆动力学模型从连续帧推断潜在动作,使世界模型训练无需动作标签成为可能,并通过一个针对机器人游戏单独训练的解码器将其基于机器人指令。技能策略基于无动作人类演示的潜在动作进行训练,并通过想象力强化学习加以改进。在 CALVIN 基准测试中,仅由人类演示训练的 DiWALI 策略显著优于用机器人演示训练的基线策略。将人类游戏纳入世界模型训练,经过微调后比仅使用同等量的机器人游戏更能产生更强的策略。对物理机器人的实验表明,主要通过人类演示学到的技能,在想象力中得到进一步提升,而世界模型则训练为人类和机器人游戏。综合来看,这些结果表明,人类经验既能支持操控技能的习得,也能支持后续策略改进,从而减少对带动作标记的机器人数据的依赖。视频、数据集和代码可在此 https URL 获取。
Keyword: diffusion policy
SPICE: Stochastic Policy for Interaction in Cluttered Environments
SPICE:杂乱环境中交互的随机策略
- Authors: Nidhi Parayil, Benjamin Greenland, Thierry Peynot, Chris Lehnert
- Subjects: Subjects:
Robotics (cs.RO); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.23585
- Pdf link: https://arxiv.org/pdf/2611.23585
- Abstract
Agricultural tasks such as harvesting and pruning require robots to reach through dense foliage, where contact is often unavoidable and vision is obstructed. Intermittent contact creates an imbalance between free-space and contact-rich observations, complicating the learning of contact-responsive behaviours. We present SPICE, a tactileconditioned diffusion policy for reaching targets in dense plant canopies using tactile sensing as the sole environmental feedback. SPICE exhibits three behaviours: moving through compliant foliage, manoeuvring around partially obstructing stems, and manipulating branches that block the target path. To the best of our knowledge, SPICE is the first tactile-conditioned diffusion policy for target reaching through physically cluttered plant canopies. Across over 180 real-robot trials, SPICE was evaluated against tactile-conditioned, image-conditioned, and reactive-control baselines. Spatial tactile information and explicit contact conditioning proved important, with SPICE achieving 95% branch-manoeuvring success versus 45% for the strongest tactile baseline. SPICE also achieved branch manipulation beyond the reactive controller and remained effective under visual occlusion, where the image-conditioned DP failed. Zero-shot transfer across 40 trials on unseen tomato and lime plants achieved 87.5% success rate. These results demonstrate the potential of tactile-conditioned generative policies for interaction in visually occluded plant environments.
- 中文摘要
收获和修剪等农业任务需要机器人穿过茂密的树叶,在这些环境中接触往往不可避免,且视线受阻。间歇性接触造成自由空间与接触丰富观察之间的不平衡,使接触响应行为的学习变得复杂。我们介绍了SPICE,这是一种触觉条件扩散策略,利用触觉感知作为唯一的环境反馈,在密集植物冠层中达到目标。SPICE表现出三种行为:穿行顺应的叶片、绕开部分遮挡的茎,以及操控阻挡目标路径的枝条。据我们所知,SPICE是首个针对目标穿越物理杂乱植物冠层的触觉条件扩散策略。在180多项真实机器人试验中,SPICE与触觉条件、图像条件和反应式对照基线进行了评估。空间触觉信息和显性接触条件作用被证明重要,SPICE在枝条操作成功率为95%,而最强触觉基线仅为45%。SPICE还实现了超出反应控制的枝条操作,并在视觉遮挡条件下依然有效,而图像调理DP在视觉遮挡下未能实现。在40项未见到的番茄和青柠植株试验中,零剂量转移成功率达到了87.5%。这些结果表明,触觉条件生成策略在视觉遮蔽植物环境中的相互作用具有潜力。
Old Experience, New Lessons: Relabeling Autonomous Robot Rollouts in Interactive Imitation Learning
旧经验,新教训:在互动模仿学习中重新命名自主机器人推广
- Authors: Zhaoting Li, Cosimo Della Santina, Jens Kober
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.24273
- Pdf link: https://arxiv.org/pdf/2611.24273
- Abstract
Interactive imitation learning enables robots to learn from human interventions during deployment. These interactions naturally produce sparse human corrections and abundant autonomous robot experience without human action labels. Yet existing methods either leave this experience unused for action supervision or imitate historical robot actions, which can reinforce suboptimal behavior. To address this limitation, we propose Set-valued Relabeling of Autonomous experience (SeRA), a framework that learns from autonomous robot experience through set-valued supervision of policy relabels. Later policy checkpoints relabel this experience, and agreement across checkpoints filters the relabels. Since agreement does not guarantee correctness, SeRA constructs desired action sets around accepted relabels. Following Set-Supervised Diffusion Policy (SDP), it trains an action-chunking diffusion policy to increase the probability of actions within these sets. A step-masked denoising loss combines accepted relabels with available human corrections in partially supervised chunks. Experiments across five simulated manipulation tasks in online and offline settings show that SeRA improves policy performance over SDP and outperforms baselines that imitate historical robot actions or exact relabels. On a real-robot tomato-picking task, SeRA achieves 80% success, compared with 40% for SDP, using the same human feedback and training budget. These results demonstrate that relabeled autonomous robot experience can complement human corrections to improve policy performance.
- 中文摘要
交互式模仿学习使机器人能够在部署过程中从人类干预中学习。这些交互自然会产生稀疏的人类纠正和丰富的自主机器人体验,而无需人类动作标签。然而,现有方法要么将这种经验置于动作监督之外,要么模仿历史机器人的行为,从而强化次优行为。为解决这一局限,我们提出了“集合值自主经验重标”(SeRA),这是一种通过对策略重标的集合值监督从自主机器人经验中学习的框架。后续策略检查点会重新标记这种体验,而跨检查点的一致性则会过滤这些重标。由于一致性并不保证正确,SeRA 会围绕被接受的重标构建所需的动作集。遵循集合监督扩散策略(SDP),它训练一个动作分块扩散策略,以提高这些集合内发生动作的概率。步进掩码去噪损耗将被接受的重新标记与部分监督的人工修正结合起来。在在线和离线环境中进行的五个模拟操作任务实验表明,SeRA相较于SDP能提升策略性能,并优于模拟历史机器人动作或精确重新标记的基线。在真实机器人摘番茄任务中,SeRA在使用相同的人工反馈和训练预算下,成功率为80%,而SDP为40%。这些结果表明,重新标记的自主机器人体验可以补充人工修正,从而提升策略性能。