生成时间: 2026-11-18 01:29:42 (UTC+8); Arxiv 发布时间: 2026-11-17 20:00 EST (2026-11-18 09:00 UTC+8)
今天共有 79 篇相关文章
Keyword: reinforcement learning
Skill-Latent Rewards: LLM-Guided Reward Formulation for Hierarchical Reinforcement Learning
技能潜在奖励:基于LLM的奖励体系,用于层级强化学习
- Authors: Maryam Shoaeinaeini, Brent Harrison
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.18597
- Pdf link: https://arxiv.org/pdf/2611.18597
- Abstract
Long-horizon reinforcement learning under sparse and delayed feedback remains an open challenge. Existing return-decomposition methods redistribute episodic rewards to ease credit assignment, but they operate on raw state--action transitions and disregard the subgoal structure that governs meaningful progress in complex tasks. We propose Skill-LR, a hierarchical reinforcement learning framework in which a vision--language model identifies the agent's active subgoal at each timestep and a large language model generates a matching latent reward formulation through a two-stage prompting process. By conditioning reward signals on the currently active subgoal, Skill-LR produces stage-specific credit that evolves with the hierarchical plan, reducing temporal variance and improving sample efficiency. On multi-stage MiniGrid and BabyAI benchmarks, including DoorKey and Unlock-to-Unlock, Skill-LR converges faster and more reliably than sparse environmental rewards and strong credit-assignment baselines such as RUDDER and LaRe, demonstrating that subgoal-aligned latent rewards offer a principled path toward stable, interpretable credit assignment in long-horizon tasks.
- 中文摘要
在稀疏且延迟反馈下的长期强化学习仍是一个开放挑战。现有的返回分解方法通过重新分配情节奖励以简化学分分配,但它们基于原始的状态-行动转移,忽视了支配复杂任务中有意义进展的子目标结构。我们提出了 Skill-LR,这是一种层级强化学习框架,其中视觉-语言模型在每个时间步识别代理的活跃子目标,大型语言模型通过两阶段提示过程生成匹配的潜在奖励表述。通过对当前活跃子目标的奖励信号进行条件化,Skill-LR 生成阶段特定的积分,并随层级计划演进,减少时间差异并提高样本效率。在多阶段 MiniGrid 和 BabyAI 基准测试中,包括 DoorKey 和 Unlock-to-Unlock,Skill-LR 比稀疏的环境奖励和强的学分分配基线(如 RUDDER、LaRe)更快、更可靠地收敛,这表明与子目标对齐的潜在奖励在长期任务中为稳定、可理解的学分分配提供了一条有原则的路径。
FlashREINFORCE: Critic-Free Single-Rollout Asynchronous RL for Agentic Language Models
FlashREINFORCE:面向代理语言模型的无批评单次部署异步强化学习
- Authors: Jian Hu, Yifan Zhang, Hao Zhang, Binfeng Xu, Shaokun Zhang, Hongqing Peng, Zhiding Yu, Pavlo Molchanov, Jan Kautz, Yi Dong
- Subjects: Subjects:
Computation and Language (cs.CL); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.18627
- Pdf link: https://arxiv.org/pdf/2611.18627
- Abstract
Asynchronous reinforcement learning suits long-horizon agents with irregular rollout times, but group-relative updates require repeated rollouts of each prompt. However, group-based sampling limits prompt coverage at a fixed rollout budget and introduces group synchronization barriers. One rollout per prompt avoids these constraints but removes the variance-reducing group baseline, making updates noisier and more prone to instability on stale, variable-length trajectories. We present FlashREINFORCE, a critic-free, one-pass framework built on One-Batch REINFORCE, a Sequence Trust Region, and Sample-Mean Optimization. These components preserve prompt coverage and signed feedback while controlling policy drift and length-amplified negative updates. In our experiments, FlashREINFORCE remains stable through 6,000 updates on DeepSeek-R1-Distill-Qwen-1.5B at a policy lag of approximately four updates. On Qwen2.5-Math-1.5B, it reaches 38.0 mean accuracy across five benchmarks, outperforming the reported GRPO baseline by 1.7 percentage points with half the rollouts (256k versus 512k). Python-tool experiments extend to the 30B MoE model Qwen3-30B-A3B, with stable training at policy lag eight. On Qwen2.5-7B-Instruct, FlashREINFORCE maintains tool use where GRPO stops making tool calls and achieves 98.3%/96.5% seen/unseen ALFWorld success.
- 中文摘要
异步强化学习适合长期部署时间不规则的代理,但组相对更新需要每个提示的重复部署。然而,基于组的抽样限制了在固定部署预算下的提示覆盖范围,并引入组同步障碍。每个提示进行一次部署避免了这些限制,但消除了减少方差的组基线,使更新在陈旧且可变长度的轨迹上更为噪声大,更容易不稳定。我们介绍 FlashREINFORCE,这是一个无批评、单遍框架,基于 One-Batch REINFORCE、序列信任区域和样本均值优化。这些组件在控制策略漂移和长度放大的负向更新的同时,保持了提示覆盖率和签名反馈。在我们的实验中,FlashREINFORCE 在 DeepSeek-R1-Distill-Qwen-1.5B 上 6,000 次更新后保持稳定,政策时差约为四次更新。在 Qwen2.5-Math-1.5B 上,它在五个基准测试中达到 38.0 的平均准确率,在部署量减半(256k 对 512k)的情况下,表现比报告的 GRPO 基线高出 1.7 个百分点。Python工具实验扩展到30B的MoE模型Qwen3-30B-A3B,其稳定训练时策略滞后为8。在Qwen2.5-7B-Instruct上,FlashREINFORCE维持了GRPO停止工具调用的工具使用,实现了98.3%/96.5%的ALFWorld成功率。
Minari: Infrastructure for Collaborative Reinforcement Learning Datasets
Minari:协作强化学习数据集的基础设施
- Authors: Omar G. Younis, Rodrigo Perez-Vicente, John U. Balis, Alex Davey, Will Dudley, Jordan K. Terry
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.18644
- Pdf link: https://arxiv.org/pdf/2611.18644
- Abstract
Offline reinforcement learning (RL) depends on shared datasets, but sharing them is still hard. Each benchmark ships its own format, documentation is thin, and most datasets cannot be regenerated once the environment code moves on. We introduce Minari, an open-source Python library for creating, sharing, and loading RL datasets. Its API follows Gymnasium, so recording a dataset means wrapping an environment and stepping through it as usual. From there, users can attach metadata, filter, combine, and shuffle datasets, and publish or download them with one command through the HuggingFace Hub. Every dataset stores the environment specification, seeds, applied wrappers, and pinned package requirements needed to rebuild the environment and regenerate the data. TorchRL, AgileRL, and d3rlpy already load Minari datasets natively, so switching datasets in those libraries takes one line of code. Minari also hosts 338 reproducible datasets (350K trajectories, 55M steps) covering robotic locomotion and manipulation, maze and grid navigation, language-conditioned instruction following, Atari, and web navigation, including a full regeneration of D4RL with public generation scripts. We report baseline scores for six offline RL algorithms on the regenerated D4RL and MuJoCo datasets.
- 中文摘要
离线强化学习(RL)依赖于共享数据集,但共享它们仍然很困难。每个基准测试发布了自己的格式,文档稀少,大多数数据集一旦环境代码更新就无法重新生成。我们介绍 Minari,一个开源的 Python 库,用于创建、共享和加载 RL 数据集。其 API 遵循 Gymnasium,因此记录数据集意味着对环境进行封装并像往常一样逐步穿过。在此之后,用户可以附加元数据、筛选、合并和洗牌数据集,并通过 HuggingFace Hub 一个命令发布或下载它们。每个数据集都存储了重建环境和重新生成数据所需的环境规格、种子、应用的封装和固定包需求。TorchRL、AgileRL 和 d3rlpy 已经原生加载 Minari 数据集,因此在这些库中切换数据集只需一行代码。Minari还托管了338个可复现数据集(35万轨迹,5500万步),涵盖机器人移动与操作、迷宫与网格导航、语言条件指令跟随、Atari及网页导航,包括使用公开生成脚本的完整D4RL再生。我们报告了重建后的D4RL和MuJoCo数据集上六个离线RL算法的基线分数。
ECOD: Experience-Calibrated On-Policy Distillation for Multi-Turn Agents
ECOD:为多轮代理提供经验校准的策略内蒸馏
- Authors: Jie Jin, Shengqin Wang, Yan Gao, Xin Tan, Yuan Xie
- Subjects: Subjects:
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.18656
- Pdf link: https://arxiv.org/pdf/2611.18656
- Abstract
Sparse and delayed rewards make reinforcement learning challenging for long-horizon language agents. On-policy distillation (OPD) complements environmental rewards with dense teacher feedback on student-generated contexts. Standard OPD can misalign supervision with relative teacher competence, student trajectory outcomes, and the temporal structure of task execution. Recent variants adjust guidance through divergence-based weighting and temporal schedules, while joint calibration from task-completion experience remains underexplored. We introduce Experience-Calibrated On-Policy Distillation (ECOD), which allocates teacher supervision at the task, trajectory, and turn levels. ECOD compares teacher and student success-rate posteriors under matched task conditions, restricts direct supervision to failed student trajectories, and derives a temporal prior from successful teacher lengths. The resulting weight applies to teacher losses or teacher-derived advantages. We evaluate ECOD on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. Under matched teacher settings, ECOD improves standard OPD by up to 4.06% in relative ALFWorld unseen success. Across both model scales, ECOD-assisted SOD, TCOD-F2B, and ATOD configurations yield average paired relative gains of 6.20% on ALFWorld unseen and 7.05% on WebShop, with WebShop gains ranging from 2.29% to 21.60%. These results support experience-calibrated supervision allocation as a reusable mechanism across distinct distillation objectives.
- 中文摘要
奖励稀疏且延迟,使得长期视野语言代理的强化学习变得困难。策略上提炼(OPD)通过对学生生成语境的教师密集反馈,补充环境奖励。标准OPD可能使督导与相对教师能力、学生轨迹结果及任务执行时间结构不匹配。近期变体通过基于分歧的权重和时间时间表调整指导,而基于任务完成经验的联合校准仍未被充分探索。我们引入了经验校准策略提炼(ECOD),该方法在任务、轨迹和转折层级分配教师监督。ECOD比较匹配任务条件下教师与学生的成功率后验,将直接监督限制在失败学生轨迹,并从成功教师的时长推导出时间先验。由此产生的权重适用于教师损失或教师带来的优势。我们用 QWEN2.5-1.5B-Instructions 和 Qwen2.5-7B-Instructions 评估 ALFWorld 和 WebShop 上的 ECOD。在匹配教师设置下,ECOD 在相对 ALFWorld 未见成效中可将标准 OPD 提高高达 4.06%。在两个模型尺度中,ECOD 辅助 SOD、TCOD-F2B 和 ATOD 配置在未见 ALFWorld 上平均对应获得 6.20% 和 WebShop 上 7.05%,WebShop 则在 2.29% 至 21.60% 之间。这些结果支持经验校准的监督分配作为在不同蒸馏目标间可重复使用的机制。
ThinkWise: Learning When to Think in Large Multimodal Reasoning Models via Visual and Difficulty Perception
ThinkWise:通过视觉和难度感知学习何时在大型多模态推理模型中思考
- Authors: Wenkang Zhou, Chen Yang, Jiaheng Peng, Xitong Li, Hongzhi Yin, Yunke Zhang, Lu Yu, Peng-Fei Zhang, Xuhui Ren, Shijie Zhang
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.18685
- Pdf link: https://arxiv.org/pdf/2611.18685
- Abstract
Large Multimodal Reasoning Models (LMRMs) extend large language models to heterogeneous modalities, demonstrating strong potential across a wide range of open-ended tasks. Reinforcement Learning with Verifiable Rewards (RLVR) has been widely adopted to improve reasoning performance in LMRMs; however, existing approaches overlook the critical role of visual perception uncertainty in guiding reasoning processes. We term this limitation Reasoning Allocation Misalignment, where the misalignment between visual perception uncertainty and reasoning depth leads to over-allocation of deep reasoning under uncertain visual grounding, resulting in unnecessarily long reasoning trajectories and degraded performance. To address this issue, we propose ThinkWise, a perception-aware RLVR framework that jointly models task difficulty and visual perception state to regulate reasoning depth. Specifically, we introduce an instance-adaptive visual perception modeling mechanism that enables context-aware calibration of perception uncertainty, leading to more accurate perception estimation. We further design a structured reward formulation that jointly considers perception, correctness, and task difficulty. In particular, an answer reward provides supervision for correctness, a perception- and difficulty-conditioned length reward adaptively regulates reasoning depth, and a perception reward encourages trajectories with stronger visual grounding. Extensive experiments on multiple multimodal reasoning benchmarks demonstrate that ThinkWise consistently improves the trade-off between accuracy and reasoning efficiency across both 3B and 7B models. These results highlight the importance of perception-aware reasoning and establish a principled framework for advancing LMRMs.
- 中文摘要
大型多模态推理模型(LMRM)将大型语言模型扩展到异构模态,在广泛的开放式任务中展现出强大潜力。带可验证奖励的强化学习(RLVR)已被广泛采用,以提升LMRMM的推理表现;然而,现有方法忽视了视觉感知不确定性在引导推理过程中的关键作用。我们将这种限制称为推理分配错位,即视觉感知不确定性与推理深度之间的错位导致深度推理在不确定的视觉基础下过度分配,导致推理轨迹过长并降低性能。为解决这一问题,我们提出了ThinkWise,一种感知感知的RLVR框架,结合建模任务难度和视觉感知状态,以调节推理深度。具体来说,我们引入了一种实例适应视觉感知建模机制,使感知不确定性能够基于情境进行校准,从而实现更准确的感知估计。我们进一步设计了一种结构化奖励表述,将感知、正确性和任务难度共同纳入考量。特别是,答案奖励为正确性提供监督,感知与难度条件长度奖励适应性调节推理深度,感知奖励则鼓励具有更强视觉基础的趋势。在多个多模态推理基准测试上的大量实验表明,ThinkWise 在 3B 和 7B 模型中持续提升了准确性与推理效率之间的权衡。这些结果凸显了感知感知推理的重要性,并为推进 LMRM 建立了有原则的框架。
Video-as-Experience: Learning Computer Use via Audio-Visual Understanding
视频即体验:通过视听理解学习计算机使用。
- Authors: Guangzhi Sun, Zhonghao Zhai, Yixuan Li, Chao Zhang
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.18747
- Pdf link: https://arxiv.org/pdf/2611.18747
- Abstract
Multimodal large language models (LLMs) have enabled capable computer-use agents that perceive graphical user interfaces, reason over instructions, and execute actions. However, existing approaches to learning from video demonstrations typically convert videos into screenshot-based action trajectories, rather than directly leveraging their rich audio-visual information. We introduce VEX (Video as EXperience), a framework that uses audio-visual understanding to unify learning from external demonstrations and reflecting on an agent's own behavior. Following a watch-act-rewatch paradigm, the agent extracts procedural guidance from tutorial videos containing visual demonstrations and spoken explanations, executes tasks, and verifies its progress by rewatching its execution as video. We instantiate VEX with video-SALMONN-VEX, a 9B-scale audio-visual computer-use agent built on the video-SALMONN 2 architecture with a Qwen3.5-9B backbone. We further introduce Action-Grounded Guide Optimization (AGGO), a reinforcement learning method that optimizes video-derived guidance for downstream action prediction, and augment AgentNet, OSWorld, and CUA-Gym with over 24K real-world tutorial videos. Experiments on OSWorld and a complementary subset of CUA-Gym demonstrate consistent improvements, with video-SALMONN-VEX achieving a 22% relative increase in task success over its Qwen3.5 backbone on OSWorld. Moreover, its generated guidance improves Gemini-3.5-flash by 10% relatively on OSWorld. These results highlight the potential of audio-visual understanding as a unified interface for procedural learning and self-verification in computer-use agents.
- 中文摘要
多模态大型语言模型(LLM)使得能够感知图形用户界面、基于指令推理并执行动作的计算机智能体成为可能。然而,现有通过视频演示学习的方法通常将视频转换为基于截图的动作轨迹,而非直接利用其丰富的视听信息。我们介绍了VEX(视频即体验),这是一种利用视听理解,将外部演示的学习与对自身行为的反思整合起来的框架。遵循观看-行为-重看的模式,代理从包含视觉演示和口头说明的教程视频中提取程序指导,执行任务,并通过视频重看执行过程验证进度。我们用video-SALMONN-VEX实例化VEX,这是一种基于video-SALMONN 2架构、Qwen3.5-9B骨干的9B尺度视听计算机代理。我们进一步介绍了 Action-Grounded Guide Optimization(AGGO),这是一种强化学习方法,用于优化视频导出的指导以预测下游动作,并为 AgentNet、OSWorld 和 CUA-Gym 加入超过 24K 的真实世界教程视频。在 OSWorld 及其配套子集上的实验显示出持续的改进,video-SALMONN-VEX 在 OSWorld 上的任务成功率相较于 Qwen3.5 骨干网提高了 22%。此外,其生成的指导在 OSWorld 上使 Gemini-3.5-flash 相对于 OSWorld 提高了 10%。这些结果凸显了视听理解作为计算机代理中程序学习和自我验证统一界面的潜力。
LongCat-Video-Avatar 1.6 Technical Report
LongCat-Video-Avatar 1.6 技术报告
- Authors: Meituan LongCat Team: Xunliang Cai, Yiqi Chen, Feng Gao, Xiaoyu Kong, Le Li, Hongyu Liu, Xiaoming Wei, Tianyu Yang, Yong Zhang, Sijie Zhao
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.18822
- Pdf link: https://arxiv.org/pdf/2611.18822
- Abstract
Recent advances in audio-driven avatar video generation have enabled increasingly realistic synthesis, yet achieving precise lip-audio synchronization, natural, human-like motion, and visual consistency over long sequences remains challenging. We present LongCat-Video-Avatar 1.6, which builds on LongCat-Video-Avatar 1.5 to improve expressive naturalness and reduce error accumulation during extended generation, e.g., exaggerated lip movements and overly red, temporally unstable lip coloration, enhance the naturalness of facial expressions and coordinated head and body movements. These improvements are driven by a refined post-training pipeline that integrates reinforcement learning, improved classifier-free guidance strategies for DMD-based few-step distillation, long-video rollouts, and on-policy distillation. Together, these refinements improve expressive realism and enable the generation of videos exceeding 10 minutes while maintaining visual quality and temporal consistency. Arena evaluations show that LongCat-Video-Avatar 1.6 outperforms its predecessor and competing commercial audio-driven avatar models, demonstrating the effectiveness of the improved post-training pipeline.
- 中文摘要
音频驱动头像视频生成的最新进展使合成变得越来越逼真,但实现精准的唇音同步、自然、类人动作以及长序列中的视觉一致性仍具挑战性。我们介绍 LongCat-Video-Avatar 1.6,它基于 LongCat-Video-Avatar 1.5 改进表现力自然,减少长时间生成过程中的错误累积,例如夸张的嘴唇动作和过红且时间不稳定的唇色,增强面部表情的自然感以及头部和身体协调的动作。这些改进得益于精细的培训后流程,整合了强化学习、改进的无分类器指导策略,用于基于 DMD 的少数步骤蒸馏、长视频推广以及策略上提炼。这些改进共同提升了表现力的真实感,使得在保持视觉质量和时间一致性的同时,生成超过10分钟的视频。Arena 评估显示,LongCat-Video-Avatar 1.6 优于其前身及竞争对手基于音频的商业化身模型,充分展示了改进后的培训流程的有效性。
Correcting Persistent Failures in Point-Cloud Grasping Policies using Human Demonstrations in Virtual Reality
利用虚拟现实中的人工演示纠正点云抓取策略的持续性失败
- Authors: Achyuthan Unni Krishnan, Karthik Rengarajan, Bharath Narasimhan, Chris von Csefalvay
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.18919
- Pdf link: https://arxiv.org/pdf/2611.18919
- Abstract
Robots in homes, warehouses and farms must pick up objects they have never seen, usually without a 3D model or a category label. A grasping policy that works directly on the point cloud from a depth sensor needs only the geometry the robot observes, so it applies to any new object. Offline grasp generators such as Contact-GraspNet work this way, but they rank grasps by learned plausibility rather than by whether the grasp actually lifts the object. Reinforcement learning (RL) can optimize for that directly and make a point-cloud grasping policy much better on average, yet it leaves specific grasps persistently unsolved. We present a human-in-the-loop failure loop that targets those grasps: situations an RL policy keeps failing are mined automatically, sent to a virtual-reality application where a person picks the object up with their own hands, and returned to training as two grasp points. The person contributes only where to grasp; the robot chooses its own approach. In simulation, RL raises success on unseen objects from 55% with Contact-GraspNet to 83%. Adding 58 human demonstrations nearly doubles success on the hard cases RL kept failing, from 24% to 45%, and recovers performance that RL alone loses on similar cases that were never demonstrated, without reducing success on unseen objects.
- 中文摘要
家庭、仓库和农场中的机器人必须拾取从未见过的物体,通常没有3D模型或类别标签。直接通过深度传感器对点云工作的抓取策略只需机器人观察到的几何形状,因此适用于任何新物体。离线抓取生成器如Contact-GraspNet就是这样工作,但它们根据学习的合理性来排名抓握,而非按抓握是否实际抬起物体。强化学习(RL)可以直接优化这一点,使点云抓取策略平均表现更好,但具体抓握问题却持续未被解决。我们展示了一个针对这些抓取的“人为参与”失败循环:RL策略屡次失效的情况会自动挖掘,送至虚拟现实应用,由用户用手拾起物体,然后作为两个抓取点返回训练。个人只会在抓取位置做出贡献;机器人选择自己的方法。在模拟中,强化学习在接触抓网下对看不见物体的成功率从55%提升到83%。增加58次人工演示几乎能使强化学习在困难案例中成功率翻倍,从24%降至45%,并恢复了增强现实在类似案例中仅有的性能损失,且不降低对看不见物体的成功率。
Policy Optimization With Multiple Verifiers via Pareto Rankings
通过帕累托排名实现多验证者策略优化
- Authors: Kai Yamashita, Shohei Taniguchi, Hiroki Furuta, Masahiro Suzuki, Yutaka Matsuo
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19064
- Pdf link: https://arxiv.org/pdf/2611.19064
- Abstract
Reinforcement learning with verifiable rewards (RLVR) has become an important approach to improving the reasoning capabilities of large language models. In tasks such as code generation and instruction following, multiple verifiers evaluate whether a response satisfies different requirements. Their outcomes provide detailed feedback about partial success, yet this feedback is commonly aggregated into a scalar reward for policy optimization. Rewarding only complete success discards distinctions among partial solutions, while counts and weighted sums impose comparisons between responses that satisfy different subsets of requirements. We introduce Partial-Order Likelihood Policy Optimization (POLPO), which learns from the comparisons established by verification without first constructing an aggregate reward. Pareto dominance identifies responses that satisfy additional requirements without losing any already satisfied, while leaving responses with conflicting outcomes incomparable. POLPO extends the listwise likelihood underlying Direct Preference Optimization by maximizing the total probability of all complete rankings consistent with these partial orders. This objective provides supervision from verified improvements even before complete solutions are available, without requiring a prescribed ordering of incomparable responses. Experiments on code generation and instruction following demonstrate improved learning efficiency and higher rates of complete requirement satisfaction compared with scalar-reward training. These results support partial-order likelihood as an effective approach to learning from multiple verifiers.
- 中文摘要
带有可验证奖励的强化学习(RLVR)已成为提升大型语言模型推理能力的重要方法。在代码生成和指令跟随等任务中,多个验证者评估响应是否满足不同需求。他们的结果提供了关于部分成功的详细反馈,但这些反馈通常被汇总成策略优化的标量奖励。仅奖励完全成功则消除部分解决方案之间的区分,而计数和加权和则要求对满足不同要求子集的响应进行比较。我们引入了部分似然策略优化(POLPO),该方法通过验证建立的比较学习,而无需先构建总奖励。帕累托优势识别出满足额外要求但不失去已满足条件的响应,同时使结果冲突的响应不可比。POLPO 通过最大化所有与这些偏序一致的完整排名的总概率,扩展了直接偏好优化所支撑的列表似然。这一目标在尚未获得完整解之前,就对已验证的改进进行监督,而无需对不可比的回答进行规定的排序。关于代码生成和后续指令的实验显示,与标量奖励训练相比,学习效率更高,完全需求满足率更高。这些结果支持偏序似然作为从多个验证器学习的有效方法。
Assume/Guarantee Contracts for the Compositional Multi-Agent Reinforcement Learning of Heterogeneous Teams
承担/保证异构团队组合多智能体强化学习的合同
- Authors: Dominic Klukas, Cyrus Neary
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19078
- Pdf link: https://arxiv.org/pdf/2611.19078
- Abstract
Multi-agent reinforcement learning (MARL) problems for heterogeneous teams require agents to collaborate to complete team tasks while making the best use of diverse agent capabilities. We introduce assume/guarantee contracts for heterogeneous multi-agent systems (AGC-MAS) and Iterative Heterogeneous Assignment for MARL (IHA-MARL). AGC-MAS contracts are compositional objects that encode the assumptions provided to, and the guarantees required of, each subtask contributing to the overall team task. We formulate contracts as tuples that specify a subtask, an automaton, timeout and success-probability requirements, and dispatch behavior. The automaton tracks an agent's high-level progress through the subtask associated with its contract and provides a natural mechanism for synchronization across contracts. Our abstraction enables robust interaction across heterogeneous low-level subtask implementations, allows heterogeneous agents to substitute for one another in satisfying contracts, and supports reuse of previously satisfied contracts in new team contexts. Having defined our theoretical framework, we develop a decomposition algorithm that breaks down a team task into a set of contracts. We then propose an iterative algorithm for task assignment and MARL that alternates between assigning contracts representing subtasks to individual agents, training the assigned agents to satisfy their contracts, and assessing their resulting capabilities to inform the next round of task assignments until the team's success probability meets the desired threshold. Our experimental results demonstrate the versatility of contracts in enabling agent substitution and contract reuse. Across three benchmarks with five seeds each, IHA-MARL achieves observed $100\%$ final evaluation success. In CPU timing pilots, comparable baselines required approximately $7$--$11\times$ more training-loop time.
- 中文摘要
针对异构团队的多智能体强化学习(MARL)问题要求代理协作完成团队任务,同时最大限度地利用多样化的智能体能力。我们引入了异构多智能体系统(AGC-MAS)的假设/保证合同,以及为 MARL 引入迭代异构分配(IHA-MARL)。AGC-MAS 合同是组合对象,用于编码每个子任务所提供的假设和对其所需的保证,这些任务对整体团队任务有所贡献。我们将合同以元组形式制定,指定子任务、自动机、超时和成功概率要求以及派遣行为。自动机跟踪代理在其合同相关子任务中的高层次进度,并为跨合同的同步提供了自然机制。我们的抽象实现了异构低层子任务实现间的稳健互动,允许异构代理在满足合同中相互替代,并支持在新的团队环境中重用先前满足的合同。在定义了我们的理论框架后,我们开发了一种分解算法,将团队任务拆分为一组合同。随后,我们提出了一个迭代的任务分配算法和 MARL,该算法交替分配代表子任务的合同给单个代理,训练被分配的代理以满足其合同,并评估其能力以指导下一轮任务分配,直到团队成功概率达到预期阈值。我们的实验结果展示了合同在实现代理替代和合同重用方面的多功能性。在三个基准测试中,每个基准测试有五个种子,IHA-MARL 实现了观察到的 100% 美元最终评估成功率。在 CPU 计时试点中,类似基线大约需要 7 美元到 11 美元,额外的训练循环时间。
Wireless Edge-Assisted Adaptive Latent Reasoning for Vision-Language-Action Models in Physical AI
物理人工智能中视觉-语言-动作模型的无线边缘辅助自适应潜在推理
- Authors: Fatih Temiz, Melike Erol-Kantarci
- Subjects: Subjects:
Robotics (cs.RO); Networking and Internet Architecture (cs.NI)
- Arxiv link: https://arxiv.org/abs/2611.19131
- Pdf link: https://arxiv.org/pdf/2611.19131
- Abstract
Vision-language-action (VLA) models enable embodied artificial intelligence (AI) by translating visual observations and language instructions into robotic actions. Emerging use cases in physical AI are expected to drive the need for efficient inference and communication, as the computational demands of the VLA models pose challenges for real-time control with limited onboard resources. Test-time scaling through latent reasoning allows these models to refine hidden representations without generating explicit chain-of-thought (CoT) tokens, but introduces a trade-off between task success, inference latency, and energy consumption. In this paper, we address the efficient collaborative inference under a realistic communication scenario. We propose a wireless edge-assisted framework for adaptive latent reasoning (WEALR) that augments onboard computation with edge computing. A decision-making agent jointly selects the execution location and latent-reasoning convergence threshold. We formulate this joint adaptation as a constrained Markov decision process (CMDP). A Lagrangian-constrained deep reinforcement learning (CDRL) approach, namely trust region policy optimization Lagrangian (TRPO-Lag), learns to balance task performance and resource consumption under varying wireless conditions and task requirements. We evaluate the framework using LIBERO manipulation tasks, profiled VLA inference costs, and ray-traced wireless channels. Experimental results demonstrate that TRPO-Lag achieves up to 23% higher feasibility-weighted task success rate compared to benchmark CDRL methods.
- 中文摘要
视觉-语言-行动(VLA)模型通过将视觉观察和语言指令转化为机器人动作,实现具身人工智能(AI)。物理人工智能的新兴应用场景预计将推动高效推理和通信的需求,因为VLA模型的计算需求在有限的板载资源下对实时控制带来了挑战。通过潜在推理进行测试时扩展,这些模型可以在不生成显式链式思维(CoT)令牌的情况下细化隐藏表征,但也带来了任务成功率、推理延迟和能耗之间的权衡。本文探讨了在现实通信场景下高效的协作推理。我们提出了一种无线边缘辅助自适应潜在推理(WEALR)框架,通过边缘计算增强板载计算。决策代理共同选择执行位置和潜在推理收敛阈值。我们将这一联合适应定义为受限马尔可夫决策过程(CMDP)。一种拉格朗日约束深度强化学习(CDRL)方法,即信任区域策略优化拉格朗日量(TRPO-Lag),学习在不同无线条件和任务需求下平衡任务性能与资源消耗。我们利用 LIBERO 操作任务、剖面 VLA 推断成本和光线追踪无线信道进行评估。实验结果表明,与基准 CDRL 方法相比,TRPO-Lag 在可行性加权任务成功率上高出多达 23%。
Low-Latency LLM-Based Streaming Multi-Talker ASR with Verifiable Rewards
基于低延迟的 LLM 流式多话客 ASR,并提供可验证的奖励
- Authors: Jing Pan, Ruizhi Li, Ju Lin, Jing Zheng, Florian Metze
- Subjects: Subjects:
Computation and Language (cs.CL); Sound (cs.SD)
- Arxiv link: https://arxiv.org/abs/2611.19136
- Pdf link: https://arxiv.org/pdf/2611.19136
- Abstract
LLM-based architectures offer strong generalization and instruction-following capabilities for streaming automatic speech recognition (ASR), but multi-talker use introduces coupled transcription, attribution, and latency challenges. Our wearable setting requires real-time transcription of a conversation between a near-field device wearer and a far-field partner. Because partial hypotheses are rendered on persistent sides of a dialogue UI, speaker roles cannot be permuted and each speaker tag must precede its turn. These sparse tags are weakly optimized by supervised fine-tuning (SFT), while committing to a leading tag can delay the text that follows. We therefore post-train a streaming LLM-based ASR model with reinforcement learning with verifiable rewards (RLVR), combining a fixed-role Speaker-Attributed WER (SA-WER) grader with a turn-level onset-delay reward. On an eight-direction multi-talker cohort, SA-WER RL reduces SA-WER from $20.51\%$ for SFT to $13.14\%$. Adding the delay objective reduces onset $p50$ by $65\%$ (from $0.918\,\text{s}$ to $0.320\,\text{s}$) and $p90$ by $44\%$ (from $1.389\,\text{s}$ to $0.780\,\text{s}$) relative to SA-WER-only RL, while achieving $12.70\%$ SA-WER. These results demonstrate that fixed-role recognition quality and streaming responsiveness can be improved jointly.
- 中文摘要
基于 LLM 的架构为流式自动语音识别(ASR)提供了强大的泛化和指令跟随能力,但多话者使用带来了耦合转录、归因和延迟等问题。我们的可穿戴环境要求对近场设备佩戴者与远场伙伴之间的对话进行实时转录。由于部分假设是在对话界面的持久侧面上呈现,说话者角色无法置换,每个说话者标签必须先于其回合。这些稀疏标签通过监督微调(SFT)优化较弱,而承诺使用前置标签可能会延迟后续文本的发布。因此,我们通过带有可验证奖励的强化学习(RLVR)对基于流式LLM的ASR模型进行后期训练,将固定角色的说话者属性WER(SA-WER)评分器和回合级起始延迟奖励结合起来。在八向多话机队列中,SA-WER RL 将 SA-WER 从 SFT 的 20.51 美元降至 13.14 美元。添加延迟目标可将起始$p 50$减少65美元(从0.918\text{s}$降至0.320\,\text{s}$),$p 90$减少44美元(从1.389\text{s}$至0.780\,\text{s}$),同时实现12.70美元SA-WER。这些结果表明固定角色识别质量和流响应速度可以共同提升。
SuperGlance: Driving at a Glance with a Vision-Language Model
SuperGlance:用视觉语言模型一览无遗
- Authors: Jieyuan Pei, Xin Yan, Mingwei Xu, Haiyi Zhang, Siyu Zou, Zhangyi Hu, Zhuoxuan Li, Yuguang Yang, Rongjunchen Zhang, Cheng Chi, Hui Xiong, Yan Wang
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.19138
- Pdf link: https://arxiv.org/pdf/2611.19138
- Abstract
Autonomous driving is turning to vision-language-action models (VLAs). They answer and plan by decoding, one step at a time. Each step is another forward pass, so a plan can take over a second of GPU time. Embodied AI often pairs such a slow, reasoning System 2 with a fast System 1 that acts. Human drivers plan at a glance, with System 1. Faster designs usually put System 1 outside the vision-language model (VLM), in a planner without its abilities. Yet before a VLM writes its first token, it has already read the scene into one hidden state. We show that, once trained, this hidden state is enough to drive and to answer questions about what the model sees. SuperGlance reads it to turn a decoder-only VLM into an encoder-style driving policy. Its linear head scores a fixed set of 5,273 candidate trajectories. With the set fixed, rewards are computed offline, so supervised targets can be reward-weighted and reinforcement learning maximizes the exact expected reward. Before and during policy training, multiple-choice training keeps the VLM's visual abilities in one pass, so the policy still answers questions about driving scenes. Training questions are built against shortcuts. Each must be answered right in two option orders. Against drift, a KL term keeps each stage near its starting model. One pass offers VLAs a more plausible path to System 1. SuperGlance answers visual multiple-choice questions as well as its base VLM does by decoding. It plans in 84 ms of GPU time, six times faster than Qwen-Drive-1.0, and at least as well on NAVSIM, PAI-AV and WOD-E2E. In AlpaSim closed loop, km per at-fault incident holds for both planners up to a 300-ms delay and drops beyond. With GPU time charged as delay, SuperGlance's lead over Qwen-Drive-1.0 grows from 2.8 to 3.9 times. Delayed by 1.2 s, our model falls below Qwen-Drive-1.0 at its own 518 ms.
- 中文摘要
自动驾驶正在转向视觉-语言-行动模型(VLA)。它们通过解码一步一步来回答和规划。每一步都是前进,因此一个计划可能占用超过一秒的GPU时间。具象人工智能常将这样缓慢、善于推理的系统2与快速且能行动的系统1配对。人类驾驶员一目了然地规划,系统1也就是这样。更快的设计通常会将系统1置于视觉-语言模型(VLM)之外,也就是在规划器中,没有其能力。然而,在VLM写入第一个令牌之前,它已经将场景读入一个隐藏状态。我们展示了,一旦训练完成,这个隐藏状态就足以驱动并回答关于模型所看到内容的问题。SuperGlance读取它,将仅需解码器的VLM转化为编码器式的驾驶策略。其线性头会为固定的5,273条候选轨迹进行评分。在固定的集合之后,奖励会在离线中计算,因此可以对受监督的目标进行奖励加权,强化学习则最大化精确的预期奖励。在策略训练之前和过程中,选择训练将VLM的视觉能力集中在一次传递中,因此策略仍然能回答驾驶场景相关的问题。训练问题是基于捷径构建的。每个选项必须按两个选项顺序正确回答。在对抗漂移时,KL项使每个阶段保持接近其起始模型。一次通过为VLA提供了一条更合理的路径通往系统1。SuperGlance回答视觉选择题的表现与其基础VLM一样,通过解码实现。它在 GPU 上规划时间为 84 毫秒,是 Qwen-Drive-1.0 的六倍,并且在 NAVSIM、PAI-AV 和 WOD-E2E 上至少同样出色。在 AlpaSim 闭环中,每个故障事件的公里数对两个规划器来说都保持在延迟最多 300 毫秒,超过之后会下降。在将 GPU 时间计入延迟后,SuperGlance 相对于 Qwen-Drive-1.0 的领先优势从 2.8 倍扩大到 3.9 倍。延迟 1.2 秒后,我们的模型在自身 518 毫秒时低于 Qwen-Drive-1.0。
A Unified Control Framework, Mathematical Model, and Five-layer Closed-loop Case Study for Embodied Intelligence Control Systems
一个统一控制框架、数学模型及五层闭环案例研究,适用于具身智能控制系统
- Authors: Yiwei Zhou, Zhongcheng Lei, Xiaoran Dai, Wenshan Hu, Hong Zhou
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.19187
- Pdf link: https://arxiv.org/pdf/2611.19187
- Abstract
Embodied intelligence holds that an agent develops cognition and behavior through real-time body-environment interaction, so its control system must tightly couple perception, decision making, learning, optimization, constraints, and physical execution. This report asks whether a unified framework exists, how its mathematical models unify, and how the framework is realized computably. Four conclusions emerge. (1) No single framework is accepted, but the field converges on a common skeleton: from operational-space and whole-body control to MPC, reinforcement learning, active inference, and diffusion/flow policies, all reduce to constrained sequential decision making on the configuration manifold. (2) Two languages describe it-deterministic optimal control (Euler-Lagrange plus constrained OCP, reduced in real time to a QP) and probabilistic inference (MDP/POMDP plus free-energy minimization)-rigorously bridged by control as inference and KL/path-integral control. (3) Engineering unification is already achieved by constrained optimization; the new frontier is cross-morphology, cross-task shared representation. (4) The hardest gap is provable stability and safety, the entry point for spectral, geometric, differentiable-optimization, and safe-learning methods. A single-link manipulator then runs through all five layers: L4 parses the instruction into a reference, L3 identifies a Koopman lifted linear world model, L2 solves a receding constrained QP, L1 adds torque saturation and a CBF, and L0 is the true nonlinear plant. Closed-loop tracking RMSE is 0.0218 rad with all constraints satisfied and a world-model error near 5 x 10^-6 rad, reported with curves and a 3D animation.
- 中文摘要
具身智能认为,智能通过实时的身体-环境互动发展认知和行为,因此其控制系统必须紧密结合感知、决策、学习、优化、约束和物理执行。本报告探讨是否存在统一框架,其数学模型如何统一,以及该框架如何实现可计算。得出四个结论。(1)没有接受单一框架,但该领域汇聚于一个共同的骨架:从操作空间控制和全身控制到MPC、强化学习、主动推理以及扩散/流策略,所有这些都归结为对配置流形的受限顺序决策。(2)两种语言描述了确定性最优控制(欧拉-拉格朗日加约束OCP,实时还原为QP)和概率推断(MDP/POMDP加自由能最小化)——通过控制作为推断和KL/路径积分控制进行严格衔接。(3)工程统一已通过受限优化实现;新的前沿是跨形态学、跨任务共享表示。(4)最难的差距是可证明的稳定性和安全性,这是谱、几何、可微优化和安全学习方法的切入点。然后,一个单链路操作器会穿过所有五层:L4 解析指令为参考,L3 识别一个 Koopman 提起的线性世界模型,L2 求解一个后退的受约束 QP,L1 添加扭矩饱和和 CBF,而 L0 是真正的非线性装置。闭环跟踪RMSE在所有约束满足时为0.0218弧度,世界模型误差约为5 x 10^-6弧度,并以曲线和三维动画报告。
Distilling Safety Filters for Filter-free Multi-agent RL
无滤芯多剂RL的蒸馏安全过滤器
- Authors: Srikar Babu Gadipudi, Somayeh Sojoudi, Claire J. Tomlin
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.19190
- Pdf link: https://arxiv.org/pdf/2611.19190
- Abstract
Reinforcement learning (RL) can teach robot teams to navigate shared spaces, but trial and error causes collisions that real robots cannot afford. Control barrier function (CBF) safety filters prevent these collisions by correcting unsafe actions, yet in a team the filter becomes a centralized optimization whose number of constraints grows quadratically with the team size. We propose an approach that builds the filter's safety into the RL policy and examine how such a system behaves during filter-free deployment. However, the filter that keeps training safe also hides what the policy must learn: it masks the cost of unsafe actions, which actor-critic methods then overvalue, and it keeps near-collision states out of the training data. Our approach therefore learns only from executed safe actions with implicit Q-learning and distills the filter into the policy at synthetic near-collision states. We prove that filter distillation loss bounds the policy's constraint violation and hence how far agents can cross their safety margin, so errors never accumulate. When distillation states cover deployed states, the loss also bounds the collision probability at every step. Without a filter and acting only on local observations, our policies have no collisions in 1,800 test episodes with multiple agents. A policy trained with three agents transfers zero-shot to teams of twenty at constant per-agent computation and stays collision-free with up to eight agents. We test our policies on Crazyflie quadrotors with up to five flying at the same time.
- 中文摘要
强化学习(RL)可以教会机器人团队如何导航共享空间,但反复试验会导致真实机器人无法承受的碰撞。控制屏障功能(CBF)安全过滤器通过纠正不安全的行为来防止这些碰撞,但在团队中,过滤器变成了集中式优化,其约束数量随团队规模平方增长。我们提出了一种方法,将过滤器的安全性纳入强化学习策略,并研究该系统在无过滤器部署时的行为。然而,保持训练安全的过滤器也隐藏了策略必须学习的内容:它掩盖了不安全行为的成本,从而高估了这些行为,并将近碰撞状态排除在训练数据之外。因此,我们的方法仅从带有隐式Q学习的安全动作中学习,并将过滤器提炼进合成近碰撞状态下的策略中。我们证明了滤波器蒸馏损耗限制了策略的约束违规,从而限制了代理能跨越安全边界的距离,因此错误不会累积。当蒸馏状态覆盖已部署状态时,损失也会限制每一步的碰撞概率。在没有滤波器且仅基于局部观测的情况下,我们的策略在1800次测试中与多个代理不发生碰撞。由三个代理训练的策略会以恒定每个代理计算的零点任务转移到二十人小组,最多八个代理时保持无碰撞。我们在Crazyflie四旋翼机上测试策略,同时最多可同时飞行五个。
LogT: Logically Think with Images for Visual Search
LogT:用图像进行逻辑思考以进行视觉搜索
- Authors: Yanjun Fu, Quanzeng You, Jiadong Guo, Yujie Lu, Sai Charitha Akula, Haiping Wu, Miao Liu, Sanghamitra Dutta, Lu Yuan
- Subjects: Subjects:
Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.19200
- Pdf link: https://arxiv.org/pdf/2611.19200
- Abstract
Thinking with images empowers vision-language models (VLMs) to use an external Zoom-in tool for fine-grained visual search. Recent works employ supervised fine-tuning (SFT) and reinforcement learning (RL) with tool-specific designs to enable such capabilities. However, achieving logical reasoning alongside accurate and efficient tool use remains a significant challenge. To address this, we shift the focus away from increasing RL complexity and instead identify SFT data quality as a key bottleneck. We then propose LogT (short for Logically Think with Images), a fully automated SFT data engine that explicitly enforces logic-chain-guided reasoning. LogT generates an SFT dataset featuring challenging, visually grounded questions and structured tool-use traces with human-like logic. We validate our approach by fine-tuning two base models, Qwen2.5-VL-7B and InternVL3.5-8B, on the dataset and evaluating them across seven benchmark test sets. We show that LogT-SFT based on Qwen2.5-VL-7B outperforms the state-of-the-art trained with both SFT and RL, despite using 3.1$\times$ less data and $>120\times$ fewer GPU hours. Furthermore, applying a simple RL recipe without tool-specific design on LogT-SFT yields a model that surpasses prior methods with complex RL setups by a substantial 9.59-point margin. In addition to strong performance gains, our model exhibits higher tool-use accuracy and efficiency, as well as more structured and logically consistent reasoning traces, directly validating the effectiveness of our approach.
- 中文摘要
用图像思考使视觉语言模型(VLM)能够使用外部放大工具实现细粒度的视觉搜索。近期研究采用监督式微调(SFT)和强化学习(RL),并结合工具专用设计来实现此类能力。然而,在准确高效工具使用的同时实现逻辑推理仍是一个重大挑战。为解决这一问题,我们将重点从增加强化学习复杂度转向将SFT数据质量视为关键瓶颈。随后,我们提出了LogT(LogT,全自动的图像逻辑思维)数据引擎,明确执行逻辑链引导推理。LogT生成一个SFT数据集,包含具有挑战性、视觉化的问题和具结构化工具使用痕迹,逻辑类人。我们通过在数据集上微调两个基础模型 Qwen2.5-VL-7B 和 InternVL3.5-8B,并在七个基准测试集中进行评估,验证了我们的方法。我们证明,基于 Qwen2.5-VL-7B 的 LogT-SFT 在同时用 SFT 和 RL 训练的先进模型中表现优于最先进方案,尽管其使用的数据少了 3.1$\times$,GPU 小时也少了 $>120\times$。此外,在 LogT-SFT 上应用一个没有工具特定设计的简单 RL 配方,得出的模型比以往复杂 RL 设置方法高出 9.59 分。除了性能强劲提升外,我们的模型还展现出更高的工具使用准确性和效率,以及更结构化、逻辑一致的推理痕迹,直接验证了我们方法的有效性。
Temporal Attention-Enhanced Mixture-of-Experts Guidance for Multi-Revolution Low-Thrust Rendezvous
时间注意力增强专家混合指导,适用于多转低推力会合
- Authors: Chenglong Wu, Yi Zhou, Hongxu Zhu, Jialong Song, Hexi Baoyin
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.19226
- Pdf link: https://arxiv.org/pdf/2611.19226
- Abstract
Multi-revolution low-thrust rendezvous requires both long-duration orbital transfer and precise terminal control. This paper proposes a temporal attention-enhanced mixture-of-experts guidance law trained by deep reinforcement learning. A far-range expert acts on modified equinoctial elements, a terminal expert acts on the relative state in the target's local frame, and a learned gate blends their thrust commands. The terminal expert encodes its observation history with a Transformer to address partial observability, and the gate uses a separate encoder to account for the resulting history-dependent command. The experts are pretrained separately, the gate is trained with both experts frozen, and the complete guidance law is then fine-tuned end to end. On 600 test missions, it achieves rendezvous in every case at tolerances of 10 m and 1 m/s, reduces mean flight time and delta-v by 50.6% and 44.5% relative to distance-based hard switching, and remains successful without retraining under navigation errors, unmodeled orbital perturbations, eclipse-induced thrust outages, and thrust loss.
- 中文摘要
多周低推力交会需要长时间轨道转移和精确的终端控制。本文提出了一种通过深度强化学习训练的、时间注意力增强的专家混合引导定律。远距离专家对修改后的春分点元素进行作用,终端专家根据目标局部参考系中的相对状态作用,而学习过的门则混合推力指令。终端专家用变压器编码其观测历史以解决部分可观测性问题,而门则使用单独编码器来处理由此产生的历史依赖指令。专家分别预训练,门在两位专家都冻结状态下训练,然后对完整的制导定律进行端到端微调。在600次测试任务中,它在每次情况下都能在10米和1米/秒的容差下实现交会,平均飞行时间和delta-v相较于基于距离的硬切换分别减少了50.6%和44.5%,并且在导航误差、未模拟轨道扰动、日食引发推力中断和推力损失的情况下保持成功。
Spreads Don't Price a Rival Market Maker's Information
价差不会给竞争对手做市商的信息定价
- Authors: Oliver James Oberle, Damien Berriaud, Roger Wattenhofer
- Subjects: Subjects:
Computer Science and Game Theory (cs.GT)
- Arxiv link: https://arxiv.org/abs/2611.19289
- Pdf link: https://arxiv.org/pdf/2611.19289
- Abstract
When one market maker becomes better informed, do its competitors widen their bid-ask spreads? Adverse-selection reasoning suggests that they should, because the better-informed maker avoids toxic order flow and leaves more of it to the others, and the closest model of competition with an informed dealer predicts more conservative quotes by the uninformed dealers. We study this question in a stochastic quoting game in which one maker receives a private signal of varying precision about a persistent state that drives order flow. Strategies are learned with independent reinforcement learning, which we benchmark against an exact dynamic-programming solution of the single-agent case. In none of the configurations we examine does the average spread of the uninformed makers increase with the rival's precision. Where competition for order flow is weak, the slope of their spread in precision is statistically equivalent to zero within half an action tick; where competition is strong, their spread tightens. The informed maker's own spread tightens markedly in every configuration and serves as a positive control. These results hold with two, three and four makers, across an eightfold variation in the depth of the background liquidity, with a background quote that reprices after toxic fills, and under price-time priority. Our measurements point to a competitive trade-off. In the model the total toxicity of the flow does not depend on any maker's information. A better-informed maker avoids more of it, and where this leaves more toxic flow to its competitors, it also quotes more tightly, so that a wider quote would cost them more order flow than it would save in adverse selection. The uninformed makers adjust the skew of their quotes and their inventory instead, and the rival's information changes the distribution of profits across makers while quoted spreads remain nearly unchanged.
- 中文摘要
当某家做市商变得更了解情况时,其竞争对手是否会扩大买卖差?逆向选择推理表明他们应该这样做,因为信息更充分的做市商会避免有毒的订单流,把更多订单留给其他人,而最接近的竞争模型中,知情交易者会预测无知交易商会给出更保守的报价。我们在一个随机报价游戏中研究这个问题,其中一位做市商收到一个关于驱动订单流的持久状态的私密信号,其精度各异。策略通过独立强化学习学习,我们将其与单代理情形的精确动态规划解进行基准对比。在我们考察的任何配置中,没有任何一种配置会随着竞争对手的精度提升而增加,这些无知做商的平均价差会随竞争者精度的提升而增加。当订单流竞争较弱时,其精度差距斜率在半个动作周期内统计上等同于零;当竞争激烈时,其差距会收紧。知情做商自身的价差在每种配置下都会显著收紧,并作为正对照。这些结果适用于两家、三家和四家做价商,背景流动性深度有八倍变化,背景报价在有毒填充后重新定价,且价格时间优先级。我们的测量表明存在竞争权衡。在模型中,整体流量的毒性不依赖于任何做商的信息。信息更充分的做板商会避免更多此类信息,而在这让更多有害流量给竞争对手时,它也会更紧密地报价,因此更宽的报价会让他们损失的订单流多于在逆向选择中节省的费用。不知情的做板商会调整其报价和库存的偏斜,而竞争对手的信息改变了做盘商间利润分布,而报价价差几乎保持不变。
MARS: Dynamic Routing of Reinforcement-Learned Atomic Skills for Multi-Stage Robotic Manipulation
MARS:增强学得的原子技能在多阶段机器人操作中的动态路由
- Authors: Zequan Zeng, Junqin Yang, Kai Wang
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.19324
- Pdf link: https://arxiv.org/pdf/2611.19324
- Abstract
Vision-Language-Action (VLA) models have demonstrated strong capabilities in multi-task robotic manipulation. However, they still face limitations in control precision and execution stability during critical manipulation stages, such as object grasping, alignment and insertion. To combine general-purpose manipulation capabilities with task-specific fine-grained control, while enabling effective coordination between generalist and specialized policies throughout multi-stage manipulation tasks, this paper proposes MARS (Modular Atomic-skill Routing System), a dynamic routing framework for reinforcement learning-based atomic skills. MARS integrates a frozen generalist VLA policy with multiple independently trained atomic skill policies. Using a shared RL Token state representation and the reference action of VLA, specialized control policies are optimized through reinforcement learning for different cr`itical manipulation stages. For high-level policy selection, we develop a dynamic router that integrates temporal context and skill semantics. The router employs Mamba-based recurrent encoding and a historical feature window to capture state evolution during manipulation. Skill descriptions of candidate policies are encoded into semantic queries, and attention mechanisms are used to establish correspondences between skill semantics and temporal state features. Experimental results show that MARS achieves an average Macro-F1 score of 90.47% across four LIBERO task suites under fixed-trajectory evaluation, outperforming an MLP-based router by 7.08 percentage points. In the simulated Tube Transport task, the full-task success rate increases from 20.04% to 85.67%. These results demonstrate that stage-specific reinforcement learning and temporal state-aware dynamic policy routing can improve multi-policy coordination and overall task execution performance in multi-stage robotic manipulation.
- 中文摘要
视觉-语言-动作(VLA)模型已展现出多任务机器人操作的强大能力。然而,它们在关键操作阶段(如对象抓取、对齐和插入)仍面临控制精度和执行稳定性的限制。为将通用操作能力与任务特定细粒度控制相结合,同时实现多阶段操作任务中通用策略与专业策略的有效协调,本文提出了MARS(模块化原子技能路由系统)——一种基于强化学习的原子技能动态路由框架。MARS 将冻结的通用VLA策略与多个独立训练的原子技能策略整合。通过共享的 RL 令牌状态表示和 VLA 的参考动作,针对不同关键操作阶段的强化学习优化了专用控制策略。在高层策略选择中,我们开发了一个动态路由器,将时间上下文与技能语义整合在一起。该路由器采用基于 Mamba 的循环编码和历史特征窗口,捕捉操作过程中的状态演变。候选策略的技能描述被编码为语义查询,并利用注意力机制建立技能语义与时间状态特征之间的对应关系。实验结果显示,MARS 在固定轨迹评估下,四个 LIBERO 任务套件中平均 Macro-F1 得分为 90.47%,比基于 MLP 的路由器高出 7.08 个百分点。在模拟的 Tube Transport 任务中,全任务成功率从 20.04% 提升到 85.67%。这些结果表明,阶段特定强化学习和时间状态感知动态策略路由能够提升多策略协调和多阶段机器人操作中的整体任务执行性能。
Gradient-Based Co-Design of Systems and Reinforcement Learning Policies with Differentiable Simulators
基于梯度的系统与强化学习策略的协同设计,并使用可微分模拟器
- Authors: Akhil S. Anand, Xun Shen, Marc Peter Deisenroth, Sebastien Gros
- Subjects: Subjects:
Machine Learning (cs.LG); Systems and Control (eess.SY); Optimization and Control (math.OC)
- Arxiv link: https://arxiv.org/abs/2611.19329
- Pdf link: https://arxiv.org/pdf/2611.19329
- Abstract
Co-design is the problem of jointly optimizing the design of a system and the policy that operates it. Neither can be chosen independently: a system design is only as good as the policy trained for it, and the optimal policy changes with the design. When that policy is learned by reinforcement learning (RL), evaluating a candidate design requires a full policy training run, and the design is usually optimized by black-box search such as Bayesian optimization (BO) or evolution strategies (ES). These methods scale poorly with the design dimension and ignore the differentiable structure of modern simulators and, in particular, the sensitivity of the optimal policy with respect to the design. We instead derive a gradient-based co-design method, where the design gradient decomposes into a direct effect of changing the design at a fixed policy and an indirect effect through the design-dependent optimal policy. The indirect effect requires the sensitivity of the converged policy to the design, which we obtain by applying the implicit function theorem to the stationarity condition of the policy gradient. Based on this sensitivity, we develop a proof-of-concept co-design algorithm with stochastic value gradient (SVG)-style policy optimization, prove that it converges to an approximately stationary design, and demonstrate on a differentiable networked-system benchmark with high-dimensional design spaces that it consistently outperforms BO and ES.
- 中文摘要
协同设计是指共同优化系统设计及其运行策略的问题。两者都不能独立选择:系统设计的优劣取决于为其训练的策略,而最优策略会随设计变化。当该策略通过强化学习(RL)学习时,评估候选设计需要完整的策略训练运行,设计通常通过贝叶斯优化(BO)或演化策略(ES)等黑箱搜索进行优化。这些方法在设计维度上扩展性较差,且忽视了现代模拟器的可微结构,尤其是最优策略相对于设计的敏感性。我们取而代之的是基于梯度的协同设计方法,其中设计梯度分解为在固定策略下直接改变设计的影响,以及通过设计依赖的最优策略产生的间接效应。间接效应要求收敛策略对设计的敏感度,我们通过将隐函数定理应用于策略梯度的平稳条件得到。基于这种敏感性,我们开发了一种带有随机值梯度(SVG)策略优化的概念验证协设计算法,证明其收敛到近似平稳设计,并在高维设计空间的可微网络系统基准测试中证明其持续优于BO和ES。
Delay-Aware Reinforcement Learning for O-RAN Control under Stochastic Reward Feedback
在随机奖励反馈下用于O-RAN控制的延迟感知强化学习
- Authors: Xingqi Wu, Junaid Farooq, Tao Li, Juntao Chen, Ying Wang
- Subjects: Subjects:
Multiagent Systems (cs.MA); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19344
- Pdf link: https://arxiv.org/pdf/2611.19344
- Abstract
Reinforcement learning (RL) is increasingly employed for real-time control in Open Radio Access Networks (O-RANs). However, the asynchronous nature of O-RAN feedback introduces stochastic delays between actions and their observable rewards, leading to misaligned credit assignment and unstable training. This letter formulates the O-RAN control problem as a Markov decision process (MDP) with delayed and stochastic rewards, where the true reward corresponding to an action is received after an unknown delay. To address this, we propose a delay-aware deep RL framework that integrates a long short-term memory (LSTM)-based delay predictor and a temporal reward realignment module within an off-policy learning pipeline. Using empirical delay traces collected from the Open AI Cellular (OAIC) testbed, we demonstrate that the proposed framework achieves faster convergence and significantly reduces latency violations and packet drops compared to conventional RL methods that ignore feedback delay. The results confirm that reward-delay compensation is critical for reliable AI-driven O-RAN control.
- 中文摘要
强化学习(RL)正日益被应用于开放无线接入网络(O-RAN)中的实时控制。然而,O-RAN 反馈的异步特性导致动作与其可观赏奖励之间存在随机延迟,导致学分分配不匹配和训练不稳定。本信将 O-RAN 控制问题表述为一个具有延迟和随机奖励的马尔可夫决策过程(MDP),即在未知延迟后获得与某一行为相对应的真实奖励。为解决这一问题,我们提出了一个延迟感知深度强化学习框架,该框架将基于长短期记忆(LSTM)的延迟预测器和时间奖励重配模块整合在非策略学习管道中。利用从 Open AI Celluld(OAIC)测试平台收集的经验延迟痕迹,我们证明所提框架相比忽视反馈延迟的传统 RL 方法实现了更快的收敛速度,并显著减少了延迟违规和丢包。结果证实,奖励-延迟补偿对于可靠的 AI 驱动 O-RAN 控制至关重要。
SliceWise: Traffic-Aware Multi-Agent RL with Temporal Encoding for Joint Slice Admission and Resource Orchestration in O-RAN
SliceWise:带有时间编码的流量感知多代理RL,用于O-RAN中的联合切片准入和资源编排
- Authors: Xingqi Wu, Yuhui Wang, Junaid Farooq, Hakim Ghazzai, Gianluca Setti
- Subjects: Subjects:
Networking and Internet Architecture (cs.NI)
- Arxiv link: https://arxiv.org/abs/2611.19357
- Pdf link: https://arxiv.org/pdf/2611.19357
- Abstract
Ultra-reliable low-latency communication (URLLC) services in mission-critical applications such as industrial automation, remote healthcare, and autonomous systems demand stringent quality-of-service (QoS) guarantees under dynamic, resource-constrained wireless conditions. Open radio access networks (O-RAN) offer a software-defined and programmable foundation for embedding AI-driven control. This letter proposes SliceWise, a traffic-aware multi-agent deep reinforcement learning (MARL) framework for joint admission control, user-slice association, and resource orchestration in O-RAN. The control problem is modeled as a cooperative multi-agent Markov decision process, with long short-term memory (LSTM) networks capturing temporal traffic dynamics and dueling double deep Q-networks (D3QN) ensuring stable learning. Simulation results show that SliceWise substantially reduces delay violations and packet drops while improving slice-level reliability and resource efficiency, offering a scalable orchestration solution for AI-native O-RAN deployments.
- 中文摘要
在工业自动化、远程医疗和自主系统等关键任务应用中,超可靠低延迟通信(URLLC)服务在动态、资源受限的无线条件下要求严格的服务质量(QoS)保证。开放无线接入网络(O-RAN)为嵌入AI驱动控制提供了软件定义和可编程的基础。本信提出了SliceWise,一个用于O-RAN中联合准入控制、用户切片关联和资源编排的流量感知多智能体深度强化学习(MARL)框架。控制问题被建模为一个协作式多智能体马尔可夫决策过程,长短期记忆(LSTM)网络捕捉时间流量动态,双重深度Q网络(D3QN)则确保学习稳定。仿真结果显示,SliceWise 在提升切片级可靠性和资源效率的同时,大幅减少了延迟违规和丢包,为 AI 原生 O-RAN 部署提供了可扩展的编排解决方案。
SARGO: Agreement-Gated Reinforcement and Suppression via On-Policy Self-Guidance for Visual Hallucination Mitigation
SARGO:通过策略上自我指导实现的协议门槛强化与抑制,用于视觉幻觉缓解
- Authors: Jinyu Li, Zhiming Zhang, Hao Fang, Hongyao Yu, Bin Chen, Albert Ma, Shu-Tao Xia
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19388
- Pdf link: https://arxiv.org/pdf/2611.19388
- Abstract
Vision-language models remain susceptible to visual hallucinations, often mixing grounded information with unsupported claims in the same response. Response-level reinforcement learning assigns a shared advantage to all tokens in a response, causing negative updates to suppress correct information alongside unsupported claims. To reduce hallucination while preserving useful visual information, we introduce SARGO (Self-guided Advantage-Routed Grounding Optimization), a post-training framework for selective reinforcement and suppression through on-policy self-guidance. Direct and reciprocal weighting gives better-supported tokens stronger reinforcement on positive updates and weaker penalties on negative updates. Agreement between overall and hallucination-specific advantages selects when to apply this guidance. At the response level, a reference-calibrated reward couples factuality with the retention and expansion of grounded information, discouraging lower hallucination scores obtained through omission. Experiments on Qwen3-VL-4B, Qwen3-VL-8B, and Gemma3-4B demonstrate that SARGO outperforms the evaluated mitigation baselines across Obj-Hal, AMBER, THRONE, and SHR evaluations. SARGO achieves these gains while preserving useful visual information and general multimodal capabilities.
- 中文摘要
视觉语言模型仍易受视觉幻觉影响,常常在同一反应中将有根据的信息与无依据的主张混合在一起。响应级强化学习为响应中的所有标记分配共享优势,导致负向更新抑制正确信息与无支持主张。为减少幻觉同时保留有用的视觉信息,我们引入了SARGO(自我引导优势引导基础优化),这是一种通过策略内自我引导实现选择性强化和抑制的后训练框架。直接权重和互惠权重使支持较好的代币在正向更新时获得更强的强化,而对负面更新则给予较弱惩罚。整体优势与幻觉特定优势的一致性决定了何时应用这一指导。在响应层面,参照校准奖励将事实性与基于基础信息的保留和扩展结合起来,从而抑制因遗漏而获得的较低幻觉分数。对Qwen3-VL-4B、Qwen3-VL-8B和Gemma3-4B的实验表明,SARGO在Obj-Hal、AMBER、THRONE和SHR评估中优于已评估的减缓基线。SARGO在保留有用视觉信息和通用多模态能力的同时实现了这些提升。
Bellman Residual Bracketing: Resolving the Sample Complexity of Discounted and Average-Reward RL
贝尔曼残余括号:解析折现后和平均回报RL的样本复杂度
- Authors: Matthew Zurek, Yudong Chen
- Subjects: Subjects:
Machine Learning (cs.LG); Information Theory (cs.IT); Optimization and Control (math.OC); Probability (math.PR); Machine Learning (stat.ML)
- Arxiv link: https://arxiv.org/abs/2611.19566
- Pdf link: https://arxiv.org/pdf/2611.19566
- Abstract
We study the sample complexity of the empirical planning or plug-in approach for reinforcement learning (RL) with a generative model. We show the discounted plug-in approach finds an $\varepsilon$-optimal policy with $O(SA (1-\gamma)^{-3} \varepsilon^{-2} \log(SA/\delta))$ samples, where $S, A, \gamma, \delta$ are the state and action sizes, discount factor, and failure probability, respectively. In the average-reward setting, we show that discounted plug-in and direct (undiscounted) plug-in achieve $O(SAH\varepsilon^{-2}\log(SA/\delta))$ and $O(SAD\varepsilon^{-2}\log(SA/\delta))$ complexities, respectively, where $H$ upper-bounds the optimal bias span and $D$ is the diameter. Combining with our lower bounds for nonadaptive generative models, these results resolve the minimax sample complexity of discounted RL and average-reward RL --- in both the span- and diameter-based classes --- up to constant factors. Our main technical contribution is a novel Bellman residual bracketing technique which improves the efficiency of prior absorbing-MDP-based concentration analyses.
- 中文摘要
我们研究了基于生成模型的经验规划或插件强化学习(RL)方法的样本复杂度。我们证明,折现插件方法在使用 $O(SA (1-\gamma)^{-3} \varepsilon^{-2} \log(SA/\delta))$ 样本时,找到了 $\varepsilon$ 最优策略,其中 $S、A、\gamma、\delta$ 分别为状态和动作大小、折现因子 分别是失败概率。在平均-奖励设置下,我们表明折现插件和直接(未折现)插件分别达到$O(SAH\varepsilon^{-2}\log(SA/\delta))$)和$O(SAD\varepsilon^{-2}\log(SA/\delta)$复杂度,其中$H$为最佳偏置跨度的上界,$D$为直径。结合我们对非自适应生成模型的下界,这些结果解决了基于张距和直径类别的折现RL和平均回报RL---的极小极大样本复杂度---常数因子。我们的主要技术贡献是一种新型 Bellman 残差包围技术,提高了先前基于 MDP 吸收浓度分析的效率。
Towards Probabilistically Safe Reinforcement Learning: Theory and Deployment on Humanoid and Aerial Robots
迈向概率安全强化学习:理论与在类人机器人和空中机器人上的应用
- Authors: Pol Mestres, Varun Gabbita, Jason H. K. Siu, Sergio A. Esteban, Hampus Serneke, Joel W. Burdick, Aaron D. Ames
- Subjects: Subjects:
Robotics (cs.RO); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.19616
- Pdf link: https://arxiv.org/pdf/2611.19616
- Abstract
Reinforcement learning (RL) enables robots to acquire complex skills, but standard RL formulations do not guarantee that learned policies satisfy safety constraints, which limits their use in safety-critical settings. Existing safe RL methods with provable guarantees typically support only restricted classes of constraints and require unbiased estimators of the objective and constraint functions, which prevents their direct integration with state-of-the-art algorithms such as proximal policy optimization (PPO). In this paper, we extend the recently introduced robust safe gradient flow-based reinforcement learning (RSGF-RL) framework to overcome these limitations. Our formulation accommodates a general class of trajectory-level safety constraints, including chance constraints such as collision avoidance with a prescribed probability, and our analysis allows for biased value and gradient estimators. We show that, with high probability, the constraint violation throughout training is bounded by the estimator bias plus a term that vanishes as the number of rollouts grows, and decays to this level if the initial policy is unsafe. We also establish the convergence properties of the iterates to approximate KKT points of the original safe RL problem. Finally, we show that RSGF-RL implemented with PPO-style estimates enables zero-shot sim-to-real deployment of whole-body control policies on a Unitree G1 humanoid and collision-free waypoint navigation on a Crazyflie drone.
- 中文摘要
强化学习(RL)使机器人能够习得复杂技能,但标准强化学习表述并不能保证所学策略满足安全约束,这限制了其在安全关键环境中的应用。现有具有可证明保证的安全强化学习方法通常只支持受限的约束类别,并要求对目标函数和约束函数进行无偏估计,这阻碍了它们与诸如近端策略优化(PPO)等最先进算法的直接集成。本文将新近推出的稳健安全梯度流强化学习(RSGF-RL)框架扩展以克服这些限制。我们的表述涵盖了一类轨迹级安全约束,包括以规定概率避免碰撞等偶然约束,并支持有偏值和梯度估计。我们表明,在训练过程中,约束违规的概率被估计偏误加一个随着部署次数增加而消失的项所限制,如果初始策略不安全,则会衰减到该水平。我们还建立了迭代次数的收敛性质,以近似原始安全强化学习问题的KKT点。最后,我们证明,基于PPO式估计实现的RSGF-RL实现了在Unitree G1人形机器人上零射模拟到真实的全机控制策略部署,并在Crazyflie无人机上实现无碰撞航点导航。
Toward Planning Representations That Separate Task Structure from Visual Content
迈向规划将任务结构与视觉内容分离的表述
- Authors: Aarav Srivastava, Vishwas Sathish, Rajesh P. N. Rao
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.19621
- Pdf link: https://arxiv.org/pdf/2611.19621
- Abstract
Effective planning requires representations that preserve task-relevant geometry and object identity while ignoring visual variation that does not alter the task. Existing evaluations of visual generalization often change several factors simultaneously, making the source of a failure difficult to identify. We introduce a controlled evaluation of object-goal navigation (ObjectNav), the task of reaching an instance of a named object category, in procedurally generated ProcTHOR houses. Training and evaluation houses share floor plans, object locations, goals, and start states, while visual factors are changed individually or in combination. Across model-free reinforcement learning (RL), model-based RL, and agents using frozen pretrained visual representations, our results show that appearance changes impair navigation, but their effects differ sharply by factor. Surface material changes account for most of the average change in success, whereas clutter and lighting have little effect. The magnitude of a global image statistics shift does not reliably predict performance: changing only the goal object's appearance can substantially impair an end-to-end agent despite a small image-level change. We found that a model based agent that plans over frozen features from the pretrained DINOv2 image encoder achieves the highest success (0.83) under the full shift, but remains sensitive to surface appearance. Our experiments reveal an important requirement for planning representations: they must separate invariant spatial task structure from appearance.
- 中文摘要
有效的规划需要在不影响任务的视觉变化的同时,保留任务相关几何形状和对象身份的表征。现有的视觉泛化评估往往同时改变多个因素,使得识别失败的根源变得困难。我们在程序生成的ProcTHOR房屋中引入了对象目标导航(ObjectNav)的受控评估,该任务是到达命名对象类别的实例。培训和评估房屋共享平面图、物体位置、目标和起始状态,而视觉因素则单独或组合改变。在无模型强化学习(RL)、基于模型的强化学习以及使用冻结预训练视觉表示的代理中,我们的结果表明外观变化会影响导航,但其影响因因素而异。表面材料的变化解释了成功率平均变化的主要原因,而杂乱和照明则影响有限。全局图像统计偏移的幅度无法可靠预测性能:仅改变目标对象外观,即使图像层面变化很小,也可能对端到端智能件造成显著影响。我们发现,基于模型的智能体在预训练 DINOv2 图像编码器中规划冻结特征时,在完整偏移下实现最高成功率(0.83),但对表面外观仍保持敏感。我们的实验揭示了规划表示的一个重要要求:必须将不变的空间任务结构与外观分离。
Attacker Diversity Reduction in Adaptive LLM Red Teaming
自适应大型语言模型红队中攻击者多样性的减少
- Authors: Chris Tava
- Subjects: Subjects:
Cryptography and Security (cs.CR); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19680
- Pdf link: https://arxiv.org/pdf/2611.19680
- Abstract
Reinforcement learning can make automated red-teaming more successful while narrowing the attacks it produces. Co-training an attacker and a defender with group-relative policy optimization (GRPO) has been reported to reduce attacker diversity. This experiment reproduces attack diversity reduction using three runs with a Qwen2.5-14B attacker and a Phi-4-mini defender trained by direct preference optimization (DPO). The attacker receives a success-only reward and uses a length-unbiased loss, bounded updates, warm-up and ratio clipping. The attacker is evaluated on 26 AdvBench behaviors. Training ended after 12 rounds, following a decision based on development checkpoints, so the reported intervals are nominal. Attack Diversity Retention (ADR) compares the within-behavior of the trained attacker spread with the untrained attacker. In order for an attack diversity to be consider as having been reduced, a criterion requires the ADR to be below a 0.75 target with a 95% confidence interval together with continued attacker effectiveness. Therefore, if an attacker narrows its attack vector by 25% consistently its considered less diverse. After 24 attacker and 12 defender LORA-based updates, ADR for the three runs were 0.63, 0.61 and 0.63, respectively. Each run met the criterion at round 12; the earlier checks at rounds 4 and 8 fell short. Despite attacks being less diverse, attack success continued to improve as the attacker was updated after each training run.
- 中文摘要
强化学习可以使自动红队行动更成功,同时缩小其所产生的攻击范围。据报道,通过组相对策略优化(GRPO)共同训练攻击者和防御方,可减少攻击者多样性。本实验通过三次运行,分别使用Qwen2.5-14B攻击者和Phi-4-mini防御者,再现攻击多样性的减少效果,该防御者通过直接偏好优化(DPO)训练。攻击者获得仅成功时的奖励,并采用长度无偏损失、有界更新、热身和比例削波。攻击者会根据26项AdvBench行为进行评估。训练在12轮后结束,基于开发检查点作出决策,因此报告的间隔为名义上。攻击多样性保留(ADR)比较受训攻击者与未受训练攻击者的内部行为。为了使攻击多样性被视为被降低,标准要求ADR低于0.75目标,且置信区间为95%,且攻击者持续有效。因此,如果攻击方持续将攻击向量缩小25%,则其多样性较低。经过24次攻击方和12次基于LORA的更新后,三次运行的ADR分别为0.63、0.61和0.63。每次运行在第12轮都达到了该标准;此前在第4轮和第8轮的检查未达标。尽管攻击的多样性较少,但随着每次训练后攻击者的更新,攻击成功率持续提升。
RIFT: Reward Improvement Field Transfer for Cross-Domain Offline RL
RIFT:跨域离线 RL 的奖励改进场域传输
- Authors: Jinhee Won, Mardavij Roozbehani, Munther Dahleh
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19681
- Pdf link: https://arxiv.org/pdf/2611.19681
- Abstract
Learning from limited target data is a central challenge in cross-domain offline reinforcement learning. Abundant source data can help, but dynamics differences can make direct transition reuse unreliable. We propose Reward Improvement Field Transfer (RIFT), which instead learns how low- and high-reward source transitions differ. RIFT encodes source and target transitions in a shared latent space, pairs low- and high-reward source transitions using optimal transport, and fits an affine field to their displacements. It applies this field to low-reward target anchors and decodes the results to augment the target dataset. The approach builds on the hypothesis that related domains can share useful low-to-high reward structure despite different dynamics. We bound synthesis error under shared class geometry, identify conditions favoring target anchoring over source-high transplant, and derive a conditional connection to downstream return. Experiments on dense-reward locomotion tasks under gravity, morphology, and friction shifts show improved aggregate performance over existing baselines.
- 中文摘要
从有限的目标数据中学习是跨域离线强化学习中的核心挑战。丰富的源数据可以提供帮助,但动态差异可能使直接迁移的再利用变得不可靠。我们提出奖励改进场转移(RIFT),它用于学习低回报和高回报源迁移的差异。RIFT在共享的潜空间中编码源和目标转移,利用最优传输将低回报和高回报源转移配对,并将仿射场拟合到它们的位移。它将该场应用于低回报目标锚点,并解码结果以增强目标数据集。该方法基于这样一个假设:相关领域即使动态不同,仍能共享有用的低至高奖励结构。我们将合成误差在共享类几何下界定,识别有利于目标锚定而非源高迁移的条件,并推导出与下游返回的条件联系。在重力、形态和摩擦变化条件下的高密度奖励运动任务实验显示,整体性能优于现有基线。
Where Should LLM Agents Adapt at Test Time? In Context, in Weights, or Both
LLM 代理在测试时应在哪些方面进行调整?在语境中、权重中,或两者兼有
- Authors: Parsa Mirtaheri, Mohammadreza Armandpour, Keivan Alizadeh, Iman Mirzadeh, Parshin Shojaee, Aryo Lotfi, David Harrison, Mehrdad Farajtabar
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.19700
- Pdf link: https://arxiv.org/pdf/2611.19700
- Abstract
LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure. Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. Adding GRPO to this policy achieves the highest mean success, 51.6%. While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. However, ICL also frequently skips inspection and verification for the current task. Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.
- 中文摘要
LLM代理在同一环境(如代码仓库、数据库或应用套件)中重复运行时,会遇到共享潜在结构的任务。经验可以揭示这种结构,使代理能够重用环境知识并从环境反馈中学习。代理可以在测试时通过两种方式利用这种经验进行适应:将其保留在上下文中,并更新权重。这两条通道通常单独研究,导致各自贡献了什么以及何时相互补充尚不清楚。我们比较了上下文学习(ICL)、使用GRPO提供可验证奖励的强化学习,以及使用SDPO单独或组合的自蒸馏,采用共享的在线协议和模块化框架。代理会遇到匹配的任务流,在适应任务前对每个任务进行评估,并仅从自身尝试和环境反馈中学习。我们对来自四个环境的 109 个任务进行了 Qwen3.5-9B 的无状态基线评估,这些任务来自 SWE-bench Verified、AppWorld、ScienceWorld 和 BIRD-Critic-SQLite,每个环境使用五个任务顺序。ICL 提供了最大的独立收益:其最佳策略将跨环境的平均成功率从 28.8% 提高到 49.4%。在该策略中加入 GRPO 可实现最高的平均成功率 51.6%。虽然添加 GRPO 提升了每个 ICL 策略下的整体平均值,但 SDPO 的效果则褒贬不一。痕迹分析显示,在 ICL 下查找次数更少、尝试次数更短,这与对共享结构知识的重复利用相符。然而,ICL 也经常跳过当前任务的检查和验证。在 ICL 中加入 GRPO 有助于恢复这些行为,同时保留其减少的环境查找。这些观察表明,带有可验证奖励的强化学习可以优化代理人使用保留经验的方式,从而推动对上下文管理和权重更新的联合研究。
Sample-Efficient Ensemble Reinforcement Learning at Scale
大规模样本高效集成强化学习
- Authors: Yoav Feigenbaum, Dmitrii Krylov, Roy Fox
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19733
- Pdf link: https://arxiv.org/pdf/2611.19733
- Abstract
Ensemble methods such as MeanQ have been shown to improve the sample efficiency of relatively small model-free deep reinforcement learning (RL) architectures, including DQN and similarly sized variants such as Rainbow. However, to our knowledge, the effects of ensembling on large-scale architectures such as Bigger Better Faster (BBF) have not been studied. In this work, we present MeanQ-BBF, a hybrid model-free algorithm that combines the principles of MeanQ and BBF. Through statistically rigorous evaluation on the Atari 100K benchmark, we show that in contrast to naive scaling of the network size of BBF, which degrades performance, MeanQ-BBF is able to leverage the increase in model capacity to improve performance over BBF. Moreover, MeanQ-BBF does this in a computationally efficient way by organizing the added parameters into largely independent learners. Finally, we discuss the performance-cost tradeoff in the context of practical sample-efficient RL.
- 中文摘要
集成方法如 MeanQ 已被证明能提升相对小型无模型深度强化学习(RL)架构的样本效率,包括 DQN 及类似规模的变体如 Rainbow。然而,据我们所知,集合对大型架构(如 Bigger Better Faster,BBF)的影响尚未被研究。在本研究中,我们介绍了 MeanQ-BBF,一种结合 MeanQ 和 BBF 原理的混合无模型算法。通过对 Atari 100K 基准的统计严谨评估,我们表明,与对 BBF 网络规模进行朴素扩展(降低性能)不同,MeanQ-BBF 能够利用模型容量的提升,提升性能。此外,MeanQ-BBF 通过将新增参数组织成相对独立的学习者,以计算效率的方式实现这一目标。最后,我们讨论了在实际高效样本强化学习的背景下的性能与成本权衡。
Exploration of Reinforcement Learning in Generative Recommendation Models
在生成式推荐模型中探索强化学习
- Authors: Pengfei Zheng, Wuchao Li, Shiyao Wang, Honghui Bao, Hongyi Xie, Jiaxin Deng, Xingmei Wang, Jinghao Zhang, Yufei Ye, Ruiming Tang, Guorui Zhou, Han Li, Kun Gai, Defu Lian
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19742
- Pdf link: https://arxiv.org/pdf/2611.19742
- Abstract
Traditional cascaded recommender systems balance accuracy and efficiency over large item spaces, but misaligned stage-wise objectives hinder end-to-end optimization. Generative recommendation unifies these stages in an autoregressive framework, providing a natural interface for reinforcement learning (RL) to optimize the entire generation process. However, directly applying RL often leads to unstable optimization and degraded recommendation quality. We revisit Group Relative Policy Optimization (GRPO) and other ratio-based objectives through a unified logit-gradient view. We identify three issues: (i) clipped surrogates create dead-gradient regions, under-utilizing samples; (ii) negative-advantage updates can become effectively unbounded as importance ratios grow, allowing a small subset of negatives to dominate optimization; and (iii) ratio-based control lacks sensitivity to absolute policy confidence, so updates on high-confidence cases may fail to vanish. Motivated by this diagnosis, we propose \emph{Gradient-Bounded Policy Optimization} (GBPO), a simple objective with stop-gradient normalization that is simultaneously (a) fully gradient-active, (b) sign-symmetrically bounded for both positive and negative updates, and (c) explicitly confidence-weighted under current policy probabilities. This ensures that updates gracefully vanish for $\pi!\to!1$ positives and $\pi!\to!0$ negatives. Across the evaluated settings, GBPO achieves competitive recommendation performance with more stable training than competing RL methods.
- 中文摘要
传统的级联推荐系统在大型题库中平衡了准确性和效率,但阶段目标错位会阻碍端到端优化。生成式推荐将这些阶段统一在自回归框架中,为强化学习(RL)提供了优化整个生成过程的自然接口。然而,直接应用RL往往会导致优化不稳定和推荐质量下降。我们通过统一的对数梯度视图重新审视了群体相对策略优化(GRPO)及其他基于比率的目标。我们识别出三个问题:(i) 截断代理会造成死梯度区域,导致样本利用率不足;(ii) 随着重要性比的增加,负优势更新可能实际上变得无界,使得少数负面因素主导优化;以及 (iii) 基于比率的控制对绝对策略置信度缺乏敏感性,因此高置信案例的更新可能无法消失。基于这一诊断,我们提出了 \emph{梯度边界策略优化}(GBPO),这是一个具有停止梯度归一化的简单目标,同时(a)完全激活梯度,(b)对正负更新均具有符号对称界限,且(c)在当前策略概率下显式置信加权。这确保了对 $\pi\!to\!1$ 和 $\pi\!!to\0$ 负值的更新都能优雅地消失。在评估的设置中,GBPO 通过比竞争的 RL 方法更稳定的训练,实现了具有竞争力的推荐表现。
Fine-Grained RLAIF for Visual Safety: Learning to Zoom into Hazards for Multimodal Safety Alignment
细粒度 RLAIF 用于视觉安全:学习聚焦危险以实现多模式安全对准
- Authors: Chunxue Xu, Yiwei Wang, Yujun Cai, Songze Li
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.19786
- Pdf link: https://arxiv.org/pdf/2611.19786
- Abstract
Recent multimodal safety alignment heavily relies on a text-centric paradigm, assuming risks are explicit in language or globally prominent in images. However, real-world hazards are often visually subtle or peripheral, causing existing models to fail due to their reliance on coarse, global visual representations. In this work, we study fine-grained visual safety alignment and introduce a challenging 1,200-sample benchmark across four domains, where hazards necessitate localized inspection. To address this, we propose Fine-Grained RLAIF (FGR), a reinforcement learning framework that casts image zoom-in as a learnable perception policy. FGR employs a decomposed reward design to jointly align three core safety behaviors: (i) where to look via a visual evidence reward, (ii) when to act via a tool-formatting constraint, and (iii) how to reason via visually grounded safety explanations. FGR achieves accuracy ranging from 86.71% to 93.00%, consistently outperforming strong multimodal and agent-based baselines.
- 中文摘要
近期多模态安全对齐高度依赖以文本为中心的范式,假设风险在语言中明确存在或在图像中全球显著。然而,现实世界的危害往往在视觉上很微妙或边缘,导致现有模型因依赖粗糙的全局视觉表征而失效。本研究中,我们研究细粒度的视觉安全对齐,并提出了涵盖四个领域、包含1200个样本的挑战性基准,这些领域存在危害需要局部检查。为解决这一问题,我们提出了细粒度RLAIF(FGR),这是一种强化学习框架,将图像放大视为一种可学习的感知策略。FGR采用分解奖励设计,共同对齐三大核心安全行为:(i)通过视觉证据奖励观察何处,(ii)何时通过工具格式约束采取行动,以及(iii)如何通过视觉为基础的安全解释进行推理。FGR的准确率范围在86.71%至93.00%之间,持续优于强劲的多模态和基于因子的基线。
Foundation Pareto Flow Policy for Multi-Objective Reinforcement Learning
多目标强化学习的基础帕累托流策略
- Authors: Zhanjiang Yang, Lijun Sun, Yueming Li, Meng Li, Yijun Yang
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19825
- Pdf link: https://arxiv.org/pdf/2611.19825
- Abstract
The overarching goal of reinforcement learning (RL) is to efficiently optimize the agent's policy for various environments, tasks, dynamics, and even user preferences, which often consist of multiple and potentially conflicting rewards/objectives. This necessitates the learning of Pareto-optimal policies that can navigate different trade-offs among objectives. However, existing multi-objective RL (MORL) methods, whether trained online or offline, are typically limited to task- or objective-specific policy learning and do not address generalization of learned trade-offs to unseen tasks/objectives. In this work, we propose the Foundation Pareto Flow Policy (FP2) for offline MORL in continuous robotic control, inspired by the pre-mid-post-training paradigm underlying recent foundation models. Specifically, we instantiate FP2 as a multi-modal diffusion transformer equipped with a likelihood-free flow matching loss, enabling unified, scalable, and instructable learning of diverse trade-off trajectories. We further develop a three-stage training pipeline to gradually optimize the policy: (1) pre-training on a large-scale MORL dataset, (2) mid-training on a carefully curated high-quality dataset, and (3) post-training on few collected trajectory data from new tasks and trade-offs. Empirical results demonstrate that FP2 outperforms previous state-of-the-art methods on most tasks of the D4MORL benchmark, achieving superior zero-shot interpolation and extrapolation on complex Pareto manifolds as well as efficient few-shot adaptation of the learned trade-off prior to new continuous-control tasks. Code is available at this https URL.
- 中文摘要
强化学习(RL)的总体目标是高效优化代理针对各种环境、任务、动态甚至用户偏好的策略,这些偏好往往包含多个且可能相互冲突的奖励/目标。这需要学习帕累托最优策略,以应对目标之间的不同权衡。然而,现有的多目标RL(MORL)方法,无论在线还是离线训练,通常仅限于任务或目标特定策略学习,未能解决将所学权衡推广到看不见的任务/目标。本研究中,我们提出了基于近期基础模型中中中前中期后训练范式的基础帕累托流策略(FP2),用于持续机器人控制中的离线MORL。具体来说,我们将FP2实例化为一种多模态扩散变换器,具备似然无匹配流损耗,实现对多样化权衡轨迹的统一、可扩展且可指导的学习。我们进一步开发了三阶段训练流程,逐步优化策略:(1)在大规模 MORL 数据集上进行预训练,(2)在精心策划的高质量数据集上进行中期训练,(3)对来自新任务和权衡的少量轨迹数据进行后训练。实证结果表明,FP2在D4MORL基准测试的大多数任务中优于以往最先进的方法,在复杂的帕累托流形上实现了更优的零射值插值和外推,并在新的连续控制任务前对所学权衡进行了高效的少数样本适应。代码可在此 https URL 获取。
Bridging the Terminology Gap: A Data-Centric Alignment Framework for Multilingual Translation in Cross-Border Corporation
弥合术语鸿沟:跨境企业多语种翻译的数据中心对准框架
- Authors: Jinsheng Wei, Song Cheng, Bo Xiao, Ke Zeng
- Subjects: Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19858
- Pdf link: https://arxiv.org/pdf/2611.19858
- Abstract
As globalization accelerates the demand for seamless cross-border enterprise collaboration, multilingual translation remains bottlenecked by insufficient domain-specific terminology, rigid terminology translation, weak instruction-following capabilities, and high inference latency in long-instruction scenarios. To overcome these barriers, we propose Bridging the Terminology Gap (BTG), a data-centric framework. First, BTG mines cross-lingual term alignments from domain documents and leverages LLMs to synthesize terminology-rich parallel corpora. Second, a hybrid filtering strategy combining heuristics and XCOMET scoring ensures high data fidelity. Third, a two-stage training paradigm employs supervised fine-tuning (SFT) for targeted terminology injection, followed by reinforcement learning (RL) to refine output preferences, effectively balancing domain knowledge integration with model reliability. Extensive offline evaluations (e.g., XCOMET, GEMBA, Acc@T) and online human assessments demonstrate significant improvements in terminology accuracy and instruction-following capabilities, validating the effectiveness of our framework, which is now deployed on our cross-border collaboration platform.
- 中文摘要
随着全球化加速了对无缝跨境企业协作的需求,多语言翻译仍受领域专属术语不足、术语翻译僵化、指令跟随能力薄弱以及长指令场景下高推理延迟的瓶颈。为克服这些障碍,我们提出了“弥合术语鸿沟”(BTG)——一个以数据为中心的框架。首先,BTG 从领域文档中挖掘跨语言术语对齐,并利用大型语言模型(LLM)综合富含术语的并行语料库。其次,采用结合启发式与 XCOMET 评分的混合过滤策略,确保高数据忠实度。第三,采用两阶段训练范式,采用监督式微调(SFT)进行有针对性的术语注入,随后通过强化学习(RL)优化输出偏好,有效平衡领域知识集成与模型可靠性。广泛的线下评估(如 XCOMET、GEMBA、Acc@T)以及在线人工评估显示,术语准确性和指令跟随能力显著提升,验证了我们框架的有效性,该框架现已部署在我们的跨境协作平台上。
Max-Min Throughput in RSMA-Aided SWIPT for Sustainable Internet of Underground Things: A Deep Reinforcement Learning Approach
RSMA辅助SWIPT中的最大最小吞吐量用于地下物物联网的可持续:深度强化学习方法
- Authors: Kaiqiang Lin, Mohamed-Slim Alouini
- Subjects: Subjects:
Systems and Control (eess.SY); Machine Learning (cs.LG); Signal Processing (eess.SP)
- Arxiv link: https://arxiv.org/abs/2611.19865
- Pdf link: https://arxiv.org/pdf/2611.19865
- Abstract
Unmanned aerial vehicle (UAV)-enabled wireless-powered underground sensor networks (WPUSNs) allow underground devices (UDs) to harvest radio-frequency energy from a UAV via wireless energy transfer (WET) techniques, enabling sustainable Internet of underground things (IoUT) applications. This paper integrates rate-splitting multiple access (RSMA)-aided simultaneous wireless information and power transfer (SWIPT) into UAV-enabled WPUSNs to deliver information and energy to UDs. Within this framework, we investigate a max-min fairness problem that jointly optimizes the UAV transmit power, power allocation and message splitting, power-splitting ratios, and time allocation, aiming to maximize the minimum uplink throughput across UDs subject to per-UD downlink throughput requirements and a UAV energy-consumption constraint. To address this non-convex problem under dynamic underground environments, we develop an off-policy deep reinforcement learning approach based on the primal-dual truncated quantile critics (PD-TQC), where a TQC-based reward critic evaluates the fairness objective, cost critics estimate constraint costs, and an actor learns the resource-control policy with adaptive dual-variable updates. Simulations using in-situ soil volumetric water content measurements from a center-pivot irrigation farm demonstrate that PD-TQC achieves the highest annual average minimum uplink throughput compared with four benchmarks while satisfying the per-frame constraints in over 96% of the evaluated frames, and that this throughput advantage is preserved under different Rician factors, numbers of UDs, and downlink throughput thresholds.
- 中文摘要
无人机(UAV)支持的无线地下传感网络(WPUSN)允许地下设备(UD)通过无线能量传输(WET)技术从无人机获取射频能量,从而实现可持续的地下物联网(IoUT)应用。本文将速率分割多接入(RSMA)辅助的无线信息与电力传输(SWIPT)集成到无人机支持的WPUSN中,以向UD传递信息和能量。在这一框架下,我们研究了一个最大最小公平性问题,该问题联合优化了无人机发射功率、功率分配与消息分配、功率分配比以及时间分配,旨在最大化在每个UD下行吞吐量要求和无人机能耗约束下,实现UD间最小上行吞吐量。为解决动态地下环境下的非凸问题,我们基于原始-对偶截断分位数批评者(PD-TQC)开发了一种非策略深度强化学习方法,其中基于TQC的奖励批评者评估公平性目标,成本批评者估算约束成本,行为者通过自适应双变量更新学习资源控制策略。利用中心枢轴灌溉农场的原位土壤体积水分含量测量模拟表明,PD-TQC在超过96%的评估帧中满足每帧约束,且在不同的Rician因子、UD数量和下行吞吐量阈值下实现了最高的年平均最小上行吞吐量,且在不同的Rician因子、UD数量和下行链路吞吐量阈值下保持了这一吞吐量优势。
Interactive Structured Environment Scaling for Large Language Model Agentic Reasoning
大型语言模型智能推理的交互式结构化环境缩放
- Authors: Peng Yu, Luoyi Fu, Xinbing Wang, Ying Wen
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19871
- Pdf link: https://arxiv.org/pdf/2611.19871
- Abstract
Reinforcement learning via environmental interaction is a cornerstone post-training paradigm for evolving large language models into autonomous agents capable of multi-turn tool use and exploration. Because the environment defines what the policy observes and how actions are verified, its structure directly bounds the capabilities an agent can acquire. Yet existing structured and retrieval environments rely on upstream pre-filtering---extracting a compact subgraph around the gold answer or retrieving fixed passages before the policy acts. We show that such pre-filtering introduces a fundamental measurement confound: because per-question subgraphs enclose ground-truth paths, they compress the candidate space by orders of magnitude and leak answer locations via local topology, allowing a question-blind degree heuristic to surface gold answers far above chance. To restore path selection, candidate pruning, and optimal stopping as genuine policy decisions, we propose RISE (Interactive Structured Environment), which transitions structured reinforcement learning from static subgraphs to full-scale interactive databases along two coupled axes. First, Decision-Space Scaling exposes the unpruned knowledge graph via a five-stage graph-agnostic typed intent adapter under a bounded call budget and a multiplicative grounding reward gate. Second, Learning-Frontier Scaling overcomes within-group variance collapse under group-normalized objectives by uniting model-independent structural tier progression with a self-evolving competence-frontier bandit. Across nineteen benchmarks and three model backbones, RISE substantially advances in-domain and unseen-schema knowledge-graph reasoning, transfers zero-shot to open-domain passage retrieval, and significantly enhances out-of-domain mathematical and logical deduction.
- 中文摘要
通过环境交互进行强化学习,是将大型语言模型演化为能够多回合使用和探索工具的自主智能体的基础训练后范式。由于环境定义了策略观察到的内容以及如何验证动作,其结构直接限制了代理能够获得的能力。然而,现有的结构化和检索环境依赖于上游预过滤---在金答案周围提取紧凑子图,或在策略行动前提取固定段落。我们表明,这种预过滤引入了一个根本性的测量混淆:由于每题子图包含了地面真实路径,它们以数量级的规模压缩候选空间,并通过局部拓扑泄漏答案位置,使得无问次数启发式方法能够远高于偶然率地呈现金答案。为了恢复路径选择、候选人修剪和最佳停止作为真正的政策决策,我们提出了RISE(交互式结构化环境),该方法将结构化强化学习从静态子图迁移到沿两个耦合轴的全规模交互式数据库。首先,决策空间尺度通过五阶段无关图型意图适配器,在有限调用预算和乘法基础奖励门下,暴露未修剪的知识图谱。其次,学习前沿尺度通过结合模型无关的结构层级进程与自我演化的能力前沿强化者,克服群体归一化目标下的群体内方差崩溃。在十九个基准测试和三个模型骨干框架中,RISE 大幅推进了域内和未见模式知识图谱推理,将零样本数据转化为开放域的通道检索,并显著提升了域外的数学和逻辑推理能力。
Let Execution Do the Teaching
让执行来教你
- Authors: Abhishek Mishra, Sandeep Srinivasa, Arko Chattopadhyay, Enrique Martinho Ferrao, Guda Pranav Reddy
- Subjects: Subjects:
Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19875
- Pdf link: https://arxiv.org/pdf/2611.19875
- Abstract
Improving a coding model is a sequence of decisions: find the limiting behavior, create useful practice, and determine which attempts deserve reinforcement. We investigate a practical way to make those decisions increasingly repeatable by grounding feedback in execution. Starting from GLM-4.7-Flash, the work combines whole-file supervised fine-tuning, targeted C++ repair curricula, group-relative reinforcement learning, and verifiers that turn program behavior into structured rewards. Compiler errors and failing tests provide specific contexts for repair practice, while executable checks score the resulting programs. We evaluate the trained models on 26 C++ coding tasks, measuring both initial success and completion after test feedback. The strongest reported post-training configuration achieves 45.2% success in the first test round and 61.5% within two rounds. These results come from an assisted development suite with evolving prompt contracts; the strongest configuration uses four selected evaluation trials and includes six tasks also used in training. The study describes an execution-guided workflow for improving coding models, including how repair tasks are constructed, how program behavior determines rewards, and how the resulting models are evaluated.
- 中文摘要
改进编码模型是一系列决策过程:找出限制行为,创造有益练习,并确定哪些尝试值得强化。我们探讨了一种通过在执行中扎根反馈,使这些决策越来越可重复的方法。从 GLM-4.7-Flash 开始,这项工作结合了全文件监督微调、针对性的 C++ 修复课程、群体相对强化学习以及将程序行为转化为结构化奖励的验证器。编译器错误和失败测试为修复练习提供了具体情境,而可执行检查则为最终程序评分。我们在 26 个 C++ 编码任务上评估训练好的模型,衡量测试反馈后的初始成功和完成情况。报告最强的训练后配置在第一轮测试中成功率为45.2%,两轮后成功率为61.5%。这些结果来自辅助开发套件,并伴随不断演进的提示合同;最强配置使用了四个精选的评估试验,并包括六个同样用于训练的任务。该研究描述了一种以执行为导向的工作流,用于改进编码模型,包括修复任务的构建方式、程序行为如何决定奖励,以及最终模型的评估方式。
Delayed Heuristic-Behavior-Target Coupling for Off-Policy Reinforcement Learning in Inventory Control
在库存控制中进行非策略强化学习的延迟启发式-行为-目标耦合
- Authors: Enhui Li (1), Yuan Cheng (1), Peiting Han (1) ((1) Xi'an Jiaotong University)
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19917
- Pdf link: https://arxiv.org/pdf/2611.19917
- Abstract
Many reinforcement learning methods tie data collection to the task policy, while coordinating independently optimized behavior and target policies remains challenging. Policies learned by these methods can perform unreliably in long-horizon tasks under uncertainty, such as inventory control, where failures can incur extreme costs. We introduce Heuristic Behavior Policy Optimization (HBPO), an off-policy framework that routes heuristic guidance through persistent behavior learning, coupled with target-policy, updating separately with residual-based advantage estimation and emphatic weighting. Under matched environment interaction budgets on DSIS and its risk-augmented variant DSIS-RB, HBPO recovers many high-cost initial policies and improves most non-collapse comparison groups. Cold starts achieve competitive costs without final cost collapse across all tested seeds. We identify a transmission chain in the delayed heuristic-behavior-target coupling as a key driver of recovery and improvement. Generation-level diagnostics trace heuristic uptake and subsequent selective target responses, while component ablations support the contribution of this transmission mechanism to policy recovery and improvement.
- 中文摘要
许多强化学习方法将数据收集与任务策略绑定,而协调独立优化的行为和目标策略仍然具有挑战性。这些方法学习的策略在长期任务中表现可能不可靠,例如库存控制,失败可能带来巨大代价。我们介绍启发式行为策略优化(HBPO),这是一种非策略框架,通过持续行为学习与目标策略相结合,分别通过基于残差优势估计和强调权重进行更新。在DSIS及其风险增强变体DSIS-RB的匹配环境交互预算下,HBPO恢复了许多高成本初始策略,并改进了大多数非崩溃的比较组。冷启动在所有被测种子中实现具有竞争力的成本且最终成本崩溃。我们发现延迟启发式-行为-目标耦合中的传递链是恢复和改进的关键驱动力。代级诊断追踪启发式采集及其后续选择性目标响应,而组件消融则支持这一传递机制对政策恢复和改进的贡献。
AllocFlow: End-to-End Portfolio Trading Action Generation from Raw Market Data
AllocFlow:基于原始市场数据的端到端投资组合交易行动生成
- Authors: Annan Wang, Yuchen Xie, Yu Cen, Leon Liangyu Chen, Qianyu Julie Zhu, Xin Li, Tianyi Wei, Weisi Lin
- Subjects: Subjects:
Machine Learning (cs.LG); Trading and Market Microstructure (q-fin.TR)
- Arxiv link: https://arxiv.org/abs/2611.19937
- Pdf link: https://arxiv.org/pdf/2611.19937
- Abstract
Real-world portfolio management requires converting noisy market observations into coordinated, simultaneous allocations across multiple assets under shared capital constraints. Advances in forecasting models and alpha discovery help identify opportunities. However, traditional pipelines often decouple signal forecasting from portfolio construction, potentially misaligning learning with allocation and execution objectives. We present AllocFlow, an end-to-end flow-matching framework that directly generates joint portfolio allocations from raw multi-asset market histories. To capture dependencies among portfolio positions, AllocFlow employs a cross-asset flow Transformer conditioned on temporal market representations to generate continuous, signed position weights across all instruments jointly. To address the sample inefficiency and instability of learning continuous allocation policies from scratch, we establish a two-stage training scheme: strategy-supervised imitation shapes a structured distribution of candidate portfolios, followed by group-relative reinforcement learning (RL) that refines joint decisions using portfolio-level financial rewards and a frozen policy reference. On hourly cryptocurrency perpetual futures, RL refinement raises the annualized net return of the imitation policy from $27.79\%$ to $51.38\%$ and its Calmar ratio from $0.955$ to $4.043$, surpassing the teacher strategy on every metric, and AllocFlow reaches a Calmar ratio of $4.90$ on six further months of unseen 2026 data without retraining.
- 中文摘要
现实中的投资组合管理需要将噪声市场观察转化为在共享资本约束下对多资产的协调、同步配置。预测模型和阿尔法发现的进步有助于识别机会。然而,传统管道往往将信号预测与投资组合构建脱钩,可能导致学习与配置和执行目标不匹配。我们介绍 AllocFlow,这是一种端到端的流程匹配框架,直接从多资产市场历史中生成联合组合配置。为捕捉组合部位之间的依赖关系,AllocFlow 采用基于时间市场表示的跨资产流变换器,联合生成所有工具间连续、带符号的头寸权重。为解决从零开始学习连续配置策略时的样本低效和不稳定性,我们建立了两阶段培训方案:策略监督模仿塑造候选组合的结构化分布,随后进行群体相对强化学习(RL),利用投资组合层面的财务回报和冻结政策参考来优化联合决策。在每小时加密货币永续期货中,RL优化将仿制政策的年化净回报从27.79美元提升至51.38美元,其Calmar比率从0.955美元提升至4.043美元,在所有指标上超过教师策略;AllocFlow则在未重新培训的情况下,在6个月未公开的2026年数据中达到4.90美元。
Discovering Programmatic Policies from Reinforcement Learning-Based Traffic Signal Controllers
从基于强化学习的交通信号控制器中发现程序化策略
- Authors: Lindong Xie, Yang Zhang, Beiyu Song, Zeren Xing, Flora D. Salim, Edward Chung
- Subjects: Subjects:
Emerging Technologies (cs.ET); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19951
- Pdf link: https://arxiv.org/pdf/2611.19951
- Abstract
Traffic signal control (TSC) is critical for mitigating urban congestion. Deep reinforcement learning (DRL) has achieved strong performance by learning adaptive policies from traffic observations, but learned neural controllers are often opaque, structurally complex, and difficult to generalize across scenarios. We propose PPL-TSC, a programmatic policy learning framework that distills compact and human-readable TSC policies from DRL teachers. PPL-TSC formulates policy learning as an evolutionary search over phase utility functions (PUFs), which score candidate signal phases and select the highest-utility phase for execution. Each generation consists of four steps: policy generation, policy review, policy calibration, and policy evaluation. In the generation step, a large language model (LLM) acts as a semantics-aware evolutionary operator to propose and refine candidate PUFs. Generated PUFs are screened through explicit checks for syntactic validity, mathematical soundness, and consistency with TSC principles, with invalid ones repaired through iterative feedback. Valid PUFs are then calibrated to improve behavioral fidelity to the DRL teacher, and evaluated using a joint score combining teacher fidelity and traffic-control performance to select candidates for the next generation. Extensive experiments across diverse traffic scenarios and representative DRL-based controllers show that PPL-TSC produces compact and human-readable policies, achieves competitive performance for most teacher families, improves transfer generalization, and substantially reduces computational cost. Code is available at this https URL.
- 中文摘要
交通信号控制(TSC)对于缓解城市拥堵至关重要。深度强化学习(DRL)通过通过从交通观测中学习自适应策略取得了优异性能,但学习到的神经控制器往往不透明、结构复杂,且难以跨场景推广。我们提出了 PPL-TSC,这是一种程序化的政策学习框架,能够从DRL教师那里提炼出简洁且易于人类阅读的TSC策略。PPL-TSC将策略学习构建为对相位效用函数(PUF)的进化搜索,PUF对候选信号阶段进行评分并选择效用最高的阶段进行执行。每一代由四个步骤组成:策略生成、策略审查、策略校准和策略评估。在生成步骤中,一个大型语言模型(LLM)作为语义感知的进化运算符,用于提出和优化候选PUF。生成的 PUF 通过明确检查语法效度、数学合理性以及与 TSC 原则的一致性进行筛查,不符的则通过迭代反馈进行修复。有效的 PUF 随后被校准以提升对 DRL 教师的行为忠实度,并通过结合教师忠实度与交通控制性能的联合评分进行评估,以筛选下一代候选人。在多种交通场景和代表性的基于 DRL 控制器的大量实验表明,PPL-TSC 能够生成紧凑且易于人类阅读的策略,为大多数教师家庭实现竞争性表现,提升迁移泛化,并大幅降低计算成本。代码可在此 https 网址获取。
Value and Policy Calibration from Execution Feedback for Socially Compatible Driving
通过执行反馈进行价值和政策校准,实现社会兼容驾驶
- Authors: Sen Shen, Ning Liu, Xiangrui Kong, Yingwei Li, Dongkun Han
- Subjects: Subjects:
Robotics (cs.RO); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.19969
- Pdf link: https://arxiv.org/pdf/2611.19969
- Abstract
Autonomous vehicle (AV) decisions at intersections affect the delay, braking, and motion smoothness of surrounding human-driven vehicles (HDVs). Learning socially compatible behavior requires connecting imagined interaction outcomes with feedback from executed actions. We propose a world-model reinforcement learning framework combines imagined policy improvement with direct value and policy calibration from finalized interactions, balancing AV progress with HDV interaction burden. An interaction-aware latent model supports imagination, while off-policy correction and replay updates enable observed rewards and transitions to supervise both the critic and actor. Simulation experiments at an unsignalized intersection achieve a 100% success and reductions in following-HDV acceleration variation of 65.5% and 44.0% relative to PIWM and DreamerV3, respectively. Ablations identify complementary roles for value calibration in reducing HDV burden and policy calibration in improving passage efficiency. Tests under out-of-distribution traffic shifts further show lower ego jerk and HDV acceleration variation than PIWM at comparable passage times.
- 中文摘要
路口的自动驾驶车辆(AV)决策会影响周围人驾驶车辆(HDV)的延迟、制动和运动平滑度。学习社会兼容行为需要将想象中的互动结果与执行行为的反馈相结合。我们提出一个世界模型强化学习框架,将想象中的策略改进与最终互动中的直接价值和策略校准相结合,在 AV 进展与 HDV 交互负荷之间取得平衡。一个具备交互感知的潜在模型支持想象,而非策略修正和回放更新则使观察到的奖励和转换能够同时监督批评者和行为者。在无信号灯交叉口进行的模拟实验,相较于 PIWM 和 DreamerV3,实现了 100% 的成功率,并使后续 HDV 加速度变异分别减少了 65.5% 和 44.0%。消融技术识别了价值校准在降低 HDV 负担方面的作用,以及策略校准在提升通行效率方面的互补作用。在非分配流量变换下的测试进一步显示,在类似通过时间下,自我震动和HDV加速度变化低于PIWM。
D'BEAST: Logged Bad Experiences as Explanation for Autonomous Driving Agents
D'BEAST:记录的糟糕经历作为自动驾驶代理的解释
- Authors: Ahmad Alfan Alfian Irfan, Nur Ahmad Khatim, Mansur Arief
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.20019
- Pdf link: https://arxiv.org/pdf/2611.20019
- Abstract
Reinforcement learning agents learn from mistakes, yet their decisions can be difficult to trace back to those experiences. We investigate whether a tabular agent's decisions can be explained by its bad experiences, defined as the training transitions associated with safety or task events. A fixed training log is replayed from the same initial Q-table under three conditions. FULL retains all updates, SKIP omits bad-experience updates, and NEUTRALIZE retains those updates without their event penalties. We illustrate the method on six classic Markov decision processes and evaluate it in closed-loop driving in CARLA. On the full driving task, FULL reached the goal from 33 of 40 test starts, whereas SKIP and NEUTRALIZE reached it from none, indicating that the bad experiences carry decision-relevant information. At the local level, for 47 decisions shortly before a failure of SKIP or NEUTRALIZE, removing or neutralizing a small group of bad experiences, with a median size of one, caused the trained agent to select the failing policy's action in 46 cases. For this agent, the results indicate that its decisions can be explained by its bad experiences.
- 中文摘要
强化学习代理会从错误中学习,但他们的决策往往很难追溯到这些经历。我们研究表格代理的决策是否可以用其不良经历来解释,这些不良经历被定义为与安全或任务事件相关的训练转换。固定训练日志会从同一初始Q表中重放,条件为三种。FULL保留所有更新,SKIP省略不良体验更新,NEUTRALIZE则保留这些更新而不影响事件惩罚。我们在六个经典马尔可夫决策过程上演示了该方法,并在CARLA闭环驾驶中进行评估。在完整驾驶任务中,FULL在40次测试启动中有33次达到目标,而SKIP和NEUTRALIZE则从零次达到目标,表明不良体验包含与决策相关的信息。在本地层面,在 SKIP 或 NEUTRALIZE 失败前的 47 项决策中,移除或中和一小群不良经历(中位数为 1)导致受过训练的代理在 46 个案例中选择了失败策略的行动。对于该代理,结果表明其决策可以用其不良经历来解释。
EchoSpec: Training Search Agents to Weigh Evidence Above Echoes
EchoSpec:培训搜索探员权衡证据,而非回声
- Authors: Shihao Yang, Can Zeng, Xinzhe Li
- Subjects: Subjects:
Information Retrieval (cs.IR)
- Arxiv link: https://arxiv.org/abs/2611.20034
- Pdf link: https://arxiv.org/pdf/2611.20034
- Abstract
Search agents trained with reinforcement learning decide what to retrieve, when to search again, and when to answer. In real corpora, one claim is copied and paraphrased across many pages while the primary source settling a question appears once. We show that reinforcement-learned search agents grant echoes the credence of independent sources and lose their ability to abstain. Across five base models from 3B to 32B and seven released RL agents on multi-hop Wikipedia questions with primary evidence intact, eight paraphrased false-claim echoes overturn 58-73% of previously correct answers, while exact-match RL drives deleted-evidence abstention below 15%. Conditioning on retrieval localizes the failure to evidence acquisition: once primary evidence enters context, agents overrule eight echoes as reliably as one, whereas echoes fill top ranks and halt search early, so larger budgets, forced search turns, or passive passage filtering alone fail to recover displaced evidence. We introduce EchoSpec, specifying reliability across five corpus transformations with known target answers and training in two stages. Verification warm-start teaches multi-hop bridge-entity recovery past echoes, after which group RL optimizes an evidence-gated $\lambda$-reward with an outcome-independent acquisition bonus and scale-preserving advantages retaining the optimal rule $\lambda/(1+\lambda)$. Across Qwen2.5-3B, Qwen2.5-7B, and Llama-3.1-8B backbones on nine evaluation suites, EchoSpec cuts eight-echo deference by 30-44 points on Qwen2.5-7B down to 29-31%, nearly quadruples eight-echo reach to 26-27% with zero-shot transfer to unseen BGE and BM25 retrievers, and raises standalone QA precision to 47-55%. Code is publicly available at this https URL.
- 中文摘要
经过强化学习训练的搜索代理决定检索什么、何时再次搜索以及何时回答。在真实语料库中,一个主张会被复制并转述到多个页面,而解决问题的原始来源只出现一次。我们表明,强化学习的搜索代理给予回声会获得独立来源的可信度,并失去弃权能力。在五个基础模型中,从3B到32B,以及七个在多跳维基百科问题中释放的强化学习代理,且保留了原始证据,八个改写虚假主张回忆推翻了58%-73%的先前正确答案,而完全匹配的强化学习则将删除证据的弃权率降低到15%以下。条件反射性将证据获取的失败定位:一旦原始证据进入语境,代理人能像一个一样可靠地覆盖八个回声,而回声占据前列并提前停止搜索,因此仅靠更大预算、强制搜索或被动通道过滤无法恢复错位证据。我们引入了 EchoSpec,在五种语料库转换中规定可靠性,并分两个阶段进行训练。验证热启动教导多跳桥-实体恢复,完成回声,随后组 RL 优化基于证据门槛的 $\lambda$-奖励,同时获得与结果无关的获取加成和规模保值优势,保留最优规则 $\lambda/(1+\lambda)$。在九个评估套件上的 Qwen2.5-3B、Qwen2.5-7B 和 Llama-3.1-8B 骨干网中,EchoSpec 将 Qwen2.5-7B 的八回声尊重度降低了 30-44 个百分点,降至 29-31%,八回声覆盖率几乎提升四倍至 26-27%,零发射传输至 BGE 和 BM25 检索器,并将独立 QA 精度提升至 47-55%。代码可在此 https 网址公开获取。
Prompt Re-Specification and Target-Mask Availability in Mask-Conditioned Visual Reinforcement Learning for Moving-Platform Landing
在移动平台着陆的掩体条件视觉强化学习中,提示重新指定和目标掩码可用性
- Authors: Ho Tim Tsui, Jongyun Kim
- Subjects: Subjects:
Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.20075
- Pdf link: https://arxiv.org/pdf/2611.20075
- Abstract
Visual reinforcement learning (RL) policies that land a drone on a moving platform can fail when the landing pad looks different from the one used in training. Using binary target masks as the visual input removes most pixel-level appearance information while preserving the spatial representation of the target, and lets a text prompt specify the target. In simulation, we test whether a frozen (fixed-weight) mask-based DrQ-v2 policy can handle a new pad when only the prompt of its fine-tuned SAM 3 detector changes. A banana-themed pad lowered landing success from 75.6% to 12.0%, and SAM 3 rarely detected it under the original prompt. A revised prompt, chosen after testing several candidates, restored success to 68.4% with no weight updates. In a separate experiment with a policy trained on ground-truth masks, filling in masks that were missing while the pad was visible helped landing far more than the tested mask-centroid and area corrections. Tracking the pad over time supplied masks more often and also kept producing them after the pad was confirmed absent from the rendered image, but it did not establish a reliable improvement in landing success. Whether a usable mask reaches the policy at all thus appears to be an important limiting factor for this approach.
- 中文摘要
视觉强化学习(RL)策略,即无人机降落在移动平台上,当着陆台外观与训练时不同时,可能会失败。使用二进制目标掩模作为视觉输入,可以去除大部分像素级外观信息,同时保留目标的空间表现,并允许文本提示指定目标。在模拟中,我们测试了冻结(固定权重)基于掩罩的 DrQ-v2 策略是否能在仅微调 SAM 3 探测器提示发生变化的情况下处理新垫子。香蕉主题的垫子将着陆成功率从 75.6% 降至 12.0%,而 SAM 3 在原始提示下几乎未检测到该目标。经过测试多个候选方案后选择的修订提示,成功率恢复到 68.4%,且未进行权重更新。在另一项针对地面真实面罩训练策略的实验中,填补垫子可见时缺失的掩模,比测试的面罩中心和区域修正更有助于着陆。随着时间推移跟踪垫子,能更频繁地提供掩模,并且在确认垫子不在渲染图像后仍能持续产生,但并未确立可靠提升着陆成功率的结论。因此,是否能达到该策略的有效掩模似乎是该方法的重要限制因素。
Infrastructure-Based Multi-Agent Reinforcement Learning with Attention for Highway Ramp Merging
基于基础设施的多智能体强化学习,关注高速公路匝道合流
- Authors: Chenhao Tong, Steffen Müller
- Subjects: Subjects:
Machine Learning (cs.LG); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.20078
- Pdf link: https://arxiv.org/pdf/2611.20078
- Abstract
On-ramp merging is a typical safety and throughput problem on highways. Without proper coordination, human drivers and self-interested autonomous vehicles may make risky manoeuvres and cause accidents or congestion. With recent advances in connected autonomous vehicles (CAVs) and multi-agent reinforcement learning (MARL), vehicles can be trained to coordinate for safe and efficient merging. However, in dense traffic, forming an effective observation of the traffic state is challenging. For example, observing only the neighbouring vehicles may lead to locally optimal decisions, while observing the entire traffic can introduce much irrelevant information. In this paper, we propose a transformer-based MARL vehicle control model to address these problems. A roadside infrastructure controller receives the states of all CAVs via vehicle-to-infrastructure (V2I) communication and sends control commands back to them. An attention-based observation encoder is designed where each vehicle can filter the relevant vehicle information from the global traffic state. The proposed model is evaluated in SUMO against three baselines: a model that observes only the neighbouring vehicles, a model that encodes the global traffic information, and the rule-based SUMO model. The results show that our model achieves the highest traffic flow, safety and smoothness. Furthermore, in a four-hour simulation with dynamic traffic demand, our model successfully avoids traffic jams, while other baseline models fail at high traffic flow. We also show that our model remains effective when human-driven vehicles are present. We further show that our model, trained in ideal communication conditions, can still remain collision-free and maintain throughput under realistic V2I communication errors. Finally, we visualise the relevance weights that an ego vehicle assigns to other vehicles in multiple traffic situations.
- 中文摘要
匝道合流是高速公路上常见的安全和吞吐量问题。如果缺乏适当的协调,人类驾驶员和自利的自动驾驶车辆可能会做出高风险的操作,从而引发事故或拥堵。随着联网自动驾驶汽车(CAVs)和多智能体强化学习(MARL)的最新进展,车辆可以被训练以协调安全高效地并入。然而,在交通密集中,建立有效交通状态观察是有挑战性的。例如,仅观察邻近车辆可能带来局部最优决策,而观察整个交通则可能引入大量无关信息。本文提出基于变压器的MARL车辆控制模型来解决这些问题。路边基础设施控制器通过车辆对基础设施(V2I)通信接收所有CAV的状态,并向它们发送控制指令。设计了一个基于注意力的观察编码器,使每辆车辆都能从全局交通状态中过滤相关车辆信息。所提模型在 SUMO 中基于三个基线进行评估:一个仅观察邻近车辆的模型,一个编码全球交通信息的模型,以及基于规则的 SUMO 模型。结果表明,我们的模型实现了最高的交通流量、安全性和顺畅性。此外,在一个为期四小时、具动态交通需求的模拟中,我们的模型成功避免了交通拥堵,而其他基线模型在高流量时则失效。我们还证明,当有人类驾驶车辆存在时,我们的模型依然有效。我们还进一步证明,在理想通信条件下训练的模型,在真实的V2I通信错误下仍能保持无碰撞并保持吞吐量。最后,我们可视化了自我车辆在多种交通情境下赋予其他车辆的相关权重。
Cleanly-DINO: Efficiently Learning Visual Task Spaces for Robotics
Cleanly-DINO:机器人领域可视化任务空间的高效学习
- Authors: Adrian Röfer, Melika Shekarriz, José Arce y de la Borbolla, Iman Nematollahi, Abhinav Valada
- Subjects: Subjects:
Robotics (cs.RO); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.20137
- Pdf link: https://arxiv.org/pdf/2611.20137
- Abstract
Learning manipulation tasks from few demonstrations requires abstractions that capture the changes relevant to task completion, such as object poses in rigid-body manipulation. However, tasks such as cleaning are defined by changes in visual state rather than object pose, making the choice of task representation less clear. In this work, we propose Cleanly-DINO to learn visual task spaces from few demonstrations. Given a mask of a task-relevant region and frozen ViT features, Cleanly-DINO fits a projection to isolate a task space and fit a progression metric to it. We train the model on diverse cleaning examples and evaluate its ability to localize incomplete task regions and transfer across visual domains. We also study its utility for robot learning by using it as an observation encoder and reward signal in a reinforcement learning setting. Finally, we deploy our fitted model within a real-world robotic wiping pipeline to evaluate its use in guiding task execution.
- 中文摘要
从少量演示中学习操作任务需要抽象,以捕捉与任务完成相关的变化,例如刚体操作中的物体姿态。然而,清理等任务是通过视觉状态的变化而非物体姿态来定义的,使得任务表示的选择不那么明确。在本研究中,我们提出 Cleanly-DINO 通过少量演示学习可视化任务空间。给定任务相关区域的掩码和冻结的 ViT 特征,Cleanly-DINO 拟合一个投影以隔离任务空间并拟合进度度量。我们基于多样化的清理示例训练模型,评估其定位未完成任务区域及跨视觉领域转移的能力。我们还研究其在机器人学习中的实用性,将其作为观察编码器和强化学习环境中的奖励信号。最后,我们将拟合模型部署在现实世界中的机器人擦除流水线中,评估其在指导任务执行中的应用。
GNN-Based Deep Reinforcement Learning for Coastal Emergency Response under Extreme Water Levels
基于GNN的深度强化学习,用于极端水位下的沿海应急响应
- Authors: Marcello Sano, Samuele Casagrande, Davide Ferrario, Sebastiano Vascon, Silvia Torresan, Andrea Critto
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.20268
- Pdf link: https://arxiv.org/pdf/2611.20268
- Abstract
When a storm surge approaches a coastal town, the emergency plan must be put into action quickly using water-level forecasts, flood extent, road access, vulnerable residents and shelters. Bringing this information together under time pressure is slow and depends on the duty officer's expertise. This paper is one step in a wider research line from data to safe decisions: machine learning to understand the hazard, reinforcement learning and planning to choose actions, large language models (LLMs) to explain and cross-check them, and an assurance layer that keeps people in charge. Here we address the action step on a real place. We ask whether a policy can learn the timing of emergency actions from decades of real events. Lido di Venezia is represented as a heterogeneous graph built from public elevation, road, building and census data and driven by hourly tide-gauge records from 1983 to 2024, with seven actions taken from the municipal emergency plan. A PPO policy is trained on this graph and compared, on storm windows unseen in training, with a rule-based system simulating the officer's discretionary decisions. On 18 historical test scenarios, PPO achieves a similar reduction in population exposure to the rule. With uncertain forecasts and limited protection it again performs similarly but does not outperform the tuned rule, and removing graph message passing does not substantially reduce performance. PPO therefore learns an effective protective response from an outcome-based reward alone: it is given the plan's actions and when each is allowed, but not the thresholds that tell the comparator when to act. Next steps test harder edge cases, move closer to a digital twin of the territory, and replace the simulated officer with assessment by emergency-management stakeholders, supported by LLM-based interpretation of each recommendation.
- 中文摘要
当风暴潮逼近沿海城镇时,必须利用水位预报、洪水范围、道路通达情况、弱势居民和避难所迅速实施应急预案。在时间压力下整合这些信息进展缓慢,且依赖值班官员的专业能力。本文是从数据到安全决策的更广泛研究线索中的一步:机器学习理解危害,强化学习与规划以选择行动,大型语言模型(LLMs)用于解释和交叉核对,以及确保人员掌控全局的保障层。这里我们在真实地点讨论行动步骤。我们探讨一项政策是否能从数十年的真实事件中学习应急行动的时间点。威尼斯利多以一个异质图表示,由公共高程、道路、建筑和人口普查数据构建,并由1983年至2024年的每小时潮位计记录驱动,其中七项行动取自市政应急计划。PPO政策基于该图表进行训练,并以培训中未见过的风暴窗口与模拟官员自由裁量决策的基于规则的系统进行比较。在18个历史测试场景中,PPO实现了类似的减少人群暴露于该规则的效果。在预测不确定且保护有限的情况下,它同样表现类似,但并未超过调优规则,移除图消息传递并不会显著降低性能。因此,PPO仅从基于结果的奖励中学习到有效的保护性响应:它得到计划的行动及其允许的时间,但不包括告诉比较者何时采取行动的阈值。下一步是测试更难的边缘案例,更接近区域的数字孪生,并用应急管理利益相关者的评估取代模拟官员,并辅以基于大型语言模型的每项建议的解读。
DexFormer 2.0: Towards a Cross-Embodiment Foundation Policy for Dexterous Grasping
DexFormer 2.0:迈向灵巧抓握的跨实体基础政策
- Authors: Lixin Xu, Ke Zhang, Xinyu Liu, Jie Tan, Senming Tan, Jiaju Yin, Yating Feng, Xiaoyi Lin, Xiyue Dong, Renjing Xu
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.20273
- Pdf link: https://arxiv.org/pdf/2611.20273
- Abstract
Dexterous grasping policies are usually trained for one arm paired with one hand. Every new pairing then needs its own policy, and the number of policies grows with every arm and hand added. We ask whether a single policy can control many arm-hand combinations and transfer to pairings and hands it has never seen. This is hard because robots differ in the number, layout, and dynamics of their joints, so their observations and actions share no common interface. We present DexFormer 2.0, a single reinforcement learning policy for cross-embodiment dexterous grasping. It maps every robot to a canonical joint interface with validity masks, encodes each joint's morphology with attention biased by the kinematic tree, and uses a Transformer-XL memory to infer embodiment-dependent dynamics from recent interaction. Trained on 18 combinations of four arms and six hands, it reaches 89% stable success on seen combinations and 51% on six held-out arm-hand pairs, outperforming GRU and DexFormer 1.0 baselines. Without fine-tuning, it also grasps with four new hand types and, through retargeting, with parallel grippers. Ablations show that memory, the kinematic-tree bias, and broader pairing coverage each matter for transfer. On a physical Flexiv arm, distilled students transfer zero-shot from simulation with two hands, including a held-out pair. Project website: this https URL.
- 中文摘要
灵巧抓握策略通常训练为一只手臂配一只手。每组新的配对都需要自己的策略,且随着每增加一只手臂和手,策略数量也随之增加。我们探讨一个策略是否能控制许多手臂-手的组合,并迁移到它从未见过的配对和手型。这很难,因为机器人在关节数量、布局和动态上各不相同,因此它们的观察和动作没有共同接口。我们介绍 DexFormer 2.0,一种用于跨身体灵巧抓取的单一强化学习策略。它将每个机器人映射到带有有效性掩码的典型关节接口,以运动树为主的注意力编码每个关节的形态,并利用 Transformer-XL 内存从近期互动中推断具身依赖的动态。在训练中,它在18种四臂六手组合上训练,在已见组合上达到89%的稳定成功率,在六对伸出的手臂-手组合上达到51%,优于GRU和DexFormer 1.0基线。无需微调,它还能使用四种新手型抓取,并通过重新定位使用平行握持器。消融显示,记忆、运动树偏向以及更广泛的配对覆盖对转移都至关重要。在物理 Flexiv 臂上,精炼学员用双手(包括伸出的一对)从模拟中零射转移。项目网站:此 https URL。
Fuzzy Explainable Continual Hierarchical Reinforcement Learning for Robotic Navigation
用于机器人导航的模糊可解释连续层级强化学习
- Authors: Peng Qin, Tao Zhao
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.20427
- Pdf link: https://arxiv.org/pdf/2611.20427
- Abstract
Continual robot navigation requires reinforcement learning policies to continuously adapt to new environments while retaining previously acquired knowledge. Meanwhile, explainable reinforcement learning aims to learn meaningful behavioral representations through policy interaction. However, these objectives are inherently conflicting, as continual learning emphasizes policy stability whereas explainable behavior learning requires adaptive exploration. Existing methods rarely address both simultaneously. To resolve this issue, this paper proposes a Fuzzy Explainable Continual Hierarchical Reinforcement Learning (FECHRL) method. A two-stage behavior cloning strategy is introduced to balance rapid policy adaptation and explainable behavior evolution by progressively transitioning from strong imitation guidance to autonomous reinforcement learning. Furthermore, a fuzzy critic based on a Takagi--Sugeno fuzzy inference system is proposed to alleviate value extrapolation errors during the transition between imitation-guided learning and autonomous exploration, thereby improving continual training stability. Experimental results on sequential multi-scenario navigation tasks demonstrate that the proposed method effectively unifies continual learning capability and policy explainability within a single reinforcement learning framework, achieving stable knowledge retention and meaningful behavioral explanation.
- 中文摘要
持续机器人导航要求强化学习策略在保留先前获得知识的同时,持续适应新环境。与此同时,可解释强化学习旨在通过策略交互学习有意义的行为表征。然而,这两者本质上存在冲突,因为持续学习强调策略稳定性,而可解释行为学习则需要适应性探索。现有方法很少同时处理两者。为解决这一问题,本文提出了一种模糊可解释持续层级强化学习(FECHRL)方法。本文引入了一种两阶段行为克隆策略,通过逐步从强模仿引导过渡到自主强化学习,平衡快速策略适应与可解释行为演化。此外,基于高木-菅野模糊推理系统,提出了一种模糊批评器,以缓解模仿引导学习与自主探索过渡过程中的价值外推误差,从而提升持续训练稳定性。在顺序多场景导航任务上的实验结果表明,所提方法有效将持续学习能力和策略可解释性统一在单一强化学习框架内,实现稳定的知识保留和有意义的行为解释。
Stochastic Self-Distillation for Flow-Based Vision-Language-Action Models
基于流的视觉-语言-行动模型的随机自蒸馏
- Authors: Jingru Guo, Alessandra Russo, Joey Bose, Christos Ziakas
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.20460
- Pdf link: https://arxiv.org/pdf/2611.20460
- Abstract
Flow-based vision-language-action (VLA) models achieve strong performance in robotics through imitation learning, yet fine-tuning them with on-policy reinforcement learning (RL) is limited. In group-based RL methods like Group Relative Policy Optimization (GRPO), every rollout step requires multi-step ODE integration, and the sampler's limited stochasticity leads to rollout groups with low variance. We propose VLA-SSD, which addresses both high inference latency and limited stochasticity. In particular, our method self-distills a pretrained flow-matching action head into a stochastic one-step policy by conditioning a single-jump generator on a Brownian path. We derive a conditional action density via our stochastic one-step policy, enabling efficient post-training via likelihood-ratio RL algorithms. We show that distilling VLA-SSD from the $\pi_0$ VLA reduces the action head's forward-pass time by a factor of 7.7, at the cost of a 5.5% relative decrease in the success rate on the LIBERO benchmark. However, by leveraging the increased sample diversity of VLA-SSD during group generation, post-training a $\pi_{0.5}$ VLA using GRPO yields a 20.6% relative improvement.
- 中文摘要
基于流程的视觉-语言-动作(VLA)模型通过模仿学习在机器人技术中实现了强劲性能,但通过策略强化学习(RL)对其进行微调的效果有限。在群组相对策略优化(GRPO)等基于群体的强化学习方法中,每一个部署步骤都需要多步常微分方程集成,而采样器有限的随机性导致展开组的方差较低。我们提出了VLA-SSD方案,既解决了高推理延迟又满足有限随机性的问题。特别是,我们的方法通过对单跳生成器在布朗路径上进行条件,将预训练的流程匹配动作头自我提炼为随机一步策略。我们通过随机一步策略推导出条件动作密度,从而通过似然比RL算法实现高效的后期训练。我们证明,从 $\pi_0$ VLA 中提炼 VLA-SSD 可以将动作头的前向传递时间减少 7.7 倍,但代价是 LIBERO 基准测试成功率相对下降 5.5%。然而,通过利用 VLA-SSD 在组生成过程中增加的样本多样性,使用 GRPO 对 $\pi_{0.5}$ VLA 进行后训练,可获得 20.6% 的相对提升。
Learning Design Taste through Verifiable Synthetic Environments and Agentic Reinforcement Learning
通过可验证的合成环境和智能体强化学习来学习设计品味
- Authors: Sujay choubey, Siddhant Saxena, Sarthak Kapil, Suhas Motwani
- Subjects: Subjects:
Multiagent Systems (cs.MA); Human-Computer Interaction (cs.HC)
- Arxiv link: https://arxiv.org/abs/2611.20612
- Pdf link: https://arxiv.org/pdf/2611.20612
- Abstract
Code-generating language models produce user interfaces that build and run, but the pages are often weak in hierarchy, spacing, typography and composition. We study whether a small code model can learn design taste through reinforcement learning in verifiable synthetic environments. The method has three components. First, a layered verification framework splits design quality into hard gates, render-grounded metrics, rubric judges, within-group relative preference and a blinded expert panel. A signal enters the reward only if it passes an admission protocol that tests agreement with experts, causal sensitivity, resistance to hack traps and stability. Second, following SCALER [1], synthetic design environments are parameterised generators with measured difficulty ladders: base pages are written once by an agentic coder, and instances are produced by syntax-tree perturbation operators and constraint samplers, scheduled by an adaptive curriculum with a quarantine rule against reward over-optimisation. Third, a 4B code model is trained with agentic reinforcement learning over multi-turn episodes in which it edits and builds a repository and reads a textual report of its own render, while hidden graders compute the reward. The admission protocol admits 7 of 16 render signals and 6 of 10 rubric dimensions. On TD-Bench, agentic training raises the Taste Index from 0.630 to 0.795 in-family and from 0.598 to 0.700 on held-out families, and wins 0.600 of blinded expert comparisons against the base model and 0.569 against a single-turn control. Most of the advantage depends on the inference-time loop, and cross-brief distinctiveness falls slightly below that of the base model, the only success criterion not met.
- 中文摘要
代码生成语言模型能生成构建和运行的用户界面,但页面在层级结构、间距、排版和组成方面往往薄弱。我们研究一个小型代码模型是否能在可验证的合成环境中通过强化学习学习设计品味。该方法包含三个组成部分。首先,分层验证框架将设计质量划分为硬门槛、基于渲染的指标、评分标准、组内相对偏好和盲专家面板。只有当信号通过了与专家的一致性、因果敏感度、抗黑客陷阱能力以及稳定性的入场协议时,才会进入奖励。其次,遵循 SCALER [1],合成设计环境是参数化生成器,并设有可测量的难度阶梯:基页由代理编码器编写一次,实例由语法树扰动算符和约束采样器生成,由自适应课程调度,并带有防止奖励过度优化的隔离规则。第三,4B 代码模型通过多回合的过程进行代理强化学习训练,在此过程中编辑并构建仓库,并读取自身渲染的文本报告,而隐藏评分器则计算奖励。准入协议允许16个渲染信号中的7个和10个评分标准中的6个。在TD-Bench上,代理训练将味觉指数从家族内的0.630提升到0.795,保留家族的味觉指数从0.598提升到0.700,并且在盲法专家比较中对基础模型赢得0.600,在单回合对照中赢得0.569。优势大部分依赖于推理时间循环,跨案情描述的独特性略低于基础模型,这是唯一未达到的成功标准。
Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment
对比激活引导的零阶偏好优化,实现内存高效的大型语言模型对齐
- Authors: Weiyun Huang, Alessio Galatolo
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.20635
- Pdf link: https://arxiv.org/pdf/2611.20635
- Abstract
Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59\% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.
- 中文摘要
直接偏好优化(DPO)简化了从人类反馈进行强化学习,但仍依赖传统的反向传播及其相关的激活记忆成本。零阶(ZO)优化避免了反向传播,但各向同性扰动可能对由所选反应与被拒绝响应差异定义的偏好目标产生噪声更新。我们引入了对比激活引导的零阶直接偏好优化(CA-ZODPO),该方法基于每个偏好对中所选与被拒绝响应之间的激活差异构建低秩搜索子空间。扰动被限制在该子空间内,而在低秩坐标中保持动量,以保持ZO训练的记忆优势。在Qwen2.5-1.5B搭配Anthropic HH-RLHF的基础上,CA-ZODPO优于常见的ZO基线,并在成对偏好评估中与一阶DPO保持竞争力。它还在序列长度为2048时,将峰值GPU内存减少约59%,同时保持零阶训练的序列长度缩放行为。
Competent Autonomous Driving Through Fine Tuning Vision Language Action Models with Symbolic Traffic Rules
通过精细调动带有符号交通规则的视觉语言行动模型,实现专业自动驾驶
- Authors: Barkin Dagda, Vinal Asodia, Iman Sharifi, Peng Wei, Saber Fallah
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.20648
- Pdf link: https://arxiv.org/pdf/2611.20648
- Abstract
Vision-language-action (VLA) models offer a unified approach to perception, reasoning, and control for autonomous driving, but accurate action prediction alone does not establish compliance with traffic regulations. This paper presents a rule-guided VLA framework that incorporates symbolic traffic-rule feedback into policy optimisation. The framework combines supervised fine-tuning on expert demonstrations with reinforcement learning (RL) refinement using a reward that encourages encoded-rule compliance while penalising deviations from expert actions. Selected provisions of the United Kingdom Highway Code are formalised as executable first-order logic rules using PyReason, covering speed adaptation, following distance, and road-user priority. A frozen vision-language model extracts scene predicates, which are combined with vehicle-state information to assess the projected consequences of candidate control actions. Offline evaluation on held-out data collected in the CARLA simulator compares three group-based RL methods. The results show improvements in both action prediction accuracy and encoded-rule compliance over the supervised baseline. The best-performing variant increases the overall rule pass rate from 92.9% to 97.6%, with zero observed violations of the encoded road-user-priority check in the evaluated samples. These findings support symbolic rule feedback as a means of improving VLA policy alignment with explicit traffic-rule criteria.
- 中文摘要
视觉-语言-行动(VLA)模型为自动驾驶提供了一种统一的感知、推理和控制方法,但仅靠准确的动作预测无法确定是否符合交通法规。本文提出了一个规则引导的VLA框架,将符号交通规则反馈纳入策略优化中。该框架结合了专家演示的监督微调与强化学习(RL)细化,采用奖励鼓励遵守编码规则,同时惩罚偏离专家行为的行为。英国《公路交通法》的部分条款被形式化为使用 PyReason 的可执行一阶逻辑规则,涵盖速度适应、跟车距离和道路使用者优先级。冻结的视觉语言模型提取场景谓词,并将其与车辆状态信息结合,以评估候选控制行为的预期后果。对CARLA模拟器中收集的保留数据进行离线评估,比较了三种基于群体的RL方法。结果显示,在动作预测准确性和编码规则合规性方面,相较于监督基线均有所提升。表现最佳的变体将整体规则通过率从 92.9% 提高到 97.6%,在评估样本中观察到编码道路使用者优先检查的违规行为为零。这些发现支持符号规则反馈作为提升VLA策略与明确交通规则标准对齐的手段。
SpeCal: Spectral Calibration for Unsupervised RLVR
SpeCal:无监督RLVR的光谱校准
- Authors: Kai Tang, Xing Zheng, Haobo Wang, Guangcheng Zhu, Shenzhi Yang, Zhengqing Zang, Yingfan Ma, Cheng Peng, Bowen Song, Weiqiang Wang
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.20691
- Pdf link: https://arxiv.org/pdf/2611.20691
- Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning abilities of large language models, but its dependence on ground-truth answers or external verifiers limits its ability to exploit abundant unlabeled reasoning data. Unsupervised RLVR alleviates this bottleneck by deriving scalar rewards from the model's own rollouts. However, the reliability of such self-generated supervision varies substantially across rollout groups and individual responses, while existing methods construct or refine their rewards mainly from decoded outcomes and reward statistics. These endpoint signals reveal what answers the model produces, but provide limited information about whether the underlying rollout representations form a reliable structure for learning. Our key observation is that hidden-state spectra offer a complementary view of supervision reliability by characterizing both the organization of a rollout group and the support received by each rollout within that group. Based on this observation, we propose SpeCal, a bi-level spectral calibration framework for unsupervised RLVR. At the group level, SpeCal combines spectral mass with a Gaussian band-pass gate over normalized entropy-based effective rank, favoring rollout groups with substantial representation variation whose effective dimensionality lies in a preferred regime. At the rollout level, the same group effective rank determines how many dominant directions are retained in a leave-one-out spectral subspace formed by the remaining rollouts, within which the conditional support of each held-out rollout is measured. The resulting signals calibrate the original reward-induced advantages at the group and rollout levels without redefining the underlying unlabeled reward.
- 中文摘要
带有可验证奖励的强化学习(RLVR)显著提升了大型语言模型的推理能力,但其对真实答案或外部验证器的依赖限制了其利用大量未标记推理数据的能力。无监督RLVR通过从模型自身的推广中推导出标量奖励,缓解了这一瓶颈。然而,这种自我生成的监督的可靠性在不同展开组和个别响应之间存在显著差异,而现有方法主要通过解码结果和奖励统计数据构建或优化其奖励。这些终点信号揭示了模型产生的答案,但对底层展开表征是否构成可靠的学习结构提供了有限的信息。我们的关键观察是,隐藏状态谱通过描述一个部署组的组织方式以及该组内每个部署所获得的支持,提供了对监督可靠性的互补视角。基于这一观察,我们提出了SpeCal,一种用于无监督RLVR的双能级谱校准框架。在组层面,SpeCal将谱质量与高斯带通门结合,基于归一化的熵有效秩,偏向具有显著代表性变异且有效维度位于偏好区间的展开组。在展开层面,同一组有效秩决定了在由剩余展开构成的“遗留一”谱子空间中保留多少主导方向,在此空间内测量每个保留展开的条件支持。所得信号在组和展开层面校准了原始奖励诱导的优势,而无需重新定义底层未标记的奖励。
GPRM: Global Perspective Process Reward Model
GPRM:全球视角过程奖励模型
- Authors: Zifan Zhang, Zhengbo Jiao, Shaobo Wang, Jing Zhou, Guoyi Li, Jiayi Chen, Wei Wang, Bing Zhao, Cheng Fang, Xiaoxiao Xu, Hu Wei, Linfeng Zhang, Lin Qu
- Subjects: Subjects:
Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.20755
- Pdf link: https://arxiv.org/pdf/2611.20755
- Abstract
Process Reward Models (PRMs) are widely used for step-level evaluation of multi-step reasoning in large language models, but existing methods are often limited by weak historical dependency modeling and insufficient global awareness. They typically fail to leverage prior step evaluations for coherent judgment and lack visibility into subsequent reasoning, leading to suboptimal assessment of intermediate steps. To address this issue, we propose the Global Perspective Process Reward Model (GPRM), which jointly models the problem, previous reasoning steps and their evaluations, the current step, and future steps to enable history-aware and globally informed process evaluation. We further introduce a structured evaluation framework along four dimensions---look back, look ahead, self-check, and goal alignment---to assess prior consistency, future plausibility, step validity, and task alignment. Based on this framework, we construct datasets for supervised fine-tuning (SFT) and reinforcement learning (GRPO). Experiments show that, despite using less training data and smaller model sizes, GPRM outperforms strong baseline models by 7.8 and 5.4 points on PRMBench and ProcessBench, respectively. GPRM also generalizes well across tasks, surpassing the GPT-4.1 baseline on Agent Error Bench, improving strong baselines by around 4 points under test-time scaling (TTS), and achieving competitive performance on PPE.
- 中文摘要
过程奖励模型(PRM)被广泛用于大型语言模型中多步推理的步骤级评估,但现有方法常受历史依赖建模薄弱和全局认知不足的限制。它们通常未能利用先前步骤评估实现连贯判断,且缺乏对后续推理的可见性,导致中间步骤评估不佳。为解决这一问题,我们提出了全球视角过程奖励模型(GPRM),该模型联合建模问题、先前推理步骤及其评估、当前步骤和未来步骤,以实现历史感知和全球知情的过程评估。我们进一步介绍了基于四个维度---回顾、展望、自我检查和目标对齐---结构化评估框架,以评估先前的一致性、未来合理性、步骤效度和任务一致性。基于该框架,我们构建了用于监督微调(SFT)和强化学习(GRPO)的数据集。实验显示,尽管使用更少的训练数据和更小的模型规模,GPRM 在 PRMBench 和 ProcessBench 上分别比强基线模型高出 7.8 分和 5.4 分。GPRM在各任务上的推广能力也很好,在Agent Error Bench上超过了GPT-4.1的基线,在测试时间缩放(TTS)下强基线提升约4分,并在PPE上实现了竞争性表现。
STA: A Split Temporal Adversary Method for Evaluating Robustness of Multimodal RL-based Driving Policies
STA:一种用于评估基于多模态RL驱动策略稳健性的分时对抗方法。
- Authors: Shenghao Xiong, Xiaolin Chang
- Subjects: Subjects:
Cryptography and Security (cs.CR); Machine Learning (cs.LG); Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.20765
- Pdf link: https://arxiv.org/pdf/2611.20765
- Abstract
Multimodal reinforcement learning (MRL) offers a promising paradigm for autonomous driving by jointly leveraging complementary sensor observations to learn effective decision-making policies. However, its effectiveness critically depends on temporal alignment across modalities, making temporal inconsistency a fundamental vulnerability. Even subtle misalignment or latency can corrupt fused representations and ultimately lead to decision failures. Existing studies have explored adversarial attacks to induce cross-modal temporal inconsistencies, thereby enabling systematic evaluation of the temporal robustness of MRL policies. However, existing methods remain limited in generality, realism, and adaptivity. To address these limitations, we propose the Split Temporal Adversary (STA), which formulates multimodal temporal attacks as a unified decision process that jointly optimizes attack timing and modality selection under realistic policy-aware and budget-constrained settings. STA includes two mechanisms: a) a unified multimodal temporal perturbation mechanism that delays observations via historical states, removing the need for modality-specific noise; b) a split multimodal adversarial decision mechanism that jointly optimizes attack timing and modality selection through adaptive optimization while leveraging the victim policy's clean actions to identify sensitive decision patterns. We provide a theoretical analysis of the induced distribution shift and a modality selection principle. Experiments in the CARLA simulator show that STA more than doubles the collision rate of the strongest baseline against PPO while using 17.8\% fewer attack attempts, and that it generalizes to a VLM-based driving policy. These results reveal temporal consistency as a critical vulnerability, highlighting temporal robustness as essential for safe autonomous driving.
- 中文摘要
多模态强化学习(MRL)通过共同利用互补的传感器观测数据来学习有效的决策策略,为自主驾驶提供了有前景的范式。然而,其有效性关键在于不同模态之间的时间对齐,这使得时间不一致成为根本性的脆弱性。即使是细微的错位或延迟也可能损坏融合表征,最终导致决策失败。现有研究已探索对抗攻击以诱导跨模态时间不一致,从而系统评估MRL策略的时间稳健性。然而,现有方法在一般性、现实性和适应性方面仍有限。为应对这些局限,我们提出了分时对抗者(STA),该方法将多模态时间攻击构建为统一决策过程,在现实的政策感知和预算约束的条件下共同优化攻击时机和模态选择。STA 包含两种机制:a)统一的多模态时间扰动机制,通过历史状态延迟观测,消除模态特异噪声的需求;b) 一种分裂多模态对抗决策机制,通过自适应优化共同优化攻击时机和模态选择,同时利用受害者策略的干净动作识别敏感决策模式。我们对诱导分布偏移进行了理论分析,并提出了模态选择原则。CARLA 模拟器中的实验表明,STA 在攻击尝试次数减少 17.8% 的情况下,将最强基线对 PPO 的碰撞率提高一倍以上,并且可推广至基于 VLM 的驾驶策略。这些结果揭示了时间一致性是关键漏洞,强调时间稳健性对安全自动驾驶至关重要。
Co-simulation-Based Safety-Assessment of Black-Box Maritime Cyber-Physical Systems
基于共仿真的黑匣子海事网络物理系统安全评估
- Authors: Andreas R. G. Sitorus, Melih Akdag, Hoang Anh Tran, Etienne Purcell
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.20775
- Pdf link: https://arxiv.org/pdf/2611.20775
- Abstract
Modern maritime cyber-physical systems consist of tightly interconnected subsystems ranging from multiple engineering domains, making exhaustive safety verification impractical. Simulation-based testing offers a scalable alternative, while co-simulation enables heterogeneous subsystem models, potentially developed by different stakeholders, to be integrated without requiring access to their internal implementations. Adaptive Stress-Testing further improves testing efficiency by formulating failure discovery as a sequential decision-making problem, guiding the search toward the most likely failure scenarios, and giving the counter safety examples using reinforcement learning. However, black-box co-simulation may provide only partial access to the system state required by conventional Markov decision process formulations. To address this limitation, we proposed Event-Based Adaptive Stress-Testing version 2, a co-simulation-based safety-assessment method formulated as a partially observable Markov decision process. This method employs Recurrent Proximal Policy Optimization to exploit observation histories when selecting adversarial actions. The method is evaluated on a ship-in-transit operation in which a collision-avoidance system is stress-tested against multi-adversarial target ships. An extended Intermediate Waypoint Sampling mechanism enables independent multi-adversary trajectory control and supports the generation of realistic collision-inducing scenarios. The experiment demonstrate that the learned stress-testing policy generalizes across different encounter cases, can be further improved through continued training, and can be made effective when the collision- avoidance system under test is replaced by another version. These results demonstrate the method's potential for scalable black-box safety assessment of complex maritime cyber-physical systems.
- 中文摘要
现代海事网络物理系统由多个工程领域的紧密互联子系统组成,使得穷尽的安全验证变得不切实际。基于仿真的测试提供了可扩展的替代方案,而协同仿真则使可能由不同利益相关者开发的异构子系统模型能够集成,而无需访问其内部实现。自适应应力测试通过将故障发现表述为顺序决策问题,引导搜索最可能的失效场景,并通过强化学习提供反安全示例,进一步提高测试效率。然而,黑箱协同仿真可能只能部分访问传统马尔可夫决策过程所要求的系统状态。为解决这一局限,我们提出了基于事件的自适应应力测试第二版,这是一种基于协同模拟的安全评估方法,作为一种部分可观察的马尔可夫决策过程。该方法采用重复近端策略优化,在选择对抗行动时利用观察历史。该方法在船舶航行中进行压力测试,针对多敌对目标舰船进行碰撞避免系统压力测试。扩展的中间航点采样机制实现独立的多敌对轨迹控制,并支持生成逼真的碰撞诱发情景。实验表明,所学的压力测试策略可推广至不同遭遇案例,通过持续训练可进一步改进,且当被测碰撞避免系统被替换为其他版本时,策略可使其有效。这些结果表明该方法在复杂海事网络物理系统中具备可扩展的黑箱安全评估潜力。
QTOP: Scaling Humanoid Motion Tracking with Quantile Critics
QTOP:用分位数批评者调整人形运动追踪
- Authors: Markus Peschl, Dana Kianfar, Daniel Dijkman
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.20791
- Pdf link: https://arxiv.org/pdf/2611.20791
- Abstract
Learning human motion tracking policies through reinforcement learning (RL) has emerged as a core capability that underpins a majority of recent progress in humanoid whole-body control. However, achieving generalization to unseen motions at high fidelity requires jointly scaling training to thousands of GPU and dataset hours. This work hypothesizes that motion tracking is inherently compute-bound because training on a diverse set of motions introduces learning interference, similar to multi-task RL. To tackle this, we propose Quantile Tracking Optimization (QTOP), which improves sample efficiency by introducing a distributional advantage estimate conditioned on state and reference pairs. QTOP outperforms on and off-policy RL baselines both in simulation and real-world tracking scenarios on multiple datasets by up to 10\%. Furthermore, we establish a scaling law for QTOP and find that, aside from scaling predictably, dataset size has minimal impact on performance for moderately sized GPU hour regimes.
- 中文摘要
通过强化学习(RL)学习人体运动跟踪策略已成为支撑人形全身控制最新进展的核心能力。然而,要实现对高保真度下的未可见动作的推广,需要联合扩展训练至数千GPU小时和数据集小时。本研究假设运动跟踪本质上受计算约束,因为对多样化运动进行训练会引入学习干扰,类似于多任务RL。为解决这一问题,我们提出了分位数跟踪优化(QTOP),通过引入基于状态和参考对的分布优势估计来提高样本效率。QTOP在多个数据集的模拟和现实跟踪场景中,均比策略外RL基线高出多达10%的优势。此外,我们为 QTOP 建立了扩展定律,发现除了可预测地扩展外,数据集大小对中等 GPU 小时数范围内的性能影响极小。
Driving the Style: Velocity-Conditioned Whole-Body Control from Motion Demonstrations
驾驭风格:通过运动演示实现速度条件的全身控制
- Authors: Murad Dawood, Oezge Drama Berkenkamp, Antonio Gonzales Marin, Nutan Chen
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.20888
- Pdf link: https://arxiv.org/pdf/2611.20888
- Abstract
Achieving human-like behavior on humanoid robots has advanced rapidly by leveraging motion-capture demonstrations rather than hand-crafted cost functions. Adversarial imitation supplies a style signal that lets a policy fulfill a task while adhering to a desired style, but is difficult to transfer to hardware. Whole-body tracking through reinforcement learning instead deploys reliably and imitates demonstrations faithfully, but the resulting policies are conditioned on the future reference, so they require a reference trajectory at runtime and can fail when the motion differs from the training data. We train a humanoid policy directly with whole-body tracking rewards while restricting its actor to compact locomotion commands and robot observations. Reference motion supervises the reward and privileged critic, without supplying reference joint trajectories to the actor or requiring teacher distillation. The demonstrated style is thereby encoded in the policy through the tracking reward, so the robot reproduces the whole-body style under commanded velocities. Trained on a single random-walk motion and commanded with velocity profiles from held-out motions, our policy reaches at least 98\% success in simulation and completes every trial on the Unitree G1. On held-out slow walking, whole-body tracking drops to 2\% success and command-conditioned distillation to 56\%, and both fail on the real robot. On the trained motions, the joint position error of our policy stays within 8\% of whole-body tracking, and it has a higher style coverage than adversarial motion priors (AMP), which fell on the real robot without a safety crane.
- 中文摘要
通过利用动作捕捉演示而非手工制作的成本函数,实现类人机器人的类人行为取得了快速进展。对抗性模仿提供了风格信号,使策略在遵守期望样式的同时完成任务,但难以转化到硬件上。通过强化学习实现全身跟踪则可靠地部署并忠实模拟演示,但最终策略依赖于未来参考,因此需要运行时参考轨迹,且当动作与训练数据不同时可能失败。我们直接通过全身追踪奖励训练类人策略,同时限制行为者执行紧凑的运动指令和机器人观察。参考动作监督奖励和特权批评者,不向行为者提供参考关节轨迹,也不要求教师提炼。演示的风格通过跟踪奖励编码进策略中,机器人在指令速度下能再现全身样式。基于单次随机行走训练,并结合保持动作的速度曲线指令,我们的策略在模拟中至少达到98%的成功率,并在Unitree G1上完成所有试验。在缓行时,全身追踪成功率降至2%,指令条件化蒸馏降至56%,且在真实机器人上两者均失败。在训练好的动作中,我们策略的联合位置误差保持在全身追踪的8%以内,且其风格覆盖率高于对抗运动先验(AMP),后者落在没有安全起重机的真实机器人上。
Comfort as a Constraint or as a Cost in Reinforcement Learning and Model Predictive Building Control
在强化学习和模型预测性建筑控制中,舒适度作为约束还是成本
- Authors: Joram Liebeskind, Valentin Bhend, Varsha N. Behrunani, Alisa Rupenyan
- Subjects: Subjects:
Systems and Control (eess.SY); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.20889
- Pdf link: https://arxiv.org/pdf/2611.20889
- Abstract
A building controller that keeps a zone within comfort bounds at minimum energy can treat the bounds as a soft constraint, by adding the weighted violation to its cost, or as a hard constraint. We analyze the effect of the hard and the soft formulation of the same problem and isolate the effect of the formulation from the effect of the algorithm. To this end, we measure the difference in energy use between the two formulations on the BOPTEST bestest_hydronic_heat_pump case, in model predictive control (MPC) and in reinforcement learning (RL). The soft formulations are a weighted MPC and weighted SAC and DDPG learners, each at twelve weights. The hard formulations are an MPC with an exact penalty and the Lagrangian learners SAC-Lagrangian and primal-dual DDPG. We find that the hard formulations use more energy than the soft formulations at the largest weight, but violate the comfort bounds less. For a large weight, the soft formulation approaches the behavior of the hard formulation. The frontiers of the energy-comfort trade-off of the three weighted controllers lie close to each other, and the weight selects an operating point on this frontier. Under the soft formulation, the three algorithms reach very similar operating points, although at somewhat different weights. For this HVAC problem, the formulation and the weight thus affect the energy use more than the choice of algorithm.
- 中文摘要
一个建筑控制器如果能将某个区域保持在舒适界限内且能量最低,可以将该界限作为软约束,将加权违规加到成本上,或者将该界限视为硬约束。我们分析同一问题的硬表述和软表述的影响,并将表述的影响与算法的影响隔离开来。为此,我们在 BOPTEST bestest_hydronic_heat_pump、模型预测控制(MPC)和强化学习(RL)中测量两种公式之间的能量使用差异。软表述包括加权 MPC 和加权 SAC 和 DDPG 学习器,每个学器权重为十二个。硬性表述为带有精确惩罚的 MPC,拉格朗日学习者为 SAC-拉格朗日和原始对偶 DDPG。我们发现,硬表述在最大权重下消耗的能量比软表述更多,但违反舒适界限的行为较少。对于较大权重,软表述趋近于硬表述的行为。三种加权控制器在能量与舒适度权衡上的边界相近,权重选择该前沿的一个操作点。在软表述下,三个算法达到非常相似的工作点,尽管权重略有不同。对于该 HVAC 问题,公式和权重对能耗的影响大于算法的选择。
Jev-as-a-Reward: Validating Probabilistic Safety Judges for Reinforcement Learning
Jev-as-a-reward:验证概率安全判断者以实现强化学习
- Authors: Eunkyeong Lee, Junhyung Moon, Seunggwan Hong, Dongsuk Oh
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.20955
- Pdf link: https://arxiv.org/pdf/2611.20955
- Abstract
Probabilistic safety judges return the probability of a safety label and cheaply score reinforcement learning rollouts. A confidence gate sends only uncertain verdicts to a stronger LLM judge, which leaves confidently passed harm in a blind spot. Policies can exploit this blind spot, and commercial judges release new versions. We propose a protocol that validates such a judge before and during training, separates the share of confidently passed harm from its amount, and re-validates it after each judge change. We apply it to two versions of Jev, a commercial decision-only judge, and to an open guard classifier. Under the probability reward of Jev, passed harm shrinks in all 11 runs under the primary reference, mostly more slowly than caught harm, and concentrates in the blind spot. Binarizing this judge meets the escape criterion on Llama, and the open guard classifier shows an escape signal in one seed under a second reference. Guard classifiers share this blind spot, and even frontier LLM judges catch at most one of 10 human-labeled confident misses. Without a trained reward model, the probability reward with a helpfulness term lowers HarmBench attack success on Llama-3.1-8B from 0.287 to at most 0.033 at the final checkpoint.
- 中文摘要
概率安全裁判返回获得安全标签的概率,并廉价地为强化学习推广打分。信心门只将不确定的判决发送给更强的 LLM 裁判,从而将自信已通过的伤害留在盲区。政策可以利用这一盲区,商业裁判会发布新版本。我们提出一种协议,在训练前和训练期间验证此类裁判,将确信通过的伤害份额与其数量分离,并在每次裁判更换后重新验证。我们将此方案应用于两种 Jev:一种仅商业决策的裁判,以及一种开放式守护分类器。在 Jev 的概率奖励下,在主要参考下,所有 11 次运行中,通过伤害都会缩小,且多半比受伤时慢,并集中在盲区。将该裁判二元化即可满足 Llama 的逃脱标准,开放守卫分类器在第二个参考点下显示一个种子中的逃脱信号。守卫分类器也有同样的盲点,甚至前沿的 LLM 裁判最多只能抓到 10 个人工标记的自信失误中的一个。在没有训练好的奖励模型的情况下,带有帮助项的概率奖励会将 HarmBench 对 Llama-3.1-8B 的攻击成功率从 0.287 降至最终检查点最多 0.033。
What Makes a Good Move? Learning to Act from Experience in Visual Worlds
什么才是明智的选择?从视觉世界中的经验学习行动
- Authors: Cehao Yang, Ziyi He, Jing Yi, Jian Guo, Hui Xiong
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.21065
- Pdf link: https://arxiv.org/pdf/2611.21065
- Abstract
Learning visual policies requires turning interaction experience into guidance for individual decisions. Trajectory outcomes obscure action quality, while unguided exploration can revisit familiar situations. We present \textsc{LightAct}, an experience-learning framework that connects skill-guided exploration with policy internalization. Interaction segments are summarized into experience entries and consolidated into a game-specific skill that guides root collection toward broader state coverage. At each root, candidate actions are compared through short rollouts under a shared frozen policy. Environment-reported score gains are averaged across continuations and normalized into relative action rewards, producing reusable supervision for reinforcement learning. Policy training uses a compact dataset of just 1,024 decision roots collected in a single source game and paired with their action rewards. Across 170 GameWorld tasks spanning 34 games, \textsc{LightAct} improves overall progress by 23.6, 29.2, and 13.0 percentage points at the 2B, 4B, and 9B scales, respectively, with our strongest model competitive with reported proprietary-model results. Ablations show broader root-state coverage with skill guidance and consistent gains across three policy optimizers. Case studies show how rollouts distinguish actions with identical immediate gains.
- 中文摘要
学习可视化策略需要将交互体验转化为个人决策的指导。轨迹结果会掩盖行动质量,而无引导探索则可能重访熟悉的情境。我们介绍了 \textsc{LightAct},这是一种将技能引导探索与策略内化相结合的体验学习框架。交互片段被总结为体验条目,并整合成游戏专属技能,引导根收集实现更广泛的状态覆盖。在每个根节点,候选行动通过共享冻结策略下的短暂部署进行比较。环境报告的分数增长在连续过程中被平均化,并归一化为相对行动奖励,从而为强化学习提供可重复使用的监督。策略训练使用仅收集在单一源游戏中的 1,024 个决策根的紧凑数据集,并与其行动奖励配对。在 170 个 GameWorld 任务中,跨越 34 个游戏,\textsc{LightAct} 在 2B、4B 和 9B 尺度上分别提升了 23.6、29.2 和 13.0 个百分点,我们最强的模型与已报道的专有模型结果相抗衡。消融分析显示出更广泛的根状态覆盖,并附有技能指导,并在三种策略优化器中持续获得持续的收益。案例研究展示了推广如何区分具有相同即时收益的行动。
CoopDistill: Equilibrium-Guided Multi-Agent Policy Distillation for Cooperative Driving
CoopDistill:均衡引导多主体政策蒸馏以实现合作驱动
- Authors: Shiyu Fang, Sheng Zhao, Jiarui Zhang, Chengkai Xu, Xiaocong Zhao, Peng Hang, Jian Sun
- Subjects: Subjects:
Robotics (cs.RO); Multiagent Systems (cs.MA)
- Arxiv link: https://arxiv.org/abs/2611.21069
- Pdf link: https://arxiv.org/pdf/2611.21069
- Abstract
Cooperative driving extends autonomous driving from individual vehicle decisions to joint maneuvers that improve traffic safety and efficiency. Optimization-based methods impose safety constraints but often fix paths to simplify multi-vehicle planning, limiting spatial flexibility. Multi-agent reinforcement learning (MARL) explores alternatives through interaction, but coordinated learning and safe execution remain challenging. Their integration nevertheless faces challenges in exploration efficiency, experience allocation, and execution reliability. To address these challenges, this paper proposes CoopDistill, an equilibrium-guided multi-agent policy distillation framework. A potential-game (PG) formulation provides a fixed-path longitudinal reference, around which a residual MARL policy learns longitudinal acceleration, conflict-approach timing, and lateral path corrections. A hard-case curriculum concentrates experience on interactions that PG leaves unresolved or substantially delayed. A comparative rollout selector evaluates the joint consequences of learned corrections against PG before choosing the executed action. Experiments in merging, roundabout, and intersection scenarios compare CoopDistill with six baselines at four connected automated vehicle penetration levels. Relative to PG, success improves by 2.1--27.1 percentage points and delay decreases by 7.3--27.3\%, with no observed collisions. Code is available at this https URL
- 中文摘要
协同驾驶将自主驾驶从单个车辆决策扩展到联合操作,从而提升交通安全和效率。基于优化的方法会施加安全约束,但通常固定路径以简化多车规划,限制空间灵活性。多智能体强化学习(MARL)通过交互探索替代方案,但协调学习和安全执行仍具挑战性。然而,其整合在探索效率、经验分配和执行可靠性方面面临挑战。为应对这些挑战,本文提出了CoopDistill,一种均衡引导的多智能体策略提炼框架。潜在博弈(PG)表述提供了固定路径的纵向参考,剩余MARL策略在此基础上学习纵向加速、冲突接近时机和横向路径修正。硬性案例课程集中经验于PG未解决或大幅延迟的交互。一个比较性展开选择器会在选择执行动作前,评估学习到的修正与PG的联合后果。在并行、环岛和交叉场景中的实验将CoopDistill与六条基线在四个互联自动驾驶车辆渗透层级下进行比较。与PG相比,成功率提升了2.1-27.1个百分点,延迟减少了7.3%--27.3%,且未观察到碰撞。代码可在此 https URL 获取。
Conflict-Aware Multi-Teacher On-Policy Distillation in Aligned Action Spaces for Cross-Game Generalization
冲突感知的多教师策略提炼在对齐行动空间中实现跨游戏泛化
- Authors: Jing Yi, Cehao Yang, Hualei Zhang, Haitao Li
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.21094
- Pdf link: https://arxiv.org/pdf/2611.21094
- Abstract
Zero-shot cross-game generalization aims to consolidate accessible source-game experts into a single vision-language model (VLM) that controls unseen games from raw pixels without target-game parameter updates. While multi-teacher on-policy distillation (MOPD) mitigates offline exposure bias and single-game reinforcement learning (RL) overfitting, scaling it to autoregressive VLMs faces two bottlenecks. First, token-to-action misalignment from subword length bias distorts discrete action probabilities and uncertainty margins. Second, multi-teacher policy conflict arises as source teachers under incompatible dynamics issue contradictory closed-loop corrections, causing negative transfer. We propose CAMOPD (Conflict-Aware Multi-Teacher On-Policy Distillation), a post-training framework that resolves multi-teacher policy conflict within an aligned semantic action space. CAMOPD first uses complete-action alignment to normalize command completion likelihoods over each game's legal action set, forming a calibrated categorical policy that provides reliable uncertainty margins for selective on-policy retention and visual specialist probes that can be evaluated across source games. To resolve policy conflict without target-game data, conflict-aware multi-teacher consolidation cross-evaluates these specialists across source games to capture both visual identifiability and inter-teacher compatibility, upweighting compatible teachers while suppressing conflicting corrections. On GameWorld's temporal and directional transfer suites with Qwen3.5-4B, CAMOPD achieves 26/30 (86.7%) and 24/30 (80.0%) Task-1 zero-shot success on unseen GeoDash and Doodle Jump, outperforming prompting, weight-merging, and multi-task baselines.
- 中文摘要
零样本跨游戏泛化旨在将可访问的源游戏专家整合到一个单一视觉语言模型(VLM),该模型能从原始像素控制未见的游戏,无需更新目标游戏参数。虽然多教师策略提炼(MOPD)缓解了离线暴露偏差和单博弈强化学习(RL)过拟合,但将其扩展到自回归VLM时面临两个瓶颈。首先,子词长度偏误导致的令牌与动作错位扭曲离散动作概率和不确定性边际。其次,多教师策略冲突发生,源教师在不兼容动态下发出矛盾的闭环纠正,导致负迁移。我们提出了CAMOPD(冲突感知多教师策略提炼),这是一种培训后框架,可在对齐语义行动空间内解决多教师策略冲突。CAMOPD 首先采用完全动作对齐法,对每个游戏的法律行动集进行指令完成概率进行规范化,形成一个经过校准的分类策略,为选择性策略保留和可视化专家探针提供可靠的不确定性余地,并可跨源游戏进行评估。为了在没有目标游戏数据的情况下解决策略冲突,冲突感知的多教师整合会在源游戏中交叉评估这些专家,以捕捉视觉识别性和教师间兼容性,对兼容教师进行加权,同时抑制冲突纠正。在 GameWorld 的 Qwen3.5-4B 时间和方向传输套件中,CAMOPD 在未见的 GeoDash 和 Doodle Jump 上,任务 1 零射程成功率为 26/30(86.7%)和 24/30(80.0%),优于提示、权重合并和多任务基线。
WinDEX: Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning
WinDEX:用于灵活远程操作和政策学习的度量级三维手部运动重建技术
- Authors: Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Kai Shi, Jiahao Ai, Xin You, Peng Yin
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.21143
- Pdf link: https://arxiv.org/pdf/2611.21143
- Abstract
Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for world-frame reconstruction; (2) WinDEX-Net, an end-to-end, fisheye-native network combining calibrated stereo, local detail and temporal context, with a data-model flywheel of model-assisted annotation and retraining; (3) distilled models for offline reconstruction and causal streaming, supporting reinforcement learning and teleoperation; and (4) low-cost CyberCap stereo-inertial capture, forming the WinDEX-Hand dataset with 1,000 hours of training annotations and held-out human-annotated in-the-wild 2D and independent optical motion-capture evaluation sets. Offline reconstruction runs at 43.61 FPS and causal streaming at 171 FPS. On held-out data, offline reconstruction achieves state-of-the-art accuracy: 14.5 pixels in-the-wild reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error. We achieve 7.6 and 3.6 times the task success rates of reinforcement learning policies trained with WiLoR and HaWoR motion references, respectively, and demonstrate real-time arm-hand teleoperation. Project website: this https URL.
- 中文摘要
人类操控数据被广泛用于训练机器人达到类似人类的灵巧度。然而,在野外进行低成本、大规模采集仍然具有挑战性。现有方法在几何基础的度量尺度重建、现实世界的泛化和推断速度方面仍然有限。我们介绍 WinDEX,它集成了 (1) 用于野外自我中心立体视频的度量 3D 手部注释的几何-时间优化,并结合视觉惯性 SLAM 进行世界帧重建;(2) WinDEX-Net,一个端到端、鱼眼原生网络,结合校准立体、局部细节和时间上下文,并配有模型辅助标注和重训练的数据模型飞轮;(3) 用于离线重建和因果流式的精炼模型,支持强化学习和远程操作;以及(4)低成本的CyberCap立体惯性捕获,构成WinDEX手数据集,包含1000小时的训练注释和显示的人类注释的野外2D和独立光学动作捕捉评估集。离线重建速度为43.61 FPS,因果流式为171 FPS。在保留数据上,离线重建实现了最先进的精度:野外重投影误差为14.5像素,结合视觉惯性里程计,世界帧联合误差为37.29毫米。我们分别实现了使用 WiLoR 和 HaWoR 运动参考训练的强化学习策略任务成功率的 7.6 倍和 3.6 倍,并展示了实时手臂手部远程操作。项目网站:此 https URL。
Heteroscedastic Gaussian Process Q-learning for Sample Efficient Experimentation
异方差高斯过程 Q 学习以实现样本高效实验
- Authors: Maximilian Bloor, Max Mowbray, Calvin Tsay, Ehecatl Antonio del Rio Chanona
- Subjects: Subjects:
Machine Learning (cs.LG); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.21249
- Pdf link: https://arxiv.org/pdf/2611.21249
- Abstract
Self-driving laboratories must learn to control a process from tens to hundreds of expensive experiments, too few for deep reinforcement learning (RL). We learn the Q-function directly with a Gaussian process (GP), which avoids learning the dynamics, and show that the GP posterior supplies the learning rate of Q-learning. If each Bellman target is treated as a noisy measurement of the Q-function, the learning rate that minimises the mean squared error is the Kalman gain, and applying it to every observed transition is equivalent to GP regression. Heteroscedastic Gaussian Process Q-learning (HGPQ) sets the noise variance of each target with a heteroscedastic GP, which learns how unreliable the targets are across the states and inputs. Targets from noisy operating conditions, or from regions where the value function has large gradients, then move the estimate less than targets from low-noise regions, so HGPQ learns quickly where the data are reliable without being misled where they are not. For kernel value iteration with per-target variances that are assigned when a target is collected, we prove a high-probability bound on the error of the value estimate, uniform over the explored region. Pooled over five simulated benchmarks, which include a chemical reactor and a bioprocess, HGPQ approaches the best return found by any method in fewer episodes than GP, kernel and deep RL baselines.
- 中文摘要
自动驾驶实验室必须从数十次到数百次昂贵的实验中学习控制一个过程,而这些实验数量不足以进行深度强化学习(RL)。我们直接通过高斯过程(GP)学习Q函数,避免学习动力学,并证明GP后验提供了Q-学习的学习速率。如果将每个贝尔曼目标视为Q函数的噪声测量,最小化均方误差的学习率即为卡尔曼增益,将它应用于每一个观测到的转变等价于GP回归。异方差高斯过程 Q-learning(HGPQ)通过异方差 GP 设置每个目标的噪声方差,该 GP 学习目标在状态和输入上的不可靠性。来自噪声运行条件或值函数梯度较大的区域的目标,则估计值的移动幅度小于低噪声区域的目标,使 HGPQ 能够快速学习数据在哪些地方可靠,而不会在不可靠的地方被误导。对于在收集目标时分配的每个目标方差的核值迭代,我们证明了值估计误差的高概率界限,且在所探索区域内均匀。HGPQ汇集在五个模拟基准上,包括化学反应器和生物过程,在比GP、核和深度RL基线更少的集数内,接近任何方法中获得的最佳回报。
Deep Learning for Physicists: From Linear Regression to Reasoning Models
物理学家的深度学习:从线性回归到推理模型
- Authors: Masanori Hanada
- Subjects: Subjects:
Machine Learning (cs.LG); High Energy Physics - Lattice (hep-lat); High Energy Physics - Theory (hep-th); Computational Physics (physics.comp-ph)
- Arxiv link: https://arxiv.org/abs/2611.21295
- Pdf link: https://arxiv.org/pdf/2611.21295
- Abstract
These lecture notes are based on an introductory course on neural networks and deep learning that the author taught in the School of Mathematical Sciences at Queen Mary University of London in 2023/24 and 2024/25. The author, however, is a physicist, and it shows: wherever possible, ideas are explained with physical intuition rather than heavy formalism. Readers with a background in physics or mathematics who want a self-contained, introductory route into modern machine learning may therefore find these notes particularly friendly. The notes start from maximum likelihood, least squares, linear regression and (stochastic) gradient descent. They then cover neural networks, backpropagation, the vanishing gradient problem, batch and layer normalization, overfitting and regularization, adversarial examples, and the kernel method. The second half turns to natural language processing: word embeddings, recurrent networks and LSTMs, the attention mechanism and Transformers, and large language models, including scaling laws, the history of GPT, instruction tuning, reinforcement learning from human feedback, hallucination, alignment faking, chain-of-thought reasoning and test-time scaling. The emphasis on large language models reflects requests from the students. The notes are aimed at third- and fourth-year students in the UK system. Because the course was also taken by students from other departments, very little is assumed: even basic material such as matrix multiplication is explained from scratch, so readers familiar with it can safely skip such parts. No originality is claimed; the aim is a coherent, intuitive path through the material. Developments after spring 2025 are not covered, but the basic ideas emphasized here are unlikely to go out of date soon.
- 中文摘要
这些讲义基于作者于2023/24和2024/25学年在伦敦玛丽女王大学数学科学学院教授的神经网络与深度学习入门课程。然而,作者是一位物理学家,这表明:只要可能,思想都以物理直觉而非繁重的形式主义来解释。因此,具有物理或数学背景且希望获得现代机器学习入门路径的读者,可能会觉得这些笔记特别友好。这些笔记从最大似然、最小二乘法、线性回归和(随机)梯度下降开始。随后涵盖神经网络、反向传播、梯度为零问题、批次和层归一化、过拟合与正则化、对抗例题以及核方法。下半部分转向自然语言处理:词嵌入、递归网络和LSTMs、注意力机制和变换器,以及大型语言模型,包括缩放定律、GPT的历史、指令调优、基于人类反馈的强化学习、幻觉、比对伪造、思维链推理和测试时间尺度。对大型语言模型的重视反映了学生的需求。这些笔记面向英国体系中的三、四年级学生。由于该课程也由其他系的学生修读,因此假设内容非常有限:即使是基础内容如矩阵乘法,也要从零开始讲解,熟悉该内容的读者可以放心跳过这些部分。不声称有原创性;目标是为材料提供一条连贯、直观的路径。2025年春季之后的发展未被涵盖,但这里强调的基本观点不太可能很快过时。
ALIVE: Making Streaming Video Models Interactive
活着:让流媒体视频模型变得互动
- Authors: Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao Liu
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.21366
- Pdf link: https://arxiv.org/pdf/2611.21366
- Abstract
Streaming video LLMs can continuously observe the world, yet often handle user requests in isolation. We introduce ALIVE, an interactive streaming video model supporting proactive monitoring, streaming task coordination, and feedback-seeking interaction. We construct ALIVE-SFT-288K and ALIVE-RL-3K for two-stage training. Contrastive supervised fine-tuning teaches when and how to respond, while reinforcement learning with Behavior-Aligned Credit Assignment (BACA) combines trajectory-level and local advantages to refine decisions, responses, and silence across multi-turn interactions. We also introduce FSI-Bench to evaluate feedback seeking and subsequent task completion. ALIVE achieves the highest overall scores among the compared models on four public benchmarks and FSI-Bench, demonstrating strong streaming understanding and interactive capabilities.
- 中文摘要
流式视频LLM可以持续观察世界,但通常会单独处理用户请求。我们介绍了 ALIVE,一种交互式流式视频模型,支持主动监控、流式任务协调和反馈寻求交互。我们构建了 ALIVE-SFT-288K 和 ALIVE-RL-3K,用于两阶段训练。对比式监督式微调教导何时以及如何作出响应,而行为对齐学分分配(BACA)的强化学习结合了轨迹级和局部优势,优化多轮互动中的决策、响应和静默。我们还引入了FSI-Bench,用于评估反馈寻求及后续任务完成情况。ALIVE在四个公开基准测试和FSI-Bench中获得最高整体评分,展现出强大的流式理解和交互能力。
Federated TD Learning in Heterogeneous Environments with Average Rewards: A Two-timescale Approach with Polyak-Ruppert Averaging
异构环境中的联合TD学习与平均回报:采用Polyak-Ruppert平均法的两时间尺度方法
- Authors: Ankur Naskar, Gugan Thoppe, Abbasali Koochakzadeh, Vijay Gupta
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.21370
- Pdf link: https://arxiv.org/pdf/2611.21370
- Abstract
Federated Reinforcement Learning (FRL) provides a promising way to speedup training in reinforcement learning when one has access to multiple edge devices that can operate in parallel. Recently, it has been shown that even when these edge devices have access to different dynamic models---such as cars or robots with differing system configurations---an optimal convergence rate that has a linear speedup proportional to the number of devices is achievable. However, this result requires that the algorithm's stepsize be chosen in a manner that depends on the unknown model parameters. Also, it applies only to a discounted setting, which has been argued to fit episodic tasks better than continuing control tasks. In this paper, we obtain finite-time bounds for heterogeneous FRL with average rewards. We show that optimal convergence rate and linear speedup are achievable even with a universal stepsize choice, independent of the underlying dynamics. To achieve our result, we modify the existing one-timescale FRL method to a novel two-timescale variant that additionally incorporates iterate averaging. We remark that this idea also extends to other settings already studied in the literature, such as the homogeneous FRL or the heterogeneous case with a discounted cost function.
- 中文摘要
当拥有多个可并行运行的边缘设备时,联邦强化学习(FRL)提供了一种有前景的方法,可以加快强化学习训练的速度。最近,研究表明,即使这些边缘设备能够访问不同的动态模型---例如具有不同系统配置的汽车或机器人---仍然可以实现与设备数量成正比的线性加速的最佳收敛率。然而,这一结果要求算法的步长必须依赖未知模型参数来选择。此外,它仅适用于折现设置,而有人认为这种设置比持续控制任务更适合情节任务。本文中,我们获得了具有平均奖励的异构 FRL 的有限时间界限。我们证明,即使采用通用步长选择,且不受底层动力学影响,也能实现最佳收敛率和线性加速。为实现我们的结果,我们将现有的单一时间尺度 FRL 方法修改为一种新的双时间尺度变体,并额外包含迭代平均。我们指出,这一理念也延伸到文献中已研究的其他情形,如齐次 FRL 或带有折现成本函数的异质情形。
WeVisForensics: Self-Evolving Agent for Omni AI-Generated Video Detection
WeVisForensics:用于全域AI生成视频检测的自我进化代理
- Authors: Yipeng Du, Yuxi Li, Ying Zhang, Chong Sun, Chen Li, Jing Lyu, Ying Tai
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.21373
- Pdf link: https://arxiv.org/pdf/2611.21373
- Abstract
Recent advances in video generation call for reliable detection grounded in explicit forensic evidence. However, current methods primarily depend on single-pass chain-of-thought reasoning that overlooks brief or localized artifacts, or on predefined multi-agent workflows that offer limited flexibility in selecting relevant inspections. Even with adaptive investigation, optimizing the investigation policy alone does not directly improve the forensic capabilities of fixed experts. We introduce WeVisForensics, an agentic framework in which a main investigator agent coordinates a self-evolving library of forensic inspector sub-agents spanning visual and audio cues. The investigator learns which artifact type to examine, over which window, and when to conclude, and these interactions provide shared feedback for both levels of agents. Our dual-timescale policy optimization strategy, DTPO, closes this loop through fast reinforcement learning of the investigator and slower refinement of the inspectors, guided by annotated artifact intervals, to report subtle artifacts they once dismissed. Furthermore, we curate ForenCue-30K, combining role-specific forensic annotations and interaction trajectories to supervise individual inspections and their coordination, with an accompanying reinforcement-learning pool. Experiments show that WeVisForensics averages 95.2% detection accuracy across in-domain, out-of-domain and agentic benchmarks, 9.0% above the strongest prior detector, while grounding its verdicts in localized evidence.
- 中文摘要
视频生成领域的最新进展要求基于明确的法医证据进行可靠的检测。然而,当前方法主要依赖单次思路推理,忽略了短暂或局部的伪影,或依赖预定义的多代理工作流程,这些工作流在选择相关检查方面灵活性有限。即便采用自适应调查,单靠优化调查策略也无法直接提升固定专家的取证能力。我们介绍了WeVisForensics,这是一种代理框架,主调查员协调一个自我演进的法医检查员子代理库,涵盖视觉和听觉线索。调查员学习应检查哪种文物类型、在哪个窗口内以及何时结束,这些互动为两级代理提供共享反馈。我们的双时间尺度策略优化策略DTPO,通过对调查员的快速强化学习和对检查员的较慢细化进行优化,并以注释化的伪件区间为指导,来报告他们曾经忽略的细微伪影,从而封闭这一循环。此外,我们策划了ForenCue-30K,结合角色特定的取证注释和交互轨迹,以监督个别检查及其协调,并配套强化学习池。实验显示,WeVisForensics 在域内、域外和代理基准测试中的平均检测准确率为 95.2%,比最强先验检测器高出 9.0%,同时其判决基于局部证据。
A Reinforcement Learning Method to Navigate SMEFT Operator Space
一种强化学习方法,用于在 SMEFT 操作员领域中导航
- Authors: Jacky Kumar
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); High Energy Physics - Experiment (hep-ex); High Energy Physics - Phenomenology (hep-ph)
- Arxiv link: https://arxiv.org/abs/2611.21439
- Pdf link: https://arxiv.org/pdf/2611.21439
- Abstract
In particle physics, discrepancies between experimental measurements and Standard Model (SM) predictions are commonly interpreted within new theoretical models. In this context, the Standard Model Effective Field Theory (SMEFT) provides a general framework encompassing a broad class of models. However, exploring the enormous space of possible SMEFT operators rapidly becomes a challenging combinatorial problem: many operators can contribute simultaneously to different observables, their effects are correlated, and distinct operator combinations can provide comparably good descriptions of the data. In this work, we formulate the exploration of the SMEFT operator space as a sequential decision-making problem using reinforcement learning (RL). Our RL agent employs a lightweight actor-critic architecture with approximately 40K trainable parameters in total. Rather than searching for a single best-fit theory, our method is designed to discover a diverse set of high-quality theory hypotheses. We test the method in a complex phenomenological environment comprising 132 observables across several sectors of particle physics, including 24 measurements that deviate from their SM predictions by more than $1.5\sigma$. We show that the learned RL policy efficiently constructs multi-operator theories that substantially improve the global fit. Comparisons with independent random-subset and random-sequential searches demonstrate the ability of the RL policy to navigate this large combinatorial space. Our results establish RL as a systematic framework for exploring diverse patterns of new physics (NP) interactions within SMEFT.
- 中文摘要
在粒子物理学中,实验测量与标准模型(SM)预测之间的差异通常在新的理论模型中被解释。在此背景下,标准模型有效场论(SMEFT)提供了一个涵盖广泛模型类别的通用框架。然而,探索可能的SMEFT算符的巨大空间很快成为一个具有挑战性的组合问题:许多算符可以同时贡献于不同的可观测量,它们的效应相互关联,不同算符组合也能提供相当优越的数据描述。在本研究中,我们将SMEFT算符空间的探索提出为利用强化学习(RL)的顺序决策问题。我们的RL代理采用轻量级的演员-批评者架构,总共约4万个可训练参数。我们的方法不寻求单一最佳拟合理论,而是旨在发现多样化的高质量理论假设。我们在一个复杂的现象学环境中测试该方法,该环境涵盖多个粒子物理领域,包含24项与SM预测偏差超过1.5美元的测量值。我们表明,所学的RL策略能够高效构建多算符理论,显著提升全局拟合度。与独立随机子集和随机顺序搜索的比较显示,RL策略能够在这一庞大的组合空间中导航。我们的结果确立了RL作为探索SMEFT中新物理(NP)交互模式的系统框架。
Warm Starting Neural Dynamics Models From Few Jacobian Samples
从少数雅可比样本中生成的温起始神经动力学模型
- Authors: Maxime Rochkoulets, Joppe De Jonghe, Mariya Ishteva
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.21465
- Pdf link: https://arxiv.org/pdf/2611.21465
- Abstract
Model-based reinforcement learning (MBRL) leverages a model of the environment, used for planning and generating samples without interacting with the real world. Motivated by evidence that informed initialization significantly improves the efficiency of MBRL, we propose TDec-Init, an algorithm to warm-start neural dynamics models using transition Jacobians. Using a decoupling algorithm based on the decomposition of a tensor of stacked Jacobian samples, we recover weights and flexible nonlinearities, and use them to initialize a feedforward neural network. In the low-data regime, we show on several benchmark environments that our approach outperforms random initialization and value training, and either matches or outperforms Sobolev training, while requiring fewer training steps between episodes.
- 中文摘要
基于模型的强化学习(MBRL)利用环境模型,用于规划和生成样本,而无需与现实世界交互。基于证据表明知情初始化显著提升 MBRL 效率,我们提出了 TDec-Init 算法,一种利用转移雅可比矩阵进行神经动力学模型预热启动的算法。利用基于叠加雅可比样本张量分解的解耦算法,我们恢复权重和灵活非线性,并用它们初始化前馈神经网络。在低数据环境下,我们在多个基准环境中表明,我们的方法优于随机初始化和值训练,且能匹敌甚至优于索博列夫训练,同时每集之间所需的训练步骤更少。
Algorithm Distillation for Safe In-Context Reinforcement Learning
用于安全上下文强化学习的算法提炼
- Authors: Minjae Kwon, Amir Moeini, Shangtong Zhang, Lu Feng
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.21524
- Pdf link: https://arxiv.org/pdf/2611.21524
- Abstract
Safe in-context reinforcement learning (Safe ICRL) asks a policy with fixed parameters to improve on an unseen constrained task from its interaction history while aiming to keep its cost within a budget. Prior work studied Safe ICRL on five benchmarks in three families: a grid world with a hidden goal and obstacles (DarkRoom), continuous navigation where goals and hazards stay invisible until reached (PointGoal, CarGoal), and locomotion with a hidden velocity limit (HalfCheetah, Ant). Safe Algorithm Distillation (Safe AD), trained on the learning histories of a safe RL algorithm, has shown adaptation failures in prior evaluations. For example, in the locomotion tasks its return stays below zero, and it rarely reaches the goal in CarGoal. We revisit two parts of Safe AD. First, we distill a budget-aware teacher with two phases. It searches for the hidden task, exploring more when more budget remains. After its first reward or cost, an expert policy exploits the task within the remaining budget. We collect the teacher's histories under fixed or dynamic budgets. Dynamic budgets increase the occupancy entropy of the return and budget inputs in every environment, and at test time they improve return and lower cost in DarkRoom and HalfCheetah. Second, we propose A-Lag, a test-time adjustment that reweights the distilled policy's actions by their predicted cost, with an adaptive Lagrange multiplier that is updated from the costs observed. In our experiments, A-Lag had lower mean episode cost than Safe AD in all five environments, with lower mean final return in four.
- 中文摘要
安全情境强化学习(Safe ICRL)要求一个具有固定参数的策略,从交互历史中改进一个看不见且受限的任务,同时力求将其成本控制在预算之内。此前的研究研究了基于三个家族的五个基准测试了安全ICRL:带有隐藏目标和障碍的网格世界(DarkRoom)、目标和危险在到达前保持隐形的连续导航(PointGoal、CarGoal)以及具有隐藏速度限制的移动(HalfCheetah、Ant)。基于安全强化学习算法的学习历史训练的安全算法蒸馏(Safe AD),在先前评估中显示出适应失败。例如,在移动任务中,其返回率始终低于零,且很少达到CarGoal中的目标。我们回顾了Safe AD的两个部分。首先,我们提炼出一位具有两个阶段的预算意识型教师。它会寻找隐藏任务,当预算剩余时会探索更多。在完成第一个奖励或成本后,专家策略会在剩余预算内利用该任务。我们收集教师在固定或动态预算下的历史记录。动态预算会增加每个环境中的回报和预算输入的占用熵,测试时它们能提升回报并降低 DarkRoom 和 HalfCheetah 的成本。其次,我们提出了 A-Lag,即测试时调整方法,根据预测成本重新加权提炼策略的行为,并配备根据观察到的成本更新的自适应拉格朗日乘数。在我们的实验中,A-Lag在所有五种环境中的平均每集成本都低于Safe AD,在四个环境中的平均最终回报更低。
Procedure without Judgment: What Small Language Models Learn under an Exact Verifier
无判断过程:小型语言模型在精确验证器下学习的内容
- Authors: Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R., Arun Prakash, S. Senthilkumar
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.21529
- Pdf link: https://arxiv.org/pdf/2611.21529
- Abstract
When an authorization policy blocks an agent's tool call, the audit log should name the facts that forced the decision and nothing more. For an executable policy over an enumerable context space this minimal sufficient reason can be computed exactly, so an explanation is scored against ground truth rather than a proxy for it. We use that exactness to ask what a model can learn from a verifier that has no error in it. The explainer sees one request and one decision, never the policy, and has thirty turns in which it may call the policy on altered requests and ask whether a candidate set of facts is sufficient. Frontier models find the exact reason about half the time with the tools alone, and one sentence telling them to shrink the answer lifts Haiku 4.5 from 58% to 75%. A 4B model fine-tuned on 2000 traces of the deletion algorithm reaches 92% on the policy it was trained on. Teacher kind is what moves this: at matched data size the algorithm-taught student leads the frontier-taught one in all three training seeds, and the two arms never overlap. What the student learns is a fixed schedule of sufficiency checks, and the schedule does not travel. On three public policies it never saw, its answers stay sufficient (94%) and stop being minimal (48%), while Haiku under the same sentence is exact on all 47 episodes. Reinforcement learning from the exact reward leaves held-out accuracy where it was and inverts the strategy, from probing the policy to spending the entire budget on sufficiency checks. The oracle made the effect of training legible, but it did not make training better.
- 中文摘要
当授权策略阻止代理工具调用时,审计日志应列出促使该决策的事实,仅此而已。对于可枚举上下文空间下的可执行策略,这个最小充分理由可以精确计算,因此解释是基于真实事实而非代理评分。我们利用这种精确性来询问模型能从没有错误的验证者那里学到什么。解释者只看到一个请求和一个决策,从不看到策略本身,并且有三十个回合,在这期间它可以在修改请求时调用策略,并询问候选事实集是否足够。前沿模型仅用工具大约一半的时间能找到确切原因,一句让它们缩小答案的话,就能将 Haiku 4.5 从58%提升到75%。一个4B模型在2000次删除算法痕迹上微调后,在训练策略上达到92%。推动这一进程的是教师类型:在匹配的数据量下,算法学的学生在前沿教导的三个训练种子中领先一个,且两条训练种子从不重叠。学生学到的是固定的充分性检查时间表,且时间表不会随波逐流。在三项它从未见过的公共政策上,其回答保持足够(94%),不再是最微(48%),而同一句话下的俳句在全部47集中完全一致。从精确奖励进行强化学习,会让原有的准确性被搁置,并颠倒了策略,从深入政策到将全部预算花在充分性检查上。预言机使训练效果变得清晰可读,但并未让训练变得更好。
Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle
乐观的验证者被黑:奖励在精确流程下的黑客攻击 Oracle
- Authors: Arunkumar V, Manoranjan Gandhudi, Gangadharan G. R., Arun Prakash, S. Senthilkumar
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.21538
- Pdf link: https://arxiv.org/pdf/2611.21538
- Abstract
Reinforcement learning from verifiable rewards inherits whatever its verifier gets wrong. Outside mathematics and code the verifier is a proxy, a rubric, a learned judge or a hand-written checker, and its distance from the true objective is never measured, so a rising reward cannot be told apart from a hack. We build a setting where the true reward is exactly computable at the process level: executable authorization policies over 124,416 contexts admit an oracle that lists every minimal sufficient reason for every decision. We compile these policies into exact process rewards, degrade the verifier along seven realistic axes (outcome-only, sufficiency, recall-only, extraction noise, coarse taxonomy, stale policy, LLM judge), and train against each while the exact verifier scores every rollout. Four phenotypes appear. Optimistic verifiers are hacked by padding: the sufficiency proxy ends at 0.940 while true faithfulness falls from 0.731 to 0.667 on 96 held-out episodes. Outcome-only verifiers saturate and stop teaching, noise dilutes, and ranking-preserving gaps do no harm. A model-free audit of the verifier, its selection error and gap sign, together with a short saturation pilot, anticipates which phenotype a run will produce. Across nine configurations spanning two frameworks, two model vendors, two adapter sizes, three precisions, two initialisations and two seeds, the phenotypes recur, sufficiency damage is positive in all 10 replication cells and beyond two standard errors in 7, and optimistic gaps remain the steeper branch. The damage magnitude does not replicate, and five pre-registered attempts to explain it are reported as they fell. We release the testbed, the training instrumentation, and 14 real public policies (Cedar, OPA, Gatekeeper) ported to executable oracles.
- 中文摘要
从可验证奖励中进行强化学习,会继承验证者所犯的错误。在数学和代码之外,验证者是代理、评分标准、学习的裁判或手写检查器,其距离真实目标的距离从不被测量,因此奖励上升无法区分为黑客攻击。我们构建了一个环境,使得真正的奖励在过程层面完全可计算:覆盖124,416个上下文的可执行授权策略,允许一个预言机列出每个决策的每一个最低充分理由。我们将这些策略汇编成精确的过程奖励,沿七个现实轴(仅结果、充分、仅召回、提取噪声、粗分类法、陈旧策略、LLM裁判)降低验证者,并针对每个轴进行训练,而精确验证者则对每次部署进行评分。出现了四种表型。乐观验证器会被填充攻击:充分代理最终在 0.940,而真实忠实度在 96 集被保留的集数中从 0.731 降至 0.667。仅为结果的验证器会饱和并停止教学,噪声稀释,而保持排名的差距并无害处。对验证器、其选择误差和间隙符号的无模型审计,以及短饱和试验,可以预测一次运行将产生哪种表型。在九种配置中,跨越两个框架、两个模型供应商、两种适配器大小、三种精度、两次初始化和两个种子,表型反复出现,充分损害在全部10个复制单元中均为正,超过2个标准误在7个单元格中,乐观缺口仍为较陡的分支。损害幅度不重复,报告了五次预注册的尝试,试图解释它时发生。我们发布了测试平台、训练工具,以及14个真正的公共策略(Cedar、OPA、Gatekeeper),并移植到了可执行的预言机中。
Mitigating Catastrophic Forgetting with Physics-Indexed Specialist Memory: the Evolutionary Neuroplastic Agent
利用物理索引的专业记忆缓解灾难性遗忘:进化神经可塑性因素
- Authors: Blas Eugenio Vicco Zarate, Miguel Hernán Vicco
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
- Arxiv link: https://arxiv.org/abs/2611.21598
- Pdf link: https://arxiv.org/pdf/2611.21598
- Abstract
Continual reinforcement learning requires adaptation to changing conditions while preserving performance under previously encountered dynamics. It was investigated whether specialists generated by evolutionary search can provide a reusable memory when combined with environment detection and failure-driven search. The Evolutionary Neuroplastic Agent (ENA) evolves small neural networks without gradients, archives selected policies by detected dynamic regime, and retrieves them through Bayesian policy reuse. The architecture was evaluated in non-stationary CartPole and HalfCheetah environments. In the 30-replicate CartPole ablation, the complete architecture achieved a mean success rate of 83.2% across the five test environments. Among variants that learned the harder CartPole environments, the complete architecture showed the most consistent retention. A selection ablation also showed that the retrieval rule affected performance. In the evaluated sequences, ENA adapted more slowly than PPO on CartPole and achieved lower returns than SAC on HalfCheetah. The evidence concerns the complete architecture and does not isolate gradient-free optimization as the cause of retention or establish superiority over other continual-learning methods.
- 中文摘要
持续强化学习需要适应变化的环境,同时保持先前遇到的动态环境下的性能。研究了进化搜索生成的专家在结合环境检测和故障驱动搜索时,是否能提供可重复使用的内存。进化神经可塑性代理(ENA)演化无梯度的小型神经网络,按检测到的动态区域存档选定策略,并通过贝叶斯策略重用进行检索。该架构在非固定的 CartPole 和 HalfCheetah 环境中进行了评估。在 30 个重复的 CartPole 消融中,完整架构在五个测试环境中的平均成功率为 83.2%。在学习更难 CartPole 环境的变体中,完整架构的保留率最为一致。选择消融还显示检索规则影响性能。在评估序列中,ENA 在 CartPole 上的适应速度比 PPO 慢,且在 HalfCheetah 上获得的回报低于 SAC。证据主要关注完整架构,并未将无梯度优化单独定位为保留率的原因,也未确立其优于其他连续学习方法的优势。
ViRA: Toward Ultra-High-Resolution Visual Agents via Cross-Input Relative Policy Optimization
ViRA:通过交叉输入相对策略优化实现超高分辨率视觉代理
- Authors: Huawei Lin, Jinting Liu, Tongyan Wang, Xuyang Chen, Ruihua Shi, Zhichao Xu, Jiaxin Qin, Guoshenghui Zhao, Baijian Yang, Yingjie Victor Chen, Tan Yu, Weijie Zhao
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.21605
- Pdf link: https://arxiv.org/pdf/2611.21605
- Abstract
Visual agents answer questions about ultra-high-resolution photos by zooming into the relevant regions. However, existing agents often answer from common sense instead of reading the evidence. Reinforcement learning (RL) on the final answer cannot correct this, because a guess earns the same reward as reading. We propose Cross-Input Relative Policy Optimization (CrossRPO), which trains agents on contrast sets. A contrast set asks one question about several versions of an image that differ only in a small region with the evidence and have different answers, so only reading the evidence solves the whole set. CrossRPO rewards solving the whole set, and we prove that it does not reinforce a set answered from common sense. We build contrast sets for training (ViRA-Train) and evaluation (ViRA-Bench) by editing one small object in images of 21 to 120 megapixels, and we photograph further sets with real changes (ViRA-Bench-Real) to exclude editing traces. We train two Visual Reasoning Agents, ViRA-9B and ViRA-27B, with CrossRPO on only about 3,500 items of ViRA-Train each. CrossRPO achieves 63.3% set accuracy on ViRA-Bench at the maximum resolution, against 59.3% for GRPO. ViRA-9B is the most accurate open model of its size on all benchmarks, and ViRA-27B outperforms Claude Haiku 5.5 and GPT-6-Luna, both at their highest thinking level, on most benchmarks.
- 中文摘要
视觉智能体通过放大相关区域来回答有关超高分辨率照片的问题。然而,现有智能体往往基于常识回答,而非阅读证据。对最终答案进行强化学习(RL)无法纠正这一点,因为猜测与阅读获得相同的奖励。我们提出了交叉输入相对策略优化(CrossRPO),它通过对比集进行训练。对比集会就一个图像的多个版本提出一个问题,这些版本与证据仅在一个小区域内不同,且答案不同,因此仅通过阅读证据即可解决整个数据集。CrossRPO 奖励解决整集,我们证明它并不会强化基于常识答案的集合。我们通过编辑一个小物体在2100万像素到120万像素的图像中进行编辑,构建训练(ViRA-Train)和评估(ViRA-Bench)的对比度集,并拍摄带有实际变化(ViRA-Bench-Real)以排除编辑痕迹的后续集。我们训练了两个视觉推理代理ViRA-9B和ViRA-27B,每个在大约3,500个ViRA-Train项目上使用CrossRPO。CrossRPO在ViRA-Bench最高分辨率下实现了63.3%的集准率,而GRPO为59.3%。ViRA-9B是同尺寸中所有基准测试中最精确的开放模型,ViRA-27B在大多数基准测试中都优于Claude Haiku 5.5和GPT-6-Luna(两者在最高思维水平下)。
Keyword: diffusion policy
Behavioral Denoising: A New Perspective on Nonlinear Behavioral Control Inspired by Closed-Form Diffusion Policies
行为去噪:受封闭式扩散策略启发的非线性行为控制新视角
- Authors: Lucas Y. Jia, John Z. Zhang, Zachary Manchester
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.19712
- Pdf link: https://arxiv.org/pdf/2611.19712
- Abstract
Diffusion policies have become a dominant approach to imitation learning in robotics. Recent work suggests that, in the low-data regime, they largely retrieve or combine the demonstrated actions, which the closed-form diffusion policy does explicitly. This closely resembles behavioral control, where future inputs are formed as linear combinations of recorded trajectories. In this work, we make this connection precise: we show that the weights with which the closed-form diffusion policy combines the demonstrated actions are the unique solution of a regularized behavioral control problem. Building on this connection, we propose an algorithm, termed $\chi^2$-projected behavioral denoising, which is a training-free method that combines nearby trajectory data with a limit on extrapolation when the data offer little support. It shows encouraging results on four nonlinear control tasks compared to recent nonlinear behavioral control methods. We hope that this connection offers a useful perspective for understanding both data-driven control and diffusion policies. Code and more visualizations can be found at this https URL.
- 中文摘要
扩散策略已成为机器人学中模仿学习的主流方法。最新研究表明,在低数据环境中,扩散策略主要检索或组合已展示的动作,而封闭形式扩散策略则明确完成了这一点。这与行为控制非常相似,在行为控制中,未来输入是由记录轨迹的线性组合形成的。在本研究中,我们明确了这一联系:我们展示了闭式扩散策略将已展示动作组合的权重,是规范化行为控制问题的唯一解决方案。基于这一联系,我们提出了一种算法,称为 $\chi^2$-投影行为去噪,这是一种无需训练的方法,将附近轨迹数据结合,并在数据支持有限时限制外推。与近期非线性行为控制方法相比,它在四项非线性控制任务上显示出令人鼓舞的结果。我们希望这一联系能为理解数据驱动的控制和扩散策略提供有用的视角。代码及更多可视化内容可在此 https 网址找到。
Conditional Optimal Transport and Triangular Maps for Imitation Learning
用于模仿学习的条件最优传输和三角映射
- Authors: Mohammad Al-Jarrah, Yuhang Mei, Karen Leung, Bamdad Hosseini, Amirhossein Taghvaei
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.19815
- Pdf link: https://arxiv.org/pdf/2611.19815
- Abstract
We introduce a unified formulation of imitation learning from demonstrations as a conditional sampling problem solved by triangular transport: a policy is the second block of a triangular map that transports a reference distribution to the conditional distribution of the action chunk given the observation history. Unlike in Diffusion Policy, where it is a fixed Gaussian, the reference becomes a design variable; we take it to be the empirical distribution of the demonstrated action chunks and introduce three instances of the formulation: an optimal transport policy (OTP), a flow matching policy (FMP), and a stochastic interpolant policy (SIP). On the Push-T and robomimic Square tasks, the three policies are competitive in task performance with Diffusion Policy and Consistency Policy while using considerably smaller networks and shorter training, and the informative reference accounts for a substantial part of their performance when few network evaluations are used. The three instances offer different trade-offs rather than a single best method: OTP generates an action chunk with a single evaluation of the smallest network (0.23 million parameters on Push-T) and is suitable for settings in which inference time and model size matter, whereas SIP attains the highest scores among the proposed policies at a higher cost.
- 中文摘要
我们引入了一种统一的表述,将从演示中模拟学习作为一个由三角形传输解决的条件抽样问题:策略是三角形映射的第二个块,将参考分布传递到给定观测历史的动作块的条件分布。与扩散策略中为固定高斯分布不同,参考成为设计变量;我们将其视为已展示动作块的经验分布,并引入了三种表述实例:最优传输策略(OTP)、流匹配策略(FMP)和随机插值策略(SIP)。在 Push-T 和机器人模拟方形任务中,这三种策略在任务性能上与扩散策略和一致性策略具有竞争力,同时使用更小的网络和更短的训练时间,且在使用较少网络评估时,参考参考解释了其性能的相当一部分。这三种实例提供不同的权衡,而非单一最佳方法:OTP 通过对最小网络(Push-T 上的 23 万参数)进行单次评估生成一个动作块,适用于推理时间和模型规模重要的环境,而 SIP 在所有拟议策略中得分最高且成本更高。