生成时间: 2026-07-07 19:02:28 (UTC+8); Arxiv 发布时间: 2026-07-07 20:00 EDT (2026-07-08 08:00 UTC+8)

今天共有 77 篇相关文章

Keyword: reinforcement learning

A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges

大语言模型驱动渗透测试概览：分类学、共演化与开放挑战

• Authors: Zheyuan He, Jiaxun Dong, Zihao Li, Ting Chen, Gelei Deng, Feng Luo, Jinkun Ji, Yuanlong Cao, Xiapu Luo
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.02605
• Pdf link: https://arxiv.org/pdf/2607.02605
• Abstract Agents4Pentest, an emerging class of LLM-based autonomous penetration testing systems, has become a rapidly growing area in security research. Despite this growth, the field still lacks a unified taxonomy, a systematic understanding of how agent architectures and evaluation benchmarks have co-evolved, and a clear characterization of remaining capability and reliability gaps. This survey addresses these gaps through a systematic analysis of 81 papers between 2023 and 2026. We organize the literature into six categories: evaluation benchmarks, general-purpose systems, domain-specific frameworks, CTF-based systems, defense-oriented research, and surveys. We further trace a four-phase architectural evolution from text-only reasoning agents to agents trained with Reinforcement Learning with Verifiable Rewards (RLVR), showing that each transition is driven by a distinct capability bottleneck. Our analysis yields several key findings. First, RLVR marks a shift in capability acquisition from imitation of expert demonstrations to reward-driven self-improvement, enabling agents to discover previously undocumented attack strategies. Second, CTF platforms have evolved from evaluation testbeds into dual-purpose infrastructure for both agent evaluation and RL training. Third, domain-specific frameworks improve efficiency through recurring specialization mechanisms, but their gains remain largely confined to narrow task classes and are difficult to compare across domains because existing evaluations rely on different benchmarks. Fourth, the field is expanding beyond offensive automation toward adversarial defense and security compliance. Across these categories, we identify three structurally linked open challenges: evaluation reliability, limited performance on multi-stage attack scenarios, and scarcity of high-quality training data.
• 中文摘要 Agents4Pentest 是一类新兴的基于大型语言模型的自主渗透测试系统，已成为安全研究中快速发展的领域。尽管如此，该领域仍缺乏统一的分类法、对代理架构与评估基准如何共同演化的系统理解，以及对剩余能力和可靠性差距的清晰刻画。本调查通过对2023年至2026年间81篇论文的系统分析，解决了这些空白。我们将文献分为六类：评估基准、通用系统、领域特定框架、基于夺旗的系统、国防导向研究和调查。我们还进一步追溯了从纯文本推理智能体向接受可验证奖励强化学习（RLVR）训练的智能体的四阶段架构演进，表明每一次转换都由不同的能力瓶颈驱动。我们的分析得出了几个关键发现。首先，RLVR标志着能力获取从模仿专家演示转向以奖励为驱动的自我提升，使智能体能够发现此前未被记录的攻击策略。其次，CTF平台已从评估测试平台演变成了代理评估和强化学习训练的双重基础设施。第三，领域特定框架通过周期性专业化机制提升效率，但其收益主要局限于狭窄的任务类别，且难以跨领域比较，因为现有评估依赖于不同的基准测试。第四，该领域正从进攻自动化扩展到对抗性防御和安全合规领域。在这些类别中，我们识别出三个结构性相关的开放挑战：评估可靠性、多阶段攻击场景表现有限以及高质量训练数据的稀缺性。

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

暗中问答：部分可观察性下的不确定门控大型语言模型协助

• Authors: Juarez Monteiro, Nathan Gavenski, Guilherme Lima, Francisco Galuppo, Odinaldo Rodrigues, Adriano Veloso
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.02686
• Pdf link: https://arxiv.org/pdf/2607.02686
• Abstract Reinforcement learning agents operating under partial observability must act on incomplete information, making them natural candidates for guidance from small language models (SLMs) that carry broad reasoning priors. Yet integrating SLM guidance into this setting has proven difficult: across all test environments, vanilla uncertainty-gated approaches achieve an overwrite rate at or near zero, meaning the SLM almost never contributes an independent action. We trace this failure to the bare egocentric prompt, which provides insufficient context for genuine reasoning, and identify it as a context problem rather than a capacity problem. We propose ASK+, which supplies the SLM with trajectory-aware context (a partially revealed map, visited positions, and action history) and structured chain-of-thought reasoning, converting it from a passive redundancy check into a more informative consultant that occasionally corrects the policy. We further establish that the predictive entropy signal used for selective querying measures action uncertainty rather than state uncertainty and remains informative in POMDPs, making uncertainty-gated assistance viable beyond fully observable settings. The stateful prompt drives substantial gains: on DoorKey, where vanilla ASK matches PPO (both 89%), ASK+ reaches 93% success; on FourRooms, success climbs from 53% to 70%; on HigherLower, accuracy reaches 73.7%, matching the SLM-only upper bound. Across all environments, Qwen3.5-2B matches or exceeds Qwen3.5-4B, confirming that prompt design and selective gating dominate the impact of model scale, enabling guidance without large models.
• 中文摘要 在部分可观察性下运行的强化学习代理必须对不完整信息采取行动，因此它们自然成为带有广泛推理先验的小型语言模型（SLMs）指导的候选者。然而，将SLM指导整合到这一环境中却很困难：在所有测试环境中，原版的不确定门控方法覆盖率接近零，这意味着SLM几乎从不贡献独立动作。我们将这种失败归因于裸露的自我中心提示，该提示词缺乏足够的背景支持真实推理，并将其视为语境问题而非容量问题。我们提出ASK+，它为SLM提供轨迹感知上下文（部分揭示的地图、访问位置和行动历史）和结构化的思维链推理，将其从被动冗余检查转变为更具信息量的顾问，偶尔纠正策略。我们进一步确定，用于选择性查询的预测熵信号测量的是动作不确定性而非状态不确定性，并且在POMDP中依然具有信息量，使得不确定性门控辅助在完全可观测环境之外依然可行。有状态提示带来了显著的提升：在DoorKey上，原版ASK匹配PPO（均为89%），ASK+成功率达93%;在FourRooms，成功率从53%上升到70%;在HigherLower中，准确率达到73.7%，与仅SLM的上限相符。在所有环境中，Qwen3.5-2B与Qwen3.5-4B相当甚至超过，证实提示设计和选择性门控主导模型规模的影响，使得无需大规模模型即可获得指导。

Reinforcement Learning for Data-Efficient Code-Switched ASR

数据高效代码切换ASR的强化学习

• Authors: Ziwei Ye, Peter Vickers
• Subjects: Subjects: Computation and Language (cs.CL); Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2607.02757
• Pdf link: https://arxiv.org/pdf/2607.02757
• Abstract Audio-language models can be prompted for code-switched speech, but their decoding is not optimized for code-switching and often fails at language boundaries. We propose a practical reinforcement learning with verifiable rewards recipe for data-efficient adaptation of audio-language models to code-switched ASR using group relative policy optimization, combining an error rate reward with a script fidelity reward that penalizes wrong writing systems and a two-pass draft-and-refinement procedure. Using Qwen2-Audio as a reproducible testbed across 10 language pairs, training on only TTS code-switched speech, we show that RLVR with 10% of the data matches LoRA supervised fine-tuning trained on the full dataset, with the largest gains on typologically distant pairs. The error rate reward eliminates translation errors while the script fidelity reward separately reduces script contamination without degradation. These gains transfer zero-shot to a human-recorded code-switching corpus.
• 中文摘要 音频语言模型可以提示进行代码切换语音，但其解码未针对代码切换进行优化，且常在语言边界处失败。我们提出了一种实用的强化学习，采用可验证的奖励配方，利用组相对策略优化，将音频语言模型数据高效适应到代码切换ASR，结合错误率奖励与惩罚错误书写系统的脚本忠实度奖励，以及两次草稿与精炼过程。利用Qwen2-Audio作为可重复测试平台，覆盖10对语言，仅在TTS语码切换语音上训练，我们证明RLVR在10%的数据中与LoRA监督的微调训练完全匹配，且在类型学上相距较远的对上获得最大收益。错误率奖励消除翻译错误，而脚本忠实度奖励则单独减少脚本污染且不破坏数据。这些增益将零射击数据转移到人工录制的代码转换语料库中。

CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

CLEAR：面向端到端自动驾驶的大规模闭环强化学习

• Authors: Yunxiao Shi, Hong Cai, Mohammad Ghavamzadeh, Fatih Porikli
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.02841
• Pdf link: https://arxiv.org/pdf/2607.02841
• Abstract End-to-end autonomous driving (E2E-AD) aims to directly map raw sensor information to driving actions. Recently, with the rapid advancement of multi-modal large language models (MLLMs), researchers have proposed the paradigm of Vision-Language-Action (VLA) models for E2E-AD, where it seeks to integrate visual perception, language understanding and action prediction within a single policy. However, existing VLA-based policies largely adopts imitation learning, where it only learns to drive by optimizing distance-based metrics w.r.t. logged expert trajectories. Such distribution shift between open-loop training and closed-loop inference leads to suboptimal performance in closed-loop planning. To close this gap, we present CLEAR, a system that enables closed-loop training using Reinforcement Learning (RL) at scale for E2E-AD. We propose to learn a novel residual waypoint policy around the waypoint prior from pretrained VLA policies, effectively harnessing the knowledge within. On another front, one of the key challenges to scale up RL for vision-based policies is the number of parallel simulation environments since RL is data hungry. To that end, we design a heterogeneous pipeline that places the simulator and the VLA learner on distinct compute groups, which allows us to dramatically increase the number of simulation environments running in parallel while avoiding resource contention and maintaining training stability. We show that with a simple reward, CLEAR significantly outperforms previous methods and sets new state-of-the-art performance on the challenging benchmarks of CARLA longest6 v2 and Bench2Drive.
• 中文摘要 端到端自动驾驶（E2E-AD）旨在直接将原始传感器信息映射到驾驶行为上。近年来，随着多模态大型语言模型（MLLM）的快速发展，研究人员提出了面向E2E-AD的视觉-语言-行动（VLA）模型范式，旨在将视觉感知、语言理解和动作预测整合到单一策略中。然而，现有基于VLA的策略大多采用模仿学习，仅通过优化基于距离的指标来学习驾驶，这些指标已记录的专家轨迹。开环训练与闭环推断之间的分布偏移导致闭环规划表现不佳。为弥补这一差距，我们介绍了CLEAR系统，该系统能够大规模实现基于强化学习（RL）的E2E-AD闭环训练。我们计划从预训练的VLA策略中学习一种围绕前置路径点的新型残留航点策略，有效利用其中的知识。另一方面，扩展基于愿景的策略的强化学习（RL）面临的一个关键挑战是并行仿真环境的数量，因为强化学习对数据的需求很大。为此，我们设计了一个异构流水线，将模拟器和VLA学习器放置在不同的计算组上，这使我们能够显著增加并行运行的模拟环境数量，同时避免资源争用并保持训练稳定性。我们证明，通过简单的奖励，CLEAR显著优于以往方法，并在CARLA longest6 v2和Bench2Drive这两个高难度基准上创造了新的最先进性能。

Trading Confidence: Comprehensive Uncertainty Estimation in Algorithmic Trading

交易信心：算法交易中的全面不确定性估计

• Authors: Lin Li, Li Rong Wang, Hsuan Fu, Xiuyi Fan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.02864
• Pdf link: https://arxiv.org/pdf/2607.02864
• Abstract Reinforcement Learning (RL) has emerged as a powerful approach in financial trading, enabling agents to learn optimal strategies through direct market interaction. However, financial markets are highly uncertain, with price fluctuations driven by stochastic volatility, model limitations, and regime shifts. Traditional RL models struggle in dynamic environments, often failing to adapt to sudden market disruptions, leading to suboptimal trading decisions. To address this challenge, we propose an uncertainty-aware RL framework that integrates distributional, epistemic, and aleatoric uncertainty estimations. Our approach enhances uncertainty estimation using SHAP-weighted reconstruction uncertainty, MC Dropout, and an LSTM-based technical indicator consensus mechanism. Experimental results on five major U.S. stock indices demonstrate that RL agents equipped with uncertainty estimation significantly outperform traditional models in return and risk management. This study advances uncertainty estimation in RL-based financial trading, with future research extending its application to other asset classes and alternative RL architectures for greater adaptability.
• 中文摘要 强化学习（RL）已成为金融交易中的一种强大方法，使代理能够通过直接的市场互动学习最优策略。然而，金融市场高度不确定，价格波动受随机波动、模型局限性和格局转移驱动。传统的强化学习模型在动态环境中表现不佳，常常无法适应突发的市场变动，导致交易决策不够理想。为应对这一挑战，我们提出了一个不确定性感知强化学习框架，整合了分布、认识论和偶然不确定性估计。我们的方法通过SHAP加权重建不确定性、MC退出以及基于LSTM的技术指标共识机制，增强了不确定性估计。五个主要美国股票指数的实验结果表明，配备不确定性估计的强化学习代理在回报和风险管理方面显著优于传统模型。本研究推动了基于强化学习的金融交易中的不确定性估计，未来研究将将其应用于其他资产类别和替代强化学习架构，以提升适应性。

Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models

RLVR中的奖励粒度：比较小语言模型中数学推理的过程与结果奖励结构

• Authors: Anagha Radhakrishna Palandye, Rebecca Glick, Osheen Kaul
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.02869
• Pdf link: https://arxiv.org/pdf/2607.02869
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving mathematical reasoning in language models. Yet most RLVR work rewards only the final answer (outcome-based rewards), leaving the impact of step-level process supervision (process rewards) underexplored especially for small models that lack the capacity to self-correct under sparse feedback. We systematically compare five reward conditions applied to Qwen2.5-0.5B fine-tuned with Group Relative Policy Optimization (GRPO) on GSM8K: a no-RL baseline, process-only, outcome-only, and three hybrid weightings ($\lambda \in {0.9, 0.5, 0.1}$ process weight). Process-only supervision achieves 63.73% test accuracy versus 53.75% for outcome-only, a nearly 10-percentage point gap while yielding reasoning traces with higher step validity and lower deviation from ground-truth chain length. Hybrid rewards generally correlate positively with process weight, with one notable anomaly: the low-process / high-outcome configuration ($\lambda=0.1$) underperforms pure outcome supervision, suggesting conflicting optimization signals. Error analysis using GPT-4o as a judge reveals distinct failure mode distributions: process models generate structurally inconsistent but arithmetically grounded traces, while outcome models produce concise but derivation-error-prone chains. Our results demonstrate that reward granularity is a first-order design decision for RLVR, with process-level supervision substantially improving both accuracy and trace fidelity in small language models.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为提升语言模型数学推理的有前景范式。然而，大多数RLVR工作只奖励最终答案（基于结果的奖励），导致对缺乏自我纠正能力的小型模型，步骤级过程监督（过程奖励）的影响被忽视。我们系统地比较了在GSM8K上用组相对策略优化（GRPO）微调的五个Qwen2.5-0.5B奖励条件：无强化基础、仅过程、仅结果和三种混合权重（$\lambda \in {0.9， 0.5， 0.1}$ 过程权重）。仅过程监督的测试准确率为63.73%，仅结果检测为53.75%，差距近10个百分点，同时产生更高的步骤效度和更小偏离真实链长度的推理迹迹。混合奖励通常与进程权重呈正相关，但有一个显著的异常：低过程/高结果配置（$\lambda=0.1$）在纯结果监督下表现不足，表明优化信号存在冲突。以GPT-4o为评判的错误分析揭示了不同的失败模式分布：过程模型生成结构不一致但算术基础的痕迹，而结果模型则生成简洁但易产生推导错误的链条。我们的结果表明，奖励粒度是RLVR的一阶设计决策，过程级监督显著提升了小语言模型中的准确性和痕迹准确性。

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

牡蛎-II：大型语言模型中建设性安全对齐的强化学习

• Authors: Jiyang Guan, Yong Xie, Jun Chen, Jiexi Liu, Zipeng Ye, Defeng Li, Jiayu Shen, Jialing Tao, Hui Xue
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.02914
• Pdf link: https://arxiv.org/pdf/2607.02914
• Abstract Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning this http URL across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.
• 中文摘要 大型语言模型（LLMs）在多种应用中展现出卓越的能力，但确保其同时的安全性、实用性和可信度仍是一个持续的挑战。传统的拒绝导向对齐策略减少有害内容生成，但系统性地未能满足合法用户需求，常常隐瞒那些能够安全且建设性地回应敏感查询意图的信息。基于Oyster-I开创的建设性安全范式，该范式超越了一刀切拒绝，转向深思熟虑、以响应为导向的安全对齐，我们指出其基于监督微调（SFT）方案的两个关键局限：对分布外场景的安全泛化不足，以及我们称之为安全链思维（CoT）过度泛化的现象，即安全导向的推理模式过度应用于无害查询， 降低了帮助性和用户体验。为解决这些局限性，我们提出了Oyster-II，一种基于强化学习（RL）的建设性安全对齐框架，结合零强化学习范式和多阶段强化学习（http URL），在广泛的基准测试中，Oyster-II在安全维度上全面超越Qwen3-14B及其前身Oyster-I，实现跨尺度性能可与Qwen3-Max和Qwen3.5-397B相媲美。

VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learning

视频搜索器：通过强化学习赋能视频深度研究，实现多工具代理推理

• Authors: Zhenkun Gao, Yicheng Bao, Jinlong Peng, Xueheng Li, Theo Huang, Bangwei Liu, Kunquan Li, Zhenye Gan, Tao Hu, Chengjun Xie, Mingqian Yang, Xuanhua He, Zhizhong Zhang, Xin Tan, Chengjie Wang, Yuan Xie
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.02927
• Pdf link: https://arxiv.org/pdf/2607.02927
• Abstract Video understanding is moving beyond closed-context perception toward open-world evidence exploration, a paradigm formalized as Video Deep Research (VDR). However, existing multimodal search agents primarily target static images, and the current VDR benchmark relies on text-centric retrieval that discards crucial visual information. To address these limitations, we propose VideoSearcher, a closed-loop agentic framework that empowers Vision-Language Models with multi-tool reasoning for VDR. VideoSearcher unifies temporal localization, spatial focusing, and multimodal search within a single reasoning trajectory, enabling agents to progressively ground visual clues, retrieve relevant evidence, and synthesize answers. To optimize knowledge-intensive reasoning trajectories, we propose Bi-branch Sequence Policy Optimization (BiSPO), a reinforcement learning algorithm that decouples tool-invocation optimization from answer-accuracy optimization. This design provides stable learning signals for both evidence-grounded reasoning and purposeful tool use. Furthermore, we construct VideoSearch-QA, the first benchmark designed to evaluate open-world video information grounding and multimodal search-based reasoning. Extensive experiments demonstrate that VideoSearcher significantly outperforms prior open-source agentic baselines across various search-oriented and multimodal understanding benchmarks.
• 中文摘要 视频理解正从封闭上下文感知转向开放世界证据探索，这一范式被正式化为视频深度研究（Video Deep Research，简称VDR）。然而，现有的多模态搜索代理主要针对静态图像，而当前的VDR基准测试依赖于以文本为中心的检索，丢弃了关键的视觉信息。为解决这些局限性，我们提出了VideoSearcher，一个闭环智能体框架，赋能视觉语言模型，支持多工具推理VDR。VideoSearcher 将时间定位、空间聚焦和多模态搜索统一在单一推理轨迹中，使智能体能够逐步扎根视觉线索，检索相关证据，并综合答案。为了优化知识密集型推理轨迹，我们提出了双分支序列策略优化（BiSPO），这是一种强化学习算法，将工具调用优化与答案准确性优化解耦。该设计为基于证据的推理和有目的的工具使用提供了稳定的学习信号。此外，我们构建了VideoSearch-QA，这是首个旨在评估开放世界视频信息基础和多模态搜索推理的基准测试。大量实验表明，VideoSearcher 在多个面向搜索和多模态理解的基准测试中显著优于以往的开源代理基准。

A Sliding-Window-Based Reinforcement Learning for Dynamic Assembly Flow Shop Scheduling with Multi-Product Delivery

基于滑动窗口的强化学习，用于多产品交付的动态装配流程车间排程

• Authors: Junhao Qiu, Jianjun Liu, Ting Liu, Rongjie Liao, Zhantao Li, Qingfu Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.02941
• Pdf link: https://arxiv.org/pdf/2607.02941
• Abstract Multi-product kitting delivery imposes significant challenges for real-time scheduling in hybrid manufacturing systems that integrate processing and assembly, as dynamic order arrivals simultaneously alter supply dependencies and the set of feasible job-machine assignments. This paper proposes a sliding-window-based reinforcement learning (SWRL) framework for end-to-end online scheduling in the flexible assembly flow shop scheduling problem with complex kitting constraints. The problem is formulated as a heterogeneous graph-based Markov decision process that captures the dual-layer kitting structure and the tail-product bottleneck dynamics that produce a sparse reward landscape. To address the resulting challenges, SWRL integrates a sliding-window filtering mechanism that filters inactive nodes and prioritizes kitting-critical operations, a spatiotemporal graph encoding network that tracks bottleneck shifts across consecutive decision states, and a dynamic action mapping module with a constrained waiting strategy that adapts to the changing action space under variable topologies. Experiments on real-world instances from a home appliance manufacturer demonstrate that SWRL achieves consistent tardiness reductions over classical dispatching rules and existing deep reinforcement learning methods, and exhibits robust performance across varying resource configurations, order loads, and arrival concentrations.
• 中文摘要 多产品套装交付在集成加工与组装的混合制造系统中，实时调度面临重大挑战，因为动态订单到达同时改变供应依赖关系和可行的作业-机器分配组合。本文提出了一种基于滑动窗口的强化学习（SWRL）框架，用于在灵活装配流车间调度问题中实现端到端在线调度，且具有复杂套件约束。该问题被表述为一种异构的基于图的马尔可夫决策过程，捕捉了双层组合结构和尾积瓶颈动态，从而产生了稀疏的奖励景观。为应对由此产生的挑战，SWRL集成了滑动窗口过滤机制，用于过滤非活跃节点并优先处理关键操作;一个时空图编码网络，追踪连续决策状态间瓶颈变化;以及动态动作映射模块，具备受限等待策略，适应可变拓扑下变化的动作空间。家用电器制造商的实际实验表明，SWRL在传统调度规则和现有深度强化学习方法上实现了持续的延迟减少，并在不同资源配置、订单负载和到达集中条件下表现出稳健的性能。

Incentivizing Vision Language Models to Search for Long Video Question Answering

激励视觉语言模型搜索长视频问答

• Authors: Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.02959
• Pdf link: https://arxiv.org/pdf/2607.02959
• Abstract We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process. VSeek utilizes a natural language-driven search to identify relevant context within long videos and is post-trained with reinforcement learning (RL) to jointly formulate targeted search queries and reason over retrieved clips for LVQA. While RL post-training has revolutionized reasoning in symbolic domains such as mathematics and code, its application to long-video understanding remains hindered by a lack of verified rewards. To ensure that the retrieved context is relevant, we propose a novel neuro-symbolic approach that bridges open-ended natural language with discrete visual verification. Specifically, complex user queries are compiled into formal temporal logic specifications for systematically decomposing natural language questions into a definitive checklist of required atomic visual primitives, such as key objects and activities, along with their temporal ordering. These systematically derived grounding events provide the critical feedback signal for RL post-training, enabling dense, verifiable rewards based on the successful retrieval of these specific visual elements rather than relying entirely on outcome-only answer accuracy. By explicitly optimizing for this verifiable evidence-seeking behavior, VSeek improves Pass@1 scores by up to 8% and Pass@4 scores by 15% on long-video understanding benchmarks compared to base models. We open-source our code at this https URL.
• 中文摘要 我们介绍了VSeek，一种能动框架，将长视频问答（LVQA）从被动的单次感知任务转变为多回合检索过程。VSeek利用自然语言驱动的搜索在长视频中识别相关上下文，并通过强化学习（RL）进行后期训练，共同制定有针对性的搜索查询，并对检索到的片段进行LVQA推理。虽然强化学习后训练在数学和代码等符号领域革新了推理，但其在长视频理解中的应用仍受限于缺乏经过验证的奖励。为确保检索的上下文相关性，我们提出了一种新颖的神经符号方法，将开放式自然语言与离散的视觉验证相结合。具体来说，复杂的用户查询被汇编成形式时序逻辑规范，系统地将自然语言问题分解为必需原子视觉原语（如关键对象和活动）及其时间顺序的明确清单。这些系统性推导的接地事件为强化学习后训练提供了关键反馈信号，使得基于成功检索这些特定视觉元素而产生的密集且可验证的奖励，而非完全依赖结果准确性。通过明确优化这种可验证的证据寻求行为，VSeek在长视频理解基准测试中Pass@1分数提升了最多8%，Pass@4分数提升了15%，相比基础模型。我们将代码开源于这个 https URL。

Reinforcement Learning for Evidence-Seeking Diagnostic Reasoning with Large Language Models

基于大型语言模型进行证据寻求诊断推理的强化学习

• Authors: Shengyi Hua, Kangzhe Hu, Conghui He, Xiaofan Zhang, Shaoting Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.02983
• Pdf link: https://arxiv.org/pdf/2607.02983
• Abstract Recent reasoning-centric Large Language Models (LLMs) have made significant strides, yet they predominantly operate on a passive-inference pattern that assumes complete information. In contrast, real-world clinical intelligence is inherently an iterative investigative process requiring strategic evidence acquisition. To bridge this gap, we formalize medical diagnosis as an Iterative Evidence-Seeking Task. We leverage Reinforcement Learning with Verifiable Rewards (RLVR) to elicit intrinsic reasoning within a closed-loop environment, guided by a novel suite of rewards that enforce diagnostic precision and examination consistency. To facilitate this, we introduce the Retrieval-Augmented Generation-based Examination Simulator (RAGES), a high-fidelity clinical oracle that provides realistic, knowledge-grounded follow-up evidence. Empirical results across diverse datasets demonstrate that our framework enables LLMs to transition from passive responders to autonomous assistants. Notably, our model demonstrates comparable performance to larger and reasoning-enhanced baselines, while RAGES proves superior to vanilla LLMs in generating biologically plausible clinical feedback.
• 中文摘要 近年来以推理为中心的大型语言模型（LLMs）取得了显著进展，但它们主要采用被动推理模式，假设信息完整。相比之下，现实世界的临床情报本质上是一个需要战略性证据获取的迭代调查过程。为了弥合这一差距，我们将医学诊断正式化为一个迭代证据寻求任务。我们利用可验证奖励强化学习（RLVR）在闭环环境中激发内在推理，并由一套新颖的奖励引导，强化诊断的准确性和检查一致性。为此，我们引入了检索增强生成检测模拟器（RAGES），这是一种高精度的临床预言机，能够提供真实、基于知识的后续证据。跨越多种数据集的实证结果表明，我们的框架使大型语言模型能够从被动响应者转变为自主助手。值得注意的是，我们的模型在生成生物学上合理的临床反馈方面表现优于普通大型语言模型。

Human-Centric Reflective Architecture for Human-AI Collaborative Decision-Making

以人为本的反思架构，用于人机协作决策

• Authors: Andreas Kouridakis, Dimitrios Patiniotis Spyropoulos, George Vouros
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.03025
• Pdf link: https://arxiv.org/pdf/2607.03025
• Abstract The use of Large Language Models (LLMs) across diverse areas of human activity-ranging from everyday tasks to safety-critical applications-aims to enhance decision-making effectiveness with minimal human feedback. Concurrently, it seeks to align decisions with human expectations, preferences, and needs while mitigating risks associated with AI non-determinism. However, humans frequently over- or under-rely on AI recommendations, and current AI systems remain poorly calibrated to human expectations. To address these challenges, we introduce a human-AI collaborative decision-making framework designed to augment human capabilities and align AI agents with human preferences and expectations. Specifically, this paper (a) formulates the collaborative decision-making task as a stochastic game between an AI agent and a human player, and (b) proposes the Human-Centric Reflective Architecture (HCRA), which integrates human-calibrated models with reinforcement learning agents that leverage linguistic feedback in an iterative, reflective process. Evaluation results demonstrate that HCRA enhances decision-making effectiveness and delivers high-quality recommendations.
• 中文摘要 大型语言模型（LLMs）在人类活动的多个领域——从日常任务到安全关键应用——旨在以最小的人类反馈提升决策效率。同时，它旨在使决策与人类的期望、偏好和需求保持一致，同时降低人工智能非确定性带来的风险。然而，人类经常过度或不足依赖人工智能的建议，现有的人工智能系统仍然未能很好地适应人类的期望。为应对这些挑战，我们引入了人机协作决策框架，旨在增强人类能力，使AI代理符合人类偏好和期望。具体来说，本文（a）将协作决策任务表述为人工智能代理与人类玩家之间的随机博弈，（b）提出以人为中心的反思架构（HCRA），该架构将人类校准的模型与利用语言反馈的强化学习代理集成为迭代反思过程。评估结果表明，HCRA提升了决策效率并提供高质量的建议。

Spectral Rewiring for Exploration, Purification, and Model Merging

用于探索、纯化和模型合并的光谱重接

• Authors: Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu, Yuxuan Song, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03065
• Pdf link: https://arxiv.org/pdf/2607.03065
• Abstract Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected by premature saturation of test-time scaling, and interference when consolidating multiple capabilities through multi-domain training or model merging. We show that the reasoning-effective component of these updates is largely concentrated in the base model's spectral space, motivating Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains this spectral core while removing orthogonal components. SAR therefore preserves reasoning gains and filters residual update directions that suppress performance or amplify cross-domain interference. Across several model families and scales, SAR extracts compact reasoning cores using as little as approximately 0.58% of total parameters: it preserves over 99% of post-training performance and improves high-k exploration in mathematical reasoning, and generalizes to agentic coding by improving six of seven open benchmarks on an in-house model. SAR also purifies mixed-domain training updates by releasing suppressed coding capability while maintaining math reasoning and instruction following. It further enables model merging across experts, yielding cross-domain generalization that surpasses previous merging baselines and even the best single-domain experts. Overall, SAR shows that extracting reasoning-effective updates from parameter geometry can serve as a training-free mechanism to improve reasoning and multi-domain performance.
• 中文摘要 强化学习已成为大型语言模型训练后标准的方案，但密集的全参数更新会造成两个部署相关的瓶颈：推理性能受抑制，通常表现为测试时间扩展过早饱和;以及通过多域训练或模型合并整合多能力时的干扰。我们表明，这些更新中有效的推理成分主要集中在基础模型的谱空间，这促使了亚空间对齐重接线（SAR）的后续编辑方法，该方法保留了谱核，同时去除正交成分。因此，SAR保留了推理增益，并过滤了抑制性能或放大跨域干扰的残余更新方向。在多个模型家族和尺度中，SAR仅用约0.58%的总参数提取紧凑推理核心：它保留了99%以上的训练后性能，提升数学推理中的高k探索能力，并通过改进内部模型中的六个开放基准测试，推广到代理编码。SAR还通过释放抑制编码能力来净化混合域训练的更新，同时保持数学推理和指令跟踪。它进一步支持跨专家的模型合并，实现跨领域泛化，超越以往合并基线甚至顶尖单域专家。总体而言，SAR表明，从参数几何中提取有效的推理有效更新可以作为一种无需训练的机制，提升推理和多域性能。

Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System

在1-DoF直升机系统中整合物理知情神经网络以实现安全强化学习

• Authors: Georg Schäfer, Jakob Rehrl, Stefan Huber
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03125
• Pdf link: https://arxiv.org/pdf/2607.03125
• Abstract Deep reinforcement learning (DRL) offers powerful control for industrial cyber-physical systems (ICPSs), but its "black-box" exploration risks violating strict hardware safety limits. Typically, these constraints are managed through complex reward shaping. In this work-in-progress paper, we embed a differentiable physics model directly into the proximal policy optimization (PPO) actor loss function. By simulating short-horizon future trajectories during training, the policy is penalized for anticipated safety violations independent of the task-reward signal. Evaluated on a simulated 1-degree-of-freedom helicopter testbed with strict pitch constraints, our physics-informed soft regularizations substantially reduce constraint violations while maintaining reliable target tracking.
• 中文摘要 深度强化学习（DRL）为工业网络物理系统（ICPS）提供了强大的控制，但其“黑匣子”探索存在违反严格硬件安全限制的风险。通常，这些约束通过复杂的奖励塑造来管理。在这篇正在进行的论文中，我们直接将一个可微物理模型嵌入到近端策略优化（PPO）演员损失函数中。通过在训练期间模拟短期未来轨迹，该政策因预期的安全违规而受到惩罚，而这些违规与任务奖励信号无关。在模拟1自由度直升机测试平台上进行严格俯仰约束的评估，我们基于物理的软正则化显著减少了约束违规，同时保持了可靠的目标跟踪。

ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

ACPO：通过细粒度替代熵优化自适应信贷政策

• Authors: Zijun Xie, Yuyang You, Yongzhi Li, Enlei Gong, Zeyu Chen, Quan Chen, Yanhua Cheng, Peng Jiang, Yadong Mu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03126
• Pdf link: https://arxiv.org/pdf/2607.03126
• Abstract Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.
• 中文摘要 强化学习（RL）显著提升了大型语言模型（LLM）的推理能力，但稀疏的结果奖励仍使代币级学分分配变得困难。现有可扩展的强化学习方法通常在各代币间均匀分配轨迹级奖励，而近期的熵感知方法要么依赖粗熵分离启发式，要么直接优化真熵，这可能导致与抽样代币策略更新不匹配的非局部梯度分量。我们提出了自适应信贷政策优化（ACPO），这是一种基于模式-局部替代熵的代币级信用分配框架。ACPO通过强调成功推销中的不确定决策和失败推销中的过度自信的代币，非对称地调节政策更新。我们证明代理体允许确定性熵界限，并且在模态对齐和近端更新下，保持策略梯度向主导顺序。数学推理和编码基准测试（包括AIME 2025和HumanEvalPro）的实验显示，ACPO在强强强化学习基线如DAPO、GTPO和SAPO中持续提升。

Anticipatory Reinforcement Learning for Trajectory Tracking

轨迹追踪的预期强化学习

• Authors: Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03132
• Pdf link: https://arxiv.org/pdf/2607.03132
• Abstract Deep reinforcement learning (DRL) in industrial control often suffers from lag and overshoot due to purely reactive control based on the current tracking error. To achieve anticipatory control without high computational overhead, we introduce a predictive formulation that augments the DRL state space with target velocities and future reference horizons. Evaluating eight configurations using proximal policy optimization (PPO) on a 1-degree-of-freedom (1-DoF) helicopter testbed, simulation results showed a 9-fold error reduction, lowering the mean absolute deviation from 2.73° to 0.31°. However, zero-shot transfer to physical hardware revealed a sim-to-real gap. Interestingly, a simpler configuration using a single, further look-ahead horizon matched the real-world top performance of the most complex model (1.11°). Overall, evaluating various combinations of prediction horizons and target velocities demonstrated that highly granular predictive data is not necessarily required for physical transfer.
• 中文摘要 工业控制中的深度强化学习（DRL）常因纯反应式控制基于当前跟踪误差而出现滞后和超冲问题。为了实现在不高计算开销的预判控制，我们引入了一种预测公式，通过目标速度和未来参考视野来增强DRL状态空间。在1自由度（1-DoF）直升机测试平台上，使用近端策略优化（PPO）评估八种配置，模拟结果显示误差减少了9倍，平均绝对偏差从2.73°降至0.31°。然而，零发射器传输到实体硬件后，模拟与现实之间存在差距。有趣的是，使用单一进一步前瞻视距的更简单配置，其实际最高性能与最复杂模型的1.11°相当。总体而言，评估预测视野和目标速度的多种组合表明，物理传输不一定需要高度细粒度的预测数据。

Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization

利用贝叶斯优化进行能量感知强化学习的样本高效帕累托前缘建模

• Authors: Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03140
• Pdf link: https://arxiv.org/pdf/2607.03140
• Abstract Industrial automation increasingly demands control strategies that balance operational performance with strict energy efficiency requirements. A common approach to solving this multi-objective problem, particularly within the framework of reinforcement learning (RL), is to formulate a single, scalar reward function that linearly combines the competing objectives. However, the manual weighting of these different objectives is heavily reliant on domain intuition, incredibly time-consuming, prone to human bias, and frequently fails to uncover optimal trade-off solutions. This work addresses the critical challenge of automating the weight selection process to systematically and efficiently discover the Pareto front of optimal trade-off policies. We formulate the weight selection process as a multi-objective Bayesian optimization (MOBO) problem and evaluate its sample efficiency against a standard uniform grid search baseline. Using a physical Quanser Aero 2 testbed configured for 1-DoF pitch control, our results demonstrate that the MOBO approach, utilizing the expected hypervolume improvement (qEHVI) acquisition function, consistently outperforms uniform grid sampling. MOBO achieves superior hypervolume and maximum spread, successfully identifying high-quality, diverse trade-off policies with a reduced evaluation budget, thereby enabling highly efficient energy-aware control in complex mechatronic systems.
• 中文摘要 工业自动化越来越需要在运营性能与严格的能源效率要求之间取得平衡的控制策略。解决这一多目标问题的常见方法是，尤其是在强化学习（RL）框架内，建立一个单一的标量奖励函数，线性地组合了竞争目标。然而，对这些不同目标的人工权重高度依赖领域直觉，耗时极长，容易产生人为偏见，且常常无法找到最优权衡方案。本研究解决了自动化权重选择过程以系统高效发现最优权衡策略帕累托前沿的关键挑战。我们将权重选择过程表述为多目标贝叶斯优化（MOBO）问题，并以标准统一网格搜索基线评估其样本效率。使用配置为1-DoF俯仰控制的物理Quanser Aero 2测试平台，我们的结果表明，利用预期超体积改进（qEHVI）采集函数的主板方法，始终优于均匀网格采样。主板实现了卓越的超量和最大扩散，成功识别高质量、多样化的权衡策略，同时降低评估预算，从而实现复杂机电系统中高效的能源感知控制。

Exp2VLA: Enabling Vision-Language-Action for Drone Navigation from Expert Demonstrations

Exp2VLA：通过专家演示实现无人机导航的视觉语言行动

• Authors: Van Huyen Dang, Kabilesh Rajendran, Erdi Sayar, Erdal Kayacan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03146
• Pdf link: https://arxiv.org/pdf/2607.03146
• Abstract Vision-language-action (VLA) models open a new path toward intuitive robot control by directly linking perception, language, and action in a single end-to-end framework. Yet for UAVs, practical adoption remains difficult because existing solutions are either computationally heavy or insufficiently capable in complex environments. In this work, we propose a practical expert-distillation pipeline (Exp2VLA) for language-conditioned drone navigation. The core idea is to distill expert behavior, obtained from reinforcement learning, teleoperation, or other controllers, into training data that can be used to fine-tune compact VLA models. This allows existing control strategies to be transferred into a unified language-guided navigation model, reducing manual system integration and lowering the barrier for deploying new robot behaviors. Experiments in both sim-to-sim and simulation-in-the-loop settings across multi-object scenes show that the fine-tuned models can handle varied semantic commands and generalize to unseen target compositions. The proposed framework demonstrates how expert-policy distillation can help mechatronic systems move from specialized control modules toward more flexible and reusable robot intelligence.
• 中文摘要 视觉-语言-行动（VLA）模型通过将感知、语言和动作直接连接在单一端到端框架中，开辟了直觉机器人控制的新路径。然而，对于无人机来说，实际应用仍然困难，因为现有解决方案要么计算量大，要么在复杂环境中能力不足。本研究提出一种实用的专家蒸馏流水线（Exp2VLA），用于语言条件无人机导航。核心思想是将强化学习、远程操作或其他控制器获得的专家行为提炼为训练数据，用于微调紧凑的VLA模型。这使得现有控制策略能够转移到统一的语言引导导航模型中，减少了手动系统集成，降低了部署新机器人行为的门槛。在多对象场景的模拟对模拟和模拟环路环境中的实验表明，精细调优的模型能够处理不同的语义命令，并推广到未见的目标组合。该框架展示了专家-政策提炼如何帮助机电系统从专用控制模块迈向更灵活且可重复使用的机器人智能。

Understanding electricity consumption behaviour through Inverse Reinforcement Learning

通过逆向强化学习理解用电行为

• Authors: Enrico Cofler, Carlos Rodriguez-Pardo, Matteo Giuliani, Andrea Castelletti, Massimo Tavoni
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.03176
• Pdf link: https://arxiv.org/pdf/2607.03176
• Abstract Understanding how households consume electricity in response to socioeconomic and climatic drivers is important for decision-makers designing energy policies in a changing climate and under geopolitical tensions. Consumers respond differently to thermal stress depending on income, consumption habits and the surrounding built environment, a nonlinear behaviour that most approaches oversimplify. In this study, households are treated as agents interacting with complex environments, and Inverse Reinforcement Learning is used to represent their consumption behaviour as model implied reward functions. Specifically, we observe how these reward functions change when households undergo socioeconomic and climatic shocks. The framework is tested on different clusters of electricity consumption profiles in Italy. Clusters' reward functions are retrieved and used to understand how cooling behaviour changes from summer 2021 to summer 2022 and 2023, before, during and after the energy crisis and a heatwave. We find that these shocks reshaped cooling behaviour heterogeneously across consumer groups, in directions conditioned by their prior habits and built environment. Across the 2021 to 2023 summers, we identify a spectrum of responses: transient adjustments that receded as the shocks eased, durable shifts persisting into 2023, and consumers exhibiting negligible change. At the intradaily scale, groups comparable in socioeconomic and environmental context but differing in their daily timing of consumption responded distinctly, identifying time of use as a separate dimension of behavioural heterogeneity. Energy policies and demand-response schemes should therefore account not only for who consumers are and where they live, but for when they consume and whether their response to a shock persists.
• 中文摘要 理解家庭如何根据社会经济和气候因素消耗电力，对于在气候变化和地缘政治紧张局势下制定能源政策的决策者至关重要。消费者对热应力的反应因收入、消费习惯和周围建筑环境而异，这是一种非线性行为，大多数方法都过于简化。本研究将家庭视为与复杂环境互动的代理，并利用逆强化学习将其消费行为表示为模型隐含的奖励函数。具体来说，我们观察到当家庭经历社会经济和气候冲击时，这些奖励函数的变化。该框架在意大利不同电力消费集群中进行了测试。群集的奖励函数被检索并用于理解2021年夏季至2022年和2023年夏季期间，能源危机和热浪前后冷却行为的变化。我们发现，这些冲击在消费者群体间异质地重塑了冷却行为，方向受其先前习惯和建筑环境的影响。在2021年至2023年夏季，我们识别出一系列反应：随着冲击缓解而退去的暂时调整，持续到2023年的持续变化，以及消费者表现出几乎无变化。在日内尺度，社会经济和环境背景相似但每日消费时间不同的群体反应明显，将使用时间视为行为异质性的独立维度。因此，能源政策和需求响应方案不仅应考虑消费者身份和居住地，还应考虑他们何时消费以及对冲击的反应是否持续。

CRRL: A Causality-Based Reinforcement Learning Framework for Autonomous System Recovery

CRRL：基于因果律的自主系统恢复强化学习框架

• Authors: Safia Fatima, Kai Olav Ellefsen, Leon Moonen
• Subjects: Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.03177
• Pdf link: https://arxiv.org/pdf/2607.03177
• Abstract Traditional reinforcement learning (RL) for recovery in autonomous systems lacks causal understanding and generalizes poorly to novel failure scenarios. RL policies often stall in failure states, spending up to 70% of an episode immobilized. Rule-based recovery alone is inadequate, and adding heuristic recovery to a pretrained PPO policy worsens rewards because policies cannot coordinate well with unanticipated interventions. The issue is not missing recovery mechanisms but a lack of policies trained to collaborate with them. We introduce CRRL, a causal-guided RL framework that trains policies to work effectively with rule-based recovery. The recovery detects stalled states and assists the agent. Causal relations from driving logs shape the training signal, teaching the policy to anticipate stalls and adjust actions in recovery contexts. The framework follows MAPE-K, with sensor collection, causal model construction, and hybrid RL policy training corresponding to Monitor, Analyze, and Plan/Execute, respectively. We evaluate CRRL through a four-condition ablation study across three driving scenarios, with 20 episodes per condition. We find that causal training significantly improves reward, distance, and velocity. Moreover, 9 of 20 roundabout episodes required zero recovery intervention, confirming navigation competence. These results show that causal-guided training produces effective RL policies that cooperate with rule-based safety components.
• 中文摘要 传统的自主系统恢复强化学习（RL）缺乏因果理解，且难以推广到新颖的失效场景。强化政策常在失败状态下停滞，导致一集时间高达70%处于瘫痪状态。仅靠规则驱动的恢复是不充分的，而在预训练的PPO政策中加入启发式恢复会恶化奖励，因为政策无法与意外干预做好协调。问题不在于缺乏恢复机制，而是缺乏经过培训以配合这些机制的政策。我们介绍了CRRL，一个因果引导的强化学习框架，训练策略以有效配合基于规则的恢复。恢复检测停滞状态并协助代理。驾驶日志的因果关系塑造训练信号，教导策略预测停滞并在恢复情境中调整动作。该框架遵循MAPE-K，包含传感器收集、因果模型构建和混合强化学习策略训练，分别对应监控、分析和计划/执行。我们通过一项涵盖三种驱动情境的四种病症消融研究，每种病症共20次发作，评估了CRRL。我们发现因果训练显著提升了奖励、距离和速度。此外，20个绕行事件中有9个无需恢复干预，确认了导航能力。这些结果表明，因果引导训练能够产生有效的强化学习政策，并与基于规则的安全组成部分相配合。

Unbiased Alignment for Large Language Models with Noisy Preferences

具有噪声偏好的大型语言模型的无偏比对

• Authors: Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03248
• Pdf link: https://arxiv.org/pdf/2607.03248
• Abstract The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: this https URL.
• 中文摘要 大型语言模型与人类偏好的对齐通常通过人类反馈强化学习或直接偏好优化实现。然而，这些方法容易受到现实偏好数据集中普遍存在的显著噪声影响。为解决这一关键问题，我们提出了一个无偏比对的理论框架，介绍了无偏奖励模型（URM）损失和无偏直接偏好优化（UDPO）损失。通过数学纠正偏好噪声引起的失真，我们的新目标使得直接从噪声数据集中实现无偏的模型训练，无需干净的地面真实监督。我们通过严谨的理论分析证明我们的方法具有抗噪能力、参数向下兼容且经过分类校准。跨多元数据集的综合实验表明，我们的方法优于最先进的基线。代码可访问：此 https URL。

Hierarchical Multi-Agent Reinforcement Learning for Carbon-Aware AI Data Centers in Power Distribution Systems

电力分配系统中碳感知AI数据中心的分层多智能体强化学习

• Authors: Hyunsoo Lee, Panggah Prabawa, Dae-Hyun Choi, Joongheon Kim
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03324
• Pdf link: https://arxiv.org/pdf/2607.03324
• Abstract Eco-friendly energy management for artificial intelligence data centers (AIDCs) is crucial because of the significant increase in energy consumption-induced carbon emissions from AIDCs resulting from the rapid expansion of AI applications. This paper proposes a hierarchical carbon-aware multi-agent reinforcement learning (CA-MARL) framework for robust and efficient operations of AIDCs under uncertainties while ensuring low-carbon operation of power distribution systems. The framework comprises a workload manager (WM) agent and multiple local AIDC agents trained using a multi-agent transformer method, corresponding to a global AIDC aggregator and a local AIDC operator, respectively. Leveraging AIDC operation data along with nodal carbon intensity (NCI) calculated from the carbon emission flow-integrated distribution system operator problem, the WM agent spatially allocates AI training and inference jobs among all AIDCs. Based on the jobs allocated from the WM agent and NCI information, each AIDC agent schedules economical and eco-friendly operations of the AIDC by performing the following tasks: i) temporal shifting of training jobs, ii) spatial allocation of training graphics processing unit (GPU) blocks and inference GPUs within the AIDC, and iii) control of the supply air temperature of the cooling system. The effectiveness of the proposed framework was assessed using an IEEE 33-node power distribution system.
• 中文摘要 人工智能数据中心（AIDIC）的环保能源管理至关重要，因为AI应用的快速扩展导致的能源消耗导致的碳排放显著增加。本文提出了一种分层碳感知多智能体强化学习（CA-MARL）框架，用于在不确定性条件下实现AIDC的稳健高效运行，同时确保电力分配系统的低碳运行。该框架由一个工作负载管理器（WM）代理和多个本地AIDC代理组成，这些代理使用多代理变换器方法训练，分别对应一个全局AIDC聚合器和一个本地AIDC操作员。利用AIDC运行数据及碳排放流集成分配系统操作员问题计算的节点碳强度（NCI），WM代理空间地将AI培训和推断任务分配给所有AIDCs。基于WM代理和NCI信息分配的作业，每个AIDC代理通过执行以下任务来安排AIDC的经济且环保的运行：i）训练任务的时间转移，ii）AIDC内训练图形处理单元（GPU）块和推理GPU的空间分配，iii）控制冷却系统的送风温度。该框架的有效性通过IEEE 33节点电力分配系统进行了评估。

MUTE: Return-Preserving Communication Unlearning for Efficient Multi-Agent Coordination

MUTE：返回保持通信的高效多智能体协调去学习

• Authors: Rui Zuo, Qinwei Huang, Mingyang Li, Zhenhang Zhang, Simon Khan, Qinru Qiu
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.03473
• Pdf link: https://arxiv.org/pdf/2607.03473
• Abstract Inter-agent communication is critical for coordinating Multi-Agent Reinforcement Learning (MARL) agents under partial observability to perform effectively in cooperative games; however, real-world bandwidth constraints demand sparse interactions. Prior approaches primarily address this trade-off by optimizing information-theoretic surrogates. We argue that these statistical proxies are fundamentally misaligned with the true objective: a message can be highly informative yet irrelevant to the joint return of the task. In this work, we propose Message Unlearning for Targeted Efficiency (MUTE), a framework that views communication reduction as a value-guided machine unlearning problem. MUTE rigorously quantifies the Counterfactual Message Value using an attention-based estimator, and systematically unlearns the transmission of low-value messages from a policy trained without any communication constraints. This is achieved through a dual-objective mechanism that enforces communication sparsity while preserving the return of the original joint policy. We derive a theoretical upper bound on the performance gap induced by this sparsification, guaranteeing controlled return degradation. We also empirically evaluate MUTE on various complex multi-agent environments, achieving 80% to 90% bandwidth reduction while maintaining performance comparable to state-of-the-art baselines.
• 中文摘要 代理间通信对于协调多智能体强化学习（MARL）智能体在部分可观察性下有效参与合作游戏至关重要;然而，现实世界的带宽限制要求交互稀疏。以往的方法主要通过优化信息论替代来解决这一权衡。我们认为这些统计代理与真实目标根本不符：一条信息可能信息量很大，但对任务的联合返回无关。在本研究中，我们提出了“目标效率信息去学习”（MUTE）框架，将沟通减少视为价值导向的机器学习去学习问题。MUTE通过基于注意力的估计器严格量化反事实消息值，并系统地从无任何通信约束的策略中剔除低价值消息的传输。这通过一种双重目标机制实现，既确保通信稀疏，又保持原始联合政策的回归。我们推导出了由稀疏化引起的性能差距的理论上上限，保证了受控的回报退化。我们还实证评估了MUTE在各种复杂多智能体环境中的应用，实现了80%至90%的带宽削减，同时保持与最先进基线相当的性能。

Demonstrating Generalization Failures via Mixtures of Conditional Policies

通过多种条件策略混合演示泛化失败

• Authors: Jou Barzdukas, Jack Peck, Julian Schulz, Paulius Rauba, Steven Basart, Lennie Wells
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03478
• Pdf link: https://arxiv.org/pdf/2607.03478
• Abstract Post-training of frontier language models is conducted on curated task suites, and inevitably leaves a distribution shift between training and deployment environments. This exposes developers to generalization failures, which are relatively poorly understood. To better understand such generalization failures, we believe the community should construct clean demonstrations under simplified conditions. To facilitate this, we propose a simple and flexible way to construct language models which fail to generalize in controllable ways when subsequently trained with Reinforcement Learning (RL) on a given distribution of training tasks. Our construction uses Supervised Fine-Tuning on a dataset of a mixture of transcripts corresponding to a collection of 'conditional policies', which can each independently be assigned certain behaviors on each different task distribution, to obtain a model that is then well approximated as a 'mixture of conditional policies.' We observe that RL training then selects for policies that obtain the highest reward on the training distribution. This can produce striking behaviors: in a controlled setting with two distributions containing identical questions prepended with two different 'trigger strings', RL training on either distribution actively degrades performance on the other to zero, even though the underlying task is identical. We also use our construction to illustrate two novel ways in which generalization may fail in future language models, corresponding to distribution shifts of task coverage and temporal context respectively. While our construction is deliberately simple and may not closely resemble 'natural' generalization failures, the resulting 'model organisms' are of interest for alignment stress-testing and generalization science, and can be used as existence proofs that training success and generalization can come apart in structured ways.
• 中文摘要 前沿语言模型的后期训练是在精心策划的任务套件上进行的，这不可避免地导致训练与部署环境之间的分布转移。这使开发者面临泛化失败的风险，而这些失败相对较少被理解。为了更好地理解此类泛化失败，我们认为社区应在简化条件下构建干净的演示。为此，我们提出了一种简单灵活的方法，构建语言模型，在随后用强化学习（RL）训练给定训练任务分布时，无法以可控方式泛化。我们的构建在一个由多份“条件策略”混合的转录本组成的数据集上进行监督微调，这些转录本可以独立分配不同任务分布的特定行为，从而得到一个模型，并将其很好地近似为“条件策略的混合”。我们观察到强化学习随后会选择在训练分布中获得最高奖励的策略。这可能导致显著的行为：在受控环境中，两个分布包含相同问题，前置两个不同的“触发字符串”，在任一分布进行强化学习时，即使底层任务相同，也会主动将另一方的性能降至零。我们还利用构造说明了未来语言模型中泛化可能失败的两种新颖方式，分别对应任务覆盖分布和时间上下文的分布变化。虽然我们的构造刻意简化，且可能与“自然”泛化失败不甚相似，但所得的“模式生物体”对比对应压力测试和泛化科学具有重要价值，并可作为训练成功与泛化可以以结构化方式分离的存在证明。

High-Precision Formation Control for Heterogeneous Multi-Robot Systems via Hierarchical Hybrid Physics-Informed Deep Reinforcement Learning

通过分层混合物理知情深度强化学习，实现异构多机器人系统的高精度编队控制

• Authors: Yanzhou Li, Guangli Chen, Xiao-Meng Li, Wenjian Zhong, Yongkang Lu, Shenghuang He
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03512
• Pdf link: https://arxiv.org/pdf/2607.03512
• Abstract Existing classical control methods commonly require precise models and struggle to cope with model uncertainties and external disturbances, while end-to-end reinforcement learning (RL) approaches suffer from low sample efficiency and poor convergence. To overcome these challenges, this paper proposes a hierarchical hybrid physics-informed deep reinforcement learning (HHy-PIDRL) framework, aiming to realize high-precision, highly responsive formation control for heterogeneous multi-robot systems (HMRSs). The proposed framework contains two layers. Specifically, first, the upper layer designs an autonomous navigation policy network for Ackermann-steering leader based on the Soft Actor-Critic (SAC) deep reinforcement learning (DRL) algorithm. Second, the lower module integrates a high-fidelity physical feed-forward controller, a classical proportional-derivative (PD) controller, and an adaptive DRL residual controller to propose an effective hybrid model and DRL (HM-DRL)-based formation control policy network. Third, a unique hierarchical reward function is designed for training Omnidirectional followers, which effectively guides agents toward a refined, stable control policy. Experimental results demonstrate that, the success rate of both the upper-layer autonomous navigation policy network and the HM-DRL based formation control policy networks reach 100%. Meanwhile, ablation experiments are conducted to verify the validity and credibility of the proposed method.
• 中文摘要 现有的经典控制方法通常需要精确模型，难以应对模型的不确定性和外部干扰，而端到端强化学习（RL）方法则存在样本效率低和收敛性差的问题。为克服这些挑战，本文提出了一个分层混合物理知情深度强化学习（HHy-PIDRL）框架，旨在实现异构多机器人系统（HMRS）的高精度、高响应性编组控制。该框架包含两层。具体来说，首先，上层基于软演员-批判者（SAC）深度强化学习（DRL）算法设计了阿克曼引导领导者的自主导航策略网络。其次，下模块集成了高精度物理前馈控制器、经典比例导数（PD）控制器和自适应日间移动残差控制器，提出了一种有效的混合模型和基于日程加速（HM-DRL）的编队控制策略网络。第三，设计了独特的层级奖励函数，用于训练全向追随者，有效引导智能体朝向精细、稳定的控制策略。实验结果表明，上层自主导航政策网络和基于HM-DRL的编队控制策略网络的成功率均达到100%。同时，还进行了消融实验以验证所提方法的有效性和可信度。

Anchored Self-Play for Code Repair

代码修复的锚定自玩

• Authors: Caroline Choi, Zeyneb Kaya, Shirley Wu, Tengyu Ma, Tatsunori Hashimoto, Ludwig Schmidt
• Subjects: Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.03523
• Pdf link: https://arxiv.org/pdf/2607.03523
• Abstract Code repair is an important capability for language models (LMs): given a buggy program and unit tests, an LM must produce a fixed program that passes the tests. Because code repair data is limited, we aim to scale supervision by using an LM to generate bug--fix tasks. We propose generator--fixer self-play, in which a single model is trained with reinforcement learning to generate bugs and fix them. As the fixer improves, the generator adapts to produce more difficult bugs, yielding an automatic curriculum. To test whether this curriculum generalizes, we introduce BugSourceBench, a repair benchmark spanning realistic bug sources: bugs in human-written code, LM-generated code, and human-edited LM-generated code. On BugSourceBench, we find that self-play drifts toward difficult but unrealistic bugs, improving on synthetic bugs but degrading on human-authored ones. We propose Anchored Self-Play (ASP), which anchors self-play with a small reference set by adding a code-embedding similarity reward for generation and mixing reference bugs into fixer training. Across bug sources, ASP achieves the best fix rates, improving average fix rate over standard self-play by $+24\%$ relative / $+7.0$ pp absolute, with gains on bugs from both LMs and humans.
• 中文摘要 代码修复是语言模型（LM）的重要能力：在有缺陷的程序和单元测试的情况下，LM必须生成一个通过测试的固定程序。由于代码修复数据有限，我们旨在通过使用LM生成缺陷修复任务来扩大监督。我们提出__generator——修复器自play__，即通过强化学习训练单个模型生成并修复错误。随着修复器的进步，生成器会适应以产生更难的漏洞，从而产生自动课程。为了测试该课程是否具有普遍适用性，我们引入了 BugSourceBench，这是一个涵盖真实错误源的修复基准测试：人类编写代码、LM 生成代码和人类编辑 LM 生成代码中的漏洞。在BugSourceBench上，我们发现自玩模式往往偏向于困难但不现实的bug，虽然比合成bug有所改进，但在人工编写的bug上则会有劣质。我们提出了锚定自玩（ASP），通过添加代码嵌入相似性奖励，用于生成引用错误并将引用错误混入修复器训练，从而锚定自玩。在各种漏洞来源中，ASP实现了最佳修复率，平均修复率相较标准自玩提升了$+24\%$ / PP absolute $+7.0$，同时在LM和人类的漏洞上都有收益。

Aligning Language Models with Selective Prediction

语言模型与选择性预测的对齐

• Authors: Gaoxiang Luo, Yifan Wu, Sinian Zhang, Aryan Deshwal, Ju Sun
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.03528
• Pdf link: https://arxiv.org/pdf/2607.03528
• Abstract Large language models (LLMs) are increasingly deployed as critical decision-making components in high-stakes real-world AI systems, rendering LLM reliability a foremost practical concern. In this paper, we focus on enhancing LLM reliability through selective prediction (SP), a strategy that allows an LLM to only predict for inputs where it is likely to be correct (i.e., coverage) and hence reduce the error rate (i.e., risk) on that portion of inputs -- flagging the remaining inputs for future human discretion. In other words, SP improves LLM reliability by balancing the risk-coverage trade-off and enabling seamless human-AI collaboration. To integrate SP into LLMs, we focus on the LLM post-training alignment stage and propose to align LLMs with SP performance metrics, in contrast with existing LLM alignment methods that focus primarily on correctness or calibration metrics. Specifically, we propose a novel alignment framework, Reinforcement Learning for Selection Reward (RLSR), which targets the area under the risk-coverage curve (AURC) -- a popular SP performance metric -- as its alignment objective. RLSR achieves substantially better risk-coverage trade-off compared to multiple alignment baselines on both in-domain and out-of-domain tasks.
• 中文摘要 大型语言模型（LLM）正日益作为高风险现实世界人工智能系统中的关键决策组件被部署，使得LLM的可靠性成为最实际的关注点。本文重点通过选择性预测（SP）提升LLM的可靠性，该策略使LLM仅预测其可能正确的输入（即覆盖率），从而降低该部分输入的错误率（即风险），并标记剩余输入供未来人类自行判断。换句话说，SP通过平衡风险覆盖权衡和实现无缝人机协作，提升了LLM的可靠性。为了将 SP 集成到 LLM，我们关注 LLM 训练后对齐阶段，并提出将 LLM 与 SP 性能指标对齐，与目前主要关注正确性或校准指标的 LLM 对齐方法形成对比。具体来说，我们提出了一种新颖的对齐框架——选择奖励强化学习（RLSR），其对齐目标是风险覆盖曲线下区域（AURC）——一个流行的SP绩效指标。RLSR在域内和域外任务中，相较于多个对齐基线实现了显著更好的风险覆盖权衡。

MentalThink: Shaping Thoughts in Mental SVG World

MentalThink：在Mental SVG世界中塑造思想

• Authors: Kangheng Lin, Jisheng Yin, Dingming Li, En Yu, Yana Wei, Han Zhou, Liang Zhao, Hongyu Zhou, Hongbo Peng, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03530
• Pdf link: https://arxiv.org/pdf/2607.03530
• Abstract We introduce MentalThink, a visual-symbolic reasoning paradigm that equips Multimodal LLMs (MLLMs) with an executable mechanism for "mental" visualization. The core of MentalThink is a think-with-SVG pipeline, where the model learns to generate, render, and interpret scalable vector graphics (SVG) code as an intermediate visual representation for multi-turn reasoning. By creating structured vector sketches, the model can externalize spatial hypotheses, inspect them through deterministic rendering, and reason within a constrained geometric space, effectively mimicking the human process of mental imagery. We instantiate this paradigm through a two-stage training framework, combining Supervised Fine-Tuning (SFT) for SVG syntactic alignment with multi-turn Reinforcement Learning (RL) to encourage iterative inspection, revision, and refinement of intermediate visual hypotheses. Extensive evaluations demonstrate that MentalThink achieves superior performance on spatial understanding and reasoning benchmarks (e.g., 55.1% on VSIBench, 76.0% on MindCube), showing that executable vector graphics provide a verifiable visual workspace for dynamic perspective taking, visual reflection, and compositional scene construction.
• 中文摘要 我们介绍了 MentalThink，一种视觉符号推理范式，为多模态大型语言模型（MLLM）配备了可执行的“心理”可视化机制。MentalThink的核心是一个带有SVG的思考流程，模型学习生成、渲染和解释可扩展矢量图形（SVG）代码，作为多回合推理的中间视觉表示。通过创建结构化的矢量草图，模型可以将空间假设外部化，通过确定性渲染进行检查，并在受限的几何空间内推理，有效模拟人类的心理意象过程。我们通过两阶段培训框架实现这一范式，结合了SVG语法对齐的监督微调（SFT）与多回合强化学习（RL），鼓励对中间视觉假设进行迭代检查、修订和完善。大量评估表明，MentalThink在空间理解和推理基准测试中表现优异（例如VSIBench为55.1%，MindCube为76.0%），表明可执行矢量图形为动态透视、视觉反射和构图场景构建提供了可验证的可视化工作空间。

Cross-Embodiment Robot Manipulation via a Unified Hand Action Space

通过统一手部动作空间实现跨身体机器人操作

• Authors: Luis Felipe Casas, Robert Teal, Keval Shah, Abhijit Tadepalli, Wanxin Jin, Yu Xiang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03570
• Pdf link: https://arxiv.org/pdf/2607.03570
• Abstract Robot manipulation policies are typically tied to specific robotic hand embodiments, limiting the transfer of learned behaviors across platforms with different kinematic structures. In this work, we propose the Unified Hand Action Space (UHAS), a sphere-based unified action representation for cross-embodiment dexterous manipulation. UHAS represents robotic hand actions as geometric deformations of a canonical sphere and uses a Cascade Inverse Kinematics (CIK) algorithm to map the shared representation to embodiment-specific joint configurations. Using reinforcement learning, we train dexterous manipulation policies directly in the proposed action space for in-hand cube reorientation tasks. We evaluate our method in both simulation and real-world experiments across multiple robotic hands, including the Allegro Hand, LEAP Hand, Shadow Hand, and MANO Human Hand. Experimental results demonstrate effective dexterous manipulation, zero-shot transfer to unseen hands, rapid finetuning across embodiments, and successful real-world deployment. Our experiments show that the proposed UHAS representation enables stable dexterous control and cross-embodiment policy transfer across robotic hands.
• 中文摘要 机器人操作策略通常与特定的机器人手型绑定，限制了不同运动学结构平台上所学行为的转移。在本研究中，我们提出了统一手部动作空间（UHAS），这是一种基于球体的统一动作表示，用于跨身体灵巧操作。UHAS将机器人手部动作表示为标准球体的几何变形，并使用级联逆运动学（CIK）算法将共享表示映射到具体体型的关节配置。通过强化学习，我们直接在拟议的动作空间中训练灵巧操作策略，用于手握立方体重新定向任务。我们在模拟和现实实验中评估了我们的方法，涵盖多种机器人手，包括快板手、LEAP手、影手和MANO人手。实验结果显示了有效的灵巧操作、零射击转移至看不见的手、跨体形的快速微调以及成功的实际部署。我们的实验表明，所提拟的UHAS表示能够实现机器人手之间稳定的灵巧控制和跨身体策略转移。

A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning

通过微调和强化学习迈向稳健的无监督域适应迈向一步

• Authors: Sushant Dagaji Desale, Rahul Mishra, Ashutosh Kumar Sinha
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03600
• Pdf link: https://arxiv.org/pdf/2607.03600
• Abstract Adversarial robustness in Unsupervised Domain Adaptation (UDA) remains a significant challenge due to noisy pseudo labels and inherent distributional shifts between the clean source and adversarially perturbed target domains. Existing approaches often fail to achieve an optimal trade-off between robustness and accuracy, as pseudo-labels generated by domain-adapted models tend to introduce classification errors under adversarial attacks. In this work, we propose \textbf{SFT+RL}, a two-stage robust UDA framework that integrates Supervised Fine Tuning (SFT) and Reinforcement Learning (RL) on top of CLIP's pre-trained visual encoder. In the SFT stage, we adversarially fine-tune a linear classifier using PGD-based perturbations over the labelled source domain while partially unfreezing CLIP's projection layer. It allows adaptation to adversarial noise while preserving CLIP's rich semantic priors. We introduce a confidence-guided pseudo-labeling strategy in the RL stage to annotate unlabeled target samples progressively. Pseudo labels are filtered using a decaying confidence threshold to balance quality and coverage, and the model is trained on a composite dataset formed by combining clean source samples with high-confidence target samples. Adversarial training is applied to mixed batches of clean and adversarial examples to enhance cross-domain robustness. Comprehensive evaluations on three benchmark datasets OfficeHome~\cite{tomm-ude}, PACS~\cite{pacs}, and VisDA~\cite{visda} demonstrate the effectiveness of our approach. Notably, \textbf{SFT+RL} achieves average improvements of \textbf{10.2\%} in clean accuracy and \textbf{15.8\%} in adversarial robustness across all three datasets, outperforming existing state-of-the-art methods.
• 中文摘要 由于伪标记噪声以及干净源与被对抗扰动目标域之间固有分布偏移，非监督域域适应（UDA）中的对抗性鲁棒性仍是重大挑战。现有方法常常无法在鲁棒性和准确性之间取得最佳权衡，因为由领域适应模型生成的伪标签在对抗攻击下往往会引入分类错误。本研究提出 \textbf{SFT+RL}，一个两阶段稳健的 UDA 框架，集成了监督微调（SFT）和强化学习（RL）在 CLIP 预训练视觉编码器之上。在SFT阶段，我们利用基于PGD的微扰对标注源域进行对线性分类器进行对抗性微调，同时部分解冻CLIP的投影层。它允许对对抗噪声进行适应，同时保持CLIP丰富的语义先验。我们在强化学习阶段引入了一种基于置信度引导的伪标记策略，逐步注释未标记的目标样本。伪标签通过衰减置信阈值进行过滤，以平衡质量与覆盖范围，模型则在由干净源样本与高置信度目标样本结合形成的复合数据集上训练。对抗训练应用于混合批次的干净和对抗样本，以增强跨域的鲁棒性。对三个基准数据集OfficeHome~\cite{tomm-ude}、PACS~\cite{pacs}和VisDA~\cite{visda}的全面评估展示了我们方法的有效性。值得注意的是，\textbf{SFT+RL}在三个数据集中均实现了\textbf{10.2\%}的净化精度提升和\textbf{15.8\%}的对抗鲁棒性，均优于现有最先进的方法。

Explainable Reinforcement Learning for Adaptive Traffic Signal Control

自适应交通信号控制的可解释强化学习

• Authors: Dickens Kwesiga, Nishu Choudhary, Angshuman Guin, Michael Hunter
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.03703
• Pdf link: https://arxiv.org/pdf/2607.03703
• Abstract Reinforcement Learning (RL) has emerged as a powerful paradigm for adaptive traffic signal control. However, in safety-critical infrastructure like traffic control, the opaque, black-box nature of deep RL models poses challenges for transportation agency acceptance, regulatory compliance, operational trust, troubleshooting, and fine-tuning. To bridge this gap between high-performance optimization and human-comprehensible interpretability, this effort introduces a novel, explainable entity centric RL framework for safe and transparent traffic signal control. Rather than processing traffic states through monolithic, flat vectors, the proposed architecture disaggregates real-time intersection observations into distinct, high-dimensional lane entities and phase temporal configurations to inherently preserve the structural topology and geometric configurations of the intersection. Relational dependencies and inter-lane conflicts are dynamically extracted via a dual-stage attention network featuring sequential multi-head cross-attention and self-attention blocks. This design yields a real time affinity matrix that quantifies the direct influence of signal phases on specific approach volumes and queues, providing full visual and analytical interpretability. To ensure strict operational reliability, a deterministic action-masking interface is integrated directly into the Proximal Policy Optimization pipeline, explicitly blocking invalid phase transitions to guarantee absolute compliance with established signal timing and safety constraints. Evaluated in a microscopic simulation environment, outperforms state-of-the-art baselines in delay minimization. More importantly, the emergent attention weights align precisely with established traffic engineering principles, offering an auditable, trust-enabling, and deployable architecture for next-generation adaptive traffic control systems.
• 中文摘要 强化学习（RL）已成为自适应交通信号控制的强大范式。然而，在交通控制等安全关键基础设施中，深度强化学习模型的不透明、黑箱特性为交通机构的接受度、合规性、运营信任、故障排除和微调带来了挑战。为了弥合高性能优化与人类可理解性之间的差距，本项目引入了一个新颖且可解释的实体中心强化学习框架，用于安全透明的交通信号控制。所提架构不是通过单一、平坦的向量处理交通状态，而是将实时交叉口观测分解为不同的高维车道实体和相位时间配置，以本质上保留交叉口的结构拓扑和几何配置。关系依赖和车道间冲突通过双阶段注意力网络动态提取，该网络具有顺序多头交叉注意力和自我注意力块。该设计产生了实时亲和矩阵，量化信号相位对特定进近体积和队列的直接影响，提供完整的视觉和分析可解读性。为确保严格的操作可靠性，确定性动作掩蔽接口直接集成到近端策略优化流水线中，明确阻止无效的相变，确保绝对符合既定的信号时序和安全约束。在显微模拟环境中评估时，在延迟最小化方面优于最先进的基线。更重要的是，涌现注意力权重与既定的交通工程原则高度契合，为下一代自适应交通控制系统提供了可审计、信任启用和可部署的架构。

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

将交错多模态推理作为统一决策过程的桥接

• Authors: Zican Hu, Xuyang Hu, Yiming Liu, Zuwei Long, Wei Liu, Yunzhuo Hao, Jiawei Gu, Linjie Li, Yu Cheng, Zhenhong Sun, Weibo Gu, Xing Sun, Zhi Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03748
• Pdf link: https://arxiv.org/pdf/2607.03748
• Abstract Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge. Existing approaches apply RL exclusively to text steps, relegating image generation to supervised surrogates, preventing policy gradients from propagating through the full interleaved trajectory across heterogeneous modalities. This leaves the potential of RL for UMMs largely untapped. In the paper, we introduce \textbf{BRAID} (\textbf{B}ridging inte\textbf{R}le\textbf{A}ved mult\textbf{I}-modal reasoning as a unified \textbf{D}ecision process), a simple framework that casts multi-turn text-image-text reasoning as a unified Markov decision process (MDP), enabling joint optimization of textual and visual generation via a single, principled RL objective. BRAID computes a shared trajectory-level advantage and propagates it coherently into both text tokens and image denoising paths, each optimized through its modality-native policy gradient mechanism. To further address long-horizon credit assignment, BRAID employs a vision-language model (VLM) judge that scores each intermediate image on its reasoning utility, supplying dense turn-level feedback to sharpen learning at critical visual branches. Experiments on spatial reasoning and visual perception benchmarks show that BRAID consistently outperforms various baselines, confirming that a unified MDP formulation with vision-thinking guidance is essential for effective multi-modal reasoning.
• 中文摘要 统一多模态模型（UMM）已展现出有前景的交错文本-图像推理能力，但通过强化学习（RL）有效优化此类多回合生成仍是一个开放挑战。现有方法仅将强化学习应用于文本步骤，将图像生成交由监督替代，防止策略梯度在异质模态间完整交错传播。这使得强化学习对UMM的潜力基本未被充分利用。本文介绍了 \textbf{BRAID}（\textbf{B}ridging inte\textbf{R}le\textbf{A}ved mult\textbf{I}-模态推理作为统一的 \textbf{D}ecision 过程），这是一个简单的框架，将多回合文本-图像-文本推理定位为统一的马尔可夫决策过程（MDP），通过单一原则性的强化学习目标实现文本与视觉的联合生成。BRAID计算共享轨迹层优势，并将其连贯传播到文本标记和图像去噪路径中，每个路径都通过其模态原生策略梯度机制进行优化。为了进一步解决长视野的学分分配问题，BRAID采用视觉语言模型（VLM）评判器，对每个中间图像的推理效用进行评分，提供密集的转向级反馈，以提升关键视觉分支的学习。空间推理和视觉感知基准测试的实验表明，BRAID始终优于各种基线，证实了统一的MDP表述与视觉思维指导对于有效的多模态推理至关重要。

Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models

三思而后行：将树搜索提炼为行动评估，适用于冻结的VLA模型

• Authors: Xinyi Xie, Zican Hu, Zhanyu Liu, Yicheng Dong, Wenhao Wu, Zhenhong Sun, Haoran Li, Chunlin Chen, Zhi Wang, Pichao Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03751
• Pdf link: https://arxiv.org/pdf/2607.03751
• Abstract Vision-Language-Action (VLA) models acquire broad embodied capabilities through large-scale pretraining, yet their generalization remains far more fragile than that of LLMs and VLMs. The prevailing remedy, post-training via supervised fine-tuning or reinforcement learning, improves task-specific performance but narrows the generalist capability that makes pretraining valuable. We identify a key bottleneck: VLA failures stem not only from action generation but also from action evaluation. A diagnostic pass@k study confirms that frozen VLAs already contain competent behaviors in their output distribution, with overall success rates rising from 33% at pass@1 to 92% at pass@32. Inspired by this, we propose SVA (Search, Value, and Act), a simple framework that equips frozen VLA policies with long-term consequence awareness. SVA first uses Monte-Carlo tree search in simulation to fully explore the VLA's output distribution and collect diverse trajectories annotated with empirical returns; this knowledge is then distilled into a lightweight Q-value model that predicts the expected consequence of candidate actions; at deployment, the frozen VLA proposes multiple candidates and the evaluator selects the one with the highest uncertainty-regularized Q-value, requiring no simulator access. By decoupling action proposal from consequence evaluation, SVA preserves the generalization capacity of the VLA backbone while substantially improving task success rates. Experiments across embodied benchmarks show that SVA consistently improves generalization on unseen tasks and exhibits strong test-time scaling behavior. Strikingly, SVA enables a 9B VLA to outperform a 27B VLA by 7 points at 27% lower inference latency, suggesting that scaling test-time evaluation is more cost-effective than scaling model size.
• 中文摘要 视觉-语言-行动（VLA）模型通过大规模预训练获得了广泛的具象能力，但其泛化远比大型语言模型和大型语言模型脆弱得多。主流的解决方法是通过监督微调或强化学习进行训练后，提升任务特定表现，但限制了使预训练具有价值的通才能力。我们识别出一个关键瓶颈：VLA失效不仅源于动作生成，也源于动作评估。一项诊断pass@k研究证实，冷冻的VLA在输出分布中已经表现出有能力的行为，整体成功率从pass@1时的33%上升到pass@32时的92%。受此启发，我们提出了SVA（搜索、价值与行动）框架，这是一个为冻结的VLA政策提供长期后果意识的简单框架。SVA首先在模拟中使用蒙特卡洛树搜索，全面探索VLA的输出分布，并收集带有经验收益的多样轨迹;这些知识随后被提炼成一个轻量级Q值模型，用于预测候选行为的预期结果;部署时，冻结的VLA会提出多个候选方案，评估者选择不确定性正则化Q值最高的候选，无需模拟器访问。通过将动作提案与结果评估解耦，SVA保留了VLA骨干的泛化能力，同时显著提升了任务成功率。跨实例基准测试的实验表明，SVA持续提升对未见任务的泛化能力，并表现出强烈的测试时间缩放行为。值得注意的是，SVA使9B VLA在推理延迟降低27%的情况下，性能优于27B VLA 7个百分点，表明测试时间评估的扩展性比模型规模的扩展更具成本效益。

CineMobile: On-Device Image-to-Video Diffusion for Cinematic Camera Motion Generation

CineMobile：用于电影摄影机运动生成的设备内图像到视频扩散

• Authors: Xuyao Huang, Zelai Deng, Xu Wang, Xizhong Xiao, Zhijie Deng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.03803
• Pdf link: https://arxiv.org/pdf/2607.03803
• Abstract The growing demand for image-to-video creation on mobile devices has increasingly focused on cinematic motion effects like bullet time, dolly zoom, slow motion, etc. While Diffusion Transformers (DiTs) exhibit strong performance in video generation, their large parameter sizes and multi-step iterative denoising processes lead to substantial computational overhead, making efficient generation on mobile devices challenging. We propose CineMobile to bridge the gap. In particular, CineMobile adopts a three-fold optimization strategy: (1) leveraging a distillation-guided pruning approach to derive a compact yet efficient model that retains the essential video generation capabilities required for cinematic effects; (2) optimizing the compressed model into a 4-step generator via a combination of diffusion distillation and reinforcement learning; (3) employing a hybrid post-training quantization strategy to compress the model footprint to under 1 GB. Experimental results show that compared to the teacher model with the Wan 2.1 architecture, CineMobile achieves a 40x speedup in generation while maintaining comparable visual quality. Specifically, CineMobile generates 49-frame 480p videos with a per-step denoising latency of 0.6s on an NVIDIA H200 GPU and 20s on the MediaTek Dimensity 8400 Ultimate 5G platform, with a peak memory usage of 1.8 GB, demonstrating its practical applicability for mobile-based image-to-video creation.
• 中文摘要 移动设备上对图像转视频创作的需求日益增长，越来越多地聚焦于电影级的动态效果，如子弹时间、推轨变焦、慢动作等。虽然扩散变换器（DiT）在视频生成方面表现出优异的性能，但其大参数尺寸和多步迭代去噪过程导致计算开销巨大，使得移动设备上的高效生成具有挑战性。我们提议通过CineMobile来弥合这一差距。特别是，CineMobile采用了三重优化策略：（1）利用蒸馏引导剪枝方法，构建一个紧凑高效且保留电影效果所需的视频生成能力的模型;（2）通过扩散蒸馏和强化学习的结合，将压缩模型优化为四步生成器;（3）采用混合训练后量化策略，将模型占地压缩至1 GB以下。实验结果显示，与采用万万2.1架构的教师模型相比，CineMobile在生成速度提升了40倍，同时保持了相当的视觉质量。具体来说，CineMobile在NVIDIA H200 GPU上生成49帧480p视频，每步去噪延迟为0.6秒，在联发科天玑8400终极5G平台上为20秒，峰值内存使用量为1.8 GB，展示了其在基于移动的图像转视频制作中的实用性。

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

ObjRetarget：一个具备拟人化臂约束和多面体手部建模的对象感知运动重定向框架

• Authors: Yuanchuan Lai, Qing Gao, Ziyan Liang, Junjie Hu, Zhaojie Ju
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.03828
• Pdf link: https://arxiv.org/pdf/2607.03828
• Abstract Learning robot dexterous manipulation from human manipulation videos requires reliably retargeting human intent to executable robot actions while maintaining stable hand-object contact, which remains a key challenge in embodied intelligence. Existing retargeting methods often ignore explicit contact modeling or rely on reinforcement learning, resulting in limited accuracy and generalization. To address this, we propose ObjRetarget, a human-to-robot motion retargeting framework for learning robot dexterous manipulation from human videos, which integrates anthropomorphic arm trajectory constraints with structured hand-object geometric modeling. For arm motion, reference trajectories extracted from human videos are used for initialization, followed by anthropomorphic constraints and redundancy-aware optimization to generate natural and accurate movements. For hand manipulation, ObjRetarget represents multi-finger contacts using polytope clusters and preserves contact structure through geometric invariants to improve stability. Experiments on real robots show that ObjRetarget improves manipulation success rates and contact stability across multiple dexterous tasks, and generalizes well to different demonstrations, object poses, and task settings.
• 中文摘要 从人类操作视频中学习机器人灵巧操作，需要可靠地将人类意图重新定位到可执行的机器人动作，同时保持稳定的手与物体接触，这仍是具身智能中的关键挑战。现有的重定向方法常常忽视显式接触建模或依赖强化学习，导致准确性和泛化有限。为此，我们提出了ObjRetarget，一种人对机器人运动重定向框架，用于从人类视频中学习机器人灵巧操作，将拟人化手臂轨迹约束与结构化手-物体几何建模相结合。对于手臂运动，初始化使用从人类视频中提取的参考轨迹，随后采用拟人约束和冗余意识优化，以生成自然且准确的运动。对于手操作，ObjRetarget用多胞体簇表示多指接触，并通过几何不变量保持接触结构以提升稳定性。在真实机器人上的实验表明，ObjRetarget能提升操作成功率和接触稳定性，适用于多种灵活任务，并且能很好地推广到不同的演示、物体姿态和任务设置。

Beam Hopping Low Earth Orbit Satellite Resource Allocation for Differentiated Services and Robustness Analysis under Model Attacks

波束跳跃：低地球轨道卫星资源分配，用于差异化服务和模型攻击下的鲁棒性分析

• Authors: Shuang Zheng, Xing Zhang, Quan Z. Sheng, Haixu Wang, Wenbo Wang
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2607.03859
• Pdf link: https://arxiv.org/pdf/2607.03859
• Abstract Beam hopping (BH)-enabled Low Earth Orbit (LEO) satellites play a pivotal role in next-generation communication networks by providing global coverage, improving spectrum efficiency, and supporting flexible adaptation to heterogeneous service demands. To fully exploit these capabilities, artificial intelligence (AI) techniques are increasingly employed for dynamic resource allocation and power management. However, limited onboard resources and potential adversarial perturbations pose challenges to both efficiency and robustness. To address these issues, we leverage digital twin technology to accurately capture the spatio-temporal dynamics of user-satellite visibility, thereby providing precise state information for decision-making. Building on this, we formulate a joint optimization framework for BH scheduling and power allocation as a Markov Decision Process and propose BRIDGE, i.e., BH with Reinforcement learning incorporating Integrated Dirichlet and Gumbel-TopK Exploration, which integrates a quality of service (QoS)-driven subchannel scheduling mechanism to ensure efficient and differentiated resource allocation. The robustness of the model is systematically evaluated under three classical adversarial attacks. Simulation results demonstrate that the proposed approach achieves superior energy efficiency, service throughput, and fairness, while the robustness analysis shows stable performance under the considered bounded adversarial perturbations.
• 中文摘要 支持波束跳跃（BH）的近地轨道（LEO）卫星在下一代通信网络中发挥着关键作用，提供全球覆盖、提升频谱效率，并支持对异构服务需求的灵活适应。为了充分发挥这些能力，人工智能（AI）技术越来越多地被用于动态资源分配和电力管理。然而，有限的机载资源和潜在的对抗干扰对效率和稳健性都构成挑战。为解决这些问题，我们利用数字孪生技术准确捕捉用户卫星可见性的时空动态，从而为决策提供精确的状态信息。基于此，我们制定了作为马尔可夫决策过程的联合优化框架，并提出了BRIDGE，即结合集成狄利克雷和甘美-TopK探索的强化学习BH，结合了服务质量（QoS）驱动的子通道调度机制，确保资源分配高效且差异化。该模型的鲁棒性通过三种经典对抗攻击系统性评估。模拟结果表明，所提方法实现了更优的能效、服务吞吐量和公平性，而鲁棒性分析则显示在考虑的有界对抗扰动下表现稳定。

AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

AdaptiveSD 一个具备稳定性意识、运行时自适应的推测解码框架，支持多策略编排，用于CPU约束的LLM推断

• Authors: Sadra Saremi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.03876
• Pdf link: https://arxiv.org/pdf/2607.03876
• Abstract With the rise of small quantized GGUF-based language models and their increasing use for on-device inference tasks, we have seen the growing need for an approach capable of reliably delivering these models at scale even under severe memory bandwidth constraints such as those imposed by pure CPU implementations. Fixed-depth speculative decoding has emerged as one promising technique, but in practice, it often leads to performance degradation due to either bandwidth saturation, instability, or even catastrophic resource exhaustion resulting in system failure. To overcome this problem, we introduce AdaptiveSD, a fully runtime-adaptive speculative decoding framework aimed at ensuring robust, reliable execution across the spectrum of model types and workloads. Our solution consists of four tightly-coupled components working together in a continuous feedback loop: a Runtime Monitoring Engine tracking multiple signals relevant to ongoing computation, an Adaptive Draft Controller enforcing an eleven rule policy hierarchy prioritizing system resource preservation over raw draft count, a Dynamic Policy Engine employing a suite of heuristic and reinforcement learning techniques to dynamically modify policies depending upon workload behavior, and finally, a KV Cache Coordination Layer managing cache states with fine-grained control through INT8 shadow buffers and position aware evictions. While conventional approaches focus solely on maximizing throughput, we instead assess the effectiveness of our approach based on several key metrics including wasted drafted compute and inter-token latency dispersion alongside standard measures of speculative efficiency.
• 中文摘要 随着基于 GGUF 的小型量化语言模型的兴起及其在设备端推理任务中的日益广泛应用，我们看到了一种能够在严格内存带宽限制（如纯 CPU 实现）下，也可靠地大规模交付这些模型的方法的需求日益增长。固定深度推测解码已成为一种有前景的技术，但实际上，由于带宽饱和、不稳定性，甚至灾难性资源枯竭导致性能下降，系统故障。为解决这一问题，我们引入了AdaptiveSD，一个完全运行时自适应的推测解码框架，旨在确保在各种模型类型和工作负载范围内都能稳健可靠地执行。我们的解决方案由四个紧密耦合的组件组成，形成连续反馈循环：一个运行时监控引擎，跟踪与持续计算相关的多重信号;一个自适应草案控制器，执行十一条规则策略层级结构，优先保留系统资源而非原始草稿计数;一个动态策略引擎，采用一套启发式和强化学习技术，根据工作负载行为动态修改策略， 最后，是KV缓存协调层，通过INT8阴影缓冲区和位置感知驱逐，对缓存状态进行细粒度控制。传统方法专注于最大化吞吐量，而我们则基于多个关键指标评估方法的有效性，包括浪费的草稿计算和令牌间延迟分散，以及标准的投机效率指标。

CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning

CDCP：带上下文提示的条件扩散模型，用于多任务离线安全强化学习

• Authors: Jiayi Guan, Tianle Zhang, Li Shen, Ruiqi Zhang, Ao Zhou, Lusong Li, Guai Chen, Mengjie Li, Alois Knoll, Xiaodong He, Changjun Jiang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.03903
• Pdf link: https://arxiv.org/pdf/2607.03903
• Abstract Multi-task offline safe reinforcement learning (RL) promises to learn a shared optimal safe policy from offline data across multiple tasks. This paradigm provides an effective means for the widespread application of RL in multi-task scenarios with high risk and interaction costs. However, the triple challenges of multi-tasking, safety constraints, and out-of-distribution (OOD) actions pose a significant hurdle for existing methods to ensure safety while maximizing reward returns. In this work, we propose a Conditional Diffusion model with Contextual Prompts (CDCP) to address these challenges. Concretely, we first rethink the requirements and challenges in current multi-task decision-making and control scenarios and establish the objectives of multi-task offline safe RL. Subsequently, we transform the multi-task constrained optimization problem into a conditional generation problem using the diffusion model. Based on this, we design a classifier-free guided cost-constraint strategy to provide flexible cost constraints and eliminate extrapolation errors from OOD actions via supervised learning. Additionally, we introduce a novel contextual prompting method to enhance multi-task representation accuracy and adaptability to unseen tasks. A gradient loss synchronization strategy is also introduced to eliminate gradient interference, improving training stability. Finally, extensive experiments demonstrate that the CDCP algorithm exhibits higher performance and safety in multi-task scenarios than the current state-of-the-art baseline methods. It meets different cost constraints without further training, providing a more flexible cost-constraint solution for the multi-task safe RL.
• 中文摘要 多任务离线安全强化学习（RL）承诺从多个任务的离线数据中学习共享的最优安全策略。这种范式为强化学习在高风险和高交互成本的多任务场景中广泛应用提供了有效手段。然而，多任务处理、安全限制和分销外（OOD）行动的三重挑战，为现有方法在保障安全同时最大化奖励回报构成了重大障碍。本研究提出带上下文提示的条件扩散模型（CDCP）以应对这些挑战。具体来说，我们首先重新思考当前多任务决策和控制场景中的需求和挑战，并确立多任务离线安全强化学习的目标。随后，我们将多任务约束优化问题利用扩散模型转换为条件生成问题。基于此，我们设计了一种无分类器的引导成本-约束策略，通过监督学习提供灵活的成本约束，消除OOD动作的外推误差。此外，我们还引入了一种新型情境提示方法，以提升多任务表示的准确性和对未见任务的适应性。还引入了梯度损耗同步策略以消除梯度干扰，提升训练稳定性。最后，大量实验表明CDCP算法在多任务场景中表现优于当前最先进的基线方法。它满足不同的成本约束，无需进一步培训，为多任务安全强化学习提供了更灵活的成本约束解决方案。

TokAN: Accent Normalization Using Self-Supervised Speech Tokens

TokAN：使用自我监督语音标记进行口音规范化

• Authors: Qibing Bai, Shuai Wang, Yuhan Du, Bohan Li, Yannan Wang, Haizhou Li
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2607.03928
• Pdf link: https://arxiv.org/pdf/2607.03928
• Abstract Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.
• 中文摘要 口音归一化（AN）旨在将非母语（L2）带口音的语音转换为标准（L1）语音，同时保持说话者的身份。现有技术要么需要自然录制的L1-L2平行语音进行训练，要么在合成靶监督下质量下降。本文介绍了TokAN，一种基于令牌的重音规范化框架，运行于从L1-L2联合训练向量量化（VQ）分词器中提取的自监督离散语音令牌，无需合成监督语音。自回归编码器-解码器模型执行令牌间转换，将带L2重音的令牌序列转换为标准语音的令牌。我们还引入基于群体相对策略优化（GRPO）的强化学习（RL），使用词错误率和重音分类器置信度作为补充奖励。非自回归的流匹配合成器会从转换后的标记中恢复Mel频谱图，条件是基于源扬声器嵌入。我们还开发了支持总时长感知综合的流量匹配时长预测器，使TokAN适用于对时长至关重要的任务，如配音和现场直播。对七种英语口音的实验表明，TokAN在监督微调后将词误率从12.40%降至9.89%，强化学习后进一步降至9.23%，在口音减少和理解度方面持续优于帧对帧、直接流匹配和基于提示的令牌转换基线。

Target-Aware Interaction-Guided Reinforcement Learning for Black-Box Node Injection Attacks on Graph Neural Networks

针对图神经网络上黑箱节点注入攻击的目标感知交互引导强化学习

• Authors: Yi Lan, Ye Yuan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.04091
• Pdf link: https://arxiv.org/pdf/2607.04091
• Abstract Graph Neural Networks (GNNs) have achieved remarkable performance in graph representation learning, yet their inherent vulnerability to adversarial attacks poses severe security risks. Especially, black-box node injection attacks have become a major threat to GNNs since they inject malicious nodes without altering the original graph topology. However, they typically decouple the generation of malicious node features and edge connections, thereby resulting in suboptimal attack efficacy under stringent budgets. To address this critical issue, this study proposes a novel Target-aware Interaction-guided Reinforcement learning for Black-box node injection Attacks on GNNs (TIRBA), which formulates the attack as a Markov Decision Process and jointly optimizes node feature generation and edge construction in a heterogeneous action space. Firstly, TIRBA designs a target-aware interaction encoder to fuse information of node features and edges. Further, it introduces a class-center guidance mechanism to utilize prior class distribution information, thereby guiding efficient exploration of the high-dimensional feature space. Finally, a topology difference-aware state value evaluation is adopted to explicitly capture local structural anomalies caused by injected nodes, thereby stabilizing the reinforcement learning training process. Experimental results demonstrate that the proposed TIRBA significantly outperforms state-of-the-art black-box node injection attack methods.
• 中文摘要 图神经网络（GNN）在图表示学习方面取得了显著性能，但其对对抗性攻击的固有脆弱性带来了严重的安全风险。尤其是，黑箱节点注入攻击已成为GNN面临的主要威胁，因为它们注入恶意节点而不改变原始图拓扑。然而，它们通常会解耦恶意节点特征和边缘连接的生成，导致在严格预算下攻击效率不理想。为解决这一关键问题，本研究提出了一种针对GNNs黑箱节点注入攻击（TIRBA）的新型目标感知交互引导强化学习方法，该方法将攻击表述为马尔可夫决策过程，并在异构动作空间中共同优化节点特征生成和边缘构建。首先，TIRBA设计了一个目标感知交互编码器，用于融合节点特征和边缘的信息。此外，它引入了类中心引导机制，利用先验类分布信息，从而指导高效探索高维特征空间。最后，采用拓扑差分感知状态值的评估，以显式捕捉由注入节点引起的局部结构异常，从而稳定强化学习训练过程。实验结果表明，所提的TIRBA在最先进的黑匣子节点注入攻击方法中表现显著优异。

Forethought: Verifiable Reasoning from Neurosymbolic Primitive Programming

前瞻性思考：神经符号原始编程的可验证推理

• Authors: Vishvesh Bhat, Jay Vaghasiya, Emmanuel Anaya Gonzalez
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04096
• Pdf link: https://arxiv.org/pdf/2607.04096
• Abstract Current agentic workflows usually involve decomposing user requests into sequences of tool calls with correctly resolved parameters, the results of which are processed through reasoning traces in the language model's context window. The prevailing route to improve such reasoning is test-time scaling, which trains models to search over long chains of thought; but the resulting capability is entangled in model weights, is not verifiable step-by-step, and is costly at inference. We present Forethought, a neurosymbolic reasoning system that instead treats reasoning as an explicit, verifiable program, that builds from a library of symbolic and neural primitives which are composed through a domain-specific language. The result are reasoning programs, which are concrete representations of the model's work, and as such can be inspected and modified before deployment. Instantiated as a tool-calling execution kernel and evaluated across five benchmarks, Forethought improves base-model accuracy by about 30% relative and outperforms vanilla prompting, reinforcement learning scaffolds, and prompt-evolution methods, enabling small models to match or exceed frontier models capabilities. In a direct comparison, a non-reasoning model augmented with Forethought competes with a dedicated reasoning model while requiring roughly three orders of magnitude less post-training investment, and remains model-agnostic and auditable.
• 中文摘要 当前的代理工作流通常涉及将用户请求分解为参数正确解析的工具调用序列，这些调用的结果通过语言模型上下文窗口中的推理轨迹进行处理。改进此类推理的主流途径是测试时间尺度，它训练模型在长长的思维链中进行搜索;但所得能力与模型权重纠缠，无法逐步验证，且推理成本高昂。我们介绍了Forethought，一种神经符号推理系统，将推理视为一个显式且可验证的程序，构建自一个通过领域特定语言组成的符号和神经原语库。结果是推理程序，这些程序是模型工作的具体表示，因此可以在部署前进行检查和修改。Forethought 作为工具调用执行内核，并在五个基准测试中进行了评估，其相对基础模型准确率提升约 30%，并优于原版提示、强化学习支架和提示演化方法，使小模型能够匹敌甚至超越前沿模型的能力。直接对比中，非推理模型加以前瞻思考（Forethought）增强后，与专门推理模型竞争，且训练后投入需求约少三个数量级，且保持模型无关性和可审计性。

SOV-CAD: Stepwise Orthographic Views Guided CAD Modeling Sequence Reconstruction

SOV-CAD：逐步正交视图引导式CAD建模序列重建

• Authors: Zhaopeng Feng, Chen Zhi, Xuhong Zhang, Zhengwen Feng, Xinkui Zhao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04119
• Pdf link: https://arxiv.org/pdf/2607.04119
• Abstract Reconstructing Computer-Aided Design (CAD) modeling sequences from images is crucial for preserving design intent and supporting parametric editing. However, existing methods typically generate full CAD sequences holistically, overlooking the iterative, feedback-driven nature of human design workflows. We address this limitation by introducing the rich stepwise visual supervision: at each modeling step, the system observes the target's orthographic projections, the projections of the incrementally constructed model, and the active sketch, enabling informed action selection. To effectively leverage this on-the-fly feedback, we propose SOV-CAD, a framework that formulates CAD reconstruction as a sequential decision-making task and employs offline reinforcement learning with a Decision Transformer architecture. This design incorporates continuous visual feedback guided by geometric alignment rewards, resulting in a more accurate and human-like modeling process. Extensive experiments show that SOV-CAD surpasses state-of-the-art methods in CAD sequence reconstruction while exhibiting strong data efficiency. Code of SOV-CAD is available at: this https URL
• 中文摘要 从图像中重建计算机辅助设计（CAD）建模序列对于保持设计意图和支持参数化编辑至关重要。然而，现有方法通常整体生成完整的CAD序列，忽视了人工设计工作流程中迭代、反馈驱动的特性。我们通过引入丰富的逐步视觉监督来解决这一限制：在每个建模步骤中，系统观察目标的正交投影、增量构建模型的投影以及主动草图，从而实现有根据的动作选择。为了有效利用这种即时反馈，我们提出了SOV-CAD框架，该框架将CAD重建构建为顺序决策任务，并采用决策转换器架构的离线强化学习。该设计采用了由几何对齐奖励引导的持续视觉反馈，使建模过程更加准确且人性化。大量实验表明，SOV-CAD在CAD序列重建方面超越了最先进的方法，同时展现出强大的数据效率。SOV-CAD代码可在以下链接获取：此 https URL

Mask-based Predictive Representations for Reinforcement Learning

基于掩膜的预测表征用于强化学习

• Authors: Kai Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04153
• Pdf link: https://arxiv.org/pdf/2607.04153
• Abstract Vision-based deep reinforcement learning involves dealing with high-dimensional inputs of image information. It is crucial to abstract effective states from high-dimensional image inputs and limited samples for sample-efficient reinforcement learning. To address this challenge, inspired by fields such as natural language processing and computer vision, we propose a self-supervised task based on mask prediction as an auxiliary task for reinforcement learning. This non-reconstruction method uses the sequence information collected by the agent from the environment and the context information in the sequence to predict the masked information, thereby strengthening the agent's understanding of the task and learning effective representations. Combined with transformers, we find that the model reconstructs the masked input sequence in the latent space. By feeding the compressed representations learned by this method into reinforcement learning models, we observe an improvement in the sample efficiency of reinforcement learning. Moreover, the model outperforms state-of-the-art sample-efficient reinforcement learning methods on multiple continuous and discrete control benchmarks.
• 中文摘要 基于视觉的深度强化学习涉及处理高维图像信息输入。从高维图像输入和有限样本中抽象有效状态对于样本高效强化学习至关重要。为应对这一挑战，受自然语言处理和计算机视觉等领域的启发，我们提出了基于掩码预测的自监督任务，作为强化学习的辅助任务。这种非重建方法利用代理从环境中收集的序列信息和序列中的上下文信息来预测掩蔽信息，从而增强代理对任务的理解并学习有效表征。结合变换器，我们发现模型能够在潜空间中重建蒙蔽输入序列。通过将该方法学习到的压缩表征输入强化学习模型，我们观察到强化学习样本效率的提升。此外，该模型在多个连续和离散控制基准测试中优于最先进的样本高效强化学习方法。

BrownoutMoE: Structure-Aware Expert Grouping for Efficient and Accurate LLM Web-based Services

BrownoutMoE：结构感知专家组，提供高效且准确的基于网络的大型语言模型服务

• Authors: Yi Ding, Minxian Xu, Zhengxin Fang, Kejiang Ye, Chengzhong Xu
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2607.04164
• Pdf link: https://arxiv.org/pdf/2607.04164
• Abstract Mixture-of-Experts (MoE) large language models (LLMs) are increasingly deployed in Web-facing services, where inference must be both accurate and responsive under bursty demand. Although MoE models improve parameter efficiency through sparse expert activation, efficient MoE inference remains challenging in practice. A major reason is the highly imbalanced expert access pattern during inference: a few hot experts process most routed tokens, while many cold experts are rarely activated, leaving GPU parallelism underutilized. Existing systems mainly optimize runtime execution, such as scheduling, communication overlap, and kernel fusion, but usually preserve the original expert organization and therefore do not address the structural inefficiency caused by fragmented expert usage. In this paper, we present \textbf{BrownoutMoE}, a structure-aware optimization framework for efficient and accurate MoE inference services. Inspired by the brownout paradigm in service computing, BrownoutMoE reorganizes experts into groups to improve utilization and system efficiency while maintaining service quality. Specifically, we formulate layer-wise expert grouping as a learning problem and employ reinforcement learning to discover grouping strategies that minimize accuracy degradation. We further introduce a grouping-consistent distillation process to produce deployable models that are compatible with standard inference pipelines. Experimental results demonstrate that BrownoutMoE reduces accuracy degradation by up to 71.4% and improves throughput by up to 2.24x over baselines.
• 中文摘要 专家混合（MoE）大型语言模型（LLMs）越来越多地被部署在面向网络的服务中，推理必须在突发需求下既准确又响应迅速。尽管MoE模型通过稀疏的专家激活提高了参数效率，但高效的MoE推断在实际中仍然具有挑战性。一个主要原因是推理过程中专家访问模式高度不平衡：少数热专家处理大多数路由令牌，而许多冷专家很少被激活，导致GPU并行性未被充分利用。现有系统主要优化运行时执行，如调度、通信重叠和内核融合，但通常保留原始专家组织，因此未能解决专家使用分散带来的结构性低效问题。本文介绍了 \textbf{BrownoutMoE}，一个结构感知优化框架，用于高效且准确的 MoE 推断服务。受服务计算中棕色化范式的启发，BrownoutMoE将专家重新划分为小组，以提升利用率和系统效率，同时保持服务质量。具体来说，我们将分层专家分组作为学习问题提出，并运用强化学习发现能够最大限度降低准确率下降的分组策略。我们还引入了分组一致性的蒸馏过程，以生成兼容标准推理流水线的可部署模型。实验结果显示，BrownoutMoE可将精度下降降低多达71.4%，吞吐量提升幅度可达基线的2.24倍。

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

CritiqueDriveVLM：从验证器引导强化学习到自动驾驶潜在思维提炼

• Authors: Zhaohong Liu, Hao Ye, Xianlin Zhang, Mengshi Qi
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04179
• Pdf link: https://arxiv.org/pdf/2607.04179
• Abstract End-to-end Vision-Language Models (VLMs) show immense potential in autonomous driving. However, standard Supervised Fine-Tuning (SFT) often suffers from reasoning hallucinations and conservative biases. While traditional tool-augmented frameworks and Chain-of-Thought (CoT) approaches mitigate these issues, they incur exorbitant token consumption and unacceptable latency, rendering real-time deployment impractical. To resolve this reliability-efficiency trade-off, we propose CritiqueDriveVLM, a novel unified three-stage framework internalizing reasoning directly into the VLM. First, we introduce Critique-Driven Multi-Turn Reinforcement Learning (RL) guided by a multi-dimensional verifier. By providing granular scalar feedback and a multi-turn penalty, we force the policy to internalize logical deduction, cultivating a robust System-2 Teacher that achieves high accuracy without fragile external tools. Subsequently, we propose Latent Thought Distillation to overcome the latency bottleneck. By aligning the Student's latent representations with the Teacher's fully converged reasoning states, we compress deep logical capabilities into a fast, CoT-free System-1 Student. Extensive experiments on the widely-used DriveLMM-01 benchmark demonstrate remarkable improvements. Compared to the base model, our tool-free Teacher significantly boosts Multiple Choice Quality (MCQ) from 55.54% to a state-of-the-art 76.54%. Crucially, our distilled Student preserves competitive reasoning depth while drastically minimizing generation length to an average of merely 28 tokens. This slashes inference latency by 88% (from 3482 ms to 416 ms), paving a highly robust pathway for low-latency autonomous this http URL source code is available at this https URL.
• 中文摘要 端到端视觉语言模型（VLMs）在自动驾驶领域展现出巨大潜力。然而，标准的监督微调（SFT）常常存在推理幻觉和保守偏见。虽然传统的工具增强框架和思维链（Chain-of-Thought，CoT）方法缓解了这些问题，但它们会消耗大量代币和不可接受的延迟，使实时部署变得不切实际。为了解决这种可靠性与效率的权衡，我们提出了CritiqueDriveVLM，这是一种新颖的统一三阶段框架，将推理直接内化到VLM中。首先，我们引入了由多维验证器引导的批判驱动多回合强化学习（RL）。通过提供细致的标量反馈和多回合惩罚，我们迫使政策内化逻辑推理，培养出一个稳健的System-2教师，在不依赖脆弱的外部工具的情况下实现高准确性。随后，我们提出潜在思想蒸馏以克服延迟瓶颈。通过将学生的潜在表征与教师完全收敛的推理状态对齐，我们将深度逻辑能力压缩为快速且无CoT的System-1学生。广泛使用的 DriveLMM-01 基准测试显示出显著的改进。与基础模型相比，我们的无工具教师显著提升了选择质量（MCQ）从55.54%提升到最先进的76.54%。关键是，我们精炼的学生模式保留了竞争推理深度，同时大幅缩短了生成时间，平均仅有28个代币。这使推理延迟减少了88%（从3482毫秒降至416毫秒），为低延迟自治提供了一条非常稳健的路径。该HTTP网址源代码可在该HTTP网址获取。

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

针对代理强化学习的进度与可靠性导向的群体策略优化

• Authors: Mingxuan Fan, Peiyang Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04242
• Pdf link: https://arxiv.org/pdf/2607.04242
• Abstract Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.
• 中文摘要 基于群体的强化学习（RL）已成为提升大型语言模型代理在长期交互任务中的有效范式。为了获得比轨迹级优化更细粒度的策略更新，近期研究转向基于步骤的基于组的强化学习，即将中间步骤分组并在一个部署批次中进行比较。然而，步级优势估计对组的形成方式很敏感：按广义状态键分组可以提升覆盖范围，但可能会比较不同历史下的操作，而强制历史一致性则能带来更公平的比较，但代价是群体分散且缺少对等比较信号。本文提出ProGPO（进步与可靠性导向的群体策略优化），这是一种无批评学习的方法，用于上下文一致的步级学习。ProGPO保持精确前缀动作比较，并用基于推广状态势能的过渡信用补充稀疏的同伴比较。为可靠估计这些势，ProGPO结合语义扩展与跨历史深度的逆方差融合。我们在两个具有挑战性的代理任务ALFWorld和WebShop上评估ProGPO，辅以Qwen2.5-1.5B-Instruct。结果显示，ProGPO在相当的计算开销下优于匹配的agentic RL基线，且额外的Qwen2.5-3B-Instruct实验进一步测试了该方法的可扩展性。

HALO-WA: Hybrid-Attention Latent-Guided Online Reinforcement Learning for World-Action Models

HALO-WA：世界行动模型中的混合注意力潜在引导在线强化学习

• Authors: Angen Ye, Weijie Ke, Xiaofeng Wang, Xinze Chen, Chaojun Ni, Guosheng Zhao, Boyuan Wang, Zheng Zhu, Junjie Xie, Dapeng Zhang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04265
• Pdf link: https://arxiv.org/pdf/2607.04265
• Abstract World-action (WA) models can generate long-horizon action chunks for general-purpose robotic manipulation, but they remain vulnerable to calibration, perception, and contact-dynamics errors in real-world precision tasks, often failing in the final few millimeters of alignment or insertion. We propose HALO-WA, a hybrid-attention latent-guided online reinforcement learning (RL) framework for WA models, which leverages latent features and action priors from the WA generation process through a lightweight actor-critic adapter to enable fast online adaptation to real deployment errors. HALO-WA introduces a hybrid-attention structure that preserves the temporal consistency of action chunks while reading task-relevant information from WA latents conditioned on visual context and end-stage correction requirements, thereby producing refined action chunks. We validate HALO-WA on four real-world precision manipulation tasks, where it improves the average success rate from 26.4\% for WA-base to 87.1\%, outperforming the strongest baseline by 19.2 percentage points while requiring only 45--75 minutes of online training per task. To facilitate reproducibility, we further conduct supplementary simulation experiments in RoboTwin and release the code at this https URL.
• 中文摘要 世界动作（WA）模型可以生成用于通用机器人操作的长视距动作块，但在实际精密任务中仍易受到校准、感知和接触动力学误差的影响，常在对准或插入的最后几毫米失效。我们提出了HALO-WA，一种混合注意力潜在引导在线强化学习（RL）框架，适用于WA模型，通过轻量级actor-critic适配器利用WA生成过程中的潜在特征和动作先验，实现对真实部署错误的快速在线适应。HALO-WA引入了一种混合注意力结构，在读取基于视觉上下文和末期修正需求的WA潜在信息的任务相关信息的同时，保持动作块的时间一致性，从而产生精细的动作块。我们在四个真实世界的精密操作任务中验证了HALO-WA，其平均成功率从WA基础的26.4%提升至87.1%，比最强基线高出19.2个百分点，且每个任务只需45至75分钟的在线训练。为了促进重复性，我们在RoboTwin中进行了补充的模拟实验，并以该https URL发布代码。

On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models

关于奖励函数在强化学习中对大型语言模型置信度校准的有效性

• Authors: Chee Heng Tan, Zhuoyi Lin, Mehul Motani, Wee Sun Lee
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.04332
• Pdf link: https://arxiv.org/pdf/2607.04332
• Abstract In this paper, we consider the setting where large language models (LLMs) are trained using reinforcement learning (RL) to simultaneously improve reasoning accuracy and verbalize its confidence. Our reward scheme uses two functions for rewarding confidence verbalized by the LLM: one when the LLM is correct and a different one when the LLM is incorrect. With a poorly designed reward scheme, the LLM may be incentivized to answer incorrectly so that it can be confident that its answer is indeed incorrect, a phenomenon that we call confidence reward hacking. We propose the concept of non-hackable confidence reward schemes and define a spectrum of such reward schemes for RL confidence calibration training in LLMs. We demonstrate that selective confidence reward hacking can occur in practical datasets with reward schemes that are not designed to be non-hackable. We also demonstrate that the reward scheme with the best calibration to accuracy tradeoff depends on the dataset and the application, and propose using the reward scheme as a hyperparameter to optimize the tradeoffs in accordance to what is important for the application. The code of our experiments is available in this https URL.
• 中文摘要 本文探讨了大型语言模型（LLM）如何通过强化学习（RL）训练，以同时提高推理准确性并表达其信心。我们的奖励方案使用两种功能来奖励LLM表达的信心：一种是LLM正确时，另一种是错误。在设计不良的奖励方案中，LLM可能会被激励去错误回答，以便确信答案确实错误，这种现象我们称之为信心奖励黑客。我们提出了不可被入侵的置信奖励方案概念，并定义了用于大型语言模型中强化学习置信度校准训练的此类奖励方案的光谱。我们证明，选择性信心奖励黑客可以在实际数据集中发生，而奖励方案并非设计成不可被黑客攻击。我们还证明了在校准与准确性之间权衡最佳的奖励方案取决于数据集和应用，并建议将奖励方案作为超参数，根据应用的重点优化权衡。我们的实验代码可在该 https URL 中获取。

RL Forgets! Towards Continual Policy Optimization

现实中忘记了！迈向持续的政策优化

• Authors: Mao-Lin Luo, Zhe-Xu Wang, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, Tong Wei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.04364
• Pdf link: https://arxiv.org/pdf/2607.04364
• Abstract Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks. Recent work has increasingly favored reinforcement learning over supervised fine-tuning, driven by the belief that reinforcement learning is inherently less prone to forgetting. However, the belief remains insufficiently validated, as existing evidence is largely drawn from outdated or homogeneous benchmarks. To revisit this assumption, we introduce MRCL, a Multimodal Reasoning Continual Learning benchmark built from diverse and recently released multimodal datasets. Experiments on MRCL show that reinforcement learning can still suffer from severe catastrophic forgetting during continual post-training. To address this challenge, we propose Continual Policy Optimization (CPO), a replay-free framework grounded in the prior-task behavioral KL objective. CPO uses a theoretically justified parameter-movement regularization to limit policy drift on previous tasks. Extensive experiments across multiple model scales demonstrate that CPO consistently reduces forgetting while preserving, and in some cases improving, pretrained model capabilities. On Qwen3-VL-8B, CPO reduces forgetting by 13.7\% and improves pretrained capability by 7.0\%. The implementation code is available at this https URL.
• 中文摘要 持续的后期训练正成为适应视觉语言模型以适应不断演变任务的核心范式。近期研究越来越倾向于强化学习而非监督微调，基于强化学习本质上较少遗忘的信念。然而，这一信念尚未得到充分验证，因为现有证据大多来自过时或同质性的基准。为了重新审视这一假设，我们介绍了MRCL，这是一个由多样且近期发布的多模态数据集构建的多模态推理持续学习基准测试。MRCL实验显示，强化学习在持续的后训练过程中仍可能出现严重的灾难性遗忘。为应对这一挑战，我们提出了持续策略优化（CPO），这是一个基于先验任务行为的持续策略优化目标的无重放框架。CPO采用理论上合理的参数移动正则化来限制先前任务的策略漂移。跨多个模型尺度的广泛实验表明，CPO在保持甚至在某些情况下提升预训练模型能力的同时，持续减少遗忘。在Qwen3-VL-8B上，CPO将遗忘率降低了13.7%，预训练能力提升了7.0%。实现代码可在此 https URL 获取。

A Perception-Manipulation Robotics System for Food Cutting

一种用于切割食物的感知-操控机器人系统

• Authors: Xinyuan Luo, Wenzhen Yuan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.04367
• Pdf link: https://arxiv.org/pdf/2607.04367
• Abstract In the development of cooking robots, mastering the task of cutting is crucial. A significant challenge lies in the diverse properties of food, which necessitate distinct cutting policies and even different knives for optimal processing. This paper presents a perception-manipulation framework for food-cutting tasks. Our system features a knife selection module that utilizes force data from a preliminary fixed trial cut to select the appropriate knife for the given food. This is followed by an adaptive cutting phase using reinforcement learning (RL) to balance cutting speed and energy efficiency. In our experiments, the knife selection module achieved 100% successful rate on unseen food, and we compared the performances of fixed policy, RL policy, with human operators. Our method not only achieves high performance but also demonstrates comparable results to those of human participants.
• 中文摘要 在烹饪机器人的发展中，掌握切割这一任务至关重要。一个重大挑战在于食品的多样性，这需要不同的切割策略，甚至不同的刀具以实现最佳加工。本文提出了一种用于食物切割任务的感知-操控框架。我们的系统配备了刀具选择模块，利用初步固定试切的力数据来选择适合该食物的刀具。随后进入自适应切割阶段，利用强化学习（RL）平衡切割速度和能效。在我们的实验中，刀具选择模块在看不见的食物上实现了100%的成功率，并将固定策略和强化政策的表现与人类操作员进行了比较。我们的方法不仅实现了高性能，还展示了与人类参与者相当的结果。

Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

Nemotron-Labs-3-Puzzle-75B-A9B：压缩混合MoE大型语言模型

• Authors: Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy, Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta Stepniewska-Dziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04371
• Pdf link: https://arxiv.org/pdf/2607.04371
• Abstract We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability.
• 中文摘要 我们介绍Nemotron-Labs-3-Puzzle-75B-A9B，这是Nemotron-3-Super的压缩版本，优化为交互式部署。我们设计模型时，是在高用户吞吐量约束下最大化服务器吞吐量。在单个8xB200节点的交互式服务工作负载中，Puzzle-75B-A9B在匹配用户吞吐量约束下，服务器吞吐量约为Nemotron-3-Super的2倍。在单一 H100 GPU 上的超长上下文部署中，压缩模型将 1M 令牌并发从1请求提升到8请求。Puzzle-75B-A9B 采用多阶段流水线构建，结合了 Iterative Puzzle 压缩框架、知识提炼、强化学习、量化和多标记预测接口。压缩过程联合优化异构 MoE 剪枝、主动参数预算和 Mamba 剪枝，以提升推理效率同时保持模型质量。我们基于广泛的推理、编码、多语言、长上下文和智能基准测试来评估Puzzle-75B-A9B。尽管进行了大量压缩，该模型在多种任务中相对于母模型仍保持了较强的下游准确性。这些结果表明，大型混合MoE模型可以在保持强大下游能力的同时，大幅优化部署效率。

Auto-AEG: Scalable Data Construction for Open-Vocabulary Audio Event Grounding

Auto-AEG：用于开放词汇音频事件基础的可扩展数据构建

• Authors: Zihan Zhang, Xize Cheng, Wenhao Yan, Tong Zhang, Dongjie Fu, Boyun Zhang, Yongbo He, Tao Jin
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04383
• Pdf link: https://arxiv.org/pdf/2607.04383
• Abstract Large Audio-Language Models (LALMs) reason fluently about sound yet struggle to localize precisely when events occur, while classical Sound Event Detection attains frame-level precision only over a closed label set. At the intersection of these paradigms lies the task of Open-Vocabulary Audio Event Grounding: predicting all time intervals of a target sound event described by an arbitrary natural language query. While this task is crucial for real-world audio understanding and LALM adaptation, it is bottlenecked by data scarcity. Few large-scale resources provide open-vocabulary onset/offset supervision, and manual temporal annotation is prohibitively expensive. To address this, we introduce Auto-AEG, a scalable pipeline that constructs such supervision by automatic data construction and model fine-tuning. It pairs programmatically synthesized clips, which carry exact ground-truth intervals for supervised cold-start, with multi-model pseudo-labels on real-world audio that supply the reward signal for reinforcement learning. Training with this pipeline yields promising performance gains on both the DESED SED benchmark and AEGBench, an independent difficulty-stratified benchmark we release. Our results show that automatically constructed data, coupled with interval-aware reward function design, is an effective data-side route to expanding the temporal localization capability of LALMs.
• 中文摘要 大型音频语言模型（LALMs）能够流畅地推理声音，但难以准确定位事件发生时间，而传统的声音事件检测仅在封闭标签集上达到帧级精度。在这些范式的交汇处，有开放词汇音频事件基础的任务：预测由任意自然语言查询描述的目标声音事件的所有时间间隔。虽然这项任务对实际音频理解和LALM适配至关重要，但数据稀缺性使其成为瓶颈。很少有大型资源提供开放词汇的起始/偏移监督，且手动时间标注成本高昂。为此，我们引入了Auto-AEG，一种可扩展的管道，通过自动构建数据和模型微调构建此类监管。它将程序合成的片段与提供奖励信号的真实音频（用于监督冷启动）与现实音频上的多模型伪标签配对，后者为强化学习提供奖励信号。使用该流程进行训练，在DESED SED基准和我们发布的独立难度分层基准AEGBench上均取得了令人期待的性能提升。我们的结果表明，自动构建数据结合区间感知奖励函数设计，是扩展LALM时间定位能力的有效数据端途径。

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

LLM作为导师：针对不可验证强化学习的策略意识提示适应

• Authors: Yujin Kim, Namgyu Ho, Sangmin Hwang, Joonkee Kim, Yongjin Yang, Sangmin Bae, Seungone Kim, Jaehun Jung, Se-Young Yun, Hwanjun Song
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04412
• Pdf link: https://arxiv.org/pdf/2607.04412
• Abstract Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.
• 中文摘要 针对不可验证指令跟随的强化学习（RL）越来越依赖带有提示特定规律的LLM评判作为奖励信号。虽然最新方法在培训期间调整这些评分标准以适应不断演变的政策，但培训提示本身仍是固定的，取自固定语料库。这种静态方法常导致提示难度与策略能力之间出现严重错位，导致评委无法在提示未能引发推展间质量差异时，恢复具有辨别性的奖励信号。为解决这种不一致，我们引入了“LLM即导师”框架，将LLM的角色从评判扩展到导师：单一模型作为检验者，成对策略推展进行比较以检测非挑战性提示，同时作为生成器附加原子约束。这种仅附录设计单调地随着策略能力提升难度，产生无外部困难安排的自我校准训练信号。在三个复杂的指令跟随基准测试中，我们的方法始终优于无策略意识的基线和以往调整评分标准或重写提示的策略适应方法，表明提示适应是不可验证强化学习中策略意识缺失的轴。

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

dOPSD：扩散语言模型的策略自蒸馏

• Authors: Phuong Tuan Dat, Qi Li, Xinchao Wang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04428
• Pdf link: https://arxiv.org/pdf/2607.04428
• Abstract Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains difficult: supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods. On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacher privileged information (PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process; on Dream and LLaDA, dOPSD improves both in-domain math reasoning and out-of-domain code generation, outperforming supervised and on-policy baselines.
• 中文摘要 扩散大型语言模型（dLLMs）通过迭代去噪声掩蔽序列生成文本，提供了自回归模型的平行替代方案，但通过后训练引发强有力推理仍然困难：监督微调不符合策略且存在暴露偏差，而强化学习仅提供稀疏的序列级奖励，且难以在无可处理序列似然的情况下应用。政策自我提炼（OPSD）提供了一个有前景的替代方案，利用学生和教师的模型，提供密集的、代币级别的政策监督，但其有效性依赖于给予教师特权信息（PI）——通常是推理时无法获得的实例特定实地真理引用——因此学生最终提炼出一个薄弱的无PI共识策略，对dLLM推理几乎没有改善。我们引入了dOPSD，它直接从学生自身的去噪轨迹中获得教师特权，通过同一轨迹的后续更解码步骤来评估掩蔽位置，而非外部标签，因此教师的优势来自模型自身的解码过程;在Dream和LLaDA上，dOPSD提升了域内数学推理和域外代码生成，优于监督和策略基线。

Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world models

算子-函数补足价值-等价：潜在世界模型的规划时间诊断

• Authors: Donna Vakalis
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04464
• Pdf link: https://arxiv.org/pdf/2607.04464
• Abstract World-model evaluation for model-based reinforcement learning typically asks whether the learned model predicts reward and value well, which can leave planning-relevant errors in the model's latent rollouts unmeasured. We introduce a complementary diagnostic, operator-on-F, that compares a model's k-step latent pushforward to the environment's on an observable subset F, using the model's own predictor. On a TD-MPC2 size sweep over cheetah-run, reward-prediction error stays within [0.028, 0.091] for every model size - only about 3x variation - so an unnormalized reward-fit check has narrow resolution to distinguish them; the (unnormalized) Bellman residual and reward error themselves have weak relationships with return (Spearman -0.10 and -0.30). Operator error spans 0.28 to 2.62 over the same sizes. At 317M the operator error is 2.62 - an order of magnitude above the 0.28-0.36 cluster - and the planning return collapses to 0.9, while reward-prediction error (0.091) is the highest of the five but stays within the same small [0.028, 0.091] range as the rest of the sweep. The rank correlation between operator error and return loss is -0.90 (anchor-bootstrap 95% CI [-0.90, -0.70] at n=5 sizes; leave-one-out removal of any single size leaves it at -0.80 or stronger). The operator also returns informative, architecture-discriminating estimates in a cross-architecture comparison between TD-MPC2 and a pure-SSL latent world model. The operator diagnostic complements value-equivalence rather than replacing it.
• 中文摘要 基于模型的强化学习世界模型评估通常会询问所学模型是否能很好地预测奖励和价值，这可能导致模型潜在推广中的规划相关误差未被测量。我们引入了一个互补的诊断方法——F算子，它利用模型自身的预测变量，将模型的k步潜在推前推与环境在可观测量子集F上的推前推量进行比较。在对猎豹跑道进行TD-MPC2尺寸扫描时，奖励预测误差在[0.028， 0.091]以内——仅约3倍变化——因此非归一化的奖励拟合检验具有较窄的区分分辨率;（未归一化的）贝尔曼残差误差和奖励误差本身与回报的关系较弱（斯皮尔曼 -0.10 和 -0.30）。在相同尺寸下，操作误差范围为0.28至2.62。在317M处，操作误差为2.62——比0.28-0.36簇高出一个数量级——规划回报率缩减至0.9，而奖励-预测误差（0.091）是五者中最高的，但仍与扫描其他部分相同的小范围[0.028， 0.091]。操作误差与回波损耗的秩相关数为-0.90（锚点引导95% CI [-0.90， -0.70]，n=5个尺寸时;去除任意单一尺寸时，回波损耗为-0.80或更大）。操作员还返回了具有信息量、具有分辨性的结构估计，用于TD-MPC2与纯SSL潜在世界模型的跨架构比较。算符诊断是对价值等价的补充，而非取代它。

Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning

LLM增强型合作多智能体强化学习的方案条件稳定

• Authors: Faid Keddouri, Sohaib Houhou, Aissa Boulmerka, Nadir Farhi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.04470
• Pdf link: https://arxiv.org/pdf/2607.04470
• Abstract Large Language Models (LLMs) offer a natural interface for translating human objectives into reward signals for cooperative multi-agent reinforcement learning (MARL), yet the training-time dynamics of this integration remain poorly understood. We show that dynamically updating LLM-generated reward weights during off-policy MARL violates the stationarity assumption of Potential-Based Reward Shaping (PBRS) and contaminates the experience replay buffer, whose stored transitions carry reward labels computed under stale shaping weights. We characterise the result as a regime-dependent failure whose severity depends on how competent the unshaped baseline already is. To control it we propose two stabilisation strategies: a Phase-Based Freeze Schedule that enforces strict stationarity within training phases, and Exponential Moving Average (EMA) smoothing that bounds per-episode weight drift. We evaluate across three cooperative environments and five random seeds with QMIX, complemented by an exploratory VDN extension, yielding a three-regime taxonomy. In the augmentative regime (Simple Spread), where the baseline is functional (74.4 %), EMA significantly improves success to 86.7 % ($+12.3$ pp, $p<0.01$) while naive dynamic updates collapse it to 15.2 %. In the essential regime (Level-Based Foraging), where the baseline is broken (0.1 %), any shaping unlocks the task (95.9 % under EMA). In the supplementary regime (SMAC 3m), where the baseline is near-saturated (98.8 %), stabilised shaping preserves performance (99.9 %) while unstabilised shaping adds variance without gain. These findings establish reward-signal stationarity as a necessary design constraint and indicate that regime placement is a practical predictor of whether dynamic LLM shaping helps or harms.
• 中文摘要 大型语言模型（LLMs）为将人类目标转化为合作多智能体强化学习（MARL）的奖励信号提供了自然的接口，但这种集成的训练时间动态仍鲜为人知。我们表明，在非策略MARL期间动态更新LLM生成的奖励权重，违反了基于势能的奖励塑形（PBRS）的平稳性假设，并污染了经验回放缓冲区，后者存储的过渡带有在陈旧整形权重下计算的奖励标签。我们将结果描述为一种依赖体制的失败，其严重程度取决于未成形基线的现状。为控制其，我们提出了两种稳定策略：基于阶段的冻结计划，强制训练阶段内严格平稳;以及指数移动平均（EMA）平滑，限制每集权重漂移。我们通过QMIX在三种协作环境和五个随机种子中进行评估，并辅以探索性VDN扩展，得出三组分类法。在辅助阶段（简单差价）中，基线功能正常（74.4%），EMA显著提升成功率至86.7%（每股+12.3美元，$p<约0.01美元），而简单的动态更新则使其降至15.2%。在基本模式（基于等级采集）中，基线被打破（0.1%），任何塑形都能解锁任务（EMA下95.9%）。在补充区（SMAC 3m）中，基线接近饱和（98.8%），稳定整形保持性能（99.9%），而非稳定整形则增加方差但无增益。这些发现确立了奖励-信号平稳性作为设计的必要约束，并表明制度的分布是动态LLM塑造是有益还是有害的实用预测因子。

VLA Grounder: Language-Conditioning Space Optimization for Black-Box VLA Models

VLA Grounder：黑箱VLA模型的语言条件空间优化

• Authors: Damir Shodiev, Aleksei Staroverov, Nikita Kachaev, Alexey K. Kovalev, Aleksandr I. Panov
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04517
• Pdf link: https://arxiv.org/pdf/2607.04517
• Abstract Vision-Language-Action (VLA) models are commonly treated as end-to-end action policies conditioned on natural-language task descriptions. In practice, however, their behavior often depends sharply on how the instruction is phrased, suggesting that language is not merely a task label but an optimizable conditioning input. We study whether frozen VLA policies can be improved by optimizing language space rather than updating action weights. Our method introduces a language-conditioning space policy that translates a human instruction into a short VLA-grounded command using object appearance, spatial relations, and target-grounding cues. The language-conditioning space policy is initialized with a failure-derived command-space prior and optimized with reinforcement learning from sparse task-completion rewards, while the downstream VLA remains fully frozen. This yields language-conditioning space optimization: RL discovers which VLA-grounded commands best elicit successful behavior from the frozen action policy. Experiments on RL4VLA and VL-Think show that language-conditioning space optimization improves success on instruction-sensitive, symbolic, and multi-object manipulation tasks, demonstrating that language can serve as an optimizable variable for a robot foundation models. Website: this https URL
• 中文摘要 视觉-语言-行动（VLA）模型通常被视为基于自然语言任务描述的端到端行动策略。然而，在实际操作中，它们的行为往往极大地取决于指令的表述方式，这表明语言不仅仅是任务标签，而是一种可优化的条件反射输入。我们研究是否可以通过优化语言空间而非更新动作权重来改进冻结的VLA策略。我们的方法引入了一种语言条件空间策略，利用对象外观、空间关系和目标接地线索，将人类指令转换为短暂的VLA接地命令。语言条件空间策略以失败导出的命令空间先验初始化，并通过从稀疏任务完成奖励中强化学习进行优化，而下游的VLA则保持完全冻结。这带来了语言条件空间优化：强化学习发现哪些基于VLA的命令最能从冻结动作策略中引发成功行为。RL4VLA和VL-Think的实验表明，语言条件空间优化能提升指令敏感、符号和多对象操作任务的成功率，表明语言可以作为机器人基础模型的可优化变量。网站：这个 https URL

Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learning

超越速度需求：通过仿真引导强化学习实现能量感知代码生成

• Authors: Saurabhsingh Rajput, Tushar Sharma
• Subjects: Subjects: Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.04577
• Pdf link: https://arxiv.org/pdf/2607.04577
• Abstract Code models strictly prioritize functional correctness, leaving software energy efficiency as an unoptimized byproduct. Training models to generate energy-efficient code requires reproducible feedback at scale, which physical hardware measurement cannot reliably provide due to variance. In this paper, we replace hardware profiling with a deterministic architectural simulation harness to build Green Tea, a corpus of $3.5$ million evaluations across $1{,}474$ C++ problems. We train an energy-aware code model via supervised fine-tuning on energy-contrastive pairs, followed by closed-loop reinforcement learning (GRPO) using simulation-in-the-loop feedback. To rigorously evaluate deployment readiness, we introduce the Correctness-Adjusted Reduction in Energy Total (CARET), a metric that explicitly penalizes code that sacrifices functionality for efficiency. On $143$ held-out problems, our simulation-in-the-loop pipeline achieves $12.63\%$ CARET, nearly tripling the gain of fine-tuning alone, and successfully beats the energy efficiency of human-expert references on $58.4\%$ of its valid outputs. Furthermore, our analysis exposes the IPC trap: standard throughput proxies like Instructions-Per-Cycle (IPC) actively misrank true energy efficiency on $67.8\%$ of problems, proving the absolute necessity of direct energy simulation. By releasing our dataset and infrastructure, we bypass the $263{,}000$ CPU-hours required for reproduction, structurally empowering the community to deploy inherently energy-efficient code generation models.
• 中文摘要 代码模型严格优先考虑功能正确性，导致软件能效成为未优化的副产品。训练模型生成节能代码需要大规模可重复的反馈，而物理硬件测量由于变异性无法可靠地提供反馈。本文中，我们将用确定性架构仿真工具取代硬件分析，构建了Green Tea，这是一份价值350万美元、涵盖1美元474美元C++问题的评估语料库。我们通过对能量对比对的监督微调训练能量感知代码模型，随后采用仿真环路反馈进行闭环强化学习（GRPO）。为了严格评估部署准备度，我们引入了正确性调整总能量减少（CARET），这一指标明确惩罚那些牺牲功能性以换取效率的代码。在143美元的未解决问题上，我们的模拟环路流程实现了12.63美元（CARET）12.63%$，几乎将微调增益提升三倍，并成功击败了人类专家参考的有效输出58.4%%的能效。此外，我们的分析揭示了IPC陷阱：标准吞吐量代理如每周期指令数（IPC）在价值67.8%美元的问题上主动错误评定真实能效，证明了直接能量仿真的绝对必要性。通过发布我们的数据集和基础设施，我们绕过了复制所需的263{，}000美元CPU小时，结构性地赋能社区部署本质上节能的代码生成模型。

SILO: Simulation-in-the-Loop Sim-to-Real Transfer for Multi-Stage Cable Routing

SILO：多级电缆路由的模拟在环中模拟传输

• Authors: Stone Tao, Jie Xu, Hesam Rabeti, Yashraj Narang, Yijie Guo, Iretiayo Akinola
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04616
• Pdf link: https://arxiv.org/pdf/2607.04616
• Abstract Linear-deformable manipulation remains challenging due to the complex deformations of objects such as cables and ropes. Prior data-driven approaches, particularly imitation learning, have shown some promise in narrowly defined settings but typically require thousands of demonstrations for specific tasks and cable types, limiting scalability and generalization. We introduce a sim-to-real reinforcement learning (RL) framework for multi-stage cable routing that leverages GPU-parallelized simulation to approximate linear deformable behaviors. Training across thousands of parallel simulations enables the learned policies to generalize across diverse cable geometries and deformation patterns. To bridge the sim-to-real gap, we propose a novel deployment strategy that combines a Simulation In the LOop (SILO) execution framework, localized RL policies, and robust cable state estimation. On real-world cable routing tasks, our approach achieves higher success rates and 2x reduction in cycle times compared to prior state-of-the-art learning methods. To our knowledge, this is the first successful sim-to-real transfer of RL policies for multi-stage cable routing. Videos and additional visualizations are available at this https URL
• 中文摘要 由于电缆和绳索等物体的复杂变形，线性变形操作依然具有挑战性。以往的数据驱动方法，尤其是模仿学习，在狭窄的环境中展现出一定的潜力，但通常需要数千次演示以应对特定任务和电缆类型，限制了可扩展性和泛化性。我们引入了一个模拟到现实强化学习（RL）框架，用于多级电缆布线，利用GPU并行化仿真近似线性可变形行为。跨数千个并行仿真进行训练，使所学策略能够在不同的电缆几何形状和变形模式中进行推广。为弥合模拟与现实的差距，我们提出了一种新颖的部署策略，结合了LOop模拟（SILO）执行框架、局部强化强化策略和稳健的电缆状态估计。在现实世界的电缆布线任务中，我们的方法实现了更高的成功率和2倍的周期时间，相较于以往最先进的学习方法。据我们所知，这是多级有线路由中首次成功实现模拟到现实的强化协议策略转移。视频和更多可视化内容可在此 https 网址获取

Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning

视觉-语言-行动模型真的如其说的那样吗？关于忠诚在具身推理中的作用

• Authors: Matthew Foutter, Matteo Cercola, Lena Wild, Yunshan Wang, Michelle Li, Daniele Gammelli, Marco Pavone
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04681
• Pdf link: https://arxiv.org/pdf/2607.04681
• Abstract Embodied Chain-of-Thought has emerged as a promising mechanism to enhance robot decision-making and interpretability in black-box Vision-Language Action (VLA) models. However, whether this verbalized Chain-of-Thought truthfully reflects the policy's underlying decision process remains poorly understood. We distinguish between functional reasoning, in which reasoning improves task performance, and faithful reasoning, in which reasoning truly reflects the policy's internal decision process. We argue that SoTA alignment strategies offer a necessary but insufficient notion of faithfulness, admitting reasoning whose intermediate steps can mask the causal links in action prediction through confounding factors (e.g., reasoning that is ungrounded in the environment and internally disconnected or inconsistent), restricting policy generalization. We study this gap through a human evaluation of a SoTA reasoning model for autonomous driving, revealing an inconsistent coupling between reasoning quality and downstream trajectory improvement. We then operationalize a behavioral surrogate for embodied faithfulness through a learned critic, Pinocchio, scoring observation grounding and stepwise coherence, and use this critic as a dense reward signal in post-training an embodied policy with reinforcement learning. Across withheld driving benchmarks, our post-trained planner improves faithfulness by 4% and 18% over SoTA alignment and trajectory error post-training baselines, respectively, while maintaining competitive downstream task performance. Finally, on a synthetic out-of-distribution test set, post-training for faithfulness improves policy responsiveness to rare counterfactual scenarios by 1.6x that of a SoTA policy, suggesting that faithful reasoning traces contribute to more robust, generalizable, and interpretable embodied intelligence. Project page: this https URL
• 中文摘要 具身思维链已成为一种有前景的机制，用于增强机器人在黑箱视觉语言行动（VLA）模型中的决策和可解释性。然而，这种口头表达的思维链是否真实反映了政策的根本决策过程，仍然不清楚。我们区分功能推理（推理提升任务绩效）和忠实推理（推理真实反映策略内部决策过程）。我们认为，SoTA的对齐策略提供了必要但不足的忠实性概念，承认了某些推理的中间步骤可能通过混杂因素掩盖行动预测中的因果联系（例如，推理不符合环境、内部脱节或不一致），限制了政策的泛化。我们通过对自动驾驶SoTA推理模型的人工评估来研究这一差距，揭示了推理质量与下游轨迹改进之间的耦合不一致。随后，我们通过一位学习型批评者匹诺曹，操作化一个具身忠诚的行为替代，评分观察、扎根和逐步连贯性，并将该批评者作为带有强化学习的具身政策后期的密集奖励信号。在保留驾驶基准中，我们的培训后规划师比训练后目标匹配和轨迹误差基线分别提升了4%和18%的忠实度，同时保持了竞争性的下游任务表现。最后，在合成的分布外测试集上，忠实性后期训练使政策对罕见反事实情景的响应性提升为SoTA策略的1.6倍，表明忠实推理痕迹有助于更稳健、可推广和可解释的具身智能。项目页面：此 https URL

Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas

综合利他主义与公平偏好促进了在连续社会困境中的高级互助

• Authors: Yu Wei, Yukiko Ogura, Yoshiyuki Ohmura, Ildefons Magrans de Abril, Hoshinori Kanazawa, Yasuo Kuniyoshi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04710
• Pdf link: https://arxiv.org/pdf/2607.04710
• Abstract Inducing cooperation among distributed agents is still a difficult problem in the field of multi-agent reinforcement learning (MARL), particularly in social dilemma situations. There, individual interests are misaligned with the common good and individual rationality leads to suboptimal group outcomes. In contrast, humans are able to achieve cooperation with one another in such situations. A common explanation for such cooperative behavior is that individuals have social preferences. In order to achieve cooperation in MARL, we design a new utility function integrating altruistic preferences (incentive for other's reward) and fairness preferences (incentive for equality) from social psychology and behavioral economics, namely, Altruistic and Fairness Preference (AFP), a reward-sharing mechanism which converts one's own and other's rewards to incentives for cooperative behavior. We performed comparative experiments with standard RL and inequity aversion agents in two challenging sequential social dilemma games, and showed that AFP agents successfully achieved mutual cooperation with more collective rewards and higher equity than the baselines. To further understand the progression of AFP during training, we subsequently explore the effects of altruistic preferences and fairness preferences on agents' behavior. The results suggest that altruistic preferences encourage agents to contribute to the public goods, and fairness preferences induce mutual behavior between agents.
• 中文摘要 在多智能体强化学习（MARL）领域，尤其是在社会困境情境中，诱导分布式智能体之间的合作仍是一个困难问题。在那里，个人利益与公共利益不一致，个人理性导致次优的群体结果。相比之下，人类能够在这种情况下实现合作。对这种合作行为的常见解释是个体有社会偏好。为了实现MARL中的合作，我们设计了一个新的效用函数，整合了利他偏好（他人奖励的激励）和公平偏好（平等的激励），即利他与公平偏好（AFP），这是一种将自己和他人奖励转化为合作行为激励的奖励共享机制。我们对标准强化学习和不平等厌恶代理人在两种具有挑战性的连续社会困境游戏中进行了比较实验，结果显示AFP代理人成功实现了相互合作，获得了比基线更高的集体奖励和更高的公平性。为了进一步理解AFP在培训中的进展，我们随后探讨了利他偏好和公平偏好对代理行为的影响。结果表明，利他偏好鼓励主体为公共产品做出贡献，而公平偏好则诱导主体间的相互行为。

RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

RSPO：多回合大型语言模型代理的奖励交换策略优化

• Authors: Qiang Liu, Taian Guo, Ruizhi Qiao, Xing Sun
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04713
• Pdf link: https://arxiv.org/pdf/2607.04713
• Abstract Reinforcement learning holds significant potential for training large language models (LLMs) to handle multi-turn interactive tasks. However, in long-horizon, multi-turn tasks characterized by sparse outcome rewards, directly training with outcome rewards often results in slow convergence due to the sparsity of signals and the lack of fine-grained feedback. Furthermore, the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance. Conversely, while employing customized dense process rewards provides richer signals and accelerates convergence, these surrogate rewards may exhibit potential misalignment with the ground-truth outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model's final performance. In this work, we propose Reward-Swap Policy Optimization (RSPO), a method designed to leverage the rich information from dense process rewards to facilitate training with outcome rewards. By utilizing a reward-swap mechanism, RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards, thereby elevating the performance ceiling of the model. We conduct extensive experiments on two challenging agent benchmarks, WebShop and ALFWorld. By applying our method to various reinforcement learning algorithms, including GRPO, PPO, and GiGPO, we demonstrate that RSPO achieves consistent performance improvements across different baselines and benchmarks.
• 中文摘要 强化学习在训练大型语言模型（LLMs）以处理多回合交互任务方面具有巨大潜力。然而，在长期、多回合任务中，结果奖励稀疏，直接用结果奖励训练往往导致收敛缓慢，因为信号稀疏且缺乏细粒度反馈。此外，模型可能无法学习训练中未采样的成功轨迹，从而限制其性能。相反，虽然采用定制化的密集过程奖励能提供更丰富的信号并加速收敛，但这些替代奖励可能与实际结果奖励存在不一致。这种不一致会影响训练方向，最终降低模型的最终表现。在本研究中，我们提出了奖励-交换政策优化（RSPO），这是一种旨在利用密集过程奖励中丰富信息，促进训练并以结果奖励为基础的方法。通过采用奖励交换机制，RSPO确保了采样轨迹的多样性，同时保证优化目标与真实结果奖励之间的一致性，从而提升了模型的性能上限。我们在两个具有挑战性的代理基准测试WebShop和ALFWorld上进行了广泛的实验。通过将我们的方法应用于包括GRPO、PPO和GiGPO在内的多种强化学习算法，我们证明RSPO在不同基线和基准测试中实现了持续的性能提升。

Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment

关闭策略令牌开启策略：一种插件方法以提升LLM对齐

• Authors: Yu Li, Xiuyu Li, Mingyang Yi, Jiaxing Wang, zhangliangxu, Zhaolong Xing, Zhen Chen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.04728
• Pdf link: https://arxiv.org/pdf/2607.04728
• Abstract Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded. A natural idea is "transferring" these off-policy token into on-policy token, so that the importance scores for correction are unnecessary. Following this idea, we propose Selective Importance Sampling (SIS), which is inspired by rejection sampling. Concretely, SIS implements by viewing off-policy model as proposal distribution, and implement a token-level rejection test: accepted tokens are viewed as on-policy, so that receive unit importance score, while rejected tokens retain the standard IS correction. Our proposed SIS is theoretically proved reducing the gap between token-level and sequence-level off-policy gradient estimators. The SIS acts as a plug-in that only modifies the importance ratio in the policy loss, adding negligible wall-clock overhead, and can be combine with a vast vary of RL post-training algorithms. Experiments on dense and MoE LLMs across math and agent benchmarks show that SIS consistently improves all objectives, while providing substantially stronger robustness under off-policy data.
• 中文摘要 大型语言模型（LLM）的强化学习（RL）后训练遵循“先展开再更新”的高效范式，这不可避免地导致训练数据不符合策略。为解决这个问题，提出了重要性抽样（IS），而代币级比率在长序列上复合，导致严重的方差爆炸性增长。一个自然的想法是将这些非策略令牌“转移”为非策略令牌，这样纠正的重要性分数就不再必要了。基于这一理念，我们提出了选择性重要性抽样（SIS），其灵感来源于拒绝抽样。具体来说，SIS通过将非策略模型视为提案分配来实现，并实现了令牌级别的拒绝测试：被接受的令牌被视为策略内，因此获得单位重要性分数，而被拒绝的令牌则保留标准的IS纠正。我们提出的SIS理论上已被证明能缩小令牌级与序列级非策略梯度估计之间的差距。SIS作为插件，仅修改政策丢失的重要性比，增加的墙时钟开销可忽略不计，且可与多种强化学习后训练算法结合使用。在数学和代理基准测试中对密集型和MoE型LLMs的实验表明，SIS在非策略数据下持续提升所有目标，同时在非策略数据下提供显著更强的鲁棒性。

CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs

CARL：用于大型语言模型规划的约束感知强化学习

• Authors: Qiuyi Qi, Jinjian Zhang, Mutian Bao, Tian Liang, Guocong Li, Dongnan Liu, Wei Zhou, Jie Liu, Ming Kong, Linjian Mo, Feng Zhang, Qiang Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04854
• Pdf link: https://arxiv.org/pdf/2607.04854
• Abstract Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process. While existing approaches attempt to mitigate this by relying on external tools or task decomposition, they fail to enhance the model's intrinsic constraint awareness. To address this, we propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs' intrinsic focus on constraints. CARL introduces a constraint-aware reward by comparing the model's output distributions under constrained and unconstrained inputs, encouraging constraint focus and penalizing neglect. Compatible with various RL frameworks and requiring no external solvers or top models, CARL enables scalable, end-to-end constraint-aware planning. Extensive experiments on BlocksWorld, TravelPlanner, and T-Eval demonstrate that CARL significantly outperforms standard Reinforcement Fine-Tuning (RFT) baselines and state-of-the-art reasoning models, exhibiting a markedly increased focus on constraints.
• 中文摘要 尽管大型语言模型（LLM）具备强大的推理能力和丰富的世界知识，但它们经常生成违反任务约束的计划，削弱了它们在现实应用中的可靠性。这一缺陷源于缺乏在生成过程中系统性地整合约束信息的机制。现有方法试图通过依赖外部工具或任务分解来缓解这一问题，但未能增强模型内在的约束意识。为此，我们提出了约束感知强化学习（CARL）这一新型强化学习框架，旨在强化大型语言模型对约束的固有关注。CARL通过比较模型在受限和无约束输入下的输出分布，引入了约束感知奖励，鼓励约束聚焦并惩罚忽视。CARL兼容多种强化学习框架，无需外部求解器或顶级模型，实现了可扩展的端到端约束感知规划。在BlocksWorld、TravelPlanner和T-Eval上的大量实验表明，CARL的表现显著优于标准的强化精细调优（RFT）基线和最先进的推理模型，显著增强了对约束的关注。

RL-Ballast: Ship Ballast Water Path Planning and Clog Prediction via Reinforcement Learning

RL-压载物：通过强化学习实现船舶压载水路径规划与堵塞预测

• Authors: Ming-Kuan Lin, Yi-Chung Lai, Ming-Hsin Chiang, Tsung-Wei Pan, Jung-Hua Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.04906
• Pdf link: https://arxiv.org/pdf/2607.04906
• Abstract Under the Shipping 4.0 paradigm, autonomous and reduced-crew vessels require intelligent internal systems to maintain operational safety and structural stability. Ballast-water control is essential for ship trim and integrity, but conventional rule-based or manual approaches have limited adaptability to hydraulic anomalies such as valve failures and pipe blockages, and often depend on dense pressure or flow sensors for diagnosis. To address these limitations, this paper proposes RL-Ballast, a graph-based deep reinforcement learning framework for adaptive ballast-water path planning and sensor-frugal blockage candidate scoring. The valve-permutation problem is transformed into 54 feasible fluid-transfer routes generated using graph theory and depth-first search. The partially observable ballast environment is approximated with frame-stacked tank levels and action outcomes, allowing the agent to infer hidden blockage effects without explicitly modeling a high-dimensional POMDP. During deterministic inference, episode-level failed-action memory and dynamic action masking prevent repeated ineffective actions and support immediate rerouting. Failed transfer histories are further accumulated to rank suspicious valves or pipe segments without dense instrumentation. Monte Carlo simulations show that RL-Ballast completes all unexpected single-blockage scenarios and reduces average decision steps from 61.0 to 41.5 compared with a Dijkstra rule-based baseline. For diagnostic support, the failure-history scoring scheme achieves a 100% Top-3 hit rate, a 66.7% strict Top-1 hit rate, and an 83.3% Top-1 tie-hit rate under serially indistinguishable blockage conditions. These results suggest that RL-Ballast enables adaptive rerouting and maintenance-oriented blockage diagnosis under limited sensing conditions.
• 中文摘要 在航运4.0范式下，自主和减员船只需要智能内部系统以维持运营安全和结构稳定性。压载水控制对于船舶的平衡和完整性至关重要，但传统的基于规则或手动的方法对阀门失效和管道堵塞等液压异常的适应性有限，且诊断时常依赖高密度压力或流量传感器。为解决这些局限性，本文提出了基于图的深度强化学习框架，用于自适应压载水路径规划和传感器节约阻断候选评分。阀门置换问题被转化为利用图论和深度优先搜索生成的54条可行流体转移路径。部分可观测的压载环境通过帧堆叠槽水平和动作结果近似，使智能体无需显式建模高维POMDP即可推断隐藏的阻塞效应。在确定性推理过程中，剧集级失败动作记忆和动态动作掩蔽防止重复无效操作，并支持即时重新路由。失败的传输历史进一步累积，以对无密集仪器的可疑阀门或管道段进行排序。蒙特卡洛模拟显示，RL-Ballast 能够完成所有意外的单次阻塞场景，并将平均决策步数从 61.0 步减少到 41.5 步，相较于 Dijkstra 规则基准。在诊断支持方面，故障历史评分方案在连续无法区分的阻塞条件下实现100%前三命中率、严格前一命中率66.7%和83.3%前一平局率。这些结果表明，RL-Ballast能够在有限的感测条件下实现自适应重定路和维护导向的阻塞诊断。

Closing the Reality Gap: Zero-Shot Sim-to-Real Deployment for Dexterous Force-Based Grasping and Manipulation

缩小现实鸿沟：零机会模拟到现实部署，实现灵巧的基于原力的抓握与操控

• Authors: Zhe Zhao, Zhibin Li, Yilin Ou, Mengshi Qi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.04940
• Pdf link: https://arxiv.org/pdf/2607.04940
• Abstract Human-like dexterous hands with multiple fingers offer human-level manipulation capabilities but remain difficult to train the control policies that can deploy on real hardware due to contact-rich physics and imperfect actuation. We present a sim-to-real reinforcement learning method that leverages dense tactile feedback combined with joint torque sensing to explicitly regulate physical interactions. To enable effective sim-to-real transfer, we introduce (i) a computationally fast tactile simulation that computes distances between dense virtual tactile units and the object via parallel forward kinematics, providing high-rate, high-resolution touch signals needed by RL; (ii) a current-to-torque calibration that eliminates the need for torque sensors on dexterous hands by mapping motor current to joint torque; and (iii) actuator dynamics modeling with randomization to account for non-ideal torque-speed effects and bridge the actuation gaps. Using an asymmetric actor-critic PPO pipeline, we train policies entirely in simulation and deploy them directly to a five-finger hand. The resulting policies demonstrate two essential human-hand skills: (1) command-based controllable grasp force tracking and (2) reorientation of objects in the hand, both of which are robustly executed without fine-tuning on the robot. By combining tactile and torque in the observation space with scalable sensing and actuation modeling, our system provides a practical solution to achieve reliable dexterous manipulation. To our knowledge, this is the first demonstration of controllable grasping on a multi-finger dexterous hand trained entirely in simulation and transferred zero-shot on real hardware.
• 中文摘要 类人灵巧多指的手具备人类水平的操作能力，但由于接触丰富的物理特性和不完美的驱动，仍难以训练可部署在真实硬件上的控制策略。我们提出了一种模拟到现实的强化学习方法，结合密集的触觉反馈和关节扭矩感应，明确调节物理交互。为了实现有效的模拟到现实传输，我们引入了（i）一种计算速度极快的触觉仿真，通过并行前进运动学计算密集虚拟触觉单元与物体之间的距离，提供强化学习所需的高速、高分辨率触控信号;（ii）电流-扭矩校准，通过将电机电流映射到关节扭矩，消除灵巧手对扭矩传感器的需求;以及（iii）通过随机化实现执行器动力学建模，以考虑非理想扭矩速度效应并弥合执行间隙。通过非对称的actor-critic PPO管道，我们完全在模拟中训练策略，并直接部署到五指手中。最终的策略展示了两项关键的人手技能：（1）基于指令的可控抓握力追踪和（2）手中物体的重新定向，这两项技能均可稳健执行，无需机器人微调。通过在观察空间中结合触觉和扭矩，并结合可扩展的感应和执行建模，我们的系统为实现可靠灵巧操作提供了实用解决方案。据我们所知，这是首次在多指灵巧手上完全模拟并转制零击中，在真实硬件上实现可控抓握演示。

STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

STAPO：选择性轨迹感知策略优化，用于LLM代理培训

• Authors: Qiuyi Qi, Tian Liang, Mutian Bao, Jinjian Zhang, Dongnan Liu, Wei Zhou, Linjian Mo, Ming Kong, Jie Liu, Feng Zhang, Qiang Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.04963
• Pdf link: https://arxiv.org/pdf/2607.04963
• Abstract Reinforcement Learning (RL) is the dominant paradigm for training Large Language Model (LLM) agents on long-horizon tasks. However, sparse and delayed rewards often lead to trajectory neglect, in which agents lose focus on the task goal and interaction history at intermediate steps. Prior work has explored step-level supervision using Shannon-entropy-based uncertainty signals, which conflate inherent state complexity with agent confidence and therefore provide unreliable estimates of decision reliability. To address this issue, we propose normalized entropy, which measures confidence deviations relative to an agent's average behavior under a given state, thereby strengthening the association between low-quality actions and trajectory neglect. Building on this insight, we introduce Selective Trajectory-Aware Policy Optimization (STAPO), a hierarchical group-based RL framework. STAPO leverages normalized entropy to locate outlier steps associated with trajectory neglect and optimizes them via a joint mechanism of trajectory-aware reward and trajectory-independent penalty, enhancing trajectory awareness while preserving training stability. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that STAPO achieves state-of-the-art performance while substantially alleviating trajectory neglect, validating its effectiveness and robustness for agentic tasks.
• 中文摘要 强化学习（RL）是训练大型语言模型（LLM）代理执行长期任务的主流范式。然而，稀疏且延迟的奖励常常导致轨迹忽视，代理在中间步骤失去对任务目标和交互历史的关注。此前的研究曾探索过使用基于香农熵的不确定性信号进行阶级监督，这种信号将固有状态复杂度与代理置信度混为一谈，因此提供了不可靠的决策可靠性估计。为解决这一问题，我们提出了归一化熵，它衡量了在给定状态下，智能体相对于平均行为的置信偏差，从而加强低质量行为与轨迹忽视之间的关联。基于这一见解，我们介绍了选择性轨迹感知策略优化（STAPO），这是一个基于分层的基于群体的强化学习框架。STAPO利用归一化熵定位与轨迹忽视相关的离群步，并通过轨迹感知奖励和轨迹无关惩罚的联合机制优化这些步，提升轨迹意识同时保持训练稳定性。在ALFWorld、WebShop和搜索增强质量保证（Search-Augmented QA）上的广泛实验表明，STAPO在实现最先进性能的同时，显著减少了轨迹忽视，验证了其在代理任务中的有效性和鲁棒性。

Non-Convex Sparse Reinforcement Learning via Non-Monotone Inclusions

通过非单调包含关系进行非凸稀疏强化学习

• Authors: Kyohei Suzuki, onstantinos Slavakis
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.04990
• Pdf link: https://arxiv.org/pdf/2607.04990
• Abstract This work delivers two key contributions: one to efficient feature selection in reinforcement learning (RL), the other to the theory of non-monotone inclusions. On the RL side, the estimation bias inherent in conventional regularization schemes is addressed by augmenting classical least-squares temporal-difference (LSTD) policy evaluation with the sparsity-inducing, non-convex projected minimax concave (PMC) penalty. Because the PMC penalty is weakly convex, the resulting fixed-point problem is no longer monotone; instead, it falls under a broader class of non-monotone inclusions involving the sum of a monotone Lipschitz operator and a hypomonotone operator. On the theory side, novel convergence conditions are developed for the forward-reflected-backward splitting (FRBS) method applied to this broader class of non-monotone inclusion problems. Under mild conditions, Lyapunov stability and the existence of a limit point of the sequence of FRBS iterates are established; alternatively, under the weak Minty variational inequality assumption, exact convergence is guaranteed. Numerical tests on benchmark datasets show that the proposed FRBS iterates, applied to the non-convexly regularized LSTD problem, substantially outperform state-of-the-art feature-selection methods, especially when many noisy features are present.
• 中文摘要 这项工作带来了两个关键贡献：一是强化学习（RL）中的高效特征选择，二是非单调包含关系理论的建立。在强化学习方面，传统正则化方案固有的估计偏差通过在经典最小二乘时间差（LSTD）策略评估基础上加入稀疏性、非凸的投影极小极大凹（PMC）惩罚来解决。由于PMC惩罚是弱凸的，因此产生的不动点问题不再单调;相反，它属于更广泛的非单调包含关系，涉及单调利普希茨算符和下单调算符的和。在理论方面，针对这一更广泛的非单调包含问题，开发了前向反射后向分裂（FRBS）方法的新收敛条件。在温和条件下，可以确定Lyapunov稳定性以及FRBS迭代序列的极限点存在;或者，在弱Minty变分不等式假设下，精确收敛是保证的。基准数据集上的数值测试显示，应用于非凸正则化LSTD问题的FRBS迭代方法，在存在大量噪声特征时，显著优于最先进的特征选择方法。

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

扩散引导的不确定性感知延迟策略优化

• Authors: Junqi Tu, Zejiao Liu, Fangfei Li, Yang Tang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.05064
• Pdf link: https://arxiv.org/pdf/2607.05064
• Abstract Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.
• 中文摘要 现实环境中的强化学习常因反馈延迟而严重性能下降。现有方法通常通过构建增强态或预测真实状态来减轻观测延迟导致的性能下降。然而，这些方法常常忽视了随机MDP诱导的延迟态与真态之间的固有差异。我们理论上证明了这种差异的存在，并证明它会导致最优策略的退化。为应对这一挑战，我们提出了扩散引导不确定性感知延迟策略优化（DUPO）。我们的方法明确用扩散模型模拟延迟状态消息与当前状态之间的关系，并利用所得的差异估计对延迟策略进行权重。对多重随机延迟的连续机器人控制任务的广泛实验表明，DUPO始终优于现有方法，即使在长时间且随机延迟场景下依然有效。

TimeThink: Reasoning with Time for Video LLMs

TimeThink：视频大型语言模型的时间推理

• Authors: Handong Li, Longteng Guo, Zikang Liu, Dongze Hao, Yepeng Tang, Zijia Zhao, Jie Jiang, Zhiwei Jin, Chen Chen, Haonan Lu, Jing Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.05089
• Pdf link: https://arxiv.org/pdf/2607.05089
• Abstract Video reasoning requires models to identify and verify temporally localized evidence within long video sequences. Recent Video Large Language Models (Video-LLMs) have shown promising reasoning abilities when aligned with reinforcement learning, yet existing approaches typically rely on outcome-based rewards that supervise only the final prediction. Such supervision provides limited guidance on how models should discover the relevant temporal evidence during intermediate reasoning. In this work, we propose TimeThink, a reinforcement learning framework that explicitly guides temporal evidence discovery in Video-LLMs. Our key idea is to treat temporal clue steps as the fundamental optimization primitive of video reasoning, where each reasoning step references a candidate time interval in the video. We introduce a step-wise temporal process reward that provides localized credit assignment for these clues and a joint process--outcome optimization objective that balances reasoning fidelity with task correctness. To enable scalable training, we construct TimeThink-RFT-20K, a dataset with automatically derived temporal evidence segments. Extensive experiments across video reasoning, temporal grounding, and general video understanding benchmarks show that TimeThink consistently improves both temporal localization and reasoning performance, achieving state-of-the-art results among open-source video RL models.
• 中文摘要 视频推理需要模型识别并验证长视频序列中的时间定位证据。近期的视频大型语言模型（Video-LLMs）在与强化学习结合时展现出有前景的推理能力，但现有方法通常依赖基于结果的奖励，仅监督最终预测。这种监督对模型在中间推理过程中如何发现相关时间证据提供了有限的指导。在本研究中，我们提出了TimeThink，一种强化学习框架，明确指导视频大型语言模型中的时间证据发现。我们的核心思想是将时间线索步骤视为视频推理的基本优化原语，每个推理步骤都引用视频中的候选时间区间。我们引入了分阶段的时间过程奖励，为这些线索提供局部的功劳分配，以及一个联合过程——结果优化目标，平衡推理的忠实度与任务的正确性。为了实现可扩展的训练，我们构建了TimeThink-RFT-20K数据集，该数据集包含自动导出的时间证据片段。涵盖视频推理、时间基础和一般视频理解基准的广泛实验表明，TimeThink持续提升时间定位和推理性能，在开源视频强化模型中取得了最先进的成果。

Claim-Level Rubric Rewards for Video Caption Reinforcement Learning

视频字幕强化学习的索赔级别评分标准奖励

• Authors: Mingqi Gao, Hongyuan Dong, Yifei Chen, Zhisheng Zhong, Zheng Ruan, Wenjin Hou, Yu Chen, Han Hu, Yansong Tang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.05150
• Pdf link: https://arxiv.org/pdf/2607.05150
• Abstract In this paper, we introduce Claim-Level Rubric Rewards (CuRe), a structured reward framework designed to address the reward-design bottleneck in reinforcement learning for dense video captioning. Existing reward designs generally fall into two categories: holistic response-level judgment across heterogeneous criteria, or alignment-based evaluation against reference captions. However, both paradigms suffer from fundamental limitations. Holistic rewards struggle to ensure factual accuracy and are prone to stylistic reward hacking, while reference-based rewards overly rely on rigid textual alignment, failing to preserve the completeness and diversity inherent to open-ended generation tasks. To address these challenges, CuRe reformulates reward modeling as fine-grained claim-level verification. Specifically, CuRe decomposes captions into category-aware atomic claims through a structured rubric, converting holistic evaluation into simpler and more reliable claim-level verification.
• 中文摘要 本文介绍了申报级评分标准奖励（CuRe），这是一种结构化奖励框架，旨在解决密集视频字幕强化学习中奖励设计瓶颈。现有的奖励设计通常分为两类：跨异质标准的整体反应水平判断，或基于对齐的参考说明评估。然而，这两种范式都存在根本性的局限性。整体奖励难以确保事实准确性，且容易出现风格化的奖励篡改，而基于引用的奖励则过于依赖僵化文本对齐，未能保持开放式生成任务固有的完整性和多样性。为应对这些挑战，CuRe将奖励建模重新定义为细粒度的索赔级验证。具体来说，CuRe通过结构化评分标准将字幕分解为类别感知的原子主张，将整体评估转化为更简单、更可靠的主张层级验证。

Relational Multi-Agent Reinforcement Learning for Dynamic Pricing in High-Speed Railway Markets

高速铁路市场动态定价的关系多智能体强化学习

• Authors: Enrique Adrian Villarrubia-Martin, David Muñoz-Valero, Luis Rodriguez-Benitez, Giovanni Montana, Luis Jimenez-Linares
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.05179
• Pdf link: https://arxiv.org/pdf/2607.05179
• Abstract In liberalised railway systems, operators must set prices dynamically in an environment with partial observability, as they retain private information about their objectives and performance, where regulatory constraints prohibit communication or direct information exchange between competitors to prevent explicit collusion. Consequently, agents must learn to infer strategic interactions only from observable market data which presents a significant challenge for multi-agent reinforcement learning, where standard approaches typically treat observations as unstructured vectors, ignoring the underlying market topology that governs strategic interactions. To address this, an entity graph modelling approach is proposed, which represents the environment as a graph of operational units, rather than decision-making agents or static infrastructure, encoding competition, coordination, and connectivity relations between entities. Then, an extension of the multi-agent twin delayed deep deterministic policy gradient algorithm with graph-based representation learning processes the features of the entities through a multi-layer relational graph convolutional network and aggregates them via a learnt attention mechanism. Experimental results in a rail pricing reinforcement learning environment show that this novel framework achieves higher revenue and stability in two different settings of increasing market complexity compared to a representative selection of relational and non-relational baselines. The code is publicly available at: this https URL
• 中文摘要 在自由化的铁路系统中，运营商必须在部分可观察性的环境中动态定价，因为他们保留了关于目标和绩效的私人信息，监管限制禁止竞争者之间沟通或直接信息交换，以防止明显的串通。因此，代理必须学会仅从可观察的市场数据推断战略交互，这对多代理强化学习构成重大挑战，因为标准方法通常将观测值视为非结构化矢量，忽视支配战略交互的市场拓扑。为此，提出了一种实体图建模方法，将环境表示为操作单元的图，而非决策代理或静态基础设施，编码实体间的竞争、协调和连接关系。随后，多智能体孪生的扩展采用基于图的表示学习，利用延迟确定性策略梯度算法处理实体特征，并通过学习的注意力机制进行聚合。在铁路定价强化学习环境中的实验结果表明，该新颖框架在两种市场复杂度递增的不同环境中，比代表性选择的关系型和非关系型基线更能实现更高的收入和稳定性。代码公开可访问：此 https URL

When Claws Remember but Do Not Tell: Stealthy Memory Injection in Persistent Personal Agents

当爪子记得却不说时：持久个人代理中的隐秘记忆注入

• Authors: Yechao Zhang, Shiqian Zhao, Jiawen Zhang, Jie Zhang, Gelei Deng, Xiaogeng Liu, Chaowei Xiao, Tianwei Zhang
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.05189
• Pdf link: https://arxiv.org/pdf/2607.05189
• Abstract Persistent personal agents combine long-term memory with access to users' external environments, enabling personalized foreground assistance and proactive background execution. This integration also creates a new path to compromise: untrusted external content can be silently written into persistent memory and later reused as trusted state. We study this threat as stealth memory injection, in which a remote black-box adversary delivers a single email payload that must induce the agent to write poisoned memory, stay hidden in the agent's response to the user, and affect future behavior. We introduce WhisperBench, a 108-case benchmark spanning five risk categories and both fact and preference poisoning. Built on a real IMAP/SMTP workflow and an authentic email agent skill, it enables full-cycle evaluation of stealth memory injection attacks. To enable this black-box attack under single-email delivery and without runtime feedback, we propose MemGhost, a one-shot payload generation framework. MemGhost uses an environment proxy to emulate persistent-agent execution and an objective proxy to convert memory adoption and conversational stealth into dense rubric-based rewards, then trains the attacker policy with supervised fine-tuning and reinforcement learning. Across 56 held-out test cases, MemGhost achieves 87.5% end-to-end success on OpenClaw with GPT-5.4 and 71.4% on Claude Code SDK with Sonnet 4.6. It also transfers across personal-agent architectures (NanoClaw and Hermes Agent) and memory backends (filesystem and vector-based Mem0), and remains effective against input-level, model-level, and system-level defenses. These results suggest that persistent memory can turn ordinary external processing into a practical pathway for long-term agent compromise.
• 中文摘要 持久个人代理结合了长期记忆与对用户外部环境的访问，实现个性化的前景协助和主动的后台执行。这种集成还开辟了一条新的入侵路径：不受信任的外部内容可以被无声地写入持久存储器，之后再作为可信状态被重新使用。我们将这种威胁称为隐形内存注入，即远程黑匣子攻击者发送单一邮件负载，必须诱使代理写入有毒内存，隐藏在代理对用户的回复中，并影响未来行为。我们介绍WhisperBench，这是一个涵盖五大风险类别以及事实和偏好中毒的108个案例基准。它基于真实的IMAP/SMTP工作流程和真实的电子邮件代理技能，支持对隐形内存注入攻击的全周期评估。为了实现这种在单封邮件投递且无运行时反馈下的黑箱攻击，我们提出了MemGhost，一个一次性有效载荷生成框架。MemGhost 使用环境代理模拟持久代理执行，使用目标代理将内存采纳和对话隐蔽转化为密集的基于评分标准的奖励，然后通过监督微调和强化学习训练攻击者策略。在56个测试案例中，MemGhost在使用GPT-5.4的OpenClaw上实现了87.5%的端到端成功率，在Claude Code SDK和Sonnet 4.6上实现了71.4%。它还能跨个人代理架构（NanoClaw和Hermes Agent）和内存后端（文件系统和矢量Mem0）进行传输，并且对输入级、模型级和系统级防御依然有效。这些结果表明，持久记忆可以将普通的外部处理转化为长期代理妥协的实用途径。

Adaptive Inference Batching using Policy Gradients

利用策略梯度进行自适应推理批处理

• Authors: Ruslan Sharifullin
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Performance (cs.PF)
• Arxiv link: https://arxiv.org/abs/2607.05272
• Pdf link: https://arxiv.org/pdf/2607.05272
• Abstract Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic. We investigate whether reinforcement learning (RL) can learn adaptive batching and routing policies that outperform these heuristics, training REINFORCE and PPO agents on a discrete-event simulator validated against queuing theory and production traces (Azure Functions, BurstGPT). We formulate the problem as an MDP over queue state, request type and GPU availability, evaluating across standard Poisson traffic, extreme bursts, real-world traces and heterogeneous multi-GPU routing. Our central finding is a clear boundary condition for RL's value in systems problems. In single-GPU settings, a well-tuned static batching policy is already near-optimal under Poisson-like arrivals and RL offers only marginal gains (+0.1% to +1.0%). In multi-GPU heterogeneous routing, however, where fast and slow requests compete for shared resources, the agent discovers a workload-segregation policy that eliminates Head-of-Line blocking, yielding a 3.5x (348%) improvement over Round-Robin and a 48% improvement over the strongest heuristic baseline (Shortest-Queue), with 60% higher throughput and 25% lower latency while respecting SLA constraints. The policy generalizes to unseen bursty and real-world traffic despite training only on synthetic Poisson arrivals and an attention-augmented policy network converges roughly 20% faster than an MLP baseline. These results suggest RL's advantage over engineered heuristics concentrates in combinatorial, multi-resource decisions rather than single-resource temporal scheduling, a practical distinction for deciding where learned policies justify their engineering cost in production inference infrastructure.
• 中文摘要 推理服务系统必须在突发式、异构工作负载下平衡吞吐量和延迟，但行业标准仍是静态批处理策略，需要手动调优，无法适应流量变化。我们研究强化学习（RL）是否能学习超越这些启发式的自适应批处理和路由策略，并在经过排队理论验证和生产痕迹（Azure Functions、BurstGPT）验证的离散事件模拟器上训练REINFORCE和PPO代理。我们将问题表述为基于队列状态、请求类型和GPU可用性的MDP，评估标准泊松流量、极端突发、真实世界痕迹以及异构多GPU路由。我们的核心发现是强化学习在系统问题中价值的明确边界条件。在单GPU环境中，调优良好的静态批处理策略在泊松样到达下已接近最优，而强化学习仅带来边际收益（+0.1%至+1.0%）。然而，在多GPU异构路由中，快速和慢速请求争夺共享资源时，代理发现了工作负载隔离策略，消除了线路头阻断，比轮转提升3.5倍（348%），较最强启发式基线（最短队列）提升48%，吞吐量提升60%，延迟降低25%，同时遵守SLA约束。尽管仅对合成泊松到达进行训练，且注意力增强的政策网络收敛速度约比MLP基线快20%，该策略仍推广到看不见的爆发和现实世界流量。这些结果表明，强化学习相较于工程启发式的优势集中在组合、多资源决策上，而非单一资源的时间调度，这对于判断学习策略在生产推理基础设施中工程成本的合理性具有实际区别。

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

CompactionRL：基于上下文压缩的长视野代理强化学习

• Authors: Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, Yuxiao Dong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.05378
• Pdf link: https://arxiv.org/pdf/2607.05378
• Abstract Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-trajectory generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A30B) to achieve Pass@1 scores of 66.8% on SWE-bench Verified and 24.5% on Terminal-Bench 2.0, with absolute gains of 7.0 and 3.1 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.8 points, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0, respectively. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).
• 中文摘要 长视野代理LLM越来越受限于有限上下文窗口，因为延长的交互轨迹可能在任务完成前超过最大上下文长度。上下文压缩通过总结之前的交互状态并在压缩上下文下继续展开，提供了一种自然的解决方案，但将压缩融入强化学习方面仍然未被充分探索。我们提出了CompactionRL，一种强化学习策略，用于通过上下文压缩训练长视野代理LLMs。我们的方法结合代币级损失规范化和交叉轨迹广义优势估计，共同优化任务执行和总结生成。该设计使LLM代理能够从紧凑的长视距轨迹中学习。我们在开放模型上训练CompactionRL，并在代理编码任务中持续获得性能提升。CompactionRL使开放型GLM-4.5-Air（106B-A30B）在SWE测试台验证中获得66.8%的Pass@1，终端测试台2.0的24.5%，绝对提升分别为7.0分和3.1分。基于GLM-4.7-Flash（30B-A3B），CompactionRL在Pass@1提升了5.5个点和6.8个百分点，分别在SWE-bench Verified和Terminal-bench 2.0上达到56.0%和20.2%。因此，CompactionRL 被部署在 RL 流水线中，用于训练开放式 GLM-5.2 型号（750B-A40B）。

Weak-to-Strong Generalization via Direct On-Policy Distillation

通过直接在政策内提炼实现弱到强推广

• Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.05394
• Pdf link: https://arxiv.org/pdf/2607.05394
• Abstract Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.
• 中文摘要 带有可验证奖励的强化学习（RLVR）是提升语言模型推理的强大方法，但每个新强模型重复使用成本较高，因为目标模型在训练过程中必须生成多次扩展。随着模型规模扩大，后期训练本身就成为瓶颈。我们研究一种弱到强的替代方案：在一个更小的模型上运行强化学习，且推广成本更低，然后重复利用该强化学习学到的知识来改进更强的目标模型。直接提炼后强化学习的弱教师还不够，因为教师的最终政策将有用的强化学习收益与较小模型的局限性相结合。我们提出了直接政策提炼（Direct-OPD），即转移教师由强化学习引发的政策转变。直接OPD将后强化学习教师与自身的前强化学习参考进行比较，并将对数比视为对学生的密集隐性奖励。简单来说，检查点对告诉我们强化学习使弱模型更可能采取或更不可能采取哪些行动，直接OPD将该信号应用于强学生自身的政策状态。这直接重用弱模型的强化学习监督信号，无需训练显式奖励模型或在目标模型上运行稀疏奖励强化学习。实证上，直接门诊持续利用较弱教师来改进更强的目标模型;值得注意的是，它在 AIME 2024 上仅用 4 小时内将 Qwen3-1.7B 的表现提升到 62.4%，覆盖 8 块 A100 GPU。它优于步匹配直接强化学习，并支持多次策略切换的顺序组合。我们的结果表明，强化学习结果可以作为隐性奖励信号在不同模型尺度上再利用，而不仅仅是最终的模仿模型。

Keyword: diffusion policy

KAM-WM: Kinematic Affordance Maps from Latent World Models for Robot Manipulation

KAM-WM：来自潜在世界模型的机器人操控运动学通向图

• Authors: Xinyu Shao, Keru Zhou, Guowei Huang, Yajun Gao, Tongtong Cao, Xiu Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.04652
• Pdf link: https://arxiv.org/pdf/2607.04652
• Abstract Learning manipulation from few demonstrations requires visual priors that capture not only where to interact, but also how the interaction should begin; static priors such as segmentation masks encode only the former. We present KAM-WM, a framework that extracts a coarse directional interaction cue from a frozen latent video world model without rollout or world-model fine-tuning. KAM-WM queries a Flow Matching image-to-video backbone once and interprets its single-step latent velocity as a Kinematic Affordance Map (KAM), which provides task-conditioned interaction regions and coarse motion structure. A lightweight Perceiver compresses KAM into tokens that condition a diffusion policy together with RGB observations and proprioception. Across LIBERO and RoboTwin2.0, KAM-WM reaches 90.6% average success on LIBERO and achieves 65.7% and 22.4% success rates in the Easy and Hard settings on RoboTwin2.0, respectively. Controlled comparisons against a zero-order mask prior suggest that part of the gains comes from directional information beyond spatial localization alone. These results indicate that, in the evaluated settings, a frozen video model can provide a useful first-order visual prior for control without the test-time cost of future rollout.
• 中文摘要 从少数演示中学习操作需要视觉先验，不仅捕捉互动位置，还要识别互动应如何开始;静态先验，如分割掩码，只编码前者。我们介绍了KAM-WM，这是一个无需展开或世界模型微调的冻结潜在视频世界模型中提取粗略方向交互线索的框架。KAM-WM查询一次图像到视频的流匹配骨干网，将其单步潜在速度解读为运动学通向图（KAM），提供任务条件交互区域和粗运动结构。轻量级感知器将KAM压缩为条件化扩散策略的标记，结合RGB观测和本体感受。在 LIBERO 和 RoboTwin2.0 中，KAM-WM 在 LIBERO 上的平均成功率为 90.6%，在 RoboTwin 2.0 的简单和困难难度中分别达到 65.7% 和 22.4%。与零阶掩膜先验的受控比较表明，部分收益来自于超越空间定位的方向信息。这些结果表明，在评估的环境中，冻结视频模型可以为控制提供有用的一阶视觉先验，而无需承担未来推广的测试时间成本。