生成时间: 2026-09-04 20:34:54 (UTC+8); Arxiv 发布时间: 2026-09-04 20:00 EDT (2026-09-05 08:00 UTC+8)

今天共有 32 篇相关文章

Keyword: reinforcement learning

RL-ADA: A World-Feedback Framework for Adversarially Robust Enterprise Dialogue Agents

RL-ADA：针对对抗性强健企业对话代理的世界反馈框架

• Authors: Ram Narayanan, Harshit Rajgarhia, Abhishek Mukherji
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.02902
• Pdf link: https://arxiv.org/pdf/2609.02902
• Abstract Deploying task-oriented dialogue agents in enterprise customer support faces a persistent annotation bottleneck: robust training requires labelled interaction data at scale, yet enterprise conversational logs are privacy-sensitive and expensive to annotate, while user behaviour evolves faster than labelling pipelines can keep pace. We present RL-ADA (Reinforcement Learning with Adversarial Dialogue Agents), a co-evolutionary training framework that eliminates this bottleneck by replacing human labels with \emph{world feedback}: consequence-based reward signals derived directly from measurable interaction outcomes. A Customer Support Agent (DA, 3B parameters) and an Adversarial Customer Agent (CA, 7B parameters) co-evolve in an adversarial arena guided by a fixed automated judge: the DA is rewarded for correctly handling multi-turn customer conversations to successful resolution, while the CA is rewarded for producing realistic, intent-concealing utterances that cause misroutes, creating asymmetric adversarial pressure through opposing but independently structured rewards. An isolation gym iteratively retrains the weaker agent on prior-failure transcripts, requiring no human annotation at any stage. In a banking customer support proof of concept, tool-routing errors are eliminated and the strict end-to-end PASS rate doubles over five co-evolutionary cycles, driven solely by automated arena reward with no labelled data. We additionally observe the emergence of \textbf{Contextual Camouflage}, an adversarial strategy in which the CA learns to embed intent within dense realistic customer detail purely from reward pressure, with direct implications for enterprise red-teaming and robustness evaluation.
• 中文摘要 在企业客户支持中部署任务导向对话代理面临一个持续的注释瓶颈：稳健的培训需要大规模使用带标签的交互数据，而企业对话日志对隐私敏感且注释成本高昂，而用户行为的发展速度快于标签流水线的追随速度。我们提出了RL-ADA（与对抗性对话代理的强化学习），这是一种共进化训练框架，通过用\emph{world feedback}替代人类标签来消除这一瓶颈：基于后果的奖励信号，直接源自可测量的交互结果。客户支持代理（DA，3B参数）和对抗客户代理（CA，7B参数）在对抗环境中共同进化，由固定自动裁判引导：DA因正确处理多回合客户对话成功解决而获得奖励，而CA则因产出真实、意图隐蔽的话语导致误导，通过对立但独立结构的奖励创造非对称对抗压力而获得奖励。隔离健身房通过先前失败的转录迭代再培训较弱的代理，任何阶段无需人工注释。在银行客户支持概念验证中，工具路由错误被消除，严格的端到端通过率在五个共进化周期内翻倍，完全由自动化竞技场奖励驱动，且无标记数据。我们还观察到\textbf{情境伪装}的出现，这是一种对抗策略，CA通过奖励压力学会将意图嵌入密集且真实的客户细节中，直接影响企业红队和稳健性评估。

LLM-Guided Reinforcement Learning for Adaptive NPC Behavior in Multi-Agent Combat Games

多智能体战斗游戏中自适应NPC行为的LLM引导强化学习

• Authors: Hrithika Deepu Nair, Kayvan Karim
• Subjects: Subjects: Multiagent Systems (cs.MA); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02931
• Pdf link: https://arxiv.org/pdf/2609.02931
• Abstract Scripted and rule-based non-player characters (NPCs) in combat video games often exhibit predictable behaviors that experienced players can exploit, while reinforcement learning (RL) agents typically retain a fixed policy after training and cannot readily adapt their strategy to different opponents. We investigate a runtime strategy-selection framework in which a large language model (LLM) guides a trained RL policy without modifying its underlying behavior. To demonstrate this, we train five NPC agents with a shared PPO policy in Unity and compare a baseline configuration, in which the policy acts independently, with an LLM-augmented configuration in which a locally hosted Mistral 7B model, accessed through Ollama, reads the live game state every five seconds and assigns one of four tactical tags. We evaluate both configurations against three scripted opponent types across 600 episodes and analyze outcomes using the Mann-Whitney U test. Against a Balanced opponent that changes tactics during an episode, the LLM-augmented agents more than doubled their win rate from 11% to 24% and produced significantly longer episodes. Against an Evasive opponent, the augmented agents achieved a higher win rate and faster kills, although their shorter episode duration did not satisfy the strict hypothesis definition. Against an Aggressive opponent, the LLM's near-constant preference for encirclement was counterproductive. Analysis of 2,430 strategy selections showed that Surround was selected in 83.8% of cases regardless of opponent type, indicating limited zero-shot strategic differentiation at this model scale. These results demonstrate both the potential and limitations of LLM-guided runtime strategy selection for adaptive multi-agent game AI.
• 中文摘要 战斗游戏中的脚本化和基于规则的非玩家角色（NPC）通常表现出可预测的行为，经验丰富玩家可以利用这些行为，而强化学习（RL）代理在训练后通常保留固定策略，无法灵活调整策略以适应不同对手。我们研究了一个运行时策略选择框架，其中大型语言模型（LLM）引导训练好的强化学习策略而不改变其底层行为。为证明这一点，我们在Unity中训练了五个拥有共享PPO策略的NPC代理，并比较了一个独立行动的基线配置与一个LLM增强配置，后者由本地托管的Mistral 7B模型（通过Ollama访问）每五秒读取一次实时游戏状态，并分配四个战术标签之一。我们对这两种配置在600集的剧情化对手类型进行评估，并使用Mann-Whitney U测试分析结果。面对在回合中改变战术的平衡对手，LLM增强代理的胜率从11%翻倍至24%，且产生显著更长的发作时间。面对回避型对手，增强代理获得更高的胜率和更快的击杀，尽管其较短的发作持续时间不满足严格假设定义。面对激进对手，LLM几乎持续偏好包围则适得其反。对2430个策略选择的分析显示，无论对手类型如何，Surround在83.8%的案例中被选中，表明在该模型尺度下零射击战略差异化有限。这些结果展示了LLM引导运行时策略选择在自适应多智能体游戏AI中的潜力与局限性。

Tail-Likelihood Reinforcement Learning

尾似然强化学习

• Authors: Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar, Guanning Zeng, Qingyang Wu, Zhongzhu Zhou, Chenfeng Xu, Haiwen Feng, Yuda Song, Aarti Singh, Ruslan Salakhutdinov, J. Andrew Bagnell, Jeff Schneider, Andrea Zanette
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.02987
• Pdf link: https://arxiv.org/pdf/2609.02987
• Abstract Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-(k) gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.
• 中文摘要 强化学习通常优化平均奖励。对于生成策略，平均值可能隐藏一个重要区别：两种策略可能获得相同的平均奖励，但产生罕见但高奖励的推广概率截然不同。随着训练和推断过程中抽样增加，这一点尤为重要，因为其效益依赖于保留高奖励结果的概率质量。我们建议直接优化这种覆盖。我们不只考虑预期奖励，而是考虑其所有上尾：对于每个奖励阈值，策略超过该阈值的可能性有多大？这使连续奖励变成一类二元成功事件。我们引入尾似然强化学习（TailRL），最大化超过随机奖励阈值的对数概率。其梯度赋予稀有高奖励推广更多权重，可解释为最佳（k）梯度的混合。TailRL只需对优势函数进行简单修改，即可兼容现有的强化学习流水线。在对象定位、迷宫导航、图形界面基础化和代码优化等方面，TailRL利用罕见的高回报训练样本避免次优解，并生成在推理时更需要额外样本的模型。

GPU-Accelerated Astrodynamics World Models for Spacecraft Rendezvous and Proximity Operations

用于航天器会合和近距离操作的GPU加速天体动力学世界模型

• Authors: Duncan Eddy, Isaac R. Ward, Grace Ra Kim, Mykel J. Kochenderfer
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.03067
• Pdf link: https://arxiv.org/pdf/2609.03067
• Abstract World models are an emerging paradigm in representation learning in which an agent jointly learns state-action dynamics and observation models from offline trajectory data, enabling multi-step planning and trajectory prediction with uncertainty estimates. They have shown strong results in robotics and game environments, but, to the best of our knowledge, have not previously been applied to the space domain. This paper introduces a world model-based approach to cooperative and non-cooperative spacecraft rendezvous and proximity operations. First, we introduce an open-source, JAX-based International Space Station (ISS) docking environment supporting parallel GPU simulation of spacecraft orbit and attitude dynamics, generating the thousands of state-action transitions that world model training requires. Second, we introduce Out-of-this-World-Model, a transformer-based world model that encodes relative kinematic states and body-fixed camera imagery into a latent state and predicts its evolution under commanded thrusts and torques using one-step flow matching. It produces a distribution over future observations, capturing stochastic dynamics and per-timestep uncertainty, and outperforms DreamerV3-style posterior-correction baselines with fewer trainable parameters and hyperparameters. Third, we apply the approach to a capsule autonomously docking with the ISS under keep-out-zone constraints, demonstrating improved sample efficiency and task performance over reinforcement learning baselines (53% versus 29% docking success across ports), better out-of-distribution generalization (on held-out ports the world model more than doubles baseline success, 40% versus 17%), and detection of anomalous objects encountered during approach with 98% classification accuracy. We open-source the simulation environment and model architecture to enable further study of this paradigm.
• 中文摘要 世界模型是表征学习中一种新兴范式，智能体通过离线轨迹数据共同学习状态-动作动力学和观测模型，实现多步规划和轨迹预测，并实现不确定性估计。它们在机器人和游戏环境中取得了显著成效，但据我们所知，此前尚未应用于空间领域。本文介绍了基于世界模型的协作与非合作航天器交会与接近操作方法。首先，我们介绍了一个基于JAX的开源国际空间站（ISS）对接环境，支持并行GPU模拟航天器轨道和姿态动力学，生成世界模型训练所需的数千种状态-动作转变。其次，我们引入了基于变压器的世界模型，将相对运动学状态和固定摄像机图像编码为潜态，并通过一步流匹配预测其在指令推力和扭矩下的演变。它生成未来观测分布，捕捉随机动力学和每时间步不确定性，且在可训练参数和超参数更少的情况下优于DreamerV3式后验修正基线。第三，我们将该方法应用于一个在隔离区约束下自主对接国际空间站的舱体，展示了相较强化学习基线提升的样本效率和任务表现（53%对29%的端口对接成功率），更佳的非分布泛化（在保留端口上，世界模型基线成功率是40%对17%的两倍多），并以98%的分类准确率检测到接近过程中遇到的异常物体。我们将仿真环境和模型架构开源，以便进一步研究这一范式。

Adaptive Beam Hopping and Power Control for Dual-Layer Over-the-Air Online Federated Learning in LEO Satellite Networks

近地轨道卫星网络中双层空中在线联邦学习的自适应波束跳跃和功率控制

• Authors: Zhendong Li, Shaojie Wang, Zhou Su, Zihao Zhang, Haixia Peng, Nan Cheng, Ying Wang, Wen Chen
• Subjects: Subjects: Information Theory (cs.IT); Signal Processing (eess.SP)
• Arxiv link: https://arxiv.org/abs/2609.03202
• Pdf link: https://arxiv.org/pdf/2609.03202
• Abstract This paper investigates over-the-air (OTA) computation enabled online federated learning (FL) in low-Earth orbit (LEO) satellite networks. Specifically, we consider a dual-layer OTA aggregation architecture, where ground devices upload analog model updates to serving satellites via uplink OTA aggregation, and satellites forward the aggregated signals to a data processing center through the second round OTA aggregation. Then, we formulate a long-term data-utilization maximization problem in which devices continuously collect new data and untrained samples gradually lose freshness. The problem is subject to the satellite beam budget, transmit-power limit, and global mean squared error (MSE) constraint that governs end-to-end aggregation distortion. This yields a coupled mixed-integer nonlinear programming (MINLP) problem, involving tightly coupled discrete beam-hopping decisions and continuous power control. Due to the combinatorial action space and nonconvex constraints, the problem is NP-hard and computationally intractable. Furthermore, the time-varying satellite topology and dynamic data generation render it a sequential decision-making problem, necessitating adaptive online scheduling. To address these issues, we cast the problem as a Markov decision process and develop a proximal policy optimization (PPO)-based deep reinforcement learning framework that jointly optimizes adaptive beam hopping and power control, using an MSE-aware reward to balance data utilization and aggregation accuracy. Numerical simulation results verify that the proposed algorithm consistently outperforms other benchmark schemes, achieving superior long-term data utilization and faster FL convergence while satisfying the MSE requirement.
• 中文摘要 本文探讨了近地轨道（LEO）卫星网络中实现的空中（OTA）计算实现的在线联邦学习（FL）。具体来说，我们考虑了一种双层OTA聚合架构，地面设备通过上行OTA聚合向服务卫星上传模拟模型更新，卫星通过第二轮OTA聚合将聚合信号转发到数据处理中心。随后，我们构建了一个长期数据利用最大化问题，设备持续收集新数据，未训练样本的新鲜度逐渐丧失。该问题受卫星波束预算、发射功率极限和全局均方误差（MSE）约束，这些约束决定端到端聚合失真。这导致耦合混合整数非线性规划（MINLP）问题，涉及紧耦合的离散波束跳决策和连续功率控制。由于组合作用空间和非凸约束，该问题具有NP难和计算难解性。此外，卫星拓扑时变和动态数据生成使其成为顺序决策问题，需要自适应在线调度。为解决这些问题，我们将问题定位为马尔可夫决策过程，开发了一个基于近策略优化（PPO）的深度强化学习框架，联合优化自适应波束跳跃和功率控制，利用MSE感知的奖励平衡数据利用和聚合准确性。数值模拟结果验证，所提算法持续优于其他基准方案，实现更优越的长期数据利用率和更快的FR收敛，同时满足MSE要求。

SWIM: Student Writing Simulation via Proficiency-Conditioned Generation

SWIM：通过熟练条件生成实现的学生写作模拟

• Authors: Heejin Do, Jakub Kontak, Mrinmaya Sachan
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.03215
• Pdf link: https://arxiv.org/pdf/2609.03215
• Abstract Writing proficiency manifests in how students develop content, organize ideas, choose words, and use language. Despite growing interest in LLM-based student simulation, whether LLMs can reproduce such multidimensional variation in extended writing remains largely unexplored. In this work, we explore if language models can realistically simulate student writing, and introduce SWIM, a task that formulates Student Writing sIMulation as proficiency-conditioned essay generation. We evaluate prompting, supervised fine-tuning (SFT), and reinforcement learning (RL) methods for writing simulation using automated essay scoring as a measure of profile alignment. Extensive experiments reveal that prompting provides limited proficiency control, even for strong proprietary LLMs with rubric-grounded strategies. In particular, while models can adjust content-oriented traits, they struggle to reproduce the lexical, grammatical, and organizational variation in different proficiency levels. SFT substantially improves alignment, while RL with the proposed proficiency-alignment reward yields further gains across all writing traits and essay prompts. Our findings suggest that explicit supervision enables substantially stronger profile alignment than prompting alone, while authentic low-proficiency writing remains challenging to reproduce.
• 中文摘要 写作能力体现在学生如何发展内容、组织想法、选择词汇和使用语言。尽管对基于LLM的学生模拟越来越感兴趣，LLMs是否能在扩展写作中重现这种多维变异仍大多未被探讨。本研究探讨语言模型是否能真实模拟学生写作，并引入SWIM任务，将学生写作模拟化定义为熟练条件论文生成。我们评估了使用自动论文评分作为配置文件对齐度量的写作模拟的提示、监督微调（SFT）和强化学习（RL）方法。大量实验显示，提示法即使在基于评分标准的强大专有LLM中，也只能有限地控制熟练度。特别是，虽然模型可以调整内容导向的特质，但它们在不同熟练度水平下难以再现词汇、语法和组织的变异。SFT显著改善了对齐度，而强化学习通过所提的熟练度对齐奖励，在所有写作特质和论文题目上都取得了进一步的提升。我们的研究结果表明，明确的督导比单纯提示更能实现更强的形象对齐，而真实的低熟练度写作仍然难以复制。

Long-Horizon Consistent and Interaction-Aware World Models for Multi-Style End-to-End Driving

长视界一致且交互感知的世界模型，用于多风格端到端驾驶

• Authors: Yuxuan Han, Kunyuan Wu, Liyunong Yang, Zilu Wang, Cansen Jiang, Yi Xiao, Liang Hu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.03225
• Pdf link: https://arxiv.org/pdf/2609.03225
• Abstract End-to-end autonomous driving has increasingly adopted world model-based reinforcement learning frameworks to improve learning efficiency through \textit{imagined rollouts}. However, existing world models suffer from three key limitations: temporal inconsistency in long-horizon imagined rollouts, inadequate modeling of ego-environment interactions, and limited adaptability to diverse driving styles. To address these challenges, we propose \textit{StyleDrive}, a world-model-based learning framework that jointly enforces long-horizon consistency, explicitly disentangles interactive traffic states, and supports multi-style policy optimization within a unified learning paradigm. First, we introduce a temporal consistency regularization that integrates historical latent states through gated cross-attention, stabilizing long-horizon imagined rollouts and mitigating error accumulation. Second, we design an explicit state disentanglement module that separates ego-relevant from ego-irrelevant interactive states, enabling more interpretable and efficient decision-making in complex traffic scenarios. Third, we enable multi-style driving behaviors through Group Relative Policy Optimization, which replaces per-step reward optimization with trajectory-wise relative advantages, reducing reward variance and supporting diverse driving styles without retraining. We evaluate StyleDrive on the Bench2Drive closed-loop driving benchmark, achieving a driving score of 88.44 (+17.08 over the previous best world model-based method) and a success rate of 66.82 (+16.58). Furthermore, we deploy StyleDrive on a real automated guided vehicle platform and demonstrate promising sim-to-real transfer capability in dynamic driving scenarios.
• 中文摘要 端到端自动驾驶越来越多地采用基于世界模型的强化学习框架，通过 \textit{想象的推广}提升学习效率。然而，现有世界模型存在三大关键局限：长视野想象推展的时间不一致、自我-环境互动建模不足，以及对不同驾驶风格的适应有限。为应对这些挑战，我们提出了 \textit{StyleDrive}，这是一个基于世界模型的学习框架，能够共同强制长视野一致性，明确解开交互式交通状态，并在统一学习范式内支持多风格策略优化。首先，我们引入了一种时间一致性正则化，通过门控交叉注意力整合历史潜在状态，稳定长视野想象推送并减少错误积累。其次，我们设计了一个显式状态纠缠模块，将自我相关和无自我相关的交互状态分离开来，使复杂交通场景下的决策更具可解释性和效率。第三，我们通过群体相对策略优化实现多风格驾驶行为，该优化用轨迹上的相对优势取代了每步奖励优化，减少奖励变异，支持多样化驾驶风格而无需重新训练。我们在Bench2Drive闭环驾驶基准测试中评估StyleDrive，获得88.44分（比之前最佳世界模型方法+17.08分）和66.82分（+16.58分）的成功率。此外，我们将StyleDrive部署在真实自动引导车辆平台上，展示了动态驾驶场景下的模拟到实物转移能力。

Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification

基于强化学习和不确定性量化，风险与异常识别以实现配电网络最优运行

• Authors: Ziqi Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.03308
• Pdf link: https://arxiv.org/pdf/2609.03308
• Abstract Reliable operation of modern distribution networks requires timely identification of operational risks and anomalous events under pervasive uncertainty. In practice, operators must identify risks that are inherent in stochastic yet in-distribution conditions, and anomalies that correspond to out-of-distribution behaviors such as unusual load patterns, extreme weather or cyber-physical attacks. This paper addresses this joint risk and anomaly identification problem for optimal distribution network operation and proposes a deep reinforcement learning framework that is explicitly uncertainty aware. We integrate distributional and Bayesian deep reinforcement learning to realize a second- order uncertainty quantification scheme that decomposes total uncertainty into aleatoric and epistemic components, which are respectively used to characterize inherent risk and out-of- distribution anomalies. The resulting epistemic estimates drive both exploration during training and out-of-distribution detec- tion with fallback control during deployment, whereas aleatoric estimates are used to characterize intrinsic operational risk. Simulation results demonstrate the performance of our DRL agent and the effectiveness of the uncertainty quantification.
• 中文摘要 现代配电网络的可靠运行需要及时识别在普遍不确定性下运行风险和异常事件。在实践中，操作员必须识别存在于随机但分布内的内在风险，以及与异常负载模式、极端天气或网络攻击等非分配行为相关异常。本文探讨了这种风险与异常识别的联合问题，以实现最佳配电网络运行，并提出了一种明确感知不确定性的深度强化学习框架。我们整合分布和贝叶斯深度强化学习，实现一种二阶不确定性量化方案，将完全不确定性分解为偶然性和认识成分，分别用于描述固有风险和分布外异常。所得的认知估计既推动训练期间的探索，也推动部署期间的后备控制，同时利用偶然估计来表征内在操作风险。模拟结果展示了我们DRL代理的性能及不确定性量化的有效性。

TIPCODER: Reinforcement Learning Boosted Test-time Instruction Proposer for Code Generation

TIPCODER：强化学习增强测试时间的代码生成指令提案

• Authors: Minyu Chen, Sihao Wu, Ling-I Wu, Song Qin, Jingyang Li, Lei Ning, Jianxin Xue, Guoqiang Li
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2609.03309
• Pdf link: https://arxiv.org/pdf/2609.03309
• Abstract Test-time scaling for code generation typically explores the solution space by sampling multiple programs from a fixed instruction. We study a complementary direction: instance-level instruction-space exploration. Our observation is that many coding failures stem from missing constraints, overlooked edge cases, or misleading reasoning paths induced by the original prompt. To address this, we propose TipCoder, a test-time instruction proposer that generates problem-specific auxiliary tips before code synthesis. TipCoder distills multi-turn debugging trajectories into proactive guidance and further optimizes the Proposer with reinforcement learning using a marginal-utility reward. At inference time, it generates both a base solution and a tip-guided solution, and applies a Reward Model for post-hoc selection. This exploration-selection design allows tips to expose additional candidate potential while reducing regressions from unnecessary guidance. Across the evaluated code-generation benchmarks and target Code LLMs, TipCoder provides a consistent instruction-level test-time scaling strategy, comparing favorably with stochastic sampling and generic prompt optimization baselines under a shared reward-model-based selection protocol.
• 中文摘要 代码生成的测试时间缩放通常通过从固定指令中抽样多个程序来探索解空间。我们研究一个互补方向：实例级指令空间探索。我们的观察是，许多编码失败源于缺失约束、被忽视的边缘情况或原始提示引发的误导性推理路径。为此，我们提出了TipCoder，一种测试时指令提案器，在代码合成前生成问题特定的辅助提示。TipCoder将多回合调试轨迹提炼为主动指导，并通过边际效用奖励进一步优化提案者，进行强化学习。在推理阶段，它生成基础解和提示引导解，并应用奖励模型进行事后选择。这种探索-选择设计使提示能够暴露更多候选潜力，同时减少不必要引导带来的回归。在评估的代码生成基准测试和目标代码大型语言模型中，TipCoder 提供了一致的指令级测试时间缩放策略，在共享奖励模型选择协议下，与随机抽样和通用提示优化基线相比表现优异。

DE-Venus: A Data-Efficient RLVR Framework for Large Language Models

DE-Venus：大型语言模型的数据高效RLVR框架

• Authors: Shenzhi Yang, Guangcheng Zhu, Kai Tang, Zhengqing Zang, Xing Zheng, Haobo Wang, Yingfan Ma, Bowen Song, Bo Han, Bo An, Lei Feng, Weiqiang Wang, Junbo Zhao, Gang Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.03324
• Pdf link: https://arxiv.org/pdf/2609.03324
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves large language model reasoning, but its practical scaling is constrained by expensive on-policy rollouts and the cost of obtaining reliable targets at scale. Existing methods address sample selection, incomplete supervision, or noisy labels separately, often entangling supervision logic with distributed training and hindering controlled comparison and reuse. We present DE-Venus, a unified framework for data-efficient RLVR that treats supervision as evolving state across data preparation and policy optimization. It organizes this lifecycle into three modules: Active Data Selection allocates training and annotation budgets; Weak Supervision Construction derives learning signals from unlabeled examples; and Training-Time Supervision Refinement filters or corrects unreliable supervision. DE-Venus supports seven representative methods and a data-selection pipeline by expressing method-specific decisions as offline dataset transitions or online transformations of targets, rewards, batches, and advantages while preserving verl's distributed execution contracts. Across public benchmarks and three business scenarios, separate configurations preserve or improve model quality with only 10% of labels or as little as 13% of relevant data; selected business configurations also reduce observed convergence steps by 63%--75%. DE-Venus thus reduces annotation and training costs without sacrificing scalable RL execution.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了大型语言模型推理能力，但其实际扩展受限于昂贵的策略部署和大规模获取可靠目标的成本。现有方法分别处理样本选择、不完整监督或噪声标签，常将监督逻辑与分布式训练纠缠，阻碍受控比较和重用。我们提出了DE-Venus，一个统一的数据高效RLVR框架，将监督视为数据准备和策略优化过程中不断演变的状态。它将生命周期组织为三个模块：主动数据选择分配训练和注释预算;弱监督构建从未标记的实例中推导学习信号;训练时间监督细化过滤或修正不可靠监督。DE-Venus支持七种代表性方法和数据选择流水线，通过将方法特定的决策表达为离线数据集转换或目标、奖励、批次和优势的在线转换，同时保持Verl的分布式执行契约。在公共基准测试和三种业务场景中，单独配置仅保留或提升模型质量，且仅有10%的标签或仅13%的相关数据;选定的业务配置还将观察到的收敛步数减少63%至75%。因此，DE-Venus降低了注释和训练成本，同时不牺牲可扩展的强化学习执行。

Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

梯度知道结果不了解的事：通过梯度对齐奖励解锁LLM推理的强化学习

• Authors: Leqi Zheng, Jinbo Su, Fang Niu, Chaokun Wang, Weiping Wang, Jiajun Zhang, Shannan Yan, Jie Wu, Zhaolu Kang, Rong Fu, Hang Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.03342
• Pdf link: https://arxiv.org/pdf/2609.03342
• Abstract Reinforcement learning from verifiable rewards (RLVR) drives chain-of-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation. We propose Gradient-Aligned Reward (GAR), which operates in the policy's own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert-anchor gradient yields a dense, reasoning-aware reward with less than 9% wall-clock overhead. We prove that this cosine admits a multiplicative decomposition into prediction-error and activation-pattern factors, providing a concrete characterization of what the alignment signal measures. On Qwen3-4B and Qwen3-8B, GAR consistently improves over GRPO and other baselines on competition-level math benchmarks and transfers to GPQA Diamond and MMLU-Pro without domain-specific data. Code and data are available at this https URL.
• 中文摘要 可验证奖励强化学习（RLVR）在大型语言模型中推动思维链推理，但其二元结果奖励无法区分正确的轨迹。现有的密集奖励替代方案，从表面启发式到过程奖励模型，要么忽视训练语料库中已有的专家解，要么需要昂贵的离线注释。我们提出了梯度对齐奖励（GAR），其运行于策略自身的梯度空间：通过输出投影层的截断反向传播为每个展开提取紧凑梯度向量，且与专家锚点梯度的余弦相似性可获得密集且具推理意识的奖励，且墙时钟开销低于9%。我们证明该余弦允许对预测误差和激活模式因子的乘法分解，具体描述了比对信号所测量的因素。在Qwen3-4B和Qwen3-8B上，GAR在竞赛级数学基准测试中持续优于GRPO及其他基线，并且无需特定领域数据即可转入GPQA Diamond和MMLU-Pro。代码和数据可在此https URL获取。

StrixAE: An Intelligent Agent for Audio Enhancement under Complex Distortion Coupling in Real-World Scenarios

StrixAE：在复杂失真耦合下实现音频增强的智能代理

• Authors: Chenglin Wu, Junjie Wu, Jinhang Chen, Mingyang Chen, Zixu Lin, Jiabian Chen, Xinghao Ding, Xiaotong Tu
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.03414
• Pdf link: https://arxiv.org/pdf/2609.03414
• Abstract Audio enhancement in real-world scenarios involves complex distortion couplings and requires personalized enhancement. Existing solutions struggle to address both simultaneously. To improve robustness and enable autonomous operation in such scenarios, we propose StrixAE, an agent based on a multimodal large language model (MLLM). StrixAE leverages the MLLM as a controller to coordinate multiple audio enhancement and personalization models. To further enhance system robustness, reduce artifacts, and improve generalization across diverse real-world scenarios, StrixAE is trained through a two-stage process: first, CoT supervised fine-tuning on AcoustBench to ground basic reasoning and tool invocation; second, Audio Perception Reinforcement Learning (APRL), a reward design specifically tailored for audio restoration pipelines that jointly optimizes format validity, structural coherence, and perceptual quality. Unlike generic RL fine-tuning, APRL introduces structured rewards that enforce executable pipelines and logical section ordering, enabling the agent to produce reliable, interpretable enhancement plans without hallucinated tools. Based on real-world test datasets, our proposed method outperforms most existing open-source and proprietary solutions, achieving state-of-the-art performance across multiple perceptual metrics and demonstrating strong generalization robustness.
• 中文摘要 现实场景中的音频增强涉及复杂的失真耦合，需要个性化增强。现有解决方案难以同时满足这两者。为提升鲁棒性并实现此类场景中的自主运行，我们提出了基于多模态大型语言模型（MLLM）的StrixAE代理。StrixAE利用MLLM作为控制器协调多个音频增强和个性化模型。为进一步提升系统鲁棒性、减少伪影并改善在多样现实场景中的泛化性，StrixAE通过两阶段过程进行训练：首先，CoT监督的AcoustBench微调，将基础推理和工具调用接地;其次，音频感知强化学习（APRL），这是一种专为音频恢复流程量身定制的奖励设计，协同优化格式有效性、结构一致性和感知质量。与通用的强化学习微调不同，APRL引入了结构化奖励，强制执行可执行的管道和逻辑段排序，使智能体能够在不依赖幻觉工具的情况下生成可靠、可解释的增强计划。基于真实测试数据集，我们提出的方法优于大多数现有开源和专有解决方案，在多感知指标上达到最先进性能，并展现出强大的泛化鲁棒性。

LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL

LeanGRPO：消除扩散强化学习中的冗余重计算

• Authors: Sijie Wang, Zhiqiang Tan, Xinrui Yang, Shaohuai Shi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
• Arxiv link: https://arxiv.org/abs/2609.03528
• Pdf link: https://arxiv.org/pdf/2609.03528
• Abstract Diffusion reinforcement learning (RL) has recently achieved significant success in post-training image and video generative models. However, most diffusion RL methods, including DanceGRPO and FlowGRPO, recompute selected timesteps with gradient tracking after rollout. Under on-policy training with the same backend for rollout and update, this recomputation is mathematically redundant. Intuitively, the rollout and policy update steps can reuse the same feed-forward backbone to avoid redundant computation, but doing so can incur a large memory overhead during rollout. To address the issue, we present LeanGRPO by restructuring the data-parallel layout and introducing two recompute-free training schedules for trajectory-logprob diffusion RL: (1) LeanGRPO-Retain enables gradient tracking during rollout and directly reuses the resulting computation graphs and saved activations for backward during update, requiring no recomputation; and (2) LeanGRPO-Reweight also enables gradients during rollout, but immediately backpropagates each selected step using a provisional advantage and delays gradient synchronization, then corrects the provisional gradients with the true advantage after the trajectory is completed. These schedules target different model scales and input sizes. Across FlowGRPO/DanceGRPO with FLUX.1-dev and Wan, LeanGRPO achieves up to 1.83x end-to-end speedup while preserving the original optimization objective.
• 中文摘要 扩散强化学习（RL）最近在训练后图像和视频生成模型中取得了显著成功。然而，大多数扩散强化学习方法，包括 DanceGRPO 和 FlowGRPO，都是在部署后通过梯度跟踪重新计算选定时间步。在策略内训练中，使用相同的后端进行推广和更新，这种重新计算在数学上是冗余的。直观上，推广和策略更新步骤可以重复使用相同的前馈骨干网以避免冗余计算，但这样做在推广过程中可能会产生较大的内存开销。为解决此问题，我们通过重构数据并行布局并引入两种无重计算的轨迹-对数扩散RL训练计划来介绍LeanGRPO：（1） LeanGRPO-Retain在推展时支持梯度跟踪，并直接重用生成的计算图和保存的激活数据，在更新时反向使用，无需重新计算;（2） LeanGRPO-Reweight也支持推展时的梯度，但会立即利用临时优势反向传播每个选定步骤，延迟梯度同步，然后在轨迹完成后以真正的优势修正临时梯度。这些调度针对不同的模型尺度和输入大小。在FlowGRPO/DanceGRPO中，LeanGRPO通过FLUX.1-dev和Wan实现了最高1.83倍的端到端加速，同时保持了原始优化目标。

A Semantic-Aware Multiple Access Scheme Leveraging Spatial Redundancy for Uplink-Dominant Network Services

一种利用空间冗余实现上行主导网络服务的语义感知多址方案

• Authors: Hamidreza Mazandarani, Masoud Shokrnezhad, Tarik Taleb
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.03559
• Pdf link: https://arxiv.org/pdf/2609.03559
• Abstract The transition toward semantic-aware communication offers a paradigm shift for next-generation mobile networks, promising to decouple information significance from raw data transmission. Despite advances in semantic extraction, the integration of semantic intelligence into the Medium Access Control (MAC) layer remains underexplored, particularly in exploiting spatial correlations among users. To address this, we introduce a novel multiple access scheme designed for uplink-dominant network services. This framework optimizes the trade-off between spectrum utilization and sustainability by formulating variable-packet-length access as distinct $\alpha$-fairness and energy efficiency problems. A key innovation of our approach is the quantification of spatial redundancies through novel metrics of self-throughput and assisted-throughput, which account for the semantic correlation of data across user equipment. We analyze these formulations to identify optimal bounds before proposing PRISM (Protocol for Redundancy Identification in Semantic Multiple-access). Grounded in Model-free Multi-Agent Deep Reinforcement Learning (MADRL), PRISM enables devices to autonomously govern spectrum access using only local observations. Extensive evaluations demonstrate that PRISM successfully leverages redundancies to outperform semantic-oblivious schemes, achieving up to ({90\%}) of the centralized optimal benchmark and improving both objectives by up to ({2\times}) across diverse user-semantic association matrices. These results validate PRISM as a viable candidate for future distributed mobile network applications, complemented by orthogonal Multiple Access Schemes where signals are multiplexed in the semantic domain.
• 中文摘要 向语义感知通信的转变为下一代移动网络带来了范式转变，有望将信息的重要性与原始数据传输解耦。尽管语义提取取得了进步，将语义智能整合进媒介接入控制（MAC）层仍未被充分探索，尤其是在利用用户间空间相关性方面。为此，我们引入了一种为上行主导网络服务设计的新型多址方案。该框架通过将可变数据包长度访问定义为独立的$\alpha$公平性和能效问题，优化频谱利用与可持续性的权衡。我们方法的一个关键创新是通过自我吞吐量和辅助吞吐量等新指标量化空间冗余，这些指标考虑了用户设备间数据的语义相关性。我们分析这些表述以确定最优边界，然后提出PRISM（语义多址冗余识别协议）。基于无模型多智能体深度强化学习（MADRL），PRISM使设备能够仅凭局部观测自主管理频谱访问。大量评估表明，PRISM成功利用冗余性能优于语义无知方案，实现集中式最优基准的最高 \（{90\%}\），并在不同用户语义关联矩阵中提升两项目标多达 \（{2\times}\）。这些结果验证了 PRISM 作为未来分布式移动网络应用的可行候选者，辅以正交多路接入方案，其中信号在语义域中复用。

From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control

从先导启发式到可部署代理：加速演示驱动的强化学习以实现截止时间限制的网络控制

• Authors: Vincenzo Norman Vitale, Mohammad Solki, Antonia Maria Tulino, Andreas F. Molisch, Jaime Llorca
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.03590
• Pdf link: https://arxiv.org/pdf/2609.03590
• Abstract Timely delivery of delay-sensitive information over dynamic, heterogeneous networks is essential for NextG interactive applications, yet providing strict End-to-End (E2E) peak latency guarantees remains an open challenge. Two obstacles limit the adoption of learning-based network control in this setting: traditional volume-based routing metrics, while highly effective for general traffic management, are not designed to capture traffic urgency; and Deep Reinforcement Learning (DRL) controllers trained from scratch suffer from sample inefficiency, long training times, and early-stage exploration volatility. This paper introduces a deployment-focused network control framework that addresses both obstacles. First, we present Effective Congestion (EC), a deadline-aware metric family that quantifies interface congestion by packet urgency and proactively filters non-viable traffic, coupled with a Uniform Path Grouping (UPG) distribution heuristic promoting robust load-balancing; the resulting policies are embedded into Multi-Agent Deep Reinforcement Learning Effective Congestion ($p^$) (MADRL EC ($p^$)), a hybrid architecture combining a distributed scheduler with a centralized RL-based router. Second, we introduce a unified training objective that generalizes existing policy-learning paradigms---behavioral cloning, offline Reinforcement Learning (RL), online RL, and offline-to-online schemes---as special cases, combining a live-reward term, a pre-collected-reward term, and a policy-imitation term. From this objective, we derive the Model-Guided Annealed Reinforcement Learning (MGA-RL) protocol, instantiated on a Deep Deterministic Policy Gradient (DDPG) backbone: a deployment-oriented, demonstration-driven training approach that generalizes conventional Offline-to-Online (O2O) schemes, in which trajectories from a lightweight [...]
• 中文摘要 在动态、异构网络上及时传递延迟敏感信息对于NextG交互应用至关重要，但要实现严格的端到端（E2E）峰值延迟保证仍是一个开放挑战。在这种环境下，基于学习的网络控制有两个障碍：传统的基于卷的路由指标虽然对一般流量管理非常有效，但并不用于捕捉流量紧急情况;而从零训练的深度强化学习（DRL）控制器则存在样本效率低、训练时间长和早期探索波动性的问题。本文介绍了一个以部署为重点的网络控制框架，解决了这两个障碍。首先，我们介绍了有效拥塞（EC），这是一个基于截止时间的指标家族，通过数据包紧急程度量化接口拥塞，并主动过滤不可行流量，结合统一路径分组（UPG）分布启发式方法，促进了稳健的负载均衡;由此产生的策略嵌入多智能体深度强化学习有效拥塞（$p^$）（MADRL EC （$p^$））中，这是一种结合分布式调度器和集中式强化学习路由器的混合架构。其次，我们引入统一的训练目标，推广现有策略学习范式---行为克隆、离线强化学习（RL）、在线强化学习和离线到在线方案---作为特例，结合了 活奖励项、预收集奖励项和策略模仿项。基于这一目标，我们推导出模型引导退火强化学习（MGA-RL）协议，基于深度确定性策略梯度（DDPG）骨干：一种部署导向、演示驱动的训练方法，推广了传统的离线到在线（O2O）方案，其中轨迹从轻量级[...]

Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learning

离线多智能体强化学习中的序列模型的分布外推广

• Authors: Oussama Hidaoui, Omer Ebead, Ulrich Armel Mbou Sob, Siddarth Singh, Juan Claude Formanek, Felix Chalumeau, Omayma Mahjoub, Sasha Abramowitz, Ruan John de Kock, Wiem Khlifi, Louay Ben Nessir, Simon Verster Du Toit, Daniel Rajaonarivonivelomanantsoa, Asim Awad Osman, Arnol Manuel Fokam, Refiloe Shabe, Arnu Pretorius
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.03667
• Pdf link: https://arxiv.org/pdf/2609.03667
• Abstract Generalising to unseen tasks remains a fundamental challenge in offline multi-agent reinforcement learning (MARL). In this work, we present a principled analysis of zero-shot task generalisation in the offline setting and conduct an extensive empirical investigation into the scaling behaviour governing task diversity, dataset size, and network capacity. To facilitate this study, we extend offline sequence modelling architectures to handle multi-task observation and action spaces alongside variable agent counts across tasks. Our primary finding is that scaling task diversity---rather than sheer dataset size is the dominant factor in achieving robust zero-shot transfer. Through large-scale experiments across four challenging environments (Connector, RWARE, SMAX, and LBF), we demonstrate that our multi-task approach achieves a mean improvement of 3.2x on held-out test tasks compared to single-task models and consistently outperforms strong behaviour cloning baselines. These results suggest that the development of generalisable MARL agents should prioritise the diversity of the training distribution with varying numbers of agents, providing a roadmap for scaling offline MARL effectively.
• 中文摘要 推广到看不见任务仍然是离线多智能体强化学习（MARL）中的根本挑战。本研究提出了离线环境中零样本任务泛化的原则性分析，并对控制任务多样性、数据集规模和网络容量的尺度行为进行了广泛的实证调查。为促进本研究，我们扩展了离线序列建模架构，以处理多任务观察和动作空间以及任务间可变的代理数量。我们的主要发现是，任务多样性的缩放---而非纯粹的数据集规模是实现稳健零样本转移的主要因素。通过在四个具有挑战性的环境（Connector、RWARE、SMAX和LBF）进行大规模实验，我们证明了我们的多任务方法相比单任务模型，在未完成测试任务上平均提升3.2倍，并且持续优于强行为克隆基线。这些结果表明，开发可推广的MARL代理应优先考虑训练分布的多样性，以不同数量的代理，为有效扩展离线MARL提供路线图。

LLM4AIGQ: LLM-based AI Guidance Query Generation Framework for Multi Interest Mining

LLM4AIGQ：基于LLM的AI引导查询生成框架，用于多兴趣挖矿

• Authors: Xiangchen Pan, Jiayi Xu, Jing Wang, Xing Fang, Lingyun Zhu
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.03674
• Pdf link: https://arxiv.org/pdf/2609.03674
• Abstract Guidance queries stimulate user consumption by extracting preferences to provide search queries with guidance value, playing a crucial role in the e-commerce field. Traditional AI-generated queries (AIGQ) generation primarily relies on a two-stage "Query-to-AI-Generated-Query" (Q2AIGQ) association paradigm, first recalling user primary search queries from user profiles, historical behavior sequences, item-side information, and the current query through multi-path retrieval, then generalizing AIGQ via rule-based methods. This approach suffers from semantic drift due to information cascade loss; additionally, primary search query derivation heavily depends on "user-item" co-occurrence relationships, lacking exploration of user multi-interests, resulting in guidance queries with low value and mismatched purchase intent. To address the expressive limitations of traditional co-occurrence-based retrieval, we propose LLM4AIGQ, an LLM-based solution for generating AI guidance queries tailored to users' multi-interests. This approach segments user interests by integrating user profiles and historical interaction sequences, infers specific consumption intents for each sub-interest, and subsequently generates corresponding AIGQ. In terms of model training, we employ a post-training pipeline comprising Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), and Direct Preference Optimization (DPO) to enhance the model's capability in generating AIGQ. We also introduce a multi-level reward design to satisfy the requirements of multi-objective optimization and long-chain reasoning in practical applications. Regarding deployment, we adopt a nearline-generation and online-read architecture to meet latency constraints. Extensive experimental analyses demonstrate that our model achieves robust performance in both offline evaluations and online A/B tests.
• 中文摘要 指导查询通过提取偏好，为搜索查询提供指导价值，从而激发用户消费，在电子商务领域发挥关键作用。传统的AI生成查询（AIGQ）主要依赖于两阶段的“查询到AI生成查询”（Q2AIGQ）关联范式，首先通过多路径检索从用户资料、历史行为序列、条目侧信息和当前查询中回忆用户主要搜索查询，然后通过基于规则的方法对AIGQ进行泛化。该方法存在信息级联丢失导致语义漂移;此外，主要搜索查询的推导高度依赖“用户-项目”共现关系，缺乏对用户多重兴趣的探索，导致引导查询价值低且购买意图不匹配。为解决传统共现检索的表达限制，我们提出了LLM4AIGQ，这是一种基于LLM的解决方案，用于生成针对用户多兴趣的AI指导查询。该方法通过整合用户画像和历史交互序列来细分用户兴趣，推断每个子兴趣的具体消费意图，并生成相应的AIGQ。在模型训练方面，我们采用了包含监督微调（SFT）、强化学习（RL）和直接偏好优化（DPO）的训练后流程，以增强模型生成AIGQ的能力。我们还引入了多级奖励设计，以满足多目标优化和长链推理在实际应用中的需求。在部署方面，我们采用近线生成和在线读取架构以满足延迟限制。广泛的实验分析表明，我们的模型在离线评估和在线A/B测试中均表现出色。

WISE: World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models

WISE：世界模型引导的想象调度，实现视觉-语言-行动模型的高效后期训练

• Authors: Chenhao Zhang, Hanyu Zhao, Hang Cheng, Tengfei Pan, Long Zeng
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.03681
• Pdf link: https://arxiv.org/pdf/2609.03681
• Abstract Post-training VLA policies typically rely on supervised fine-tuning with costly expert demonstrations or reinforcement learning with expensive and potentially unstable real-world exploration. World models offer a promising alternative by evaluating candidate behaviors through imagined futures, yet effective post-training requires more than accurate prediction: imagination must be scheduled where it is useful, bounded within reliable horizons, and translated into trustworthy policy supervision. In robotic manipulation, the value of imagination varies substantially across execution stages, while extended rollouts can accumulate prediction errors and introduce unreliable learning signals. We introduce WISE (World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models), a unified framework that coordinates when and how world-model imagination is used during policy refinement. WISE selectively invokes imagination at interaction-relevant states, performs bounded multi-view rollouts, evaluates candidate futures using progress and completion signals, and uses their relative outcomes to refine actions generated from real interaction contexts. Extensive experiments with both $\pi_0$ and $\pi_{0.5}$ demonstrate consistent improvements across diverse manipulation tasks while reducing GPU computation time by approximately 80% compared with full imagination. Real-world evaluations further show substantial gains in robustness and generalization under diverse real-world distribution shifts.
• 中文摘要 训练后VLA政策通常依赖于通过昂贵的专家演示进行监督微调，或通过昂贵且可能不稳定的现实世界探索进行强化学习。世界模型通过通过想象未来评估候选行为提供了有前景的替代方案，但有效的后训练不仅需要准确预测：想象力必须安排在有用的地方，限制在可靠的视野内，并转化为可信的政策监督。在机器人操作中，想象力的价值在执行阶段间差异显著，而长期推广则可能积累预测误差并引入不可靠的学习信号。我们介绍WISE（世界模型引导想象力调度，用于视觉-语言-行动模型高效后期训练），这是一个统一框架，协调世界模型想象力在政策细化中何时以及如何使用。WISE在交互相关状态下选择性地唤起想象力，执行有界多视角展开，利用进度和完成信号评估候选未来，并利用其相对结果细化来自真实交互上下文的动作。对$\pi_0$和$\pi_{0.5}$的广泛实验显示，在多样化的操作任务中持续提升，同时GPU计算时间比完全想象减少约80%。现实世界评估进一步显示，在多种现实世界分布变化下，稳健性和泛化性显著提升。

Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoning

展开世界：在强化空间推理中分解四维属性

• Authors: Yijun Yang, Shenghe Zheng, Wenbo Li, Jianhui Liu, Haoze Sun, Yanbing Zhang, Jiaxiu Jiang, Lin Song, Haoyang Huang, Nan Duan, Lei Zhu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.03729
• Pdf link: https://arxiv.org/pdf/2609.03729
• Abstract Despite the remarkable prowess of Vision-Language Models (VLMs) in general multimodal tasks, they remain fundamentally flat'' when reasoning about the physical world. We argue that this spatial bottleneck stems from a profound dimensional mismatch: while VLMs are trained to interpret 2D projections, true spatial reasoning demands the recovery of latent 3D geometry and temporal continuity. To conquer this high-dimensional complexity, we advocate a shift from monolithic learning to adivide and conquer'' paradigm. We present FactoSR, a factorized reinforcement learning framework that explicitly interpret the dimensions collapsed by visual projection. At its core, FactoSR decomposes the monolithic problem of world-consistent reasoning into three orthogonal, geometric sub-objectives: planar correspondence ($XY$), depth consistency ($Z$), and temporal reversibility ($T$). By optimizing these verifiable constraints within a unified policy learning mechanism, we effectively transform an ill-posed projection recovery problem into a series of tangible reasoning steps. Extensive evaluations on multi-view and video benchmarks demonstrate that this elegant decomposition yields substantial gains in 3D and 4D reasoning, achieving a 5.9% boost on VSI-Bench and 4.5% on All-Angles-Bench. Our findings suggest that reinforcing explicit, factorized 4D consistency is a critical step toward evolving VLMs into robust, world-aware reasoners.
• 中文摘要 尽管视觉语言模型（VLMs）在多模态任务中表现出色，但在推理物理世界时，它们本质上仍是“平面”的。我们认为，这一空间瓶颈源于深刻的维度错配：虽然VLM被训练用于解释二维投影，但真正的空间推理要求恢复潜在的三维几何和时间连续性。为克服这种高维复杂性，我们主张从单一学习转向“分而治之”范式。我们提出了FactoSR，一种分解化强化学习框架，明确解释视觉投影所压缩的维度。FactoSR的核心将世界一致性推理这一单一问题分解为三个正交几何子目标：平面对应（$XY$）、深度一致性（$Z$）和时间可逆性（$T$）。通过在统一的策略学习机制中优化这些可验证约束，我们有效地将错误态的投影恢复问题转化为一系列切实可行的推理步骤。对多视角和视频基准的广泛评估表明，这种优雅的分解在三维和四维推理方面取得了显著提升，在VSI-Bench上实现了5.9%的提升，在All-Angles-Bench上提升了4.5%。我们的发现表明，强化显式、分解化的四维一致性是将VLM发展为强大、具世界感知能力推理器的关键一步。

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

SVG-Score：文本到SVG生成的人类对齐评估

• Authors: Marco Cipriano, Leonardo Zini, Alexandra Schild, Valentin Teutschbein, Afsana Mimi, Marcella Cornia, Lorenzo Baraldi, Gerard de Melo
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.03806
• Pdf link: https://arxiv.org/pdf/2609.03806
• Abstract Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progress, however, is held back by the lack of domain-specific evaluation protocols: current practice relies on metrics designed for natural images, most notably CLIPScore, which was never trained on vector graphics and aligns only partially with human judgment. We introduce \textbf{\ours}, a human-aligned evaluation framework for text-to-SVG generation. Through controlled caption and image perturbations, we first show that CLIP-based scores barely react to the errors SVG generators actually make, such as wrong colors, counts, and spatial relations, and that off-the-shelf Vision-Language Model (VLM) judges, while more sensitive, respond unevenly across error types and SVG styles. We then introduce a human-annotated dataset for \textit{Semantic Alignment}, measuring how faithfully a generated SVG reflects its caption. Building on it, we develop two complementary evaluators: CLIP scorers adapted to vector graphics and then aligned to human preferences, for fast large-scale evaluation, and a VLM judge trained with supervised fine-tuning and reward-shaped reinforcement learning, for more expressive and interpretable assessment. Using both, we benchmark major open-source, commercial, and optimization-based SVG generators on an independent caption set.
• 中文摘要 随着生成模型在表现力和可控性上的提升，可扩展矢量图形（SVG）生成正受到越来越多的关注。然而，进展受限于缺乏领域特定的评估协议：目前的实践依赖于为自然图像设计的指标，最显著的是CLIPScore，该系统从未训练过矢量图形，且仅部分符合人类判断。我们介绍了\textbf{\ours}，一种基于人类的文本到SVG生成评估框架。通过受控的标题和图像扰动，我们首先展示了基于CLIP的评分几乎不会对SVG生成器实际犯的错误作出反应，如错误的颜色、计数和空间关系，而现成的视觉语言模型（VLM）评判虽然更灵敏，但在错误类型和SVG样式间的响应不均。随后，我们引入了 \textit{语义对齐}的人工注释数据集，测量生成的 SVG 对其说明文字的忠实反映程度。在此基础上，我们开发了两个互补的评估器：适应矢量图形并对齐人类偏好的 CLIP 评分器，实现快速大规模评估;以及经过监督微调和奖励形强化学习训练的 VLM 评判，提供更具表现力和可理解性的评估。利用两者，我们在独立的字幕集上对主要开源、商业和优化型 SVG 生成器进行基准测试。

Multi-step Proximal Policy Improvement in Offline Reinforcement Learning

离线强化学习中的多步近端策略改进

• Authors: Soohyun Choi, Seonvin Cho, Songnam Hong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.03842
• Pdf link: https://arxiv.org/pdf/2609.03842
• Abstract Offline reinforcement learning (RL) must reconcile two competing requirements: policy updates should stay near dataset-supported actions to keep value estimates reliable, yet meaningful gains often require moving beyond the behavior distribution. We develop a geometric view of offline actor updates by modeling policies as a probability manifold endowed with a chosen metric geometry. Under this lens, a broad class of offline actor objectives can be interpreted as a single proximal policy improvement step (SPI), i.e., an implicit discretization of a manifold gradient flow induced by a critic-defined energy. Building on this insight, we propose multi-step proximal policy improvement (MPI), a plug-in refinement mechanism that composes sequential re-centered proximal steps. MPI enables controlled policy improvement beyond dataset support while retaining proximal control at each refinement. The framework accommodates multiple policy geometries and admits practical instantiations for deterministic and diagonal-Gaussian policies. Experiments on D4RL benchmarks show that small numbers of MPI refinements improve strong offline baselines, including TD3+BC, ReBRAC, and IQL, on many tasks. Focused diagnostics further distinguish re-centered refinement from fixed-objective update scheduling and characterize limitations under critic error.
• 中文摘要 离线强化学习（RL）必须调和两个相互竞争的要求：策略更新应保持接近数据集支持的动作以保持价值估计的可靠性，但有意义的收益往往需要超越行为分布。我们通过将策略建模为赋予选定度量几何的概率流形，构建了离线演员更新的几何视图。在这一视角下，广泛的离线行动者目标可以被解释为单一的近端策略改进步骤（SPI），即由批评者定义的能量引发的流形梯度流的隐式离散化。基于这一见解，我们提出了多步近端策略改进（MPI），这是一种插件式细化机制，由顺序重新中心的近端步骤组成。MPI使得在每次细化中保持近端控制的同时，实现数据集支持之外的受控策略改进。该框架兼容多种策略几何，并允许确定性和对角高斯策略的实用实例化。在D4RL基准测试上的实验表明，少量MPI细化能提升许多任务中的强离线基线，包括TD3+BC、ReBRAC和IQL。聚焦诊断进一步区分了重中心细化与固定目标更新调度，并描述了批判误差下的限制。

EF1-Constrained Nash Social Welfare with Identical Additive Valuations: Complexity, Guarantees, and Experiments

EF1约束纳什社会福利与相同加法估值：复杂性、保证与实验

• Authors: Zih-Sian Yang, Yi-Hao Chen, Yu-Te Kuan, Cheng-Jui Wu, Chuang-Chieh Lin, Po-An Chen
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.03846
• Pdf link: https://arxiv.org/pdf/2609.03846
• Abstract We study the allocation of indivisible goods among agents with identical additive valuations, focusing on envy-freeness up to one good (EF1) and Nash social welfare (NSW). Since every maximum-NSW allocation is EF1 under additive valuations, the associated threshold problem inherits the known strong NP-hardness of NSW maximization under identical additive valuations and is strongly NP-complete. We therefore focus on welfare guarantees satisfied by arbitrary EF1 allocations. Although every such allocation is known to achieve an $e^{-1/e}$-approximation to the unrestricted optimal NSW, we identify conditions yielding stronger guarantees. Under uniform valuations, every EF1 allocation is NSW-optimal. Under an $\varepsilon$-small-item condition, every EF1 allocation achieves an explicit approximation ratio $\rho_n(\varepsilon)$ satisfying $\rho_n(\varepsilon) = 1-O(\varepsilon^2)$ as $\varepsilon\to 0$ for fixed $n$. We further consider the stronger sequential requirement that EF1 be maintained after every item assignment. For this setting, we propose \emph{PriorityNet}, a deep reinforcement learning framework trained using Proximal Policy Optimization and equipped with prospective EF1 action masking. The mask restricts every decision to assignments that preserve EF1, thereby guaranteeing prefix-wise EF1 by construction without post-processing repair. Across 3,000 test instances in each of the offline and random-order online regimes ($n\in[2,20]$ and $m\in[5,100]$), PriorityNet attains mean normalized $\operatorname{NSW}$ values of $0.9911$ and $0.9701$, respectively. Relative to offline Longest Processing Time (LPT) and online least-valued-bundle baselines, it achieves instance-wise win-minus-loss rates of $+27.10\%$ and $+17.87\%$, while matching the offline baseline's mean normalized welfare to four decimal places and modestly improving the online mean from $0.9694$ to $0.9701$.
• 中文摘要 我们研究了在加法估值相同的情况下，主体间不可分割物品的分配，重点关注无嫉妒性（至1个商品，EF1）和纳什社会福利（新南威尔士州）。由于每个最大新南威尔士州分配在加法估值下均为EF1，相关阈值问题继承了在相同加法估值下新南威尔士州最大化的已知强NP硬度，并且是强NP完全的。因此，我们关注由任意EF1分配满足的福利保障。尽管每个此类分配都已知达到与无限制最优新南威尔士州的$e^{-1/e}$近似，但我们识别出产生更强保证的条件。在均匀估值下，每个EF1分配均为新南威尔士州最优。在$\varepsilon$-小项目条件下，每个EF1分配都达到显式近似比$\rho_n（\varepsilon）$，满足$\rho_n（\varepsilon） = 1-O（\varepsilon^2）$，即固定$n$时的$\varepsilon\到0$。我们还进一步考虑了每个项目分配后EF1必须保持的更强顺序要求。针对此设定，我们提出了\emph{PriorityNet}，这是一个深度强化学习框架，使用近端策略优化训练，并配备前瞻性EF1动作掩蔽。掩码限制每个决策只能保留EF1的分配，从而通过构建保证EF1在前缀层面的存在，无需后处理修复。在离线和随机顺序在线模式（$n\in[2,20]$和$m\in[5,100]$）的3000个测试实例中，PriorityNet分别获得了平均归一化$\operatorname{NSW}$值$0.9911$和$0.9701$。相较于离线最长处理时间（LPT）和在线最低价值捆绑基线，其实例胜负率分别为$+27.10\%$和$+17.87\%$，同时将离线基线的平均归一化福利匹配到小数点四位，在线平均值也从$0.9694$适度提升至$0.9701$。

Revisiting Topological Graphs for Macro Action based Closed-loop Reinforcement Learning of Vision Language Navigation in Continuous Environment

在连续环境中视觉语言导航的宏观动作闭环强化学习中重新审视拓扑图

• Authors: Shuhao Ye, Sitong Mao, Yuxiang Cui, Yufei Wei, Xuan Yu, Shichao Zhai, Wen Chen, Shunbo Zhou, Rong Xiong, Yue Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.03906
• Pdf link: https://arxiv.org/pdf/2609.03906
• Abstract Vision-Language Navigation in Continuous Environments (VLN-CE) requires an agent to follow natural language instructions through unseen environments. Existing imitation learning (IL) pipelines struggle in this closed-loop setting: behavior cloning suffers from distribution shift, and DAgger's expert actions become ambiguous upon trajectory deviation. While Reinforcement Learning (RL) offers a natural paradigm to address this, directly applying RL to micro action spaces is sample-inefficient due to reward sparsity. To overcome this bottleneck, we reformulate VLN-CE as a Hierarchical Markov Decision Process (MDP), explicitly decoupling high-level planning from low-level control. By abstracting the environment into a topological graph, our high-level policy operates on a macro action space of frontier nodes, with a training-free low-level controller acting as its state transition, which significantly compresses the decision horizon and makes closed-loop RL tractable. To support RL optimization on the macro MDP, we propose an action-aware value head to effectively evaluate state values under the dynamic frontier action space, powering a graph-based PPO. Extensive experiments demonstrate the effectiveness of our architecture. Finally, our model achieves state-of-the-art performance on the R2R-CE and RxR-CE benchmarks.
• 中文摘要 连续环境中的视觉语言导航（VLN-CE）要求智能体在看不见的环境中遵循自然语言指令。现有的模仿学习（IL）流水线在这种闭环环境中表现不佳：行为克隆存在分布偏移问题，DAgger的专家操作在轨迹偏离时变得模糊。虽然强化学习（RL）提供了解决这一问题的自然范式，但直接将强化学习应用于微动作空间因奖励稀疏性而样本效率低下。为克服这一瓶颈，我们将VLN-CE重新表述为层次马尔可夫决策过程（MDP），明确将高层规划与低层控制脱钩。通过将环境抽象为拓扑图，我们的高级策略在由前沿节点组成的宏行动空间上运行，由无训练的低级控制器作为状态转换，显著压缩决策视野，使闭环强化学习变得可操作。为支持宏MDP上的强化优化，我们提出了一个动作感知值头，以有效评估动态前沿动作空间下的状态值，支持基于图的PPO。大量实验证明了我们架构的有效性。最后，我们的模型在R2R-CE和RxR-CE基准测试上实现了最先进的性能。

Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMs

代码大型语言模型中声音和对抗性测试生成的两阶段强化学习

• Authors: Jiacheng Xu, Wentao Zhang, Zhiyi Lyu, Fuxiang Zhang, Chaojie Wang, Yang Liu, Bo An
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.03955
• Pdf link: https://arxiv.org/pdf/2609.03955
• Abstract Reinforcement learning (RL) has substantially advanced code generation with large language models (LLMs) through executable feedback. The feedback for coding problems mainly comes from specific test cases, where high-quality test cases are often scarce since they should be both sound and discriminative. We thus turn to study the auto-generation of test cases using the learned model. We find this is naturally an adversarial RL problem: the model is expected to generate effective test cases as counterexamples, depending on the solver's current failure modes. We propose Test Cases Scaling (TCS), a two-stage RL framework for effective test generation. Both stages train a test generator from a rolling policy-aligned buffer: Stage 1 generates tests consistent with the reference solution, and Stage 2 restricts the buffer to current failure modes and learns counterexample tests. Across TACO and LiveCodeBench, TCS improves both pass@1 and inference-time answer selection according to generated tests. We find the learned test generator also enables effective selection among other LLM outputs.
• 中文摘要 强化学习（RL）通过可执行反馈实现了大型语言模型（LLM）代码生成的高度先进。编码问题的反馈主要来自具体测试用例，这些测试用例往往稀缺，因为它们应当既可靠又具判别性。因此，我们转向研究利用学习模型自动生成测试用例的方法。我们发现这自然是一个对抗性的强化学习问题：模型期望根据求解者当前的失败模式生成有效的反例测试案例。我们提出了测试案例缩放（TCS），这是一个两阶段的有效测试生成框架。两个阶段都从滚动的策略对齐缓冲区训练测试生成器：第一阶段生成与参考解一致的测试，第二阶段限制缓冲区为当前故障模式并学习反例测试。在TACO和LiveCodeBench中，TCS根据生成的测试提升了pass@1和推断时间的答案选择。我们发现，学习到的测试生成器还能在其他LLM输出中有效选择。

FiMI Banking: A Sovereign Model for Indian Retail Banking

FiMI银行：印度零售银行的主权模式

• Authors: NPCI AI Research Team: Aman Kumar, Asit Desai, Chandra Bhushan, Harsh Sharma, Harshit Bhushan, Hrithik Kadam, Keyur Doshi, Kolisetty Sai Kapardheeswar, Krishanu Adhikary, Nadeem Shaik, Navya Prakash, Nitin Kukreja, Prashant Devadiga, Shamanth MH, Shantanu Pandey, Suvradip Paul, Yatharth Dedhia
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.03960
• Pdf link: https://arxiv.org/pdf/2609.03960
• Abstract Banks need conversational systems that can answer product questions, assist customers with account-related requests, and operate safely within strict operational and regulatory constraints. General-purpose language models do not reliably meet these requirements. They fall short when a task requires grounded information, correct tool use, or cautious handling of bank-specific sensitive situations. We introduce FiMI Banking, a controlled Indian retail-banking setting. We build it from vetted banking documents, structured ground truth, synthetic customer backgrounds, and banking tools. We evaluate two post-training approaches: preference optimization for response-level behavior, and reinforcement learning with verifiable rewards for multi-turn tool-use tasks. Preference optimization improves safe behavior substantially: out-of-scope refusal rises from 52% to 80%. Reinforcement learning improves edge-case performance from 0.509 to 0.718 and order-sensitive task performance from 0.590 to 0.679, while using 29% fewer generated tokens. These results show that preference optimization and verifiable-reward reinforcement learning address complementary requirements for reliable banking agents.
• 中文摘要 银行需要能够回答产品问题、协助客户处理账户相关请求并在严格运营和监管约束下安全运行的对话系统。通用语言模型无法可靠满足这些要求。当任务需要扎实的信息、正确工具使用或谨慎处理银行特定敏感情况时，它们就显得不足。我们引入了FiMI银行，这是一个受控的印度零售银行环境。我们基于经过审核的银行文件、结构化的真实数据、合成的客户背景和银行工具构建。我们评估两种培训后方法：针对响应级行为的偏好优化，以及针对多回合工具使用任务的可验证奖励的强化学习。偏好优化显著提升了安全行为：超出范围的拒绝率从52%提升至80%。强化学习将边缘情况性能从0.509提升至0.718，命令敏感任务性能从0.590提升至0.679，同时使用减少29%的生成代币。这些结果表明，偏好优化和可验证奖励强化学习满足了可靠银行代理的互补需求。

The Dually Flat Geometry of Planning as Inference

规划的对偶平面几何作为推断

• Authors: Nikola Milosevic, Asaki Kataoka, Nicolas Hinrichs, Kenji Doya, Nico Scherf
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04005
• Pdf link: https://arxiv.org/pdf/2609.04005
• Abstract We present an alternative characterization of the occupancy measure of reinforcement learning, obtained by embedding the planning criterion into the dynamics through a resetting planning process. Its stationary measure, which we term visitation measure, is the object on which the information geometry of decision making is most naturally expressed. The achievable visitation measures form a dually flat statistical manifold whose two affine charts are the visitation probabilities and the log-policies, dual under the conditional entropy. This structure makes planning-as-inference generalize from linear rewards to nonlinear functionals of the visitation, each iterate solved by one natural-gradient step, and gives the temporal-difference error the interpretation of a marginal-utility estimate. We develop the geometry and its consequences for reinforcement learning and theoretical neuroscience.
• 中文摘要 我们提出了强化学习占用度量的另一种刻画，该表征通过将规划准则嵌入动态中，通过重置规划过程获得。其平稳度量，我们称之为访问度量，是决策信息几何最自然表达的对象。可实现的访问度量形成一个对偶平坦的统计流形，其两个仿射图分别是访问概率和对数策略，在条件熵下为对偶。该结构使规划即推断从线性奖励推广到访问的非线性泛函，每次迭代通过一个自然梯度步骤解决，并将时间差误差赋予边际效用估计的解释。我们发展了几何及其对强化学习和理论神经科学的影响。

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

模型编辑过多时：关于最小代码编辑的忠实度

• Authors: Tongyao Zhu, Wei Hern Lim, Min-Yen Kan
• Subjects: Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.04061
• Pdf link: https://arxiv.org/pdf/2609.04061
• Abstract Large language models (LLMs) are increasingly used to edit existing code, but correctness alone is not enough: useful repairs should also be minimal, reviewable, and faithful to the original implementation. We study over-editing, the tendency of a model to rewrite code beyond what is required to fix a bug. We construct an evaluation framework from 400 BigCodeBench problems by injecting controlled AST-level corruptions into reference solutions, giving each repair task a known minimal patch. Across frontier LLMs, over-editing is widespread even among strong models like GPT-5.5: high Pass@1 can coexist with unnecessarily large edits and added cognitive complexity. A preservation instruction substantially reduces this behavior, lowering average excess Levenshtein distance from 0.195 to 0.131, reducing added cognitive complexity by 26.6%, and increasing Pass@1 by 2.3 points. However, these gains do not simply follow from a larger reasoning budget or larger models. We next ask whether minimal editing can be learned directly during post-training. We observe that supervised fine-tuning overfits to seen corruption patterns, whereas reinforcement learning gives the best out-of-domain edit-fidelity and performance-retention trade-off. These results position edit fidelity as a distinct axis of code-repair quality and show that it can be measured and learned.
• 中文摘要 大型语言模型（LLM）越来越多地用于编辑现有代码，但仅靠正确性还不够：有用的修复还应是最小化、可审查且忠实于原始实现的。我们研究过度编辑，即模型倾向于重写代码超出修复缺陷所需的部分。我们通过向参考解注入受控的AST级损坏，构建了400个BigCodeBench问题的评估框架，为每个修复任务提供已知的最小补丁。在前沿大型语言模型中，过度编辑现象普遍存在，即使是像GPT-5.5这样的强模型：高Pass@1可能与不必要的大编辑和增加的认知复杂性共存。保存指令显著减少了这种行为，将平均多余Levenshtein距离从0.195降至0.131，增加认知复杂度26.6%，Pass@1提升2.3个百分点。然而，这些进步并非仅仅来自更大的推理预算或更大的模型。接下来我们探讨最小编辑是否可以在训练后直接学习。我们观察到监督微调对已见损坏模式的过拟合，而强化学习则提供了最佳的域外编辑忠实度和性能-保留权衡。这些结果将编辑忠实度定位为代码修复质量的一个独立轴，并表明其可被测量和学习。

Spurious Advantage Hidden in GRPO

GRPO中隐藏的虚假优势

• Authors: Jiamian Wang, Samyadeep Basu, Koustava Goswami, Tong Yu, Zhiqiang Tao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04063
• Pdf link: https://arxiv.org/pdf/2609.04063
• Abstract Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.
• 中文摘要 群体相对策略优化（GRPO）在强化学习中被广泛研究，其优势估计器会根据组内奖励统计给每个推广分配一个幅度。在常见情况下，这个幅度奖励通过推理得出正确答案的推广。然而，被忽视的案例也存在同一表面：某个推测可能落在该案例上，但公式仍然赋予一个高幅度，我们认为这是虚假优势。这在三种情况下出现：候选集合较少的有界答案任务;承载有界子案例的开放答案集;以及预算允许多条路径通向同一答案的搜索代理。这三种情况下都误导了策略，趋向于猜测行为。我们提出SIGNBALANCE，其大小无复合：保持验证符号，使用全局刻度，并通过每类停止梯度重新标配恢复零均值平衡。在不同量表的数学和搜索代理基准中，SIGNBALANCE在开放答案数学上与GRPO匹配，并在有界答案数学和搜索代理上有所改进。代码将发布。

Subspace Inference Enables Efficient Active Reward Learning from Preferences

亚空间推理实现了从偏好中高效的主动奖励学习

• Authors: Yutai Zhou, Erdem Bıyık
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.04066
• Pdf link: https://arxiv.org/pdf/2609.04066
• Abstract Reinforcement learning from human feedback (RLHF) has emerged as a powerful yet sample-inefficient approach for learning reward models from human preferences, making active learning a critical component in synthesizing informative preference queries. However, effective uncertainty quantification required for active learning remains a key challenge for large neural network reward models. In this paper, we introduce PreferenceEKF, a sample-efficient approach that tracks reward model uncertainty by framing active preference learning as a sequential Bayesian filtering problem. Instead of relying on computationally prohibitive posterior inference over the full neural network parameter space, our method performs sequential inference via an extended Kalman filter within a low-dimensional parameter subspace, continuously updating the reward model posterior as new preference queries arrive. Our approach enables scalable sampling of neural network parameters to efficiently compute acquisition functions for active reward learning. Experiments on the D4RL and V-D4RL benchmarks demonstrate that our approach achieves better sample efficiency, runtime, scalability, and calibration compared to other Bayesian deep learning approaches, and the learned reward models lead to competitive offline reinforcement learning policy performance. This highlights the potential of scalable Bayesian methods for preference-based reward modeling in RLHF. Our code is available at this https URL.
• 中文摘要 来自人类反馈的强化学习（RLHF）已成为一种强大但样本效率低的方法，使主动学习成为综合信息偏好查询的关键组成部分。然而，主动学习所需的有效不确定性量化仍是大型神经网络奖励模型面临的关键挑战。本文介绍了PreferenceEKF，一种样本高效方法，通过将主动偏好学习框架为顺序贝叶斯滤波问题来追踪奖励模型的不确定性。我们不依赖计算上限制的整个神经网络参数空间的后验推断，而是通过在低维参数子空间内的扩展卡尔曼滤波器进行顺序推断，随着新偏好查询的到来，持续后期更新奖励模型。我们的方法实现了可扩展的神经网络参数采样，从而高效计算主动奖励学习的获取函数。D4RL和V-D4RL基准测试的实验表明，我们的方法相比其他贝叶斯深度学习方法在样本效率、运行时间、可扩展性和校准方面实现了更好的表现，且学习到的奖励模型带来了竞争性的离线强化学习策略表现。这凸显了可扩展贝叶斯方法在RLHF中基于偏好的奖励建模中的潜力。我们的代码可在该网址访问。

DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

DRACO：细粒度学分分配，动态评分标准，用于长视线特工培训

• Authors: Shubham Gandhi, Saurabh Goyal, Kiran Kate, Yara Rizk
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2609.04094
• Pdf link: https://arxiv.org/pdf/2609.04094
• Abstract Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at this https URL.
• 中文摘要 从可验证奖励中学习强化在任务有程序检查器但大多数长期代理领域没有检查器时效果良好。我们在结果盲环境中工作，该环境无法获得真实的成功信号。多标准评分标准是提供此类奖励的常用方式;它们每个轨迹评分一次，但单个标量在数十步中信号较差。我们提出DRACO：基于评分标准的优势分配以优化信用。它在培训过程中动态生成评分标准以跟踪政策能力的演变，每个完成轨迹对评分一次，并将判断重新分配到负责注释评分的步骤，以产生GRPO中的差异化每步优势。该重分配为封闭形式，不引入任何训练有素的归因模块。在AppWorld上，DRACO比基础模型提升15.9分，比使用稀疏地面真实奖励训练的GRPO提升5.3分，尽管未使用任何验证器。在域外的Tau-Bench上，即使没有边境评判，DRACO也比基础模型提升5.3分，击败了基于评分标准的训练设置。DRACO的代码可在此 https URL 获取。

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

顺序节拍联合：论政策提炼与RLVR之间的相互作用

• Authors: Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie, Chen Zhao, Xi Ye
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.04108
• Pdf link: https://arxiv.org/pdf/2609.04108
• Abstract Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level supervision to complement the sparse RL reward, fusing the two signals within a single step: either as a \emph{weighted-additive combination} or a \emph{teacher-modulated rescaling} of the RL advantage. In this paper, we show that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks. Beyond the empirical results, we further provide a systematic understanding of this through pass@$k$ behavior, learning dynamics, and parameter updates, yielding a consistent explanation: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to this http URL provide a practical recipe, we find that the OPD validation score is the key signal for when to switch to RL, and that OPD is a better cold start for RL than SFT. Together, our results establish OPD-then-RL as a simple yet strong way to combine the two methods, turning two entangled signals into complementary stages.
• 中文摘要 带可验证奖励的强化学习（RLVR）和策略提纯（OPD）已成为训练后推理LLM的两种主流方法。以往工作利用OPD密集的代币级监督来补充稀疏的RL奖励，将两种信号融合在单一步内：要么作为加权加法组合，要么作为教师调制的强化学习优势的重标度。本文展示了一个简单的两阶段方案——先是OPD-然后是RL，在逻辑和数学推理基准中始终优于纯OPD、纯RLVR及所有此类联合基线。除了实证结果外，我们还通过pass@$k$行为、学习动态和参数更新系统地理解了这一点，得出了一个一致的解释：OPD扩展了学生对教师支持解决方案的覆盖范围，而强化学习在支持中更加敏锐，同时联合优化这两个信号使它们成为一个实用的配方，我们发现OPD验证分数是何时切换到强化学习的关键信号，而OPD比SFT更适合强化学习的冷启动。综合来看，我们的结果确立了OPD-然后强化学习作为一种简单而强有力的结合两种方法的方法，将两个纠缠信号转化为互补阶段。

Keyword: diffusion policy

MulDP: Multimodal Diffusion Policy for Autonomous Quadruped Parkour Navigation across Complex Terrains

MulDP：跨复杂地形自主四足跑酷导航的多模态扩散政策

• Authors: Kangmai Hu, Yueqi Zhang, Peng Zhai, Xiaoyi Wei, Jiabin Hu, Zhixiang Liu, Quancheng Qian, Lihua Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.03984
• Pdf link: https://arxiv.org/pdf/2609.03984
• Abstract Quadruped robots have demonstrated impressive agility in parkour locomotion across complex terrains. However, most systems still rely on human intervention for high-level planning, and autonomous parkour navigation remains underexplored. The key challenges include fine-grained velocity regulation, long-horizon anticipatory behaviors, and tight coupling between perception and embodied execution. To address these challenges, we propose a Multimodal Diffusion Policy (MulDP) that integrates visual perception with robot proprioception and goal information to generate temporally coherent and anticipatory navigation velocity commands, tightly coupling perception with embodied control to enable robust autonomous navigation. To support the training of MulDP, we construct the first Quadruped Parkour Navigation Dataset (QPND), a multimodal dataset that encompasses diverse navigation behaviors and complex terrains. Extensive simulation and real-world experiments demonstrate that MulDP enables robust long-horizon autonomous navigation and effective traversal across complex terrains.
• 中文摘要 四足机器人在复杂地形的跑酷移动中展现出令人印象深刻的敏捷性。然而，大多数系统仍依赖人工干预进行高层规划，自主跑酷导航仍未被充分探索。关键挑战包括细粒度速度调节、长视野预期行为以及感知与具身执行之间的紧密耦合。为应对这些挑战，我们提出了一种多模态扩散策略（MulDP），将视觉感知与机器人本体感知及目标信息整合，生成时间上连贯且具备的导航速度指令，紧密结合感知与具象控制，实现稳健的自主导航。为支持MulDP的训练，我们构建了首个四足跑酷导航数据集（QPND），这是一个涵盖多样导航行为和复杂地形的多模态数据集。大量模拟和实际实验表明，MulDP实现了稳健的长视距自主导航和在复杂地形上的有效穿越。