生成时间: 2026-09-14 22:49:09 (UTC+8); Arxiv 发布时间: 2026-09-14 20:00 EDT (2026-09-15 08:00 UTC+8)

今天共有 33 篇相关文章

Keyword: reinforcement learning

Performance, Efficiency and Collapse -- Advantages and Challenges in Offline Post-training of Code LLMs

性能、效率与崩溃——代码大型语言模型离线后训练中的优势与挑战

• Authors: Abhinav Anand, Sanjana Reddy Pachika, Shweta Verma, Mira Mezini
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.11956
• Pdf link: https://arxiv.org/pdf/2609.11956
• Abstract Post-training with reinforcement learning (RL) is a critical phase in the development of code-generating large language models (LLMs), as it ensures adherence to instructions and the production of functionally correct code. This process typically requires computationally intensive code sample generation from Transformer-based LLMs and substantial GPU-CPU communication for sequence verification. To address these computational challenges, this work examines whether RL-based post-training can be performed entirely offline by leveraging existing datasets rather than generating new samples. The findings indicate that, with only a few hours of training, zero-shot code generation performance of LLMs can be substantially improved without online sampling. Additionally, offline RL produces performance gains across models ranging from 0.5B to 7B parameters, although the extent of improvement varies among model families.
• 中文摘要 强化学习（RL）的后训练是开发代码生成大型语言模型（LLM）的关键阶段，因为它确保了对指令的遵守和功能正确代码的生成。这一过程通常需要从 Transformer 基础的 LLM 生成大量计算量的代码样本，以及大量的 GPU 与 CPU 通信进行序列验证。为应对这些计算挑战，本研究探讨了基于强化学习的后训练是否可以完全离线完成，即利用现有数据集而非生成新样本。研究结果表明，只需几小时的训练，LLM 的零样本代码生成性能即可在不进行在线采样的情况下显著提升。此外，离线强化学习在参数范围从 0.5 亿到 7 亿的模型中实现性能提升，尽管不同模型家族的提升程度有所不同。

Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning

认证安全策展：安全离线强化学习的无分发保证

• Authors: Adam Haroon, Cody Fleming
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.12014
• Pdf link: https://arxiv.org/pdf/2609.12014
• Abstract Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget. Certified safety curation answers with a filter-then-clone pipeline: a state-only value trained from segment comparisons scores whole trajectories, Learn-then-Test calibration certifies a selection threshold under a distribution-free $(\alpha, \delta)$ bound on the unsafe fraction of the selection, and behavior cloning follows. We are not aware of prior work certifying the composition of a training set for offline RL or imitation. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories. The policies satisfy the cost budget on eleven of fifteen DSRL tasks, one short of cloning the ground-truth safe subset, which needs a label on every trajectory; the uncertified variant reaches twelve. Retrained on the certified selection, the strongest full-label method becomes safe where no setting of its own cost target rescues it. Refusal is predictable: the certificate's probability has a closed form in the purity the pool attains, which the calibration sample estimates and the scorer enters only through.
• 中文摘要 安全离线强化学习假设每个转换都有一个成本函数。我们询问当安全性只能通过比较短片段和偶尔询问某集是否超出预算来判断时，还能实现什么。认证安全策划通过筛选后克隆流程给出答案：仅通过段比较训练的状态值对整个轨迹进行评分，学习后测试校准在无分布的$（\alpha， \delta）$下认证选择阈值，且该值对不安全部分的选择进行限制，随后进行行为克隆。我们不知道之前有关于离线强化学习或模仿训练集组成的认证工作。Oracle控制为设计提供了正当性：即使精确值，重新加权单个转换也失败，因此该值选择整个轨迹。这些策略满足了十五项DSRL任务中的十一项的成本预算，距离克隆地面真实安全子集（每条轨迹都需要标签）还差一项;未认证的变体达到十二项。在认证选择上重新训练后，最强的全标签方法在没有自身成本目标设置能挽救的情况下变得安全。拒绝是可预测的：证书的概率在池中达到的纯度中有一个封闭形式，校准样本估计，评分者仅通过该纯度进行录入。

Inverting Self-Triggered Control: Adversarial Reinforcement Learning for Sparse Denial-of-Service Attacks

逆转自我触发控制：针对稀疏拒绝服务攻击的对抗强化学习

• Authors: Adam Haroon, Erick J. Rodríguez-Seda, Tristan Schuler, Cody Fleming
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.12016
• Pdf link: https://arxiv.org/pdf/2609.12016
• Abstract Self-triggered reinforcement learning control (RL-STC) learns the sparsest control schedule that preserves Lyapunov-decreasing stability under a Run-Time Assurance (RTA) override. We invert this: an adversarial RL agent learns the sparsest jamming or Denial-of-Service (DoS) schedule that destabilizes the closed loop, with a Lyapunov-increase admissibility predicate mirroring the defender's safety certificate. We prove a plant-property lower bound on the minimum jam count required for an immediate hold-last medium-access-control adversary to force a crash against a self-triggered controller (STC) satisfying a Lyapunov contract, and recover a certificate-level analog of the consecutive-grouping optimality of prior count-budget DoS scheduling as a corollary. This extends the DoS-scheduling count-budget analysis from periodic and linear-time-invariant to STC controllers. Empirically, we train against four fixed defenders per plant (one Linear Quadratic Regulator (LQR) and three RL-STC) on Pendulum, CartPole, and Quadrotor2D. The learned adversary is the only adversary that crashes every defender on every plant at $100\%$: greedy misses Quadrotor2D LQR on $42\%$ of episodes and periodic misses Pendulum LQR on $97\%$. On jam-time-per-failure it beats baselines by up to $2.8\times$, and shows its widest absolute margin on Quadrotor2D LQR. Robustness ablations show that Gaussian observation noise exceeding the initial-state magnitude and position-only observation both preserve $100\%$ failure rate and keep the learned adversary strictly ahead of both baselines on jam-time-per-failure.
• 中文摘要 自触发强化学习控制（RL-STC）学习在运行时保障（RTA）覆盖下保持李雅普诺夫降低稳定性的最稀疏控制计划。我们将此反转：对抗性强化智能体学习最稀疏的干扰或拒绝服务（DoS）调度，使闭环不稳定，且带有符合防御方安全证书的李雅普诺夫提升可采性谓词。我们证明了一个植物属性下限，适用于立即保持最后的中介访问控制对手要求与满足李雅普诺夫合同的自触发控制器（STC）崩溃所需的最小干扰计数，并作为推论恢复了先前计数预算DoS调度的连续分组最优性的证书级对应。这将 DoS 调度计数-预算分析从周期性和线性时间不变扩展到 STC 控制器。实证上，我们在 Pendulum、CartPole 和 Quadrotor2D 上对每个工厂训练四个固定防御器（一个线性二次调节器（LQR）和三个 RL-STC）。学习对手是唯一一个在每个工厂上以 $100\% 价格崩溃所有防御者的对手：贪婪者在 $42\%$ 的集数中漏掉 Quadrotor2D LQR，在 $97\%$ 时周期性漏掉 Pendulum LQR。在每次失败的 jam-time-中，它比基线多达 $2.8\times$，并且在 Quadrotor2D LQR 上显示出最宽的绝对优势。鲁棒性消融表明，高斯观测噪声超过初始状态幅度和仅位置观测均保持了100%%的失败率，并使学习对手在每次失败的干扰时间上严格领先两个基线。

Reinforcement Learning for Syndrome Extraction

综合征提取的强化学习

• Authors: John Zhuoyang Ye, Aarav Pabla, Jens Palsberg
• Subjects: Subjects: Machine Learning (cs.LG); Quantum Physics (quant-ph)
• Arxiv link: https://arxiv.org/abs/2609.12020
• Pdf link: https://arxiv.org/pdf/2609.12020
• Abstract A key subtask of quantum error correction is to extract a syndrome that, if nontrivial, signals an error. The number of possible ways to extract a syndrome grows exponentially with the syndrome size, and these implementations vary greatly in fault tolerance, as measured by their logical error rates. This creates a natural search problem: find an implementation with a low logical error rate. Previous work solves this problem but sacrifices either solution quality or scalability. In this paper, we use reinforcement learning and importance sampling to outperform previous work at all scales. Compared with the state of the art automatic scheduling tools AlphaSyndrome and PropHunt, our tool reduces the logical error rate by 25.9\% and 71.7\% on average, respectively, culminating with a reduction of 97.8\% for a surface code with distance 15.
• 中文摘要 量子纠错的一个关键子任务是提取一个如果非平凡，就能提示错误的综合征。提取综合征的可能方法数量随着综合征大小呈指数增长，这些实现的容错率差异很大，这取决于其逻辑错误率。这自然产生了一个搜索难题：寻找逻辑错误率较低的实现。以往的工作解决了这个问题，但牺牲了解的质量或可扩展性。本文利用强化学习和重要性抽样，在所有尺度上都优于以往的工作。与最先进的自动调度工具AlphaSyndrome和PropHunt相比，我们的工具平均将逻辑错误率降低了25.9%和71.7%，最终在距离为15的表面码时降低了97.8%。

Repair Before Reinforce: Context-Augmented Knowledge Graph Reasoning for Multi-Hop Question Answering

修复后强化：多跳问答中的上下文增强知识图谱推理

• Authors: Tharaka D. Fonseka, Niraj K. Jha
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12230
• Pdf link: https://arxiv.org/pdf/2609.12230
• Abstract Question-answering often requires reasoning across multiple connected facts rather than retrieving a single isolated relation. Knowledge graphs (KGs) provide a structured way to represent such facts, but training large language models (LLMs) only on isolated KG head-relation-tail triples may limit their ability to learn the surrounding context needed for multi-hop reasoning. In this work, we propose a context-augmented training framework for multi-hop question-answering. Although generally applicable, we validate the framework in the context of disease-specific KGs, extracted using a reliable KG extraction framework called GraphMERT, for Gastroparesis and Diabetes. For each primary KG triple, we attach supporting triples extracted from the same source text chunk to form a context graph (CG). This creates two supervision settings: KG-grounded supervision, which uses only the target KG triple or path, and CG-grounded supervision, which uses the target KG triple or path together with supporting context triples. We train the Qwen3-14B model using supervised fine-tuning (SFT) under both settings, producing KGModel and CGModel variants. To strengthen the lower-hop factual foundation of the models, we introduce an LLM-judged, history-aware adaptive repair pipeline that identifies unresolved one-hop failures, continually fine-tunes on targeted repair examples, and removes or quarantines problematic noisy triples. This repair stage enables the models to reach 100% accuracy on the cleaned retained one-hop validation sets. Finally, we employ reinforcement learning (RL) using lower-hop question-answer items and evaluate generalization on harder 3-hop, 4-hop, and 5-hop tasks. Across both diseases, context-augmented supervision consistently improves multi-hop performance over KG-only supervision. RL initialized from repaired SFT checkpoints yields larger and more stable gains.
• 中文摘要 问答通常需要跨多个相互关联的事实进行推理，而非检索单一孤立关系。知识图谱（KGs）提供了结构化的方式来表示此类事实，但仅用孤立的KG头尾三元组训练大型语言模型（LLMs）可能会限制它们学习多跳推理所需的周围上下文的能力。本研究提出了一种上下文增强训练框架，用于多跳问答。虽然普遍适用，但我们在疾病特异性KG的背景下验证了该框架，该框架使用名为GraphMERT的可靠KG提取框架，适用于胃轻瘫和糖尿病。对于每个主要KG三元组，我们附加从同一源文本块提取的支持三元组，形成上下文图（CG）。这创建了两种监督设置：KG基导监督，仅使用目标KG三元或路径，以及CG基地监督，使用目标KG三元或路径及支持上下文三元组。我们在两种设置下使用监督微调（SFT）训练Qwen3-14B模型，生成KGModel和CGModel变体。为加强模型的低跳事实基础，我们引入了LLM判定的历史感知自适应修复流水线，识别未解决的单跳故障，持续微调目标修复示例，并剔除或隔离有问题的噪声三重。该修复阶段使模型在清理后的保留一跳验证集上达到100%准确率。最后，我们采用强化学习（RL）使用低跳问答题，评估更难的三跳、四跳和五跳任务的泛化。在两种疾病中，情境增强监督在多跳表现上均优于仅限公斤级的监督。从修复后的SFT检查点初始化的强化学习带来更大且更稳定的收益。

DIA: Denoising Intermediate Advantage for Diffusion Policy Optimization

DIA：扩散策略优化的去噪中间优势

• Authors: Arjun Sohal, Yuchi Zhao, Miroslav Bogdanovic, Alan Aspuru-Guzik
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.12245
• Pdf link: https://arxiv.org/pdf/2609.12245
• Abstract Diffusion-based robot policies have become widely used in robotic manipulation, where they are typically trained with behavior cloning. However, policies trained purely from demonstrations are limited by the quality and coverage of the available data. Reinforcement learning can further improve the performance of these pretrained policies through interaction. A common approach is to use policy-gradient methods that formulate diffusion-policy fine-tuning as an outer environment MDP together with an inner denoising MDP. However, existing methods typically assign the same environment-level credit to all denoising steps used to construct an action chunk, without distinguishing which intermediate decisions contributed most to the final return. We introduce Denoising Intermediate Advantage (DIA), a policy-gradient method that learns a value function over partially denoised actions and uses it to construct a denoising- level advantage for each step of the generative process. DIA com- bines this inner credit signal with the standard environment-level PPO advantage, providing state-dependent credit throughout the denoising chain. Across Robomimic, FurnitureBench, Franka Kitchen, and D3IL, DIA consistently improves final performance over existing diffusion-policy fine-tuning methods. Beyond final reward, DIA reaches successful states more efficiently and can shift farther from the pretrained behavior distribution, enabling it to discover more effective and efficient task-level strategies and subtask sequences that baseline methods fail to reach.
• 中文摘要 基于扩散的机器人策略已被广泛应用于机器人操作，通常通过行为克隆进行训练。然而，仅通过演示训练的策略受限于可用数据的质量和覆盖范围。强化学习还可以通过交互进一步提升这些预训练策略的性能。一种常见方法是使用策略梯度方法，将扩散策略微调结合为外部环境MDP和一个内部去噪MDP。然而，现有方法通常将所有用于构建动作块的去噪步骤赋予相同的环境级功劳，而未区分哪些中间决策对最终收益贡献最大。我们介绍去噪中间优势（DIA），这是一种策略梯度方法，通过学习部分去噪动作的值函数，并利用该方法构建生成过程中每个步骤的去噪级优势。DIA将这种内部信用信号与标准环境级PPO优势结合起来，在整个去噪链中提供状态依赖性的信用。在Robomimic、FurnitureBench、Franka Kitchen和D3IL中，DIA持续提升了相比现有扩散策略微调方法的最终性能。除了最终奖励，DIA更高效地达到成功状态，并能更远离预训练行为分布，从而发现基线方法未能达到的更有效和高效的任务级策略和子任务序列。

Adaptive Chemotherapy Control under Tumor Heterogeneity via Reinforcement Learning

通过强化学习实现肿瘤异质性下的适应性化疗控制

• Authors: Bereket Sitotaw Kidane, Md Samiul Haque Motayed, Shuo Wang
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.12264
• Pdf link: https://arxiv.org/pdf/2609.12264
• Abstract Designing effective chemotherapy regimens is hindered by tumor heterogeneity and drug resistance, which complicate the deployment of patient-specific model-based optimal control across diverse populations. We develop and compare closed-loop deep reinforcement learning (DRL) dosing policies with continuous (TD3) and discrete (DQN) action spaces trained on a high-dimensional heterogeneous tumor model. The DRL policies are benchmarked against a Pontryagin's Maximum Principle (PMP)-derived open-loop benchmark. We assess generalization under parametric heterogeneity using a 100-patient virtual cohort with plus or minus 10 percent uniform perturbations in growth and drug-sensitivity parameters. Across this cohort, TD3 achieves higher average tumor reduction, while DQN yields tighter inter-patient dosing consistency, revealing a clear efficacy-consistency trade-off in this study. Our simulations assume full observation of all tumor subpopulations; translation to sparse and noisy clinical measurements will require partial-observability formulations and/or state estimation. Overall, the results show that simulation-trained DRL can learn state-dependent feedback dosing policies that complement open-loop optimal control benchmarks.
• 中文摘要 设计有效的化疗方案受肿瘤异质性和耐药性阻碍，这使患者特异模型最优控制在不同人群中应用复杂化。我们开发并比较闭环深度强化学习（DRL）剂量政策与基于高维异质肿瘤模型训练的连续（TD3）和离散（DQN）动作空间。DRL策略以庞特里亚金最大原理（PMP）衍生的开环基准进行基准测试。我们利用100名患者的虚拟队列评估参数异质性下的泛化，该队列在生长和药物敏感性参数中均有正负10%均匀扰动。在该队列中，TD3实现了更高的平均肿瘤减少效果，而DQN则实现了更严格的患者间剂量一致性，揭示了本研究中疗效与一致性的明显权衡。我们的模拟假设对所有肿瘤亚群进行全面观察;转化为稀疏且噪声较大的临床测量需要部分可观测性表述和/或状态估计。总体来看，结果显示模拟训练的DRL能够学习状态依赖的反馈剂量策略，以补充开环最优控制基准。

Reinforcement Learning over Patient Trajectories for Clinical Reasoning in EHR Foundation Models

EHR基础模型中对患者轨迹的强化学习，用于临床推理

• Authors: Yuxin Xiao, Sheng Zhang, Chandan Singh, Tristan Naumann, Hoifung Poon, Jianfeng Gao, Xiaodong Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computers and Society (cs.CY)
• Arxiv link: https://arxiv.org/abs/2609.12277
• Pdf link: https://arxiv.org/pdf/2609.12277
• Abstract Electronic health record (EHR) foundation models trained on longitudinal patient trajectories have demonstrated strong performance across diverse clinical prediction tasks. However, their clinical reasoning capabilities remain constrained by next-token prediction on limited and incomplete EHR data. To address this, we propose a reinforcement learning (RL) fine-tuning framework that treats EHR foundation models as generative policies over patient trajectories. We formulate common clinical prediction problems (e.g., hospital readmission) as event-conditioned, time-windowed reasoning tasks. We then design time-aware, rollout-sensitive rewards to account for finite rollout lengths and temporally inconclusive outcomes. We find that RL fine-tuning consistently improves over pre-trained backbones and strong baselines. Notably, it enables smaller models to surpass larger pre-trained models in data-limited regimes and induces positive transfer across tasks. Further analysis shows that RL fine-tuned models generate trajectories with stronger structural and semantic alignment to ground truth and greater downstream utility.
• 中文摘要 基于纵向患者轨迹训练的电子健康记录（EHR）基础模型在多种临床预测任务中表现出良好表现。然而，其临床推理能力仍受限于有限且不完整的电子健康记录数据的下一标记预测。为此，我们提出了一个强化学习（RL）微调框架，将EHR基础模型视为对患者轨迹的生成策略。我们将常见的临床预测问题（如医院再入院）制定为事件条件、时间窗口推理任务。随后设计时间感知、对扩展敏感的奖励，以考虑有限的推广长度和时间上无定论的结果。我们发现，强化学习的微调优于预训练主干和强基线，持续提升。值得注意的是，它使较小模型在数据限制的条件下超越大型预训练模型，并诱导任务间的正向转移。进一步分析显示，强化学习微调模型生成的轨迹在结构和语义上更贴合真实性，且具有更高的下游效用。

Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models

通过决策流抽样：大型语言模型中改进的无训练潜在推理路径提取

• Authors: Zhendong Mi, Shaoyi Huang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.12317
• Pdf link: https://arxiv.org/pdf/2609.12317
• Abstract A central question in LLM reasoning is whether reinforcement learning (RL) instills genuinely new capabilities or merely reshapes how existing knowledge is expressed during inference. Building on the distribution-sharpening hypothesis, which holds that RL reallocates probability mass toward high-reward trajectories already latent in base models, we ask: can we unlock those latent paths without costly RL fine-tuning? We present Decision-Flow Sampling (DF-Sample), a training-free, data-free inference-time framework that constructs a hierarchical reasoning tree, scores terminal nodes for quality, and back-propagates utilities to inform each intermediate branching decision. Unlike conventional sampling strategies that make purely local step-wise choices, DF-Sample performs explicit global trajectory evaluation before committing to a path, recovering high-quality but low-probability reasoning chains that standard decoding overlooks. On GPQA, DF-Sample achieves 45.6% accuracy, surpassing power sampling (38.9%) and GRPO (39.9%), showing that a training-free method can outperform a trained one. Across three models and four benchmarks, DF-Sample consistently outperforms baselines, indicating substantial latent reasoning potential in pretrained base models.
• 中文摘要 大型语言模型推理中的一个核心问题是，强化学习（RL）是否真正赋予了新能力，还是仅仅重塑了推理过程中现有知识的表达方式。基于分布锐化假说，即强化学习会将概率质量重新分配到基础模型中已潜伏的高回报轨迹，我们提出问题：我们能否在不进行昂贵的强化学习微调的情况下解锁这些潜在路径？我们提出了决策流采样（DF-Sample），这是一种无训练、无数据的推理时间框架，构建层级推理树，对终端节点进行质量评分，并反向传播效用以指导每个中间分支决策。与传统采样策略在执行纯局部分步选择不同，DF-Sample 在承诺路径前进行显式全局轨迹评估，恢复标准解码忽略的高质量但低概率推理链。在GPQA上，DF-Sample的准确率达到45.6%，超过了检定力抽样（38.9%）和GRPO（39.9%），表明无训练方法可以优于训练方法。在三个模型和四个基准测试中，DF-Sample持续优于基线，显示预训练基础模型具有显著的潜在推理潜力。

BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents

BlueLM-GUI技术报告：一个以真实设备为中心的移动图形界面代理的飞轮

• Authors: Tong Ye, Kunyang Han, Guozhi Wang, Longqiang Luo, Zhifeng Ding, Yongxiang Zhang, Xiaolei Shen, Yuxuan Zhang, Zhuping Zhang, Tao Xu, Yue Pan, Yucheng Zhao, Yupei Hu, Yuanjiang Ouyang, Danfeng Shen, Runqi Lin, Hongda Cai, Zhaoxiong Wang, Mengjia Yan, Yingjie Zhong, Chen Zhou, Zeyu Zhang, Xuwen Zhu, Penggang Shi, Mingcheng Luo, Ziyang Wu, Min Jin, Mingfu Shen, Zairong Xu, Fan Zhang, Hao Wang, Liang Liu, Zhulin Xie, Lijun Yao, Xiao Liang, Liangmin Wen, Liqiang Feng, Feilong Wu, Min Hu, Min Chen, Guanjing Xiong, Xiaohu Ruan, Xiaoxin Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12394
• Pdf link: https://arxiv.org/pdf/2609.12394
• Abstract Mobile GUI agents are shifting from multi-module frameworks to native models trained end-to-end, yet industrial deployment faces three persistent gaps. Sandbox training produces a distribution mismatch with production environments; expensive real-device failures remain underutilized; and fixed benchmarks saturate, losing the power to guide iteration. We present BlueLM-GUI, a 35B-A3B mobile GUI agent built as a real-device-centric flywheel that closes these gaps through three principles. Every Sample Matters: a dual-track pipeline with Heterogeneous Triple-System Consensus evaluation and an Error Correction \& Derivation Module salvages every trajectory into usable supervision. Every Rollout Is Real: a three-stage recipe---continual pre-training, supervised fine-tuning, and agentic reinforcement learning on hundreds of real phones---grounds every rollout in real production environments, so the capability the model learns transfers directly to deployment. Every Query Evolves: a quota-driven benchmark methodology with three orthogonal axes enables precise attribution and allows the benchmark to be systematically upgraded as the model improves. BlueLM-GUI achieves 87.4 on MobileGUI-VBench, surpassing the best closed-source model by 5.1 points, and 84.9 on AndroidWorld, the best result among open-source models and competitive with closed-source models. These results demonstrate that grounding model training and iterative improvement in both real devices and the three Every principles yields strong, robust, and transferable mobile GUI capability.
• 中文摘要 移动图形界面代理正从多模块框架转向端到端训练的原生模型，但工业部署面临三个持续的空白。沙盒训练导致分布与生产环境不匹配;昂贵的真实设备故障仍未被充分利用;固定基准过于饱和，失去引导迭代的能力。我们介绍BlueLM-GUI，一款35B-A3B移动图形界面代理，构建为以真实设备为中心的飞轮，通过三大原则弥合这些差距。每个样本都很重要：双轨流水线配合异构三系统共识评估和错误纠正与推导模块，将每一条轨迹挽救为可用的监督。每一次部署都是真实的：一个三阶段方案---持续的预训练、监督式微调和在数百部真实手机上的代理强化学习---将每次部署都置于真实生产环境中，因此模型学习的能力能够直接转移到部署中。每个查询都在演变：采用三条正交轴的配额驱动基准方法，使得精确归因成为可能，并允许随着模型改进系统性升级基准。BlueLM-GUI在MobileGUI-VBench上达到87.4，领先最佳闭源模型5.1个百分点，在AndroidWorld上达到84.9个百分点，是开源模型中表现最佳，并与闭源模型竞争。这些结果表明，在真实设备和Every的三大原则中，扎根模型训练和迭代改进，能够带来强大、稳健且可迁移的移动GUI能力。

VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets

VRL-Bench：在有限试验预算下对计算机控制任务进行基准测试

• Authors: Yu Bai, Yukai Miao, Dawei Wang, Li Chen, Yanyu Ren, Yuqian Shi, Dan Li, Ying Xiong, Chengqiu Tan, Run Zhou, Li Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12404
• Pdf link: https://arxiv.org/pdf/2609.12404
• Abstract Learning from trial and error is a promising way to improve language agents on complex tasks such as computer control. Reflexion introduced verbal reinforcement learning, which turns failed trials into text that guides later attempts without updating model parameters. We introduce VRL-Bench, a harness for fair evaluation of trial-and-error learning under finite trial budgets. Across three models on MiniWoB and WebShop, we evaluate updates from several prominent verbal-memory methods spanning Reflexion and later work: each improves observed success over memory-free retry in some settings but reduces it in others. Replay experiments show that using reflection can reduce success rates, revealing a trade-off between exploiting experience and continued exploration. We propose VEX$^2$, a verbal exploration--exploitation scheduler that uses a language model to jointly select policies and allocate the remaining trial budget. VEX$^2$ is the only evaluated update to achieve positive observed success-rate gains over retry in all six settings.
• 中文摘要 通过试错学习是提升语言代理在复杂任务（如计算机控制）中的一种有前景的方法。反思引入了语言强化学习，将失败的尝试转化为指导后续尝试的文本，无需更新模型参数。我们引入了VRL-Bench，这是一种在有限试验预算下公平评估试错学习的工具。在MiniWoB和WebShop的三个模型中，我们评估了多种主要语言记忆方法的更新，涵盖反思及后续工作：每种方法在某些场景中提升了无记忆重试的成功率，但在其他环境中降低了成功率。回放实验显示，使用反思可以降低成功率，揭示了利用经验与持续探索之间的权衡。我们提出了VEX$^2$，一种口头探索-利用调度器，利用语言模型共同选择策略并分配剩余试验预算。VEX$^2$是唯一一个在所有六种环境中都实现了积极观察到的成功率提升的更新。

MInTRL: Off-policy Intervention can boost On-policy RL

MInTRL：非政策干预可以提升非政策强化学习

• Authors: Mingyu Chen, Yefan Tao, Gerald Friedland, Xuezhou Zhang, Chris Kong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12419
• Pdf link: https://arxiv.org/pdf/2609.12419
• Abstract Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.
• 中文摘要 带有可验证奖励的强化学习通常在策略上进行，保持训练数据接近当前策略，但限制学习路径为策略自身能发现的轨迹。而非策略方法如监督微调则可以利用超出基础模型能力的外部知识，但可能面临较大的分布转移。因此，关键挑战是在不牺牲可学习性的情况下扩展探索。本研究介绍了最小干预强化学习（MInTRL），通过稀疏、局部的干预扩展探索前沿，推动政策内的推广。生成过程中，法官干预策略定期审查当前策略输出，用简短修正替换错误后缀，并立即将控制权归还策略。在培训过程中，MInTRL采用序列级优势回归目标，消除重要性抽样的需求。我们表明，稀疏的局部干预可以在保持整体政策轨迹的政策内性质的同时，显著提升覆盖范围，超越有限预算的政策抽样。在数学和代码基准中，MInTRL持续优于标准的政策内和非政策基线。消融显示，MInTRL在自我干预及不同法官政策中依然有效，而在中等干预强度时表现达到峰值，凸显了最小干预的重要性。这些结果确立了最小干预作为提升政策内强化学习的有效范式。

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

长视野LLM代理强化学习的粒度自适应学分赋值

• Authors: Taoran Liang, Yang Liu, Shang Luo, Yingguang Yang, Rongrong Zhang, Yingzong Min, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Kefu Xu, Congjing Ran, Bin Chong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.12424
• Pdf link: https://arxiv.org/pdf/2609.12424
• Abstract Reinforcement learning is now the standard way to train large language model agents on long-horizon tasks, where dozens of interdependent actions precede a single sparse reward. Critic-free, group-relative methods such as GRPO suit this regime, but they broadcast one trajectory-level scalar to every step and cannot say which decision drove the outcome. GiGPO recovers a step-level signal by grouping time steps that share an anchor state, yet it merges the step- and episode-level estimates under one fixed weight, spending the same resolution on a pivotal branching decision as on a routine, near-deterministic transition. We argue that the right resolution is state-dependent, and propose GACA, a critic-free estimator whose granularity follows an uncertainty-based criticality proxy. GACA scores every step by the negative log-likelihood its own rollout already records, then blends the two advantages with a per-step weight that grows with that score, so the gradient places more weight on the fine-grained signal at above-average NLL and on the episode-level signal below it. We derive an exact risk decomposition for the implemented mixture and show that sufficiently small modulation improves on fixed mixing under positive directional alignment. A separate conditional result bounds local action-value variation using expected NLL, while an error-projection analysis characterizes when mixing adds value beyond scalar uncertainty reweighting. On ALFWorld and WebShop, GACA improves task success over GRPO and GiGPO at both 1.5B and 7B scales.
• 中文摘要 强化学习现已成为训练大型语言模型代理处理长视野任务的标准方法，在这些任务中，数十个相互依赖的动作在单一稀疏奖励之前完成。无批判、群相对的方法如GRPO适合这种模式，但它们对每一步广播一个轨迹级标量，无法确定哪个决策驱动了结果。GiGPO通过分组共享锚点状态的时间步恢复步级信号，但将步级和事件级估计合并为一个固定权重，在关键分支决策上花费的分辨率与常规、近确定性转移相同。我们认为正确的分辨率依赖状态，并提出了GACA，一种无批判估计器，其粒度遵循基于不确定性的临界性代理。GACA根据自身推广已记录的负对数似然对每一步进行评分，然后将两者优势结合，并设定随分数增长的每步权重，因此梯度对高于平均NLL的细粒度信号和低于平均水平的剧集级信号赋予更多权重。我们对实现的混合推导出精确的风险分解，并证明在正向对齐下，足够小的调制比固定混合会更好。另一个条件结果利用预期NLL界定局部动作值变化，而误差投影分析则表征混合在标量不确定性重加权之外的价值。在ALFWorld和WebShop上，GACA在1.5B和7B尺度上均提升了任务成功率，优于GRPO和GiGPO。

EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning

EvoRS：开放式强化学习中奖励系统的政策自我演化

• Authors: Weiyuan Li, Aili Chen, Xintao Wang, Yikai Zhang, Qingqing Dong, Jinghan Xu, Hongru Hou, Wenxuan Zhao, Chengkun Lang, Jun Gao, Yuanli Guo, Hongcheng Guo, Yanghua Xiao, Deqing Yang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12459
• Pdf link: https://arxiv.org/pdf/2609.12459
• Abstract Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by (2.107) and (4.767) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.
• 中文摘要 开放式强化学习通常依赖基于评分标准的奖励，针对没有直接可验证答案的任务。然而，策略和奖励系统形成了一个动态反馈循环：随着策略优化当前奖励，最初有用的奖励系统可能因奖励黑客或反应辨别性降低而变得不可靠。因此，奖励系统应当在训练过程中不断演变，而非固定。现有的动态评分标准会调整评估标准，但奖励失败也可能源于评分机制或信号组合。我们介绍了EvoRS，一个自我演进的强化学习框架，将奖励系统从政策上的经验演化为可执行的奖励DAG。具体来说，代理设计者通过策略上的推广和奖励追踪更新该系统，以保持列车时间的可靠性。在写作和角色扮演方面，EvoRS在三位评审下均达到最佳质量，分别比政策高出6.107点和4.767分，同时减少了奖励黑客和覆盖失败，保持了奖励信息量。消融确认，全面的固定奖励系统在开放式任务中无法保持可靠性，必须在培训过程中不断演进。

AMDKernelVault: Large-Scale Datasets and Agentic Training for AMD GPU Kernel Optimization

AMDKernelVault：AMD GPU内核优化的大规模数据集与代理训练

• Authors: Ji Liu, Saptarshi Majumder, Yiqing Huang, Wenwen Ouyang, Umang Pandey, Zeping Li, Chushi Chen, Zihao An, Puyuan Yang, Zekai Li, Sina Rafati, Ziqiong Liu, Pratik Prabhanjan Brahma, Dong Li, Zicheng Liu, Sharon Zhou, Emad Barsoum
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.12471
• Pdf link: https://arxiv.org/pdf/2609.12471
• Abstract We introduce AMDKernelVault, an open HIP and Triton kernel corpus and training framework for recent AMD CDNA GPUs. Existing LLM-based kernel agents are largely CUDA/NVIDIA-centric and often depend on repeated frontier-LLM calls for generation, reflection, and optimization. To address this gap, we develop HIPKernelGen and TritonKernelGen, agent-driven pipelines that transform PyTorch references into HIP or Triton kernels, compile and validate candidates under ROCm, and latency-profile them on AMD hardware. The corpus contains 62,153 execution-verified HIP kernel samples, 2,377 production-grounded ROCm Libraries QA entries, and 39,893 Triton kernels. We further train Qwen3-8B with supervised fine-tuning and execution-aware reinforcement learning as a demonstration of the corpus's utility. Under fixed evaluation budgets, it achieves the highest correctness among the compared models on PyTorch-to-HIP (34.0% Pass@1), TritonBench-G (33.2% Corr@3), and ROCmBench (41.94% Corr@3), but does not uniformly lead compilation or speed metrics. The corpus and documentation are available at this https URL, and the associated training and kernel-generation code is available at this https URL.
• 中文摘要 我们介绍AMDKernelVault，这是一个面向近期AMD CDNA GPU的开放HIP和Triton内核语料库及训练框架。现有基于LLM的内核代理大多以CUDA/NVIDIA为中心，且常依赖反复前沿调用来生成、反思和优化。为弥补这一空白，我们开发了HIPKernelGen和TritonKernelGen，这些代理驱动流水线将PyTorch引用转换为HIP或Triton内核，在ROCm下编译和验证候选内容，并在AMD硬件上进行延迟剖析。该语料库包含62,153个经过执行验证的HIP内核样本，2,377个基于生产环境的ROCm库QA条目，以及39,893个Triton内核。我们还进一步通过监督微调和执行感知强化学习训练Qwen3-8B，展示语料库的实用性。在固定评估预算下，它在PyTorch-to-HIP（34.0%Pass@1）、TritonBench-G（33.2%Corr@3）和ROCmBench（41.94%Corr@3）上实现了最高的正确性，但在编译或速度指标上并未统一领先。语料库和文档可在此 https URL 获取，相关的训练和内核生成代码也在此 https URL 获取。

LettuceVisSim: A Simulator That Generates Lettuce Image Time-series for Vision-Based Reinforcement Learning

LettuceVisSim：一款生成生菜图像时间序列的模拟器，用于基于视觉的强化学习

• Authors: Ziye Zhu, Bert van 't Ooster, Congcong Sun, Eldert van Henten, Sjoerd Boersma
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Systems and Control (eess.SY); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.12505
• Pdf link: https://arxiv.org/pdf/2609.12505
• Abstract Vision-based reinforcement learning holds strong potential for decision-making in controlled environment agriculture (CEA). However, its development is hindered by the scarcity of labelled crop images. To address this gap, LettuceVisSim, a lettuce growth simulator that generates labelled time series of crop images, was developed and validated. The simulator contains a process-based model (PBM) for shoot dry weight dynamics, a canopy layout algorithm for deriving canopy layout representations from shoot dry weight, and a Unity rendering engine for image generation. Five findings support the simulator. First, the PBM reproduced shoot dry weight under dynamic plant-density management with $\mathrm{R}^{2}=0.84$. Second, a piecewise cubic regression mapped shoot dry weight to potential projected area with $\mathrm{R}^{2}=0.94$. Third, the canopy layout representation was validated using 12 experimental datasets each having different dynamic environmental and spacing conditions. It reproduced the ground coverage ratio dynamics observed in measured images, achieving $\mathrm{R}^{2}=0.84$ when driven by measured shoot dry weight and $\mathrm{R}^{2}=0.40$ (0.76 excluding one outlier) when driven by PBM-simulated values. Fourth, the Unity rendering engine converted canopy layout representations into RGB and segmentation images at less than 10~ms. Fifth, a demonstration showed that a lighting-control policy can be learned and applied by observing only crop images that were generated with LettuceVisSim, providing a proof of concept of vision-based reinforcement learning in CEA using LettuceVisSim.
• 中文摘要 基于视觉的强化学习在受控环境农业（CEA）中的决策具有强大潜力。然而，其发展受到标记作物图像稀缺的阻碍。为弥补这一空白，开发并验证了LettuceVisSim，一款生成带标签作物图像时间序列的生菜生长模拟器。该模拟器包含基于过程的模型（PBM）用于计算枝条干重动态，一个用于从枝条干重推导树冠布局表示的树冠布局算法，以及用于生成图像的Unity渲染引擎。有五项发现支持该模拟器。首先，PBM在动态植株密度管理下以$\mathrm{R}^{2}=0.84$再现了枝条干重。其次，分段立方回归将枝条干重映射为$\mathrm{R}^{2}=0.94$。第三，利用12个具有不同动态环境和间距条件的实验数据集验证了树冠布局表示。它重现了测量图像中观察到的地面覆盖比动态，在测量的枝条干重驱动时实现了$\mathrm{R}{2}^=0.84$;在由PBM模拟值驱动时，实现了$\mathrm{R}^^{2}=0.40$（扣除一个异常值0.76）。第四，Unity渲染引擎在不到10~毫秒的速度内将树冠布局表示转换为RGB和分段图像。第五，演示显示仅观察用LettuceVisSim生成的裁剪图像即可学习和应用光照控制策略，提供了利用LettuceVisSim在CEA中基于视觉强化学习的概念验证。

From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasoners

从协作到能力：将被引导的大型语言模型专家内化为紧凑的推理工具

• Authors: Frank Nie, Shuyao Wang, Ethan B. Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12578
• Pdf link: https://arxiv.org/pdf/2609.12578
• Abstract A compact controller can coordinate stronger experts by selecting whom to consult, formulating requests, and integrating their responses. We study whether learning from both the controller's decisions and the experts' reasoning and code improves its generation after expert removal. We introduce \textsc{Rivet} for \emph{collaboration internalization}: expert-augmented reinforcement learning applies a shared outcome signal to controller decisions and returned expert spans, and verified trajectory internalization consolidates complete successful interactions through format-aware supervised training. The deployed controller generates reasoning, code, and interaction structure with local Python execution and no external LLM. Across seven competition-mathematics benchmarks, RIVET-1.7B and RIVET-4B achieve average accuracies of $28.25\%$ and $44.16\%$; Stage~II improves RIVET-4B's accuracy after expert removal by $6.49$ points, and GPQA-Diamond results provide evidence of generalization to scientific reasoning. Ablations show gains from ordinary trajectory supervision and additional format weighting, supporting the effectiveness of training on the content and structure of verified collaborations.
• 中文摘要 紧凑型控制器可以通过选择咨询对象、提出请求并整合他们的回应来协调更强的专家。我们研究从控制器的决策和专家的推理和代码中学习是否能提升专家移除后的生成。我们引入了 \textsc{Rivet}，用于 \emph{协作内化}：专家增强强化学习将共享结果信号应用于控制器决策和专家返回的跨度，经过验证的轨迹内化通过格式感知的监督训练巩固了完整的成功交互。部署的控制器通过本地 Python 执行生成推理、代码和交互结构，且无外部大型语言模型。在七项竞赛数学基准测试中，RIVET-1.7B 和 RIVET-4B 的平均准确率分别为 28.25 美元和 44.16% 美元;Stage~II 在专家移除后提升了 RIVET-4B 的准确性 6.49 美元点数，GPQA-Diamond 结果也证明了对科学推理的推广。消融显示普通轨迹监督和额外格式权重带来益处，支持对已验证协作内容和结构的训练效果。

Direct Preference Density Alignment for Conversational Audio Equalization

会话音频均衡的直接偏好密度对齐

• Authors: Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe, Pablo Martinez Nuevo, Zheng-Hua Tan
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12607
• Pdf link: https://arxiv.org/pdf/2609.12607
• Abstract Large Language Model alignment typically relies on learned proxy reward models, which significantly increase the memory footprint during training and are notoriously prone to instability and reward hacking. While offline methods like Direct Preference Optimization (DPO) bypass the reward model, they lose the ability to perform online exploration. If no optimization constraints are applied, this can lead to format collapse in bounded, continuous spaces. To resolve this, we propose Direct Preference Density Alignment: An alternative framework that removes the need for a learned proxy reward model while strictly preserving the benefits of online reinforcement learning. We leverage large-scale user data (approximately 90,000 samples) to construct non-parametric preference density maps, establishing an empirical reward surface. In addition to removing the reward model, Direct Preference Density Alignment enables the combination of the online structural grounding of Group Relative Policy Optimization (GRPO) with the targeted offline refinement of DPO. We show that this GRPO+DPO combination achieves the highest performance, and in a blind audio equalization listening test, enables a 1.5B-parameter model to achieve perceptual parity with a carefully prompt-engineered GPT-4o mini baseline, using only a fraction of the inference compute.
• 中文摘要 大型语言模型对齐通常依赖于学习中的代理奖励模型，这类模型在训练期间显著增加了内存占用，且容易出现不稳定和奖励黑客攻击。而离线方法如直接偏好优化（DPO）绕过奖励模型，但它们失去了在线探索的能力。如果不应用优化约束，可能导致有界连续空间中的格式崩溃。为解决这个问题，我们提出了直接偏好密度对齐：一种替代框架，在严格保留在线强化学习优势的同时，无需学习代理奖励模型。我们利用大规模用户数据（约9万个样本）构建非参数偏好密度图，建立经验奖励曲面。除了去除奖励模型外，直接偏好密度对齐还实现了组相对策略优化（GRPO）的在线结构基础与DPO的有针对性离线优化相结合。我们证明，这种GRPO+DPO组合实现了最高性能，并在盲测音频均衡听取测试中，使1.5B参数模型能够与精心提示设计的GPT-4o迷你基线实现感知奇偶，且仅耗费了一小部分推理计算。

SteerDuplex: Steerable Duplex Speech Dialogue Models

SteerDuplex：可导向双工语音对话模型

• Authors: Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai, Sonal Kumar, Nikhil Barhate, Isabell Sagar, Steven Li, Miheer Bavare, Daniel Quigley, Fabiola Tapia Carrillo, Jose M Patron E, Diego Macías Gutiérrez, Paul Song, Ramani Duraiswami, Dinesh Manocha, Yunzhong He
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.12623
• Pdf link: https://arxiv.org/pdf/2609.12623
• Abstract Full-duplex spoken dialogue models support low-latency turn taking, interruption handling, and backchanneling, yet a key capability remains underexplored: steerability, the ability to reliably shift conversational behavior along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. We introduce a taxonomy of text- and audio-based steerability that identifies substantial gaps in current full-duplex models. To address this gap, we introduce SteerDuplex, a Moshi-based full-duplex speech model fine-tuned on natural conversations and synthetic dialogues targeting instruction following, vocal delivery, reasoning, and duplex interaction. We further apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based semantic feedback to improve timing and response continuity. To evaluate full-duplex spoken steerability, we introduce SteerBench, a benchmark with 390 spoken prompts and 1,067 human-authored binary audio and text rubrics spanning tone, persona, style/accent, and speed/length. On SteerBench, supervised training improves audio-steering average pass rate by 44.5 percentage points over the strongest evaluated open baseline. On Audio MultiChallenge, task average pass rate improves by 7 points over its strongest evaluated open baseline. RL further raises source-clean interruption response from 72.5% to 82.5% and reduces synthetic pause barge-in from 26.5% to 9%. Steering and aggregate task scores remain comparable or higher, while reward probes reveal reward hacking through incomplete responses. Our model and benchmark support systematic research on spoken steerability, with reward analysis showing why timing gains must be evaluated alongside response completeness.
• 中文摘要 全双工口语对话模型支持低延迟的转口、中断处理和反向通道，但一个关键能力仍未被充分开发：可引导性，即根据用户指令可靠地根据语气、人格、发言速度和语音风格调整对话行为的能力。我们引入了基于文本和音频的引导分类法，识别出当前全双工模型中的重大空白。为弥补这一空白，我们引入了SteerDuplex，一种基于Moshi的全双工语音模型，针对自然对话和合成对话进行微调，目标是指令跟随、语音表达、推理和双工交互。我们进一步应用了两阶段强化学习（RL）与混合奖励，结合可验证的交互检查和基于判决的语义反馈，以提升时机和响应连续性。为了评估全双工口语引导，我们引入了SteerBench基准，该基准包含390个口语提示和1,067个人工二进制音频和文本评分标准，涵盖语调、人物、风格/口音以及速度/长度。在SteerBench上，监督训练使音频引导平均通过率比最强评估的开放基线提升了44.5个百分点。在音频多挑战中，任务平均通过率比最强的开放基线提升了7个百分点。强化学习进一步将源清中断响应率从72.5%提升至82.5%，并将合成暂停突入率从26.5%降至9%。引导和综合任务得分保持相当或更高，而奖励探针通过不完整回答揭示奖励黑客行为。我们的模型和基准支持了语音引导的系统研究，奖励分析表明为何必须同时评估时机增益与反应完整性。

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

AI阵营扭曲再审：RLHF是一款不错的实用对齐器

• Authors: Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao
• Subjects: Subjects: Machine Learning (cs.LG); Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2609.12651
• Pdf link: https://arxiv.org/pdf/2609.12651
• Abstract While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the \textit{distortion} -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter $\beta$ when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data ($\mu$) and the KL reference policy ($\pi_{\mathrm{ref}}$). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is $\tilde{\Theta}(\beta B + \beta)$, where $B$ is an upper bound on the log density ratio between $\mu$ and $\pi_{\mathrm{ref}}$. In particular, when there is no distribution mismatch (i.e., $\mu = \pi_{\mathrm{ref}}$), RLHF achieves the optimal distortion of $O(\beta)$ up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to $\mu$.
• 中文摘要 虽然人类反馈强化学习（RLHF）是将大型语言模型与人类偏好对齐的标准范式，但其在多元环境中的有效性受到质疑。值得注意的是，Gölz等人（2025年）的最新工作证明，\textit{distortion}——定义为RLHF策略的平均用户效用与最优平均效用之间的乘法差距——在用户偏好异质时，可以随着Bradley-Terry温度参数$\beta$呈指数级扩展。本研究对RLHF在奖励裁断下的失真进行了细致分析，并证明这种指数级退化并非算法的基本特性，而是分布生成偏好数据（$\mu$）与KL参考策略（$\pi_{\mathrm{ref}}$）之间分布不匹配的结果。为此，我们建立了RLHF在KL正则化强度多个区间的严格上下界。我们证明，在代表性区间，根据Bradley-Terry模型，失真为$\tilde{\Theta}（\beta B + \beta）$，其中$B$是对数密度比在$\mu$与$\pi_{\mathrm{ref}}}之间上的上界。特别地，当分布不匹配（即$\mu = \pi_{\mathrm{ref}}}）时，RLHF在常数范围内实现$O（\beta）$的最佳失真。我们的结果表明，为了合理最大化RLHF的平均效用，最好使用策略抽样偏好数据，或在RLHF前对接近$\mu$的数据进行微调。

Size Doesn't Matter: Material-State Reinforcement Learning for Excavator Transferable Soil Manipulation

尺寸无关紧要：挖掘机可转移土壤操作的材料状态加固学习

• Authors: Lennart Werner, Pol Eyschen, Sean Costello, Pierluigi Micarelli, Andrei Cramariuc, Marco Hutter
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.12677
• Pdf link: https://arxiv.org/pdf/2609.12677
• Abstract Earthmoving tasks such as excavation, backfilling, or embankment construction require deliberate repositioning of deformable soil. For these tasks, human operators use all shovel faces, while autonomous systems so far are limited to excavation and dumping. Current methods often rely on heuristic models but do not incorporate soil mechanics. We address this shortcoming by using Reinforcement Learning in a GPU-parallelized Material Point Method particle simulation. Our controllers are conditioned on material state such as shape and compactness, enabling skills that use multiple contact faces of the tool and displace material both inside and outside of the shovel. To use the same learned weights across machines, our policies operate in a normalized end-effector space and are deployed through a calibrated machine interface. We evaluate this calibrated transfer on an 11.5t hydraulic excavator and a 500g tabletop robot. We validate performance through autonomous construction of a 42m long, 2.1m high embankment in 45min, executing 201 individual policy strokes without failure, retry, or operator intervention. In a direct comparison, the autonomous controller matches an expert operator's progression speed and produces a higher, more consistent embankment. Additional qualitative backfilling and compaction experiments demonstrate the material-state awareness and calibrated transfer across machines.
• 中文摘要 土方作业如挖掘、回填或筑堤需要对可变形土壤进行有意识的重新定位。在这些任务中，人工操作员使用所有铲面，而自主系统目前仅限于挖掘和倾倒。目前的方法常依赖启发式模型，但未包含土壤力学。我们通过在GPU并行化的材料点方法颗粒模拟中使用加固学习来解决这一不足。我们的控制器基于材料状态，如形状和紧凑度，使得能够使用工具的多面接触面，并在铲子内外位移材料。为了在各机器间使用相同的学习重量，我们的策略在归一化的末端执行器空间运行，并通过校准的机器接口部署。我们在一台11.5吨液压挖掘机和一台500克台式机器人上评估这种校准后的转移。我们通过自主建造长42米、高2.1米的土堤，在45分钟内验证性能，执行201次单独策略笔触，且无故障、重试或操作员干预。直接对比中，自主控制器匹配专家操作员的进度速度，产生更高且更稳定的土堤。额外的定性回填和压实实验展示了材料状态感知和机器间校准转移。

Personalized and Trust-Aware Health Recommendation Policies for a Construction Workplace

建筑工作场所个性化且信任意识的健康建议政策

• Authors: Atefeh Mollabagher, Yogesh Gautam, Houtan Jebelli, Parinaz Naghizadeh
• Subjects: Subjects: Information Retrieval (cs.IR); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.12679
• Pdf link: https://arxiv.org/pdf/2609.12679
• Abstract Construction workers face workplace risks such as fatigue, heat stress, and other physically demanding conditions that can negatively affect their health and safety. Although monitoring these risks is important, timely and personalized health interventions are also needed to help prevent negative impacts on workers' well-being and productivity. To this end, in this paper, we propose a model to capture the interactions between a trust-aware health recommender system and workers who differ in health and trust sensitivity. Specifically, in our proposed dynamic model, worker health evolves over time, worker trust is affected by both health and recommendation dynamics, and trust in turn affects compliance with future recommendations. Given this model, we characterize the recommender policy, including a health-based recommendation triggering threshold and the recommendation frequency. We do so using both model-based short-horizon control and model-free reinforcement learning. We then investigate how recommendation frequencies are adjusted for different workers to balance their health, productivity, and trust. Our findings provide insight into the design of personalized health recommendation policies in construction workplaces and beyond.
• 中文摘要 建筑工人面临职场风险，如疲劳、热应激及其他身体要求高的环境，这些都可能对其健康和安全产生负面影响。虽然监测这些风险很重要，但及时且个性化的健康干预也需要帮助防止对工人福祉和生产力的负面影响。为此，本文提出了一个模型，以捕捉信任感知健康推荐系统与健康及信任敏感度差异工人之间的互动。具体来说，在我们提出的动态模型中，工人健康随时间演变，员工信任受健康和推荐动态影响，信任又影响对未来推荐的合规性。基于该模型，我们描述推荐政策，包括基于健康的推荐触发阈值和推荐频率。我们结合基于模型的短期视野控制和无模型强化学习进行分析。随后，我们研究如何调整推荐频率以平衡其健康、生产力和信任。我们的发现为建筑工作场所及更广泛领域个性化健康推荐政策的设计提供了见解。

SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy

SCQ：用S形有界熵稳定保守Q-学习

• Authors: Xiefeng Wu, Shu Zhang, Zhaojie Chu, Mingyu Hu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.12749
• Pdf link: https://arxiv.org/pdf/2609.12749
• Abstract Offline-to-online reinforcement learning reduces interaction cost for real-world robot learning but suffers from persistent value estimation instability. Existing methods address this through pessimistic regularization, lower-bound calibration, and architectural normalization, but an overlooked source of instability lies in the entropy formulation: the standard log-entropy term can become negative, destabilizing policy updates. We introduce SCQ (Sigmoid-Bounded Conservative Q-Learning), which replaces this term with a sigmoid-bounded formulation that stays strictly positive. SCQ retains conservative Q regularization and return-based lower-bound calibration, stabilizing policy optimization without sacrificing exploration. We evaluate SCQ on D4RL (Minari) benchmarks under both single-demonstration and standard dataset settings, as well as on simulation and real-world visual tasks. SCQ matches or exceeds baseline performance while exhibiting more stable training dynamics across state-based and visual benchmarks, and transfers to four real-robot platforms including manipulation, wheeled, quadruped, and humanoid systems. A direct clipping intervention that removes negative log-probability contributions, together with gradient-matched positive-score controls, indicates that positivity rather than a particular score shape alone drives much of the improvement. Project website: this https URL.
• 中文摘要 离线到在线的强化学习降低了现实机器人学习的交互成本，但存在持续的值估计不稳定性问题。现有方法通过悲观正则化、下界校准和架构归一化解决了这一问题，但一个被忽视的不稳定性源头在于熵的表述：标准的对数熵项可能变成负值，导致策略更新不稳定。我们引入了SCQ（S形边界保守Q-学习），用严格正的S形边界表述取代了该项。SCQ保留了保守的Q正则化和基于返回的下界校准，稳定了策略优化，同时不牺牲探索能力。我们在D4RL（Minari）基准测试中，无论是单一演示还是标准数据集设置，以及模拟和现实可视化任务中评估SCQ。SCQ在基于状态和视觉基准测试的训练动态中表现更稳定，且能支持包括操作系统、轮式、四足和人形系统在内的四种真实机器人平台。直接剪裁干预去除负对数概率贡献，结合梯度匹配正分数对照，表明积极性而非单纯某个分数形状推动了大部分改进。项目网站：此 https URL。

Curriculum-Based Adversarial Heterogeneous Agent Reinforcement Learning for Autonomous Quad-Copter Landing in Maritime Settings

基于课程的对抗异构代理强化学习，用于海上环境中自主四旋翼着陆

• Authors: Allan Minh-Tam Nguyen, Sree Showrya Kotala, Stefan Banioi-Crijman, Kurt Driessens, Rico Möckel
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.12758
• Pdf link: https://arxiv.org/pdf/2609.12758
• Abstract Recovering unmanned aerial vehicles (UAVs) in maritime environments is challenging due to wind turbulence and ship-deck motion, making it a valuable test case for alternative control and learning approaches as conventional landing approaches often become unreliable. We study simulated mid-air capture of quadrotor UAVs by a ship-mounted robotic arm, learning robust cooperative control policies with Heterogeneous-Agent Proximal Policy Optimization (HAPPO) Reinforcement Learning. We train with HAPPO using a curriculum and an adversarial wind agent (HARL-AC) in NVIDIA Isaac Lab, and compare the obtained control policies against those generated through curriculum-based domain randomization and a benchmark trained on a single sea state. In-distribution evaluation on sea states $0/4/5$ shows comparable success for HARL-AC and domain randomization of up to $97.5\%$. On out-of-distribution sea states $7/8/10$, HARL-AC generalizes better, achieving up to $16\%$ higher median success rate at sea state 10, and substantially lower crash rates of up to $14\%$ compared to the domain randomization policy. Furthermore, we show that the adversarially trained policy shows more cautious behavior, slightly increasing timeouts by $<3\%$, but yields safer recovery behavior in severe, unseen conditions.
• 中文摘要 在海上环境中回收无人机（UAV）由于风流湍流和舰甲板运动，成为替代控制和学习方法的宝贵测试案例，因为传统着陆方法往往变得不可靠。我们研究了舰载机械臂对四旋翼无人机的模拟空中捕获，利用异构代理近端策略优化（HAPPO）强化学习稳健的协作控制策略。我们在NVIDIA Isaac实验室使用HAPPO训练，使用课程和对抗风力代理（HARL-AC）进行训练，并将获得的控制策略与基于课程的领域随机化和单一海况训练基准生成的策略进行比较。海况评估显示，HARL-AC和领域随机化在最高可达97.5%美元的情况下，取得了类似的成功率。在分布外的海域状态$7/8/10$上，HARL-AC的推广更佳，在海上状态10的中位成功率高出$16\%$，且相较域随机化策略的崩溃率显著降低至$14\%$。此外，我们表明对抗训练策略表现出更谨慎的行为，超时次数略微增加<3\%$，但在严重且未见的条件下恢复行为更安全。

Offline Reinforcement Learning for Wind Farm Control: A Wind Tunnel Study under Dynamic Wind Directions

风电场控制的离线强化学习：动态风向下的风洞研究

• Authors: Yuhan Su, Hongyang Dong, Simone Tamaro, Filippo Campagnolo, Carlo L. Bottasso, Xiaowei Zhao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.12905
• Pdf link: https://arxiv.org/pdf/2609.12905
• Abstract This paper addresses the wind farm power maximization problem in the presence of wind direction changes. Specifically, a model-free Modified Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning (MTD3-BC) algorithm is proposed to tackle this task through yaw control under varying wind direction conditions. MTD3-BC is an offline reinforcement learning (RL) algorithm that aims to infer good behavior from only a precollected offline dataset. Additionally, to ensure smooth and moderate yaw adjustments, a new action consistency term is introduced into the policy optimization objective. Unlike online RL methods, MTD3-BC does not require extensive interactions with a wind farm simulator during training, significantly reducing computational costs and training time. A wind tunnel experiment is conducted to validate the effectiveness of the algorithm under varying wind directions. The results demonstrate that MTD3-BC successfully mitigates wake effects, delivering farm-level power gains of approximately 10\% over the baseline greedy strategy and performance on par with a data-calibrated model-based wake-steering benchmark, while requiring no wake model and only a small fraction of the training cost of online RL. To our knowledge, this is the first time an offline RL wind farm control policy has been validated and demonstrated experimentally.
• 中文摘要 本文讨论风向变化时风电场功率最大化问题。具体来说，提出了一种无模型的修改性双延迟深确定性策略梯度（MTD3-BC）算法，通过偏航控制在不同风向条件下完成该任务。MTD3-BC是一种离线强化学习（RL）算法，旨在仅从预先收集的离线数据集推断良好行为。此外，为确保偏航调整平稳适中，策略优化目标中引入了一个新的动作一致性项。与在线强化学习方法不同，MTD3-BC在训练过程中无需与风电场模拟器大量交互，显著降低计算成本和训练时间。通过风洞实验验证算法在不同风向下的有效性。结果表明，MTD3-BC成功减轻了尾流效应，农场层面的功率提升约为基线贪婪策略10%，性能与基于数据校准的模型尾迹引导基准相当，且无需尾迹模型，且仅为在线强化学习训练成本的一小部分。据我们所知，这是首次验证并实验验证了离线强化风电场控制策略。

Groupoid-Based Internal State Representations for Reinforcement Learning with Local Symmetries

基于群的内部状态表示，用于带有局部对称性的强化学习

• Authors: Ben Opperman, Eduardo Alonso, Esther Mondragón
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13035
• Pdf link: https://arxiv.org/pdf/2609.13035
• Abstract Symmetries play a central role in reducing the complexity of reinforcement learning problems, yet most existing approaches rely on fixed group actions or predefined state abstractions. Classical reinforcement learning algorithms typically assume a globally structured Markov decision process with uniformly applicable actions and transitions, an assumption that limits their ability to exploit modularity and local, context-dependent regularities present in many realistic environments. We propose a reinforcement learning framework using groupoids to capture local, state-dependent symmetries and support the dy- namic discovery of equivalence structures during interaction. The agent maintains orbit representatives together with transporters that map raw states to canonical forms, enabling learning and decision-making to be performed in a symmetry-reduced space while preserving local distinctions. Empirical results demonstrate that the proposed groupoid-based approach improves sample efficiency and convergence in dense and large-scale environments exhibiting strong partial symmetries, yielding substantial performance gains over standard Q-learning. These findings show that dynamically exploiting local symmetry provides a practical and mathematically principled route to scalable and generalisable reinforcement learning.
• 中文摘要 对称性在减少强化学习问题复杂性方面起着核心作用，但大多数现有方法仍依赖固定群动作或预定义状态抽象。经典强化学习算法通常假设一个全局结构化的马尔可夫决策过程，具有一致适用的动作和转移，这一假设限制了它们利用许多现实环境中存在的模块性和局部、上下文依赖规律性的能力。我们提出了一个强化学习框架，利用群体捕捉局部的状态依赖对称性，并支持交互过程中动态发现等价结构。智能体维护轨道代表体和将原始状态映射为典范形式的传送体，使学习和决策能够在对称约化空间内进行，同时保持局部区分。实证结果表明，基于群的方法是在高密度且大规模且具有强烈偏对称性的环境中提升样本效率和收敛性，较标准Q学习实现显著性能提升。这些发现表明，动态利用局部对称性为实现可扩展且可推广的强化学习提供了实用且数学原理的途径。

Robust Policy Optimization via Adversarial Importance Sampling

通过对抗重要性抽样实现稳健策略优化

• Authors: Amine Andam, Jamal Bentahar, Mustapha Hedabou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13044
• Pdf link: https://arxiv.org/pdf/2609.13044
• Abstract Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns. Advis satisfies three desirable criteria not jointly achieved by prior work: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, facilitating rapid prototyping and enabling reproducible and traceable evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6-14x more configurations than prior work. Finally, we evaluate our approach on continuous control environments, demonstrating its effectiveness relative to existing baselines. The code is available at: this https URL
• 中文摘要 在保护深度强化学习（DRL）策略免受输入扰动方面取得了显著进展。开发稳健的DRL主要包含三个阶段：算法设计、实现和评估。在本研究中，我们识别并解决了每个阶段的关键限制。首先，我们引入了对抗性重要性采样（Advis），这是一种利用标准训练轨迹的重要性抽样来估算和优化可验证的最坏情况回报的方法。Advis满足了以往工作未能共同实现的三个理想标准：它无需额外的环境交互，无需辅助网络，并且能够实现长期的鲁棒性。其次，我们引入了advrl，这是一个模块化的PyTorch库，提供现有鲁棒性方法和对抗性攻击的干净单文件实现，促进快速原型制作并实现可重复和可追溯的评估。第三，我们重新审视在学习对手下的评估，显示最佳对抗超参数不会跨智能体转移，这可能导致在使用有限攻击者配置集时高估鲁棒性。因此，我们针对大量且多样化的攻击者进行策略评估，使用比以往多6倍到14倍的配置。最后，我们评估了我们在连续控制环境中的方法，展示了其相对于现有基线的有效性。代码可在以下地址获取：此 https URL

MCRL2: Multi-resource Cross-attention-based Representation Learning-augmented Reinforcement Learning for Cloud Microservice Scheduling

MCRL2：基于多资源的跨注意力表征学习增强强化学习，用于云微服务调度

• Authors: Tiangang Li, Shi Ying, Xiangbo Tian, Chuan Shi, Ding Xiao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13048
• Pdf link: https://arxiv.org/pdf/2609.13048
• Abstract Efficient microservice scheduling is crucial for maintaining load balance across nodes in data centers and ensuring high quality of service. However, achieving this in practice remains challenging due to dynamic resource imbalance under fluctuating workloads, nonlinear coupling across multiple resource dimensions, and the heterogeneity of microservice resource demands. While reinforcement learning-based approaches have shown promise, they struggle to capture the complex interdependencies among heterogeneous resources and neglect the importance of learning informative system representations. To address these limitations, we propose MCRL2, a novel reinforcement learning approach augmented with multi-resource cross-attention-based representation learning for microservice scheduling. Specifically, we first propose MCRL, a novel representation learning approach that captures structured and informative interactions among nodes, resources, and microservices via a multi-resource cross-attention mechanism. Then, MCRL2 augments reinforcement learning through MCRL-enhanced actor-critic architecture combined with a maximum entropy objective, improving system state expressiveness and leading to more stable and effective scheduling decisions. Extensive experiments on real production cluster traces demonstrate that MCRL2 significantly outperforms existing baselines in load balancing, scheduling success rate and average completion time across diverse workload patterns.
• 中文摘要 高效的微服务调度对于数据中心节点间的负载平衡和确保高服务质量至关重要。然而，由于在工作负载波动下存在动态资源不平衡、跨多维资源的非线性耦合以及微服务资源需求的异质性，实现这一目标仍具挑战性。尽管基于强化学习的方法展现出潜力，但它们难以捕捉异构资源间复杂的相互依赖关系，忽视了学习信息系统表示的重要性。为解决这些局限性，我们提出了MCRL2，这是一种新颖的强化学习方法，辅以多资源跨注意力的表示学习用于微服务调度。具体来说，我们首先提出了MCRL，这是一种通过多资源交叉注意力机制捕捉节点、资源和微服务之间结构化且信息丰富的交互的新型表征学习方法。随后，MCRL2通过MCRL增强的actor-critic架构结合最大熵目标，增强强化学习，提升系统状态表达性，实现更稳定和高效的调度决策。对真实生产集群跟踪的广泛实验表明，MCRL2在负载均衡、调度成功率和平均完成时间等多样工作负载模式中显著优于现有基线。

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

基于正则化的强健强化学习的统一与约束视角

• Authors: Amine Andam, Jamal Bentahar, Mustapha Hedabou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13050
• Pdf link: https://arxiv.org/pdf/2609.13050
• Abstract Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.
• 中文摘要 基于正则化的方法已成为训练针对对抗性输入扰动的深度强化学习策略的标准方法。本文通过推导名义策略与最坏情况策略性能差距的新上界来统一这些方法。每个上界表示为现有正则化目标加上名义与最坏情况策略之间的KL发散惩罚，这进一步解释了为何增加KL惩罚能提升实际鲁棒性。基于这些界限，我们将强健训练表述为受限优化问题，表明现有方法对应固定拉格朗日乘数的特例。我们与策略共同更新乘数，自动调整正则化权重。最后，我们在多个连续控制任务中进行了广泛的对抗评估，以验证理论分析。

Expert-Space Exploration in MoE Reinforcement Learning

专家-太空探索在工程部强化学习中的应用

• Authors: Hongyi He, Zhenghao Lin, Xiao Liu, Peng Cheng, Yan Lu, Yeyun Gong
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.13058
• Pdf link: https://arxiv.org/pdf/2609.13058
• Abstract Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models have primarily focused on improving optimization stability and training efficiency, while treating the expert selection as a fixed component. Since routing determines the sparse computation paths that induce output distributions, expert selection offers an additional source of rollout diversity. Through empirical analysis, we find that perturbing expert routing effectively alters model output and increases rollout diversity, which is similar to increasing the decoding temperature. However, direct perturbation can activate unsuitable experts and substantially degrade rollout quality. Motivated by these observations, we introduce Expert-Space Exploration Reinforcement Learning (ESRL), an architecture-aware framework that explicitly explores the expert-routing space of MoE models. ESRL preserves high-confidence experts as anchors, and restricts stochastic routing to a plausible candidate pool, thereby retaining reliable computation paths. The perturbation strength is further adapted according to router entropy to avoid over-perturbation. To mitigate the routing mismatch introduced by perturbation, ESRL records the expert paths used during rollout and replays them during policy optimization. Experiments demonstrate that ESRL achieves the best performance across MoE backbones with top-K, top-1, and shared-expert routing, as well as across mathematics, science, and code tasks without additional sampling or computational cost. Specifically, ESRL on Qwen3-30B-A3B achieves the best among all compared methods, improving average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points, respectively. Further analyses of expert utilization and training dynamics provide insights into how exploiting MoE-specific routing structure benefits RL training.
• 中文摘要 强化学习（RL）已成为大型语言模型后期训练的核心。近期在专家混合（MoE）模型中，强化学习的进展主要集中在提升优化稳定性和训练效率，同时将专家选择视为固定组成部分。由于路由决定了诱导输出分布的稀疏计算路径，专家选择提供了额外的扩展多样性来源。通过实证分析，我们发现扰动专家路由有效改变模型输出并增加展开多样性，这类似于提高解码温度。然而，直接扰动可能会激活不合适的专家，并显著降低推广质量。基于这些观察，我们引入了专家-空间探索强化学习（ESRL），这是一个架构感知框架，明确探索MoE模型的专家路由空间。ESRL保留高置信度专家作为锚点，并将随机路由限制在合理的候选池中，从而保持可靠的计算路径。扰动强度根据路由器熵进一步调整，以避免过度扰动。为减轻扰动带来的路由不匹配，ESRL记录部署时使用的专家路径，并在策略优化时重放。实验表明，ESRL在MoE骨干网中，采用顶K、顶尖和共享专家路由，以及数学、科学和代码任务中，在无需额外采样或计算成本的情况下表现最佳。具体来说，Qwen3-30B-A3B的ESRL在所有比较方法中表现最佳，平均Pass@1和Pass@8分别提升3.2个百分点和4.5个百分点。对专家利用率和训练动态的进一步分析，有助于了解利用特定MoE的路由结构如何促进强化学习训练。

CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models

CanvasAnneal：扩散语言模型的课程强化学习

• Authors: Blake Olson, Yuhang Song, Emmett McQuinn, Yuan Shangguan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13060
• Pdf link: https://arxiv.org/pdf/2609.13060
• Abstract Diffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration. In this paper, we introduce CanvasAnneal, a curriculum-guided diffusion RL framework. During the initial RL phase, we warm-start exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas. As training progresses, we gradually remove this guidance and require the model to generate more of the reasoning trajectory independently. Across mathematical reasoning and tool-use benchmarks, CanvasAnneal improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and substantially accelerates reward improvement on several tasks, while gains are task-dependent. Our results suggest that structured training-time guidance can alleviate exploration bottlenecks in diffusion RL and speed up convergence on harder tasks.
• 中文摘要 扩散语言模型（DLMs）提供了有前景的并行生成能力，但在复杂推理和工具使用任务中仍落后于自回归模型。虽然强化学习（RL）最近被应用于增强DLMs，但标准强化学习方法存在探索瓶颈。为此，我们注入更强教师模型的推理先验来指导强化学习的探索。本文介绍了CanvasAnneal，一种课程指导的扩散强化学习框架。在初始强化学习阶段，我们通过向初始扩散画布注入教师生成的推理痕迹来预热启动探索。随着训练的推进，我们逐步取消这些指导，要求模型独立生成更多推理轨迹。在数学推理和工具使用基准测试中，CanvasAnneal 在 MATH500、Countdown 和 Tau2 上优于标准 diffu-GRPO，并在多个任务中显著加速奖励提升，而收益则依赖任务。我们的结果表明，结构化训练时间指导可以缓解扩散强化学习中的探索瓶颈，加快难度任务的收敛。

Keyword: diffusion policy

RodForesight: A World Model Enhanced Diffusion Policy for Slender and Material Agnostic Rod Insertion

RodForesight：一种世界模型增强扩散政策，适用于细长且无材质的杆状插入

• Authors: Chuanbo Yu, Mingyu Yue, Yan Lyu, Chuhan Song, Peng Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.12103
• Pdf link: https://arxiv.org/pdf/2609.12103
• Abstract Slender rod insertion arises in precision manufacturing, where millimetre scale diameter and tight clearances demand accurate perception and control. Conventional peg-in-hole methods assume a rigid object whose tip pose is fixed relative to the gripper. This assumption breaks down for a high aspect ratio rod, which can bend during manipulation, making its tip motion dependent on the rod configuration, grasp, material properties, and contact. We present RodForesight, a learning framework that factorises the task into two stages: 1) coarse approaching, which uses visual servoing to map diverse initial configurations into a compact near hole hand-off region; and 2) predictive insertion, which performs fine alignment and completes the insertion. It is worth noting that the two stages can be wrapped into an end-to-end design. During insertion, a diffusion policy generates candidate action chunks, while an action conditioned world model predicts their effects on rod-hole alignment. This pre-execution evaluation enables RodForesight to select the best action chunk based on predicted tilt and radial errors before execution. Experiments investigate the performance of different stages and the end-to-end setting, where RodForesight improves the success rate from 88.9% to 96.7%, compared to baseline methods such as diffusion policy.
• 中文摘要 细长杆插入出现在精密制造中，毫米级直径和紧密间隙要求准确感知和控制。传统的插孔方法假设一个刚性物体，其尖端姿态相对于夹持器固定。但对于高纵弦比杆来说，这一假设不成立，该杆在操作过程中可能弯曲，使其尖端运动依赖于杆的配置、握把、材料特性和接触。我们介绍RodForesight，这是一个将任务分解为两个阶段的学习框架：1）粗略进近，利用视觉伺服将多种初始配置映射到紧凑的近孔交接区域;2）预测插入，执行精细对齐并完成插入。值得注意的是，这两个阶段可以集成端到端的设计。在插入过程中，扩散策略生成候选动作块，而动作条件化世界模型则预测其对杆孔对齐的影响。这种预执行评估使RodForesight能够基于预测的倾斜和径向误差在执行前选择最佳动作块。实验研究不同阶段的性能及端到端设置，RodForesight将成功率从88.9%提升至96.7%，相较于扩散策略等基线方法。

Online Material Estimation for Conditioned Diffusion Policy in Shaping Deformable Linear Objects

在塑造可变形线性物体中的条件扩散政策在线材料估计

• Authors: Ryunosuke Yamada, Tomohiro Motoda, Yukiyasu Domae, Tokuo Tsuji
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.12634
• Pdf link: https://arxiv.org/pdf/2609.12634
• Abstract Shape control of deformable linear objects (DLOs) is challenging for imitation learning because deformation behavior varies with material properties such as stiffness and elasticity, so a single policy must generate different action sequences for different objects even when the goal shape is identical. We propose a diffusion policy conditioned on material labels that are estimated online during manipulation. A recurrent estimation network predicts the material label of the grasped object from the time series of multi-view images and robot joint states, and the predicted label conditions the diffusion policy at every inference step. We collected 480 real-robot demonstrations covering four DLO materials and three groove-placement tasks, and compared per-material specialist policies, a task-conditioned policy without material labels, a policy conditioned on ground-truth material labels, and the proposed policy. Conditioning on ground-truth material labels improved the average success rate from 45.8% to 60.0% over the task-only policy, and the proposed policy reached 60.8% without any prior material information, matching the policy given ground-truth labels. A post-hoc analysis shows that the estimator extracts material-related information from the manipulation observations and that the diffusion policy responds to the resulting conditioning signal, while the one pronounced failure case is associated with persistent confusion between two similar materials.
• 中文摘要 可变形线性物体（DLO）的形状控制在模仿学习中具有挑战性，因为变形行为会随材料性质（如刚性和弹性）变化，因此即使目标形状相同，单一策略也必须为不同物体生成不同的动作序列。我们提出了一种基于材料标签的扩散策略，这些标签在操作过程中在线估计。一个循环估计网络通过多视图图像和机器人关节状态的时间序列预测所抓物体的材料标签，预测标签则在每个推断步骤中决定扩散策略。我们收集了480个真实机器人演示，涵盖四种DLO材料和三个槽位任务，并比较了每种材料的专业策略、无材料标签的任务条件策略、基于基层真实材料标签的策略以及拟议策略。条件化基于地面真实材料标签使平均成功率从45.8%提升至60.0%，而拟议策略在无任何先验材料信息的情况下达到60.8%，与给定的基础真实标签政策相匹配。事后分析显示，估计器从操作观察中提取材料相关信息，扩散策略响应条件反射信号，而明显失败案例则与两种相似材料之间的持续混淆相关。