生成时间: 2026-09-24 21:20:52 (UTC+8); Arxiv 发布时间: 2026-09-24 20:00 EDT (2026-09-25 08:00 UTC+8)

今天共有 33 篇相关文章

Keyword: reinforcement learning

On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning

关于多目标强化学习中事后诸葛亮再标注的偏好覆盖崩溃

• Authors: Baptiste Bonin, Caro Strickland, Audrey Durand
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.26918
• Pdf link: https://arxiv.org/pdf/2609.26918
• Abstract Hindsight relabeling which retroactively replacing a transition's goal with the outcome the agent actually achieved is an effective tool for improving sample-efficiency in Reinforcement Learning (RL). A natural extension to preference-conditioned multi-objective RL (MORL) relabels transitions with the preference direction the agent achieved rather than the one asked for. We show that this extension is frequently harmful: across four preference-conditioned off-policy algorithms spanning two critic backbones and two preference-sampling schemes on the continuous-control MO-Gymnasium suite, it degrades 19 of 36 algorithm-environment settings by as much as four standard deviations, improves only one, and leaves the rest unaffected. The harm is not a symptom of noisy relabels; denoising the target recovers almost nothing, and neither prioritized sampling nor any buffer-structural choice reproduces it. Instead, repeated relabeling collapses the critic's coverage onto whatever narrow region of the preference space the agent happened to visit. We name this failure mode \emph{Preference Coverage Collapse}, and quantify it with abandoned preference mass (APM), a value-aware statistic that tracks the harm ($\rho = -0.73$) where a purely structural coverage count does not. We then introduce \texttt{her_mix}, a single-parameter convex combination pulling the achieved direction back towards the requested preference. At one fixed value across every algorithm and environment, it returns 16 of the 19 harmed settings to baseline, preserves and even improves the one setting in which relabeling helps, and cuts abandoned preference mass from $69\%$ to $6\%$. Protecting coverage over the preference simplex, not filtering noisy relabels, is what makes hindsight relabeling safe for MORL.
• 中文摘要 事后诸葛亮重新标记，即用代理实际实现的结果来回溯替代转变的目标，是提升强化学习（RL）样本效率的有效工具。偏好条件多目标RL（MORL）的自然扩展是将转变重新标记为代理实现的偏好方向，而非被要求的方向。我们证明这种扩展常常有害：在连续控制的MO-Gymnasium套件上，使用跨越两个批评主干和两种偏好采样方案的四个偏好条件非策略算法，它将36个算法环境设置中的19个降低多达四个标准差，仅改善一个，其余不受影响。这种危害并非噪声重标的症状;去噪点几乎无法恢复任何东西，且无论是优先抽样还是任何缓冲-结构选择都无法重现该问题。相反，反复重新标记会将批评者的覆盖范围压缩到代理所访问的偏好空间的某个狭窄区域。我们将此失败模式命名为 \emph{偏好覆盖崩溃}，并用放弃偏好质量（APM）来量化，这是一种价值感知的统计量，追踪伤害（$\rho = -0.73$），而纯结构性覆盖计数则无法追踪。然后引入 \texttt{her_mix}，一种单参数凸组合，将实现的方向拉回请求偏好。在每个算法和环境中保持一个固定值时，它将19个受损设置中的16个返回基线，保留甚至改进唯一一个重新标记有助于的设置，并将放弃偏好质量从69%$降至6%$。保护优先单纯形的覆盖，而不是过滤噪杂的重新标记，这才是MORL事后重新标记安全的关键。

WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps

这到底是怎么回事？！无模拟强化学习，使用Wasserstein倾斜流程图

• Authors: Abbas Mammadov, Jerry Y. Huang, Justin Lin, Partha Kaushik, Sheel Shah, Kartik Nair, Yee Whye Teh, Nicholas M. Boffi
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.27033
• Pdf link: https://arxiv.org/pdf/2609.27033
• Abstract Reward fine-tuning aims to update a pre-trained flow-based generative model to improve the downstream reward of its generated samples. Existing methods typically formulate this problem as sampling from a reward-tilted distribution, the solution to a KL-regularized reward-maximization problem. Here, we introduce an optimal transport regularizer built directly from the pre-trained drift. Unlike KL reward tilting, the resulting objective transports individual samples toward higher reward rather than reweighting the base distribution. We show that the resulting problem is equivalent to a deterministic optimal control problem on the flow. Given a pre-trained flow map, this equivalence yields a simulation-free reinforcement learning algorithm for fine-tuning generative flows. We call the resulting framework Wasserstein-Tilted Flow Maps (WTF), the first end-to-end fine-tuning recipe native to flow maps. The output is a fine-tuned flow map that retains strong reward-aligned performance at few-step inference budgets without post-hoc distillation. Experiments on ImageNet-256 and text-to-image show that WTF achieves higher reward with comparable or higher diversity than baselines, while requiring up to $280\times$ less training compute. More broadly, we argue that accelerated samplers such as flow maps are essential infrastructure for efficient post-training, and that the dominant KL-regularized formulation is only one of many choices worth revisiting.
• 中文摘要 奖励微调旨在更新预训练的基于流的生成模型，以改善其生成样本的下游奖励。现有方法通常将该问题表述为从奖励倾斜分布中抽样，即对 KL 正则化奖励最大化问题的解。这里，我们引入了直接基于预训练漂移构建的最优传输正则化器。与 KL 奖励倾斜不同，所得目标是将单个样本向更高奖励方向传输，而非重新加权基础分布。我们证明所得问题等价于流上的确定性最优控制问题。给定预训练的流图，这一等价性产生了无仿真的强化学习算法，用于微调生成流。我们将所得框架称为 Wasserstein-倾斜流图（WTF），这是流图中首个端到端微调配方。输出是一个精细调优的流程图，在少步推断预算下保持强的奖励对齐性能，无需事后蒸馏。ImageNet-256和文本转图像的实验表明，WTF在与基线相当或更高多样性下能获得更高的奖励，同时训练计算量可比高达280美元/倍数。更广泛地说，我们认为加速采样器如流图是高效后训练的关键基础设施，主流KL正则化表述只是众多值得重新审视的选项之一。

Reinforcement Learning with Decomposed Subtasks

带分解子任务的强化学习

• Authors: Mattie Terzolo, Mikolaj Sacha, Ayan Sinha, Andrew Rabinovich
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27035
• Pdf link: https://arxiv.org/pdf/2609.27035
• Abstract Group Relative Policy Optimization (GRPO) and related policy-gradient methods for training language model agents collapse an entire multi-turn rollout into a single scalar trajectory reward before it enters the policy update. When the task composes distinct skills, especially under sparse and delayed environmental feedback, this collapsing is lossy: the optimizer must implicitly infer which competency drove the outcome and how that should change behavior. We argue the right primitive is not a better scalar but a decomposition: trajectory reward should be split along subtasks before it enters the policy update. We introduce Reinforcement Learning with Decomposed Subtasks (RLDS), whose core is Subtask-Decomposed Advantage Estimation (SDAE): a replacement for the scalar GRPO advantage that splits trajectory reward into per-subtask shares on a fixed taxonomy, computes a group-relative advantage per subtask, and distributes per-token credit by weighting each subtask's advantage by its importance, concentrating it around the step where a reflection marks that subtask's execution as consequential. We evaluate on four agentic benchmarks: FrozenLake (sparse grid navigation), HotpotQA (multi-hop QA, one retrieval tool), ScienceWorld (long-horizon embodied science), and DeepResearch (long-form research, four tools, composite rubric reward). Heterogeneity diagnostics emitted during training show where decomposition pays off - gains scale with subtask heterogeneity, largest on the high-heterogeneity tasks ScienceWorld (+11.5 points, paired-bootstrap 95% CI [+9.8, +13.3]) and FrozenLake (+9.8 points, [+7.0, +12.8]), and within noise on HotpotQA and DeepResearch, where the diagnostics predicted little to recover. ScienceWorld is also more compute-efficient under RLDS than scalar GRPO (-10.9% wall-clock per step), as long rollouts amortize the fixed reflect-and-grade overhead.
• 中文摘要 组相对策略优化（GRPO）及相关的策略梯度训练语言模型代理方法，在进入策略更新前，将整个多回合的部署整合为单一标量轨迹奖励。当任务包含不同技能，尤其是在稀疏且延迟的环境反馈下，这种崩溃是有损的：优化者必须隐式推断哪种能力驱动了结果，以及这应如何改变行为。我们认为正确的原语不是更好的标量，而是一种分解：轨迹奖励应在进入策略更新前沿子任务拆分。我们引入了带有分解子任务的强化学习（RLDS），其核心是子任务分解优势估计（SDAE）：替代标量GRPO优势，该优势将轨迹奖励按固定分类法按每个子任务分配，计算每个子任务的群体相对优势，并通过按其重要性加权每个子任务的优势，集中在标记该子任务执行为后果的步骤上，分配每代币的积分。我们基于四个代理基准进行评估：FrozenLake（稀疏网格导航）、HotpotQA（多跳QA，单一检索工具）、ScienceWorld（长视野具象科学）和DeepResearch（长研究，四工具，复合评分标准奖励）。训练期间发布的异构性诊断显示分解在哪些方面有效——收益与子任务异构性成比例，在高异质性任务如ScienceWorld（+11.5点，配对引导95% CI [+9.8， +13.3]）和FrozenLake（+9.8点、[+7.0， +12.8]）中最大，而在HotpotQA和DeepResearch上则处于噪声范围内，诊断预测恢复率较低。ScienceWorld在RLDS下的计算效率也高于标量GRPO（每步-10.9%墙时钟），只要长期部署能摊销固定的反射和分级开销。

HiRE: Hindsight Reward Editing for Policy Finetuning

招聘RE：政策精细化的事后诸葛亮奖励编辑

• Authors: Haoyi Niu, Zhengtao Han, Yufeng Ji, Zhongyu Li, Koushil Sreenath
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.27068
• Pdf link: https://arxiv.org/pdf/2609.27068
• Abstract Pre-trained robot policies always require finetuning to adapt to specific environments. Reinforcement Learning (RL) offers high performance potential because it improves action optimality rather than simply mimicking data. However, such potential depends heavily on reward quality. Sparse rewards lack process feedback, human-designed rewards are costly and biased, and semantic rewards from foundation representations are often not control-centric. We propose Hindsight Reward Editing (HiRE), a training-free framework to break this reward bottleneck. HiRE bridges the broad knowledge of foundation representation models with physical control awareness, by contrasting successful and failed trajectories in hindsight. It calibrates foundation representation models by identifying "trap states" that are predicted as high-rewarding states yet eventually result in failure, and vice versa. HiRE explicitly penalizes these traps while boosting rewards for critical successful states. This approach can be flexibly compatible with any foundation representations and RL algorithms. Experiments show that HiRE consistently outperforms other reward recipes by delivering dense, control-aware feedback that prevents value function collapse and reward hacking, thereby achieving superior sample efficiency, stable policy updates, and higher performance ceilings, e.g., at least 3x performance of the base policies. Qualitative results are at this https URL .
• 中文摘要 预训练的机器人策略总是需要微调以适应特定环境。强化学习（RL）具有高性能潜力，因为它提升了动作的最优性，而不仅仅是模仿数据。然而，这种潜力很大程度上依赖于奖励质量。稀疏的奖励缺乏过程反馈，人类设计的奖励成本高且有偏见，基础表征的语义奖励往往非以控制为中心。我们提出了一种无需训练的后见之明奖励编辑（HiRE）框架，以打破这一奖励瓶颈。HiRE通过对比成功与失败的事后诸葛亮轨迹，连接基础表征模型的广泛知识与物理控制意识。它通过识别被预测为高奖励状态但最终导致失败的“陷阱状态”来校准基础表征模型，反之亦然。HiRE明确惩罚这些陷阱，同时提升关键成功状态的奖励。该方法可灵活兼容任何基础表示和强化学习算法。实验显示，HiRE通过提供密集且控制意识的反馈，防止价值函数崩溃和奖励黑客行为，持续优于其他奖励配方，从而实现更优的样本效率、稳定的策略更新和更高的性能上限，例如基础策略的性能至少是其3倍。定性结果见此 https URL。

Banana Kick: Response-Informed Skill Evolution for Humanoid Soccer

香蕉踢：人形足球的反应导向技能进化

• Authors: Hao E. Zhang, Ruize Geng, Raihan Haque, Khalil Zbiss, Guanyang Luo, Hui-ping Wang, H. Eric Tseng, Ding Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.27269
• Pdf link: https://arxiv.org/pdf/2609.27269
• Abstract Humanoid kicking requires coordinated whole-body motion and precise contact, while a banana kick demands contact mechanics that generate ball spin and aerodynamic curvature. Motion imitation provides a reliable ordinary-kick prior, but reinforcement learning may improve shot speed and placement accuracy without changing the underlying kicking technique. Adapting this prior to a qualitatively different contact-rich skill can fail even when the reward is dense and optimization remains stable. The failure occurs when the task objective is locally flat over the current policy's responses. We term this condition first-order learning starvation. To address it, we propose response-informed skill evolution (RISE), a closed-loop objective-continuation method for policy adaptation. RISE ranks bounded objective changes using response sensitivity estimated from cached rollouts and accepts updates only when they produce verified response progress while preserving kicking reliability. Our analysis shows that rescaling a saturated spin reward cannot recover first-order sensitivity at zero spin, whereas adapting coupled contact responses can provide a learnable path to spin generation. We integrate RISE into a humanoid kicking pipeline under calibrated contact and Magnus-force aerodynamics, and sim-to-real transfer. Experiments show that RISE evolves the ordinary kick into a high-spin curved kick with 11.55 rad/s mean ball spin, improves the mean evaluation score by 19.8% over a learning-progress curriculum, and raises joint target attainment from 15.2% to 50.9%. Ablations and response diagnostics support the mechanism, while 30 motion-capture-recorded physical trials demonstrate consistent hardware transfer of the learned curved kick. Project website: this https URL
• 中文摘要 类人踢球需要协调的全身运动和精准的接触，而香蕉踢则需要产生球旋转和空气动力学曲率的接触力学。运动模仿提供了可靠的普通踢球先验，但强化学习可能在不改变踢球技术的情况下提高射门速度和命中精度。在适应质上不同且接触丰富的技能之前进行调整，即使奖励密集且优化保持稳定，也可能失败。当任务目标在当前策略响应上局部平稳时，失败发生。我们将此状况称为一阶学习饥饿。为此，我们提出了响应知情技能演化（RISE），一种闭环的目标-延续策略适应方法。RISE通过缓存推算的响应敏感度对有界目标变化进行排序，仅在更新产生验证响应进度且保持踢球可靠性时接受。我们的分析表明，在零旋转时，重新调整饱和旋转奖励无法恢复一阶灵敏度，而调整耦合接触响应则可提供可学习的旋转生成路径。我们将RISE集成到人形踢球管线中，基于校准接触和Magnus力空气动力学，以及模拟到真实的传输。实验显示，RISE将普通踢球演化为高旋转的弧球踢球，平均球旋转速度为11.55，学习进展课程中平均评估分数提升19.8%，并将联合目标达成率从15.2%提升至50.9%。消融和响应诊断支持该机制，30个动作捕捉录制的物理试验显示了学习曲线踢的硬件传输一致性。项目网站：此链接 URL

Hunyuan-A13B Technical Report

魂源-A13B技术报告

• Authors: Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, Dian Jiao, Dong Du, Dong Wang, Feng Zhang, Fengzong Lian, Guanghui Xu, Guanwei Zhang, Hai Wang, Haipeng Luo, Han Hu, Huilin Xu, Jiajia Wu, Jianchen Zhu, Jianfeng Yan, Jiaqi Zhu, Jihong Zhang, Jinbao Xue, Jun Xia, Junqiang Zheng, Kai Liu, Kai Zhang, Kai Zheng, Kejiao Li, Keyao Wang, Lan Jiang, Lixin Liu, Lulu Wu, Mengyuan Huang, Peijie Yu, Peiqi Wang, Qian Wang, Qianbiao Xiang, Qibin Liu, Qingfeng Sun, Richard Guo, Ruobing Xie, Saiyong Yang, Shaohua Chen, Shihui Hu, Shuai Li, Shuaipeng Li, Shuang Chen, Suncong Zheng, Tao Yang, Tian Zhang, Tinghao Yu, Weidong Han, Weijie Liu, Weijin Zhou, Weikang Wang, Wesleye Chen, Xiao Feng, Xiaoqin Ren, Xingwu Sun, Xiong Kuang, Xuemeng Huang, Xun Cao, Yanfeng Chen, Yang Du, Zhen Yang, Yangyu Tao, Yaping Deng, Yi Shen, Yigeng Hong, Yiqi Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.27284
• Pdf link: https://arxiv.org/pdf/2609.27284
• Abstract We present Hunyuan-A13B, an open-source large language model based on a Mixture-of-Experts architecture. It contains 80 billion total parameters but activates only 13 billion during inference, balancing model capability, computational efficiency, and deployment cost. The model is pretrained on a rigorously filtered 20T-token corpus with enhanced STEM data curation, improving factual reliability and reasoning ability. High-quality supervised fine-tuning and large-scale reinforcement learning further enhance its overall performance. Hunyuan-A13B also introduces a dual-mode Chain-of-Thought framework that adapts reasoning depth to task complexity: fast thinking for routine queries and slow thinking for complex, multi-step problems. Evaluations show competitive performance across mathematics, science, programming, general language understanding, and agent tasks, often approaching that of much larger models. Its high inference throughput makes it suitable for latency-sensitive applications. We release Hunyuan-A13B to support open research and practical LLM deployment.
• 中文摘要 我们介绍了Hunyuan-A13B，一个基于专家混合架构的开源大型语言模型。它包含800亿个总参数，但在推理过程中仅激活130亿个参数，平衡模型能力、计算效率和部署成本。该模型在严格过滤的20T令牌语料库上预训练，增强了STEM数据的管理，提高了事实可靠性和推理能力。高质量的监督微调和大规模强化学习进一步提升了其整体性能。Hunyuan-A13B还引入了双模式思维链框架，将推理深度调整以适应任务复杂性：快速思考用于常规查询，缓慢思考应对复杂多步问题。评估显示，数学、科学、编程、通用语言理解和代理任务等领域表现出竞争力，常常接近更大型模型的水平。其高推理吞吐量使其适合对延迟敏感的应用。我们发布Hunyuan-A13B，以支持开放研究和实用的LLM部署。

FairTest: Search-Based Fairness Testing for Multi-Agent Reinforcement Learning Systems

FairTest：多智能体强化学习系统的基于搜索的公平性测试

• Authors: Xiaotong Wang, Xuan Xie
• Subjects: Subjects: Software Engineering (cs.SE); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27309
• Pdf link: https://arxiv.org/pdf/2609.27309
• Abstract Multi-agent Reinforcement Learning (MARL) trains a team of agents that share one environment and learn their policies together. Training maximizes the team return, and a high return does not imply that the rewards are shared fairly among the agents in every episode. Testing is an established way to discover the failures of deep reinforcement learning, yet few methods address the fairness of MARL. In this work, we propose FairTest, a search-based testing approach that seeks the unfair executions of a MARL policy. The design combines search guidance with test prioritization. The guidance scores each candidate with three fitness functions. One measures the fairness of the runs already performed, another predicts the fairness from abstract states and fairness features, and the third reads the decision uncertainty from the policy. Crossover and mutation derive further candidates from the observed executions. The prioritization ranks the candidates by the predicted fairness and the decision uncertainty, so that the runs reach the candidates where failures are expected. FairTest is evaluated on three environments and two MARL algorithms, and four baselines are given the same budget. It detects the most fairness failures compared to three baselines with statistical significance and large effect sizes. The failure count exceeds that of the strongest baseline by 221% on average and coverage improves by an average of 23%.
• 中文摘要 多智能体强化学习（MARL）训练一组共享同一环境的智能体，并共同学习各自的策略。训练最大化团队回报，高回报并不意味着每次事件中奖励在智能体之间公平分配。测试是发现深度强化学习失败的成熟方法，但很少有方法能解决MARL的公平性。本研究提出FairTest，一种基于搜索的测试方法，旨在寻找MARL策略的不公平执行。该设计结合了搜索指导与测试优先级。指导为每个候选人评分三个适应度函数。一个测量已执行运行的公平性，另一个预测抽象状态和公平特征的公平性，第三个从策略中读取决策不确定性。交叉和突变从观察到的执行中推导出更多候选。优先级排序通过预测公平性和决策不确定性对候选人进行排名，以确保运行能到达预期失败的候选人。FairTest在三种环境和两种MARL算法上评估，四个基线预算相同。它检测到的公平性失败最多，相较于三个具有统计显著性和大效应量的基线。失败率平均比最强基线高出221%，覆盖率平均提升23%。

Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning

将安全转化为能力：通过安全过滤强化学习实现最小可利用的机器人政策

• Authors: Ruihan Wu, Rui Yang, Donggeon David Oh, Duy Nguyen, Haimin Hu
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.27312
• Pdf link: https://arxiv.org/pdf/2609.27312
• Abstract Robots deployed for competitive tasks must outmaneuver their opponents without sacrificing safety. Existing approaches, including safe reinforcement learning (RL), train a single policy to achieve task success and avoid failures simultaneously. This coupling can complicate training and leave the learned policy exploitable by deliberate attacks. We propose Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning. We formulate competitive interactions as safety-critical Markov games and prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers. S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training, and retains the same filter at deployment. In simulated touchdown games, S2C outperforms eight safe RL baselines, achieving the highest win rate and Elo rating, and the lowest exploitability. Hardware stress tests against a human opponent confirm S2C's competence.
• 中文摘要 用于竞争任务的机器人必须在不牺牲安全的情况下巧妙地超越对手。现有方法，包括安全强化学习（RL），训练单一策略以实现任务成功并避免失败。这种耦合可能使训练复杂化，并使学习的策略容易被有意攻击利用。我们提出了“安全到能力”（S2C），这是一个两阶段的强化学习框架，将安全综合与竞争任务学习区分开来。我们将竞争互动表述为安全关键的马尔可夫博弈，并证明当所有玩家都承诺安全机动时，完美过滤能保持策略的不可被利用性。S2C通过对抗性强化学习一个强健的安全过滤器，在任务策略训练时将其嵌入环境中，并在部署时保留相同的过滤器。在模拟达阵游戏中，S2C优于八个安全强化基础，获得最高的胜率和Elo评级，以及最低的可利用性。对人类对手进行硬件压力测试确认了S2C的能力。

Evolving Inspectable O-RAN Slicing xApps with LLMs

进化的可检验O-RAN切片x应用与LLMs。

• Authors: Faezeh Dehghan Tarzjani, Bhaskar Krishnamachari
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27337
• Pdf link: https://arxiv.org/pdf/2609.27337
• Abstract Open RAN (O-RAN) slicing xApps must adapt resource allocations to changing channel conditions and traffic demands while meeting service-level agreements (SLAs). Deep reinforcement learning can produce adaptive policies, but their allocation rules remain encoded in neural-network parameters. Our goal is to retain this adaptability while making the controller's decision logic directly inspectable and editable by operators. We use a large language model (LLM) to evolve slicing controllers as compact Python programs whose decision logic remains readable and editable after optimization. The LLM proposes and revises candidates offline, while a calibrated simulator scores them, and the selected decision module runs unchanged in the O-RAN control path. On the NSF POWDER 5G testbed, the evolved controller releases resources from a guaranteed slice whose throughput target becomes unattainable under a sustained channel fade, improving best-effort throughput from 158.2 to 228.6 Mbps, a 44.5% gain over the best static allocation. Since the controllers are readable source code, their behavior can be predicted from their equations, defects can be diagnosed by reading the code, and calibration errors can be corrected with one-line edits, reducing SLA misses from 79.9% to 2.2% in one case and more than doubling fitness in another. In a four-slice trace-driven simulation calibrated to the same testbed, evolutionary search achieves higher average evaluation scores than independent prompting at a matched proposal budget, with mean normalized gains on held-out traces of 16.3% for prompting alone, 32.1% for evolution from scratch, and 51.0% for evolution from a starting program.
• 中文摘要 开放 RAN（O-RAN）切片 xApps 必须根据信道状况和流量需求的变化调整资源分配，同时满足服务水平协议（SLA）。深度强化学习可以生成自适应策略，但其分配规则仍编码在神经网络参数中。我们的目标是在保持这种适应性同时，使操作员能够直接检查和编辑控制器的决策逻辑。我们使用大型语言模型（LLM）将切片控制器演化为紧凑的 Python 程序，其决策逻辑在优化后依然可读且可编辑。LLM 离线提出并修订候选方案，而校准的模拟器对其进行评分，所选决策模块在 O-RAN 控制路径中运行不变。在NSF POWDER 5G测试平台上，进化后的控制器从保证切片释放资源，该片的吞吐量目标在持续的信道衰落下变得无法实现，尽力而为吞吐量从158.2提升到228.6 Mbps，比最佳静态分配提升了44.5%。由于控制器是可读的源代码，可以通过方程预测其行为，通过读取代码诊断缺陷，校准错误可以通过一行编辑修正，将SLA未命中率从79.9%降至2.2%，在另一些情况下则使适度度翻倍以上。在校准同一测试平台的四片轨迹驱动模拟中，进化搜索在匹配提案预算下比独立提示获得更高的平均评估分数，单独提示时对未保留痕迹的归一化提升为16.3%，从零开始进化32.1%，从起始程序进化51.0%。

EvoAudio: Recursive Self-Improvement for Audio Understanding

EvoAudio：递归自我提升以提升音频理解

• Authors: Yuxiang Wang, Shengbo Cai, Yingda Shen, Ming-Hao Hsu, Qinke Ni, Liqiang Zhang, Teddy Sun, Steve Yevs, Zhizheng Wu
• Subjects: Subjects: Sound (cs.SD); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27389
• Pdf link: https://arxiv.org/pdf/2609.27389
• Abstract Audio language models understand what is said far better than how it sounds. Closing this gap takes more than data. Detailed acoustic annotation is costly, labels from stronger models inherit their errors and limits, and fixed data cannot adapt as the learner improves. We therefore propose EvoAudio, a recursive self-improvement system for audio understanding. To our knowledge, it is the first to evolve the model, waveforms, questions, and difficulty in one closed loop. EvoAudio uses the current model's performance to set the focus and difficulty of the next training data. A library of audio tools then constructs questions whose answers follow from how the audio was made, providing verifiable supervision without new human annotation. Reinforcement learning updates the model, and validation decides whether it enters the next evolution round. Across 13 rounds, EvoAudio improves five models with different audio encoders and language backbones on MMSU, MMAU-Pro, and MMAR. It achieves the highest average for every backbone, raising overall performance by up to 6.3 points. The improvement unfolds over successive rounds, with each stronger model starting the next round.
• 中文摘要 音频语言模型对所说内容的理解远远超过其发音。缩小这一差距不仅需要数据。详细的声学注释成本高昂，来自更强模型的标签会继承其误差和限制，且固定数据无法随着学习者的提升而适应。因此，我们提出了EvoAudio，一种递归的音频理解自我提升系统。据我们所知，它是首个在一个闭环中进化模型、波形、问题和难度的系统。EvoAudio利用当前模型的性能来设定下一个训练数据的焦点和难度。随后构建一系列音频工具，其答案源自音频制作过程，提供可验证的监督，无需新的人工注释。强化学习更新模型，验证决定是否进入下一轮演变。在13轮中，EvoAudio在MMSU、MMAU-Pro和MMAR上改进了五款拥有不同音频编码器和语言骨干的型号。它在每个骨干中取得了最高平均分，整体性能提升了最多6.3分。改进在后续轮次中逐步展开，每台更强的机型开始下轮。

Quantum Reinforcement Learning for Cost and Delay Tradeoffs in Quantum Cloud Orchestration

量子云编排中成本与延迟权衡的量子强化学习

• Authors: An N. H. Phan, Dang Van Huynh, Muhammad Usman, Hoa T. Nguyen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Emerging Technologies (cs.ET); Quantum Physics (quant-ph)
• Arxiv link: https://arxiv.org/abs/2609.27446
• Pdf link: https://arxiv.org/pdf/2609.27446
• Abstract Quantum cloud computing, delivered through the quantum-as-a-service (QaaS) model, provides access to quantum computing resources. However, applying uniform time-based pricing across fundamentally heterogeneous quantum resources significantly complicates task orchestration, particularly when addressing the tradeoff between execution costs and system performance. While heuristic methods rely on predefined scheduling rules, classical deep reinforcement learning (DRL) models may require more trainable parameters in this setting. Motivated by the potential of parameterised quantum circuits (PQCs) as compact function approximators, we propose QRLQ, a cost-delay-aware quantum cloud scheduling framework integrating PQCs with a dueling double deep Q-network (D3QN) to dynamically account for both cost and delay. Our simulation results show that QRLQ achieves lower mean cost and delay than the heuristic baselines, achieving a 5-11% lower mean cost relative to availability-based and rotation-based heuristics and reducing mean delay by 17% and 82% relative to the strongest and weakest heuristic baselines, respectively, while retaining execution fidelity within 2% of a fidelity-greedy policy. Compared with the classical DRL baseline, QRLQ achieves comparable scheduling performance while using 72% fewer trainable parameters. This work explores the feasibility of using QRL for task orchestration in quantum cloud environments and demonstrates its potential for cost-delay-aware quantum resource management.
• 中文摘要 量子云计算通过量子即服务（QaaS）模型实现，提供了对量子计算资源的访问。然而，在根本异构的量子资源中应用统一的时间定价极大地复杂化了任务编排，尤其是在处理执行成本与系统性能权衡时。虽然启发式方法依赖预定义调度规则，但经典深度强化学习（DRL）模型在该环境中可能需要更多可训练参数。受参数化量子电路（PQC）作为紧凑函数近似器的潜力激励，我们提出了QRLQ，这是一个成本延迟感知型量子云调度框架，将PQC与双重深度Q网络（D3QN）集成，动态考虑成本和延迟。我们的模拟结果表明，QRLQ的平均成本和延迟低于启发式基线，相较于可用性和轮换启发式，平均成本降低了5-11%，平均延迟分别降低了17%和82%，同时执行忠实度保持在与忠实贪婪策略相差2%以内。与经典DRL基线相比，QRLQ在使用72%的可训练参数的情况下实现了相当的调度性能。本研究探讨了在量子云环境中使用QRL进行任务编排的可行性，并展示了其在成本延迟感知量子资源管理方面的潜力。

BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models

BEE：带有视觉-语言-行动模型的干预-自适应现实世界强化学习

• Authors: Weihui Zhao, Xiaohan Yan, Zunian Wan, Xuan Du, Zhaozhan Chi, Jianbo Mao, Ruipu Wu, Rushuai Yang, Houlin Li, Shukai Yang, Jing Wu, Yuxiang Yan, Yongcheng Liu, Chuankang Li, Guanghui Ren, Wei Shan, Maoqing Yao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.27450
• Pdf link: https://arxiv.org/pdf/2609.27450
• Abstract Vision-language-action (VLA) models handle long-horizon manipulation, yet success hinges on a few precision-critical phases where millimeter-scale errors undo all prior progress. Online reinforcement learning (RL) can optimize exactly these actions, but free exploration is far too costly on real robots, which makes human corrections indispensable. However, existing online RL methods for VLAs either cannot incorporate such corrections or fold them into undifferentiated supervision. Yet human corrections are not uniformly noisy but reliable along some action dimensions and variable along others. Building on this, we introduce BEE, an intervention-adaptive framework for real-world RL on a frozen VLA that lets the policy go BEyond Expert imitation. We formulate human corrections not as actions to reproduce but as evidence about a constraint: a Correction Model predicts how a human would correct a given VLA proposal and how consistent the correction is along each action dimension. This predicted consistency sets the per-dimension tightness of a constraint on policy optimization. Where corrections are consistent the policy stays close to the human, and where they vary, the constraint relaxes. We evaluate BEE on three real-world manipulation tasks and one LIBERO-Pro simulation task at a matched online-data budget. BEE attains the highest success rate on every task, 91.2% on average against 57.5% for RLT and 42.1% for DSRL, and the lowest human intervention rate on all real-world tasks.
• 中文摘要 视觉-语言-动作（VLA）模型处理长视野操作，但成功依赖于少数精度关键阶段，毫米级误差会推翻所有先前进展。在线强化学习（RL）可以优化这些动作，但自由探索对真实机器人成本过高，使得人工修正不可或缺。然而，现有的在线强化学习方法无法整合此类修正，或将其折叠进无差异化监督中。然而，人类修正并非噪声均匀，但在某些动作维度上可靠，而在其他维度则可变。基于此，我们引入BEE，一种针对冻结VLA的现实世界强化学习干预自适应框架，使策略超越专家模仿。我们将人类修正表述为约束的证据，而非作为复制的行为：修正模型预测人类如何纠正给定的VLA提案，以及纠正在各行动维度上的一致性。这种预测一致性设定了对策略优化约束的每维约束紧密度。在修正一致时，策略保持接近人类的限制;在调整变化时，约束会放松。我们在匹配的在线数据预算下，评估了三个真实世界操作任务和一个LIBERO-Pro模拟任务。BEE在每个任务中均获得最高成功率，平均为91.2%，而RLT为57.5%，DSRL为42.1%，在所有真实任务中人工干预率最低。

MDRC: A Deployable State-Recovery Defense for Traffic Signal Control under Sensor Corruption

MDRC：一种可部署的状态恢复防御，用于传感器损坏下的交通信号控制

• Authors: Mingyuan Li, Chunyu Liu, Xiao Liu, Yanna Jiang, Guangsheng Yu, Xu Wang, Wei Ni, Ren Ping Liu
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.27528
• Pdf link: https://arxiv.org/pdf/2609.27528
• Abstract Traffic Signal Control (TSC) is a safety-critical cyber-physical system that relies on real-time sensing. Corrupted observations caused by adversarial perturbations or sensor failures can propagate from the sensing layer into the controller and degrade traffic efficiency. Existing robust Reinforcement Learning (RL)-based TSC methods often suffer from limited cross-city generalization, high inference latency, and weak recovery under partial observability. We present MDRC (Meta-Diffusion-based framework for Resilient traffic signal Control against adversarial attacks and sensor failures), a post-detection state-recovery defense inserted between sensing and control. MDRC reconstructs trustworthy traffic states before they are consumed by the controller. It combines Denoising Diffusion Implicit Models (DDIM) for efficient state recovery with Reptile meta-learning for a transferable initialization across cities. We provide an optimization-based view of the DDIM recovery dynamics and establish a recovery-error bound that separates score approximation, numerical discretization, and initialization mismatch. Across seven real-world-derived CityFlow benchmarks, MDRC reduces Average Travel Time by 6.77% under stochastic and policy-aware attacks and by 12.75% under structured sensor loss, while improving state-recovery fidelity. We further evaluate 3,600 seconds of real roadside measurements with 50% of detector channels disabled and integrate MDRC into a hardware-in-the-loop traffic-signal stack. Over a 9.16-hour run with 32,389 sensing/control cycles, the system achieves 99.79% decision availability, produces no out-of-plan recommendations, and requires approximately 38 ms of component-wise processing per one-second control interval.
• 中文摘要 交通信号控制（TSC）是一种依赖实时传感的安全关键网络物理系统。由对抗扰动或传感器故障导致的损坏观测数据可能从传感层传播到控制器，降低交通效率。现有基于强化学习（RL）的强化增强控制方法常常存在跨城泛化有限、推理延迟高以及部分可观测性下恢复较弱的问题。我们介绍MDRC（基于元扩散的基于弹性交通信号控制的对抗攻击和传感器失效框架），这是一种插入于传感与控制之间的检测后状态恢复防御。MDRC在交通状态被控制器消费前重建可信的交通状态。它结合了去噪扩散隐式模型（DDIM）实现高效状态恢复，并结合爬虫元学习实现跨城市可转移初始化。我们提供了基于优化的DDIM恢复动态视图，并建立了一个恢复-错误界限，将分数近似、数值离散化和初始化不匹配区分开来。在七个基于真实世界来源的CityFlow基准测试中，MDRC在随机和策略感知攻击下平均旅行时间减少了6.77%，在结构化传感器丢失下减少12.75%，同时提升状态恢复的保真度。我们还进一步评估了3600秒的真实路边测量，且50%的探测器通道被禁用，并将MDRC集成到硬件在环交通信号堆栈中。在9.16小时运行、32,389次感测/控制周期内，系统实现了99.79%的决策可用性，没有超出计划的建议，每秒控制间隔约需38毫秒的组件处理。

EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management

EBRL：多粒度资源管理下的异步具身强化学习

• Authors: Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao, Han Xiao, Xin Ding, Mingzhe Huang, Xin He, Lu Shi, Hao Wu, Haipeng Dai, Guihai Chen, Yunxin Liu, Ting Cao
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.27547
• Pdf link: https://arxiv.org/pdf/2609.27547
• Abstract Embodied reinforcement learning (RL) improves model capabilities with a pipeline of environment simulation, action generation, and model updates. These stages show heterogeneous CPU and GPU demands, making efficient resource utilization difficult. Recent systems overlap rollout (simulation and generation) with training for efficiency, but exclusive GPU allocation and synchronized barrier in rollout still leave substantial hardware resource waste. In this paper, we present EBRL, an asynchronous embodied RL training system with two core techniques. The asynchronous pipelined scheduler overlaps rollout and training, pipelines simulation and generation across environment groups, and carries out each environment independently, eliminating synchronization stalls. The fine-grained resource manager pools CPU cores and GPU streaming multiprocessors, and uses stage profiles and runtime feedback to adjust resource quotas and batch sizes to meet the shifting demands among stages. We implement EBRL on RLinf and evaluate it with four embodied policies and four simulation benchmarks across heterogeneous GPU testbeds. Experiments show that EBRL achieves 1.30-3.47 times the end-to-end rollout throughput and 2.5 times of training convergency compared to the SOTA embodied RL systems.
• 中文摘要 具身强化学习（RL）通过环境仿真、动作生成和模型更新的流水线提升模型能力。这些阶段显示出CPU和GPU的异构需求，使资源利用效率变得困难。近期系统将部署（模拟和生成）与提升效率的训练重叠，但专属GPU分配和同步部署障碍仍留下大量硬件资源浪费。本文介绍了EBRL，一种异步具象RL训练系统，拥有两种核心技术。异步流水线调度器在各环境组间叠加滚动与训练、流水线模拟与生成，并独立执行每个环境，消除同步停滞。细粒度资源管理器将CPU核心和GPU流式多处理器池化，利用阶段配置文件和运行时反馈调整资源配额和批处理大小，以满足各阶段变化的需求。我们在RLinf上实现EBRL，并结合四个内涵策略和四个跨异构GPU测试平台的模拟基准进行评估。实验显示，EBRL实现端到端推广吞吐量的1.30-3.47倍，训练收敛度是SOTA内含的强化学习系统的2.5倍。

DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignment

DCRL：通过策略-奖励流形对齐实现解耦与耦合强化学习

• Authors: Henan Sun, Zehua Li, Haitao Hu, Qifan Zhang, Jianfeng Zhang, Nuo Chen, Jia Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.27572
• Pdf link: https://arxiv.org/pdf/2609.27572
• Abstract Reinforcement learning (RL) has emerged as a key paradigm for improving the reasoning capabilities of large language models (LLMs). However, existing reward systems, such as rule-based and reward-model-based, often exhibit issues such as unstable optimization and reward hacking. In this work, we revisit the general reasoning of LLMs from a geometric perspective, conceptualizing it as a coupled manifold composed of three interdependent sub-manifolds: logical deduction, evaluation, and representation. Based on this perspective, response generation in RL can be interpreted as a decoupling process from the evaluation manifold, while reward estimation corresponds to a decoupling process from the logical deduction manifold. The limitations of rule-based and reward-model RL systems can be geometrically interpreted as the mismatch of policy-reward manifolds during RL process. To address the aforementioned misalignment, we propose Decoupling and Coupling Reinforcement Learning (DCRL) framework, which incorporates two key components: (1) a syllogistic logic-based prompt evolution mechanism that dynamically refines reward rubrics to enhance the expressiveness of the reward manifold; and (2) a policy-reward re-coupling mechanism that jointly updates the reward and policy models, ensuring consistent evaluation and mitigating manifold mismatch during training. Theoretical analysis and extensive experiments across multiple reasoning domains demonstrate that DCRL consistently outperforms both rule-based and reward-model baselines. Notably, a Qwen3-4B model trained under DCRL surpasses a Qwen3-32B baseline and approaches the performance of a Qwen3-235B model, highlighting superior effectiveness and generalization in RL.
• 中文摘要 强化学习（RL）已成为提升大型语言模型（LLM）推理能力的关键范式。然而，现有的奖励系统，如基于规则和奖励模型的，常常存在优化不稳定和奖励黑客等问题。在本研究中，我们从几何视角重新审视LLMs的一般推理，将其概念化为由三个相互依赖子流形组成的耦合流形：逻辑演绎、评估和表示。基于这一视角，强化学习中的反应生成可以被解释为与评估流形的解耦过程，而奖励估计则对应于与逻辑演绎流形的解耦过程。基于规则和奖励模型的强化学习系统的局限性可以几何地解释为强化学习过程中策略-奖励流形的不匹配。为解决上述错位，我们提出了解耦耦合强化学习（DCRL）框架，包含两个关键组成部分：（1）基于三段逻辑的提示演化机制，动态优化奖励评分标准以增强奖励流形的表现力;以及（2）策略-奖励再耦合机制，联合更新奖励和策略模型，确保评估一致并缓解训练过程中的流形错配。理论分析和跨多个推理领域的大量实验表明，DCRL始终优于基于规则和奖励模型的基线。值得注意的是，在DCRL下训练的Qwen3-4B模型超越了Qwen3-32B基线，性能接近Qwen3-235B模型，凸显了RL的卓越效能和泛化性。

A DRL-Driven Optimization of RAN Slice Resource Partitioning for V2X SLA Compliance in 5G Networks

基于DRL驱动的RAN切片资源分区优化，以实现5G网络中V2X SLA合规性

• Authors: M. Martínez, I. de-la-Bandera, D. E. García, P. Vera, S. Fortes, M. L. Luque, A. Mendo, J. Ramiro, R. Barco
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.27659
• Pdf link: https://arxiv.org/pdf/2609.27659
• Abstract Vehicle-to-Everything (V2X) communications impose very demanding requirements in terms of latency and reliability, which must be met in scenarios where multiple services with diverse performance targets coexist. In such scenarios, traffic-intensive services compete for limited radio resources, complicating the fulfillment of V2X service demands. Within this context, Network Slicing (NS) emerges as a key factor that enables the creation of multiple slices and the allocation of resources among them to satisfy heterogeneous service requirements. In particular, this work addresses the Radio Access Network (RAN) slicing problem from the perspective of Physical Resource Block (PRB) partitioning under high traffic demand conditions. To this end, a reinforcement learning approach based on Proximal Policy Optimization (PPO) is proposed to determine PRB allocations that satisfy the strict latency and reliability requirements of V2X services, while improving resource utilization efficiency and minimizing performance degradation of enhanced Mobile BroadBand (eMBB) services. The proposed solution is evaluated through simulation-based experiments under various traffic loads and different V2X service requirements, demonstrating its ability to adapt resource partitioning to network conditions and service demands.
• 中文摘要 车辆到所有（V2X）通信在延迟和可靠性方面要求极高，必须满足多项性能目标服务共存的场景。在这种情况量密集型服务争夺有限的无线资源，增加了V2X服务需求的满足。在此背景下，网络切片（NS）成为实现多片创建及资源分配以满足异构服务需求的关键因素。特别地，本工作从物理资源块（PRB）划分的角度，解决高流量需求下的无线接入网（RAN）切片问题。为此，提出了基于近端策略优化（PPO）的强化学习方法，以确定满足V2X服务严格延迟和可靠性要求的PRB分配，同时提高资源利用效率，并最小化增强型移动宽带（eMBB）服务的性能下降。该方案通过基于仿真的实验在不同流量负载和不同V2X服务需求下进行评估，展示了其适应网络条件和服务需求的能力。

Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization

具有风险敏感性和批判者一致性正则化的强健对抗强化学习

• Authors: Jiaxi Wu, Tiantian Zhang, Yuxing Wang, Yongzhe Chang, Xueqian Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27667
• Pdf link: https://arxiv.org/pdf/2609.27667
• Abstract Reinforcement learning (RL) achieves strong performance in sequential decision-making but remains brittle under dynamic uncertainty and distributional shifts. Robust Adversarial Reinforcement Learning (RARL) improves robustness via worst-case perturbations, but existing approaches frequently suffer from unstable optimization and degraded value estimation. In particular, overly aggressive adversaries can drive the agent toward uninformative failure states, while adversarial perturbations amplify disagreement between double critics and introduce biased value targets. We propose a unified framework, RACER (Risk-sensitive robust Adversarial critic ConsistEncy-regularized Reinforcement learning), that revisits adversarial RL from a risk-sensitive perspective. First, we introduce a state-dependent adversarial objective that adaptively regulates perturbation strength, suppressing harmful disturbances while preserving informative exploration. Second, we propose critic consistency regularization to reduce disagreement between Q-value estimators and stabilize learning. Comprehensive experiments on challenging continuous control benchmarks demonstrate that RACER consistently improves performance, robustness, and training stability over strong robust RL baselines.
• 中文摘要 强化学习（RL）在顺序决策中表现优异，但在动态不确定性和分布变化下仍显脆弱。强化对抗强化学习（RARL）通过最坏情况扰动提升鲁棒性，但现有方法常常存在优化不稳定和价值估计退化的问题。特别是，过于激进的对抗者可能使智能体陷入无信息的失败状态，而对抗性扰动则加剧双重批判者之间的分歧并引入有偏的价值目标。我们提出了一个统一框架RACER（风险敏感强健对抗性批判一致性正则化强化学习），从风险敏感视角重新审视对抗强化学习。首先，我们引入一个状态依赖的对抗目标，自适应调节扰动强度，抑制有害扰动，同时保持信息探索。其次，我们提出批判性一致性正则化，以减少Q值估计量之间的不一致并稳定学习。对挑战性连续控制基准的综合实验表明，RACER在强强的强韧强化学习基线上持续提升性能、鲁棒性和训练稳定性。

SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving

SkillGym：将人类技能内化到大型语言模型中以实现现实世界问题解决

• Authors: Zhilong Ge, Yuting Shao, Yutao Yang, Yuxuan Cai, Jie Zhou, Kai Chen, Bo Zhang, Qin Chen, Liang He
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.27717
• Pdf link: https://arxiv.org/pdf/2609.27717
• Abstract Human-written agent skills encode rich workflows for real-world problem solving, but are typically used as external inference-time instructions rather than internalized as reusable model capabilities. We introduce \texttt{SkillGym}, a framework that transforms these skills into executable, verifiable training environments for large language model agents. Its skill-to-task pipeline instantiates concrete tasks, verifies outcomes with code-based checkers, and assesses empirical skill dependence through contrastive executions. We construct and release 2,756 environments across 12 categories and collect 8,364 successful trajectories from multiple models and harnesses, averaging 49 tool calls and over 60k logged text tokens. These resources support supervised fine-tuning on verified workflows and reinforcement learning with outcome-based rewards. Under Claude Code, supervised fine-tuning improves Qwen3.5-35B-A3B by 199 Elo on GDPval-AA v2, 19.10 percentage points on Terminal-Bench 2.1, and 28.13 and 12.38 points on SkillsBench v1.1 with and without skills, respectively. Our 35B \texttt{SkillGym-Agent} reaches 51.47\% on skill-assisted SkillsBench, exceeding reported scores for Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro. Without skills, it also surpasses skill-assisted bases under Codex and Claude Code, suggesting reusable procedural competence.
• 中文摘要 人写代理技能编码丰富的工作流以实现现实问题解决，但通常作为外部推理时间指令使用，而非内化为可重用的模型能力。我们介绍了 \texttt{SkillGym}，这是一个将这些技能转化为可执行、可验证的大型语言模型代理训练环境的框架。其技能到任务流程实现具体任务，通过基于代码的检查器验证结果，并通过对比执行评估实证技能依赖性。我们构建并发布了12个类别的2,756个环境，收集了来自多个模型和工具束的8,364条成功轨迹，平均49个工具调用和超过6万个文本记录。这些资源支持对已验证工作流的监督微调和基于结果的奖励强化学习。在Claude Code下，监督微调使Qwen3.5-35B-A3B在GDPval-AA v2上提升199 Elo，Terminal-Bench 2.1提升19.10个百分点，带技能和无技能的SkillsBench v1.1分别提升28.13点和12.38个百分点。我们的35B \texttt{SkillGym-Agent}在技能辅助SkillsBench上达到51.47%，超过了Claude Sonnet 4.6、GPT-5.4 Mini和DeepSeek V4 Pro的报告分数。在无技能的情况下，它也超过了Codex和Claude Code下的技能辅助基础，表明具备可重复使用的程序能力。

Limiting-Kernel Q($λ$): Bridging Short and Long Horizons

极限核 Q（$λ$）：连接短视野与长视野

• Authors: Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.27741
• Pdf link: https://arxiv.org/pdf/2609.27741
• Abstract In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance. The widely adopted family of approximations relying on $n$-step truncation yields computationally efficient value estimators but is inherently limited to a short evaluation horizon. In contrast, methods that exploit the global structure of the transition dynamics can accelerate policy evaluation, but their memory and computational requirements often limit scalability to large or continuous state spaces. To reconcile these limitations, we introduce Limiting-Kernel Q($\lambda$) (LKQL), an off-policy value estimator that combines $n$-step truncation with a long-horizon approximation based on the limiting kernel (LK). LKQL has the same order of complexity as $n$-step estimators and integrates directly into both on- and off-policy actor-critic algorithms. We prove that, under aperiodicity and in the near-on-policy regime, the operator underlying LKQL improves the policy evaluation convergence rate over its truncated counterpart for sufficiently large $n$, and that LKQL itself converges almost surely to the optimal values in finite Markov decision processes (MDPs) under a fixed behavior policy. On the MuJoCo continuous-control benchmark, we show that LKQL improves over $n$-step baselines in most settings, particularly on long-horizon tasks.
• 中文摘要 在基于价值的强化学习中，提高策略评估的准确性已被证明能提升下游策略优化性能。广泛采用的依赖$n$步长截断的近似家族，虽然计算效率高，但本质上受限于较短的评估视野。相比之下，利用转移动态全局结构的方法可以加速策略评估，但其内存和计算需求通常限制了对大或连续状态空间的扩展性。为调和这些局限，我们引入了极限核Q（$\lambda$）（LKQL），这是一种非策略值估计器，结合了$n$步长截断与基于极限核（LK）的长视野近似。LKQL的复杂度与$n步估计器同级，并可直接集成到开策和非策略的行为者-批判算法中。我们证明，在非周期性和近策略模式下，LKQL背后的算符在足够大$n$时，比截断对应的算符提升策略评估收敛率，且在固定行为策略下，LKQL几乎必然收敛到有限马尔可夫决策过程（MDP）的最优值。在MuJoCo连续控制基准测试中，我们表明LKQL在大多数环境中，尤其是长视野任务中，在$n$基准线上有所改善。

Satisfaction Is Not Explanation: Auditing Vacuity and Training Influence in Temporal-Logic-Guided Reinforcement Learning

满意不是解释：审计空缺与训练对时间逻辑引导强化学习的影响

• Authors: Lorenzo Bacchiani
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2609.27743
• Pdf link: https://arxiv.org/pdf/2609.27743
• Abstract A reinforcement learning policy that satisfies its temporal-logic specification has passed a test, not an assurance argument. The clause that matters to a reviewer may never have mattered to training: it may have been avoided entirely, forced by the environment regardless of what the policy learned, or redundant next to the ordinary task reward. Satisfaction probability and task return cannot tell any of this apart. This paper introduces an audit layer that can. It measures whether a specification clause was actually exercised, whether that role was forced or chosen, and whether the obvious way to test causation, weakening the clause and retraining, is even valid. Often it is not: we prove that comparable weaker/stronger training objectives can share perfect optima under standard acceptance-derived rewards, show related ablation hazards across a large corpus of published specifications, and then show that a properly designed intervention detects the effect it should. Across standard reinforcement learning benchmarks and published external artifacts, the audit layer separates six regimes that a single satisfaction number collapses into one. A policy that satisfies its specification has answered whether. This paper asks why.
• 中文摘要 满足其时序逻辑规范的强化学习策略通过了测试，而非保证论证。对审查者来说重要的条款可能从未对训练产生影响：它可能被完全避免，被环境强制，无论策略学到了什么，或者在普通任务奖励旁边是冗余的。满意概率和任务返回无法区分这些。本文引入了一个审计层，可以区分。它衡量规范条款是否实际被执行，该角色是被迫还是选择的，以及测试因果关系的明显方法——削弱条款和再训练——是否有效。通常并非如此：我们证明相似的弱/强训练目标在标准接受导引奖励下共享完美最优，展示大量已发布规范中的相关消融风险，然后证明合理设计的干预检测应有效果。在标准强化学习基准测试和已发布的外部产物中，审计层将六个状态区分开来，单个满意度数可合并为一个。满足其规范的策略已回答是否。本文探讨了为什么。

Categorical Internalisation of Environmental Groupoids for Generalisable POMDP Solving

环境群类的分类内化以实现可推广的POMDP求解

• Authors: Ben Opperman, Eduardo Alonso, Esther Mondragón
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.27745
• Pdf link: https://arxiv.org/pdf/2609.27745
• Abstract This paper advocates category theory as a practical framework for structuring and improving rein- forcement learning in high-dimensional, partially observable environments. We model symmetries between environmental states by partitioning the state space into equivalence classes induced by sym- metry orbits, and organise each such class as a groupoid with a designated canonical representative. This allows the agent to share what it learns across many similar environmental states simultaneously, rather than treating every orientation or position as an entirely new problem. Learning is thus carried out on a symmetry-reduced state space with each orbit represented once, preserving structure while eliminating redundancy and improving sample efficiency. We implement this framework within standard reinforcement learning pipelines and evaluate two different approaches on partially observable benchmarks, demonstrating that orbit-based partitioning yields consistent performance improvements in environments exhibiting latent symmetry. Beyond these empirical results, our approach illustrates how categorical structure provides a principled bridge between abstract reinforcement learning formulations and their computational application, thereby establishing a pathway toward more structured and scalable learning systems.
• 中文摘要 本文倡导范畴论作为在高维、部分可观测环境中结构化和改进强化学习的实用框架。我们通过将状态空间划分为由对称轨道诱导的等价类，并将每个类组织为带有指定典范代表的群类，来建模环境状态之间的对称性。这使得智能体能够同时在多个相似环境状态中分享所学，而不必将每个方向或位置视为全新问题。因此，学习是在对称约化状态空间中进行，每个轨道都表示一次，保持结构，同时消除冗余并提高样本效率。我们将该框架纳入标准强化学习流程，并在部分可观测基准测试上评估两种不同方法，证明基于轨道的划分在表现出潜在对称性的环境中能持续带来性能提升。除了这些实证结果，我们的方法还展示了范畴结构如何在抽象强化学习表述与其计算应用之间建立原则性桥梁，从而建立通往更结构化和可扩展学习系统的路径。

From Reasoning Strings to Partial Orders: Verifier-Certified Rule Transport through Quotient Policy Optimization

从推理字符串到部分命令：通过商策略优化实现验证者认证的规则传输

• Authors: Bang Xie, Hao Liu, Zhiyuan Peng, Xin Yin, Chenhao Ying, Yuan Luo, Senjian Zhang, Wei Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27833
• Pdf link: https://arxiv.org/pdf/2609.27833
• Abstract Many computations admit several valid execution orders because independent subgoals or disjoint state updates can commute. Reinforcement learning with verifiable rewards usually treats each successful trace as a separate token sequence, so serialization choices can be mistaken for logical dependencies. We introduce Verifier-Certified Rule Transport (VCRT), which replays adjacent operation pairs with native verifiers. Pairs whose two orders are accepted and reach the same canonical state provide commutation certificates; rejected or state-changing reversals provide anti-diamonds. VCRT uses anti-diamonds to preserve genuine prerequisites and assigns policy credit to the total probability mass of each certified orbit. It also constrains post-swap consistency, source retention, and policy drift. We evaluate leave-one-environment-out transfer across ProofWriter, CLRS, and Lean through a shared anonymized relation-graph interface. All training and checkpoint decisions are frozen before held-out evaluation, which uses one greedy trajectory per item without search or verifier feedback. VCRT obtains a 77.60% macro pass rate versus 64.53% for the strongest matched baseline, a paired gain of 13.06 points (95% bootstrap CI [12.58, 13.54]). Lean accounts for most of this gain at 33.49 points, while ProofWriter and CLRS improve by 2.85 points on average. Mechanism tests consistently favor anti-diamond supervision, whereas No-Orbit is statistically indistinguishable from full VCRT. The evidence does not establish a general benefit from exact orbit aggregation.
• 中文摘要 许多计算允许多个有效执行命令，因为独立的子目标或不相交的状态更新可以交换。带有可验证奖励的强化学习通常将每个成功跟踪视为独立的令牌序列，因此序列化选择可能被误认为逻辑依赖。我们引入验证者-认证规则传输（VCRT），它通过原生验证器重放相邻操作对。两个命令被接受且达到相同典范状态的对提供交换证书;被拒绝或状态改变的反转提供反钻石。VCRT使用反钻石保留真实前提条件，并将策略积分分配给每个认证轨道的总概率质量。它还约束交换后的一致性、源保留和策略漂移。我们通过共享匿名关系图接口评估ProofWriter、CLRS和Lean之间的留一环境输出传输。所有训练和检查点决策在进行持续评估前被冻结，该评估每条使用一个贪婪轨迹，不包含搜索或验证者反馈。VCRT宏观通过率为77.60%，而最强匹配基线为64.53%，配对提升13.06分（95% 自助置信区间 [12.58， 13.54]）。精益模式贡献了大部分提升，达到33.49分，而ProofWriter和CLRS平均提升2.85分。机制测试一贯支持反钻石监督，而无轨道测试在统计上与全VCRT无异。证据未证明精确轨道聚合具有普遍优势。

Learning What to Activate: Combinatorial Capability Allocation for Long-Horizon Multimodal Agents

学习激活内容：长视野多模态智能体的组合能力分配

• Authors: Wenhao Yuan, Chenchen Lin, Jian Chen, Jinfeng Xu, Shuo Yang, Edith Cheuk-Han Ngai
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.27869
• Pdf link: https://arxiv.org/pdf/2609.27869
• Abstract Long-horizon multimodal agents rely on specialized capabilities for perception, retrieval, reasoning, verification, and execution. Existing designs typically activate a fixed capability set or invoke a predefined workflow, incurring substantial computational overhead while failing to accommodate stage-dependent capability demands. In this paper, we study the \textit{combinatorial capability allocation} problem for long-horizon multimodal agent systems, where the system selects a cost-sensitive subset of specialized capabilities at each interaction stage, which is nontrivial since capability values depend on the selected subset, while previous allocations alter the states encountered by subsequent decisions. We introduce \textsc{CoCA}, an on-policy learning framework that recovers a deployable capability-subset policy from sparse conditional comparisons. On states visited by the student policy, the stronger teacher compares the marginal net values of candidate capabilities, conditioned on the currently selected subset. Then, we adopt a conditional utility model to transform such comparisons into an autoregressive capability-subset policy, avoiding explicit enumeration. We further introduce dual-level on-policy distillation to address distribution mismatch both across environment states and within the partial subsets encountered during set construction. Finally, trajectory-level reinforcement learning refines the distilled policy toward task success, activation cost, and allocation stability. At inference time, allocation is performed solely by the lightweight student policy without teacher queries or online updates. Experiments on long-horizon multimodal environments and controlled capability-demand shifts demonstrate the superiority of our method over the state-of-the-art baseline methods.
• 中文摘要 长视野多模态代理依赖于感知、检索、推理、验证和执行等专用能力。现有设计通常激活固定的能力集或调用预定义的工作流程，导致计算开销巨大，且无法满足阶段依赖的能力需求。本文研究了长视野多模态智能体系统的\textit{组合能力分配}问题，系统在每个交互阶段选择一个成本敏感的专用能力子集，这并非简单，因为能力值依赖于所选子集，而先前分配会改变后续决策所遇到的状态。我们介绍了\textsc{CoCA}，一种策略内学习框架，通过稀疏条件比较恢复可部署的能力子集策略。对于学生策略访问的状态，较强的教师比较候选能力的边际净值，条件为当前选择的子集。随后，我们采用条件效用模型，将此类比较转化为自回归的能力子集策略，避免显式枚举。我们进一步引入双层策略内蒸馏，以解决环境状态间及集合构建过程中遇到的部分子集内的分布不匹配问题。最后，轨迹级强化学习将提炼策略优化为任务成功率、激活成本和分配稳定性。在推断阶段，分配仅由轻量级学生策略执行，无需教师查询或在线更新。在长期多模态环境和受控能力-需求变化上的实验证明了我们方法优于最先进基线方法。

PCQC: Privileged Counterfactual Question Credit for Multi-Turn Medical Dialogue

PCQC：多回合医学对话的特权反事实问题署名

• Authors: Chenxuan Li, Jiayi Wan, Xinrong Chen, Zhongyu Zhao, Xuecheng Shang, Peixing Wan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.27987
• Pdf link: https://arxiv.org/pdf/2609.27987
• Abstract Large language models (LLMs) have made substantial progress on medical question-answering, yet effective medical dialogue also requires learning to ask questions that uncover relevant patient information. To train such dialogue policies, a common pipeline combines supervised fine-tuning with reinforcement learning (RL) based on final diagnostic correctness. However, this outcome-based supervision does not directly distinguish the contributions of individual questions and provides no question-level feedback for unexecuted alternatives. To address this gap, we introduce PCQC (Privileged Counterfactual Question Credit), which uses privileged patient information during training to learn from questions never asked. During training, PCQC makes alternative questions directly comparable at the same dialogue state by using privileged patient facts to construct their answers. A frozen diagnostic scorer evaluates the diagnostic utility of each resulting question-answer pair by how strongly it supports the correct diagnosis. PCQC turns these comparisons into relative question credit that teaches the policy which questions to favor, directly supervising both executed and unexecuted questions alongside outcome-based RL without requiring complete rollouts for the unexecuted alternatives. Extensive experiments across four medical benchmarks demonstrate that PCQC achieves 63.10% mean diagnostic accuracy, outperforming GRPO and ATPO by 4.38 and 4.21 percentage points, respectively. These gains are achieved with 33.1% fewer inquiry turns than GRPO.
• 中文摘要 大型语言模型（LLM）在医疗问答方面取得了显著进展，但有效的医疗对话也需要学会提出揭示相关患者信息的问题。为了训练此类对话策略，常见的流程结合了监督微调与基于最终诊断正确性的强化学习（RL）。然而，这种基于结果的监督并未直接区分单个问题的贡献，也未为未执行的选项提供问题级反馈。为弥补这一空白，我们引入了PCQC（特权反事实问题署名），该技术在培训中使用特权患者信息，从未被提出的问题中学习。在培训过程中，PCQC通过利用特权患者事实构建替代问题的答案，使得同一对话状态下的替代问题可直接比较。冻结的诊断评分器评估每个生成的问答对的诊断效用，判断其支持正确诊断的强度。PCQC将这些比较转化为相对问题积分，教导政策支持哪些问题，直接监督已执行和未执行的问题，同时支持基于结果的强化学习，而无需对未执行的替代方案进行全面推广。在四个医学基准测试中广泛实验显示，PCQC的平均诊断准确率达到63.10%，分别比GRPO和ATPO高出4.38%和4.21个百分点。这些提升是在查询回合比GRPO少33.1%的情况下实现的。

Learning a Speed-adaptive Hip Exoskeleton Control Policy Via Sim-to-real Reinforcement Learning

通过模拟到现实强化学习学习速度自适应髋外骨骼控制策略

• Authors: Bin Li, Zhimin Hou, Jiacheng Hou, Zenian Liang, Tong Wu, Teng Ma, Chenglong Fu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.28027
• Pdf link: https://arxiv.org/pdf/2609.28027
• Abstract Providing personalized exoskeleton assistance across varying walking speeds remains challenging. Existing online optimization methods are sample-inefficient, requiring extensive human-in-the-loop (HIL) evaluations to optimize the entire assistive torque profile. Sim-to-real reinforcement learning (RL) offers a promising alternative but cannot directly account for individual user preferences. We propose a framework integrating sim-to-real RL with online preference learning for personalized exoskeleton assistance. Specifically, assistance timing is learned in simulation by training RL policies with human musculoskeletal models across varying walking speeds. The learned policies are then distilled and deployed on a physical hip exoskeleton using onboard sensory observations. Gaussian-process-based preference learning further personalizes the assistance magnitude through pairwise user comparisons. By decoupling assistance timing learning in simulation from magnitude optimization in real-world experiments, our framework substantially reduces the online optimization space. Human-subject experiments demonstrate efficient identification of personalized assistive torque profiles across varying walking speeds with fewer real-world evaluations.
• 中文摘要 在不同步行速度下提供个性化外骨骼辅助仍具挑战性。现有的在线优化方法样本效率低，需要大量人机循环（HIL）评估以优化整个辅助扭矩曲线。模拟到现实强化学习（RL）提供了有前景的替代方案，但无法直接考虑个体用户偏好。我们提出一个将模拟到现实强化学习与在线偏好学习整合的框架，实现个性化外骨骼辅助。具体来说，辅助时机通过在不同行走速度下用人类肌肉骨骼模型训练强化学习策略，在模拟中学习。随后，学习到的策略被提炼并利用机载感官观察应用于物理髋部外骨骼。基于高斯过程的偏好学习通过两对用户比较进一步个性化辅助强度。通过将模拟中的辅助时序学习与现实实验中的幅度优化解耦，我们的框架大大缩小了在线优化空间。人受试者实验展示了在不同行走速度下高效识别个性化辅助扭矩曲线的能力，且实际评估更少。

Curriculum Learning with GNN-based Reinforcement Learning for Job Shop Scheduling

基于GNN的强化学习课程学习，用于工坊排班

• Authors: Jayakrishnan K. Vasudevan (Rosenheim University of Applied Sciences), Jonathan Hoss (Rosenheim University of Applied Sciences), Noah Klarmann (Rosenheim University of Applied Sciences)
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.28085
• Pdf link: https://arxiv.org/pdf/2609.28085
• Abstract The job shop scheduling problem is a challenging combinatorial optimization problem, and recent reinforcement learning approaches using graph neural networks have shown promise for learning scheduling policies directly from problem instances. However, training on large instances remains computationally expensive, and generalization across instance sizes remains challenging. This paper studies curriculum learning for graph neural network-based reinforcement learning in the job shop scheduling problem by comparing it with single-size training across three target sizes: 20 x 20, 25 x 25, and 30 x 30. In the curriculum setting, the policy is first trained on smaller instances and then progressively adapted to larger target sizes, allowing scheduling behavior learned in earlier stages to support learning on larger instances. Models are evaluated on unseen instances from 8 x 8 to 30 x 30 using the optimality gap, considering both generalization across all evaluation sizes and specialization on the target size. Results show that curriculum learning consistently reduces wall-clock training time, with larger benefits as the target size increases. The strongest advantage is observed at 30 x 30, where curriculum learning reduces the mean optimality gap across all evaluation sizes by approximately 8.1 percentage points, reduces the target-size mean optimality gap by approximately 8.6 percentage points, and saves approximately 50 hours of training time.
• 中文摘要 作业车间调度问题是一个具有挑战性的组合优化问题，近期利用图神经网络的强化学习方法显示出直接从问题实例学习调度策略的前景。然而，在大型实例上训练仍然计算成本高昂，跨实例规模的泛化依然具有挑战性。本文通过比较在工作车间调度问题中基于图神经网络的强化学习，比较了针对三个目标大小（20 x 20、25 x 25和30 x 30）的单一规模训练，从而研究了课程学习。在课程环境中，策略先在较小的实例上训练，然后逐步适应更大的目标规模，使得早期阶段学到的调度行为能够支持更大实例的学习。模型在未见的实例中，从8 x 8到30 x 30，利用最优性差距进行评估，同时考虑了所有评估规模的推广和目标规模的专业化。结果显示，课程学习持续减少墙上计时训练时间，随着目标规模的增加，效益更大。在30 x 30中观察到最强优势，课程学习可将所有评估规模的平均最优性差距约减少8.1个百分点，目标规模平均最优性差距约8.6个百分点，并节省约50小时的训练时间。

RL Starts before RL: On Policy Distillation for Better Reinforcement Learning

强化学习先于强化学习：关于策略提炼以实现更好强化学习

• Authors: Shuai Dong, Yongfu Zhu, Yuqi Xu, Weichu Xie, Liuwenpu, Ziyue Wang, Kaiwen Tuo, Congcong Wang, Siyuan Wang, Wenqi Shao, Shuai Yang, Ji Zhao, Caoyuan Ma, Wenzheng Chang, Taiqiang Wu, Xinlei Yu, Hongrui Wu, Xiaoxuan He, Fangke Chen, Dianyi Wang, Kanghui Tian, Sirry Chen, Xingyu Liu, Xiangnan Wu, Jiawei Guo, Haowen Hou, LingHan Chen, Zhongyu Wei, Jiaqi Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.28145
• Pdf link: https://arxiv.org/pdf/2609.28145
• Abstract Reinforcement learning (RL) improves reasoning, but its performance depends on the policy from which training begins. We study on-policy distillation (OPD) as a preparation stage for RL and ask whether its benefits extend beyond improvements in the distilled model's initial accuracy. Under shared RL settings, students initialized with OPD reach higher final performance than those trained with direct RL or supervised fine-tuning followed by RL. This advantage can emerge even when OPD produces little immediate improvement in accuracy. Pre-RL Pass@k does not fully explain the benefit: similar or even higher values do not necessarily lead to better performance after RL. Behavioral analyses point to alignment with the teacher's distribution beyond top-1 agreement as a possible explanation. Such alignment may favor higher-quality reasoning paths while retaining alternatives that RL can further refine using outcome feedback. We further examine how trajectory sources and divergence objectives affect the value of distillation for subsequent RL. Standard reverse-KL OPD performs better before RL, but forward-KL OPD overtakes it afterward; with teacher-generated distillation trajectories, reverse KL remains ahead at both stages. These findings suggest that the preferred distillation objective depends on both the trajectory source and the training that follows. Our results support evaluating OPD as preparation for RL and selecting distillation choices by the performance achieved after subsequent training.
• 中文摘要 强化学习（RL）提升推理能力，但其表现取决于训练起始的策略。我们将策略提纯（OPD）作为强化学习的准备阶段，探讨其益处是否超越了提炼模型初始准确性的提升。在共享强化学习环境下，初始化OPD的学生最终表现优于直接强化学习或监督微调后强化学习的学生。即使OPD对准确率的即时提升有限，这一优势仍可能显现。前强化学习Pass@k未能完全解释其益处：相似甚至更高的数值不一定导致强化学习后表现更好。行为分析表明，与教师分布超过前1协议的一致性可能是一个解释。这种对齐可能有利于更高质量的推理路径，同时保留可通过结果反馈进一步完善的替代方案。我们还进一步探讨了轨迹源和发散目标如何影响后续强化蒸馏的价值。标准反向KL OPD在RL前表现更好，但后期正向KL OPD超过了它;而教师生成的蒸馏轨迹则在两个阶段均领先。这些发现表明，首选蒸馏目标取决于轨迹源和后续训练。我们的结果支持将OPD评估为RL的准备，并根据后续训练后的表现选择蒸馏。

Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

剖析以优势为导引的愿景-语言-行动政策培训后

• Authors: Jiahang Cao, Hanye Zhao, Hang Lai, Shenyu Zhang, Xiaoshen Han, Xinghang Li, Futeng Liu, Wanli Peng, Heyun Wang, Yunhong Wang, Jason Li, Yong Yu, Weinan Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.28161
• Pdf link: https://arxiv.org/pdf/2609.28161
• Abstract Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However, its performance depends on several coupled choices, including how critic-derived advantages are constructed, calibrated, and used for policy training. Existing recipes often combine these choices into a single end-to-end procedure, making their individual effects difficult to identify. In this work, we dissect advantage-guided VLA post-training through a controlled empirical study that separates these design choices while accounting for their distinct estimands. We develop stage-specific offline evaluation methods to screen alternative choices efficiently, without requiring extensive real-robot policy evaluations for every possible combination. The staged evaluation identifies a modular recipe that combines temporal-difference advantage construction, group-wise calibration, and continuous advantage weighting. Across four real-world bimanual tasks, the resulting recipe improves mean task progress and success over the SFT initialization by 0.42 and 0.63, respectively. Moreover, the proposed evaluation diagnostics show an overall alignment with downstream real-world performance, supporting their use for interpreting empirical outcomes and selecting advantage-guided post-training designs in practice.
• 中文摘要 优势引导强化学习提供了一种实用的方法，可以在有限的机器人数据下对后训练视觉-语言-行动（VLA）策略进行后期训练。然而，其性能依赖于多个耦合选择，包括批评者衍生优势的构建、校准和策略训练的使用方式。现有的方案通常将这些选择合并为单一的端到端流程，使其单独影响难以识别。本研究通过受控实证研究剖析优势引导VLA训练后，区分这些设计选择，同时考虑它们各自独特的时间。我们开发了阶段特定的离线评估方法，以高效筛选备选选择，而无需对每种可能组合进行大量真实机器人策略评估。分阶段评估识别出一个模块化方案，结合了时间差分优势构建、组别校准和连续优势加权。在四个真实世界双手任务中，所得的方案分别使平均任务进度和成功率在SFT初始化期间提升了0.42和0.63。此外，所提出的评估诊断显示出与下游真实表现的整体一致性，支持其在解读实证结果和选择优势导向的培训后设计中的应用。

Finite-Sample Probabilistic Safety Certification for AI-Based Grid-Edge Coordination

基于人工智能的网格-边际协调有限样本概率安全认证

• Authors: Yihong Zhou, Hanbin Yang, Thomas Morstyn
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.28182
• Pdf link: https://arxiv.org/pdf/2609.28182
• Abstract Coordinating large population of flexible grid-edge devices can alleviate the need for time-consuming and capital-intensive network upgrades, and AI-based control methods such as multi-agent reinforcement learning or imitation learning are promising in their real-time decision scalability. However, system operators still need an independent and rigorous way to decide whether a given AI system is safe enough for deployment. This paper develops a finite-sample probabilistic safety certification framework for black-box AI decision models in closed-loop grid operation. The central idea is to reduce the complete input--AI--grid evaluator workflow to a binary unsafe outcome under an operator-defined safety specification, and then use exact binomial inference to certify the corresponding unsafe operation probability. Given a set of held-out calibration scenarios, the framework returns the tightest one-sided upper certificate and an accept/reject deployment criterion that controls the probability of false safety certification. Because the certification is for the calibration distribution that may deviate from the future operation, we further combine the nominal certificate with physically interpretable sample-space adversarial attacks, a concept widely used in AI to investigate the fragility of AI models. Case studies on grid-edge flexibility coordination with 1{,}000-agent AI models (independent parameters) verify the finite-sample safety guarantee and the value of integrating adversarial attacks into a rolling-window training-certification-deployment flow.
• 中文摘要 协调大量灵活网格边缘设备可以减少耗时且资本密集的网络升级需求，基于人工智能的控制方法如多智能体强化学习或模仿学习在实时决策可扩展性方面具有前景。然而，系统操作员仍需一种独立且严谨的方法来判断某一人工智能系统是否足够安全部署。本文为闭环网格运行中的黑箱AI决策模型开发了有限样本概率安全认证框架。核心思想是将完整的输入-AI-网格评估器工作流简化为操作员定义的安全规范下的二元不安全结果，然后使用精确二项推断来认证相应的不安全操作概率。给定一组未完成的校准场景，该框架返回最严格的单边上层证书和一个控制虚假安全认证概率的接受/拒绝部署标准。由于认证针对可能偏离未来操作的校准分布，我们进一步将名义证书与物理可解释的样本空间对抗攻击结合起来，这一概念在人工智能中广泛用于研究人工智能模型的脆弱性。关于与1{，}000智能体AI模型（独立参数）进行网格边缘灵活性协调的案例研究验证了有限样本安全保障及将对抗攻击整合进滚动窗口训练-认证-部署流程的价值。

LEAP-CBF: A Safety Filter for Uncertain Systems with Least-Effort Adversarial Potentials

LEAP-CBF：针对不确定且具备最小对抗潜力的安全过滤器

• Authors: Oswin So, Eric Yu, Chuchu Fan
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.28364
• Pdf link: https://arxiv.org/pdf/2609.28364
• Abstract Control barrier functions (CBF) are a popular safety filter to ensure safety for nonlinear dynamical systems. However, when the system is subject to uncertainties and disturbances, this requires the use of robust variants of CBFs, which can be difficult to construct and can be overly conservative, especially for high-dimensional systems under input constraints. In this work, we propose a new approach to solve these challenges by introducing Least-Effort Adversarial Potentials (LEAP), a certificate that quantifies the robustness of a given state against disturbances in terms of the effort required by the disturbance to cause failure. We show that LEAP is a CBF for the undisturbed system, but can also be used to construct a safety filter that is robust to disturbances whose cumulative effort is bounded. We propose a method for constructing LEAPs with on-policy deep reinforcement learning. Next, we demonstrate LEAPs in simulation on a variety of multi-agent systems with disturbances and uncertainties. Finally, hardware experiments on a quadruped and quadrotors validate that LEAPs are well suited to tackle the disturbances and uncertainties from real-world robotic systems.
• 中文摘要 控制障碍函数（CBF）是确保非线性动力系统安全性的常用安全滤波器。然而，当系统受到不确定性和干扰影响时，就需要使用稳健的CBF变体，这些变体可能难以构建，且在输入约束下的高维系统中可能过于保守。在本研究中，我们提出了一种新方法，通过引入最小努力对抗势（LEAP）来解决这些挑战，这是一种通过量化给定状态对干扰的鲁棒性，体现扰动导致失效所需的努力。我们证明LEAP是未受扰系统的CBF，同时也可以用于构建对累计努力有界的干扰具有鲁棒性的安全过滤器。我们提出了一种基于策略深度强化学习构建LEAP的方法。接下来，我们在多种多智能体系统中演示LEAPs的模拟，这些系统存在干扰和不确定性。最后，四足和四旋翼的硬件实验验证了LEAP非常适合应对现实机器人系统带来的干扰和不确定性。

ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Control

遗忘模仿者：动作解学用于强化学习类人生物控制

• Authors: Xukun Luan, Zhongxiang Lei, Chen Gong, Shaowei Li, Yuanguo Bi, Jinyan Liu
• Subjects: Subjects: Robotics (cs.RO); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.28378
• Pdf link: https://arxiv.org/pdf/2609.28378
• Abstract Humanoid control, leveraging human demonstrations, has achieved diverse, agile, and natural locomotion behaviors through reinforcement learning (RL). While this paradigm has yielded remarkable performance in physical humanoid control, how to eliminate specific motions from learned policies remains insufficiently explored. Addressing this issue is motivated by pressing safety and privacy concerns: the removal of malicious, poisoned, or suboptimal motions, as well as copyright-protected motions subject to the right to be forgotten under regulations such as the GDPR, is of critical importance. To this end, we propose {ForgetMimic}, the first motion-level unlearning method designed specifically for physical-world humanoid control. The core idea of ForgetMimic is as follows: given a policy $\pi_\theta$ trained on $N$ motions, our method degrades performance on a target subset of $K$ motions while preserving the effectiveness of the remaining $N-K$ motions. Furthermore, we identify and resolve two key training mechanisms in robot control that lead to unlearning failure. We conduct extensive experiments on the Unitree G1 and H2 humanoid robots across 12 motions, including Dance, Fight, Flip, and others. Experimental results demonstrate that ForgetMimic effectively eliminates memory of designated motions while maintaining the normal operation of all other motions.
• 中文摘要 类人生物控制借助人类演示，通过强化学习（RL）实现了多样、敏捷和自然的运动行为。尽管这一范式在物理类人生物控制方面取得了显著表现，但如何从学习到的策略中消除特定动作仍缺乏充分探索。解决这一问题的动机是紧迫的安全和隐私问题：移除恶意、有毒或次优的动作，以及受版权保护、受《通用数据保护条例》（如GDPR保护）可遗忘权的动作，至关重要。为此，我们提出了{ForgetMimic}，这是首个专为物理世界类人生物控制设计的动作级去学习方法。ForgetMimic 的核心思想如下：给定一个以 $N$ 动作训练的策略 $\pi_\theta$，我们的方法会降低目标子集 $K$ 动作的性能，同时保持剩余 $N-K$ 动作的有效性。此外，我们识别并解决了导致复学失败的两个机器人控制关键训练机制。我们在 Unitree G1 和 H2 类人机器人上进行了广泛的实验，涵盖 12 种动作，包括舞蹈、战斗、翻转等。实验结果表明，ForgetMimic 有效消除指定动作的记忆，同时保持所有其他动作的正常运行。

When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

何时何地信任教师：通过熵校准学分作业统一政策提炼与GRPO

• Authors: Jie Zhang, Jingxiao Yang, Zhehao Huang, Yuhang Liu, Xiaolin Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.28385
• Pdf link: https://arxiv.org/pdf/2609.28385
• Abstract Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. \emph{Path-Utility Unification} (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. \emph{Entropy-Calibrated Redistribution} (ECR) then uses the signed teacher--old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average (\mathrm{Avg@12}) accuracies of 17.21\% and 65.09\% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过最终答案的正确性来监督数学推理，但对单个代币几乎没有指导。策略提炼（OPD）对学生生成的回答提供了密集的反馈，但教师偏好不必反映正确性。近期混合方案结合了OPD和验证者衍生的优势，或使用教师比率重新加权任务学分。然而，教师指导是在基于验证者的群体归一化之后进入的，代币重加权不必保留每个回答分配的总任务学分。我们引入了GRPO统一熵校准学分再分配（UECR-GRPO），该方法在响应和令牌级别的单一GRPO式更新中整合了验证者和教师信号。\emph{路径-效用统一}（PUU）将验证者奖励和教师与锚路径对数比结合在单一KL正则化目标中。其政策实现采用长度归一化教师评分，并将小组归一化前的奖励和PPO剪裁结合，使教师证据影响响应排名。\emph{熵校准再分配}（ECR）随后利用带符号的教师-旧政策令牌差距重新分配验证者衍生的部分。全词汇教师熵减弱不确定的指导，而按响应的零和预测则保留总任务学分及其按标记的符号，避免裁剪。在五个数学推理基准测试中，UECR-GRPO在Qwen3-1.7B和Qwen3-4B学生中分别实现了17.21%和65.09%的平均\（\mathrm{Avg@12}\）准确率，分别在每个量表上都比最强基线高出0.89个百分点和0.56个百分点。

Tractable Reinforcement Learning for Full Class of Signal Temporal Logic Specifications Using Spatiotemporal Tube Reward

使用时空管奖励的可处理强化学习，适用于全类信号时序逻辑规范

• Authors: Vaishnavi Jagabathula, P Sangeerth, Pushpak Jagtap
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.28396
• Pdf link: https://arxiv.org/pdf/2609.28396
• Abstract This paper addresses the control problem for robotic systems, including non-holonomic and underactuated platforms operating under unknown dynamics and strict actuator limits to satisfy complex high-level specifications. We denote these high-level specifications using Signal Temporal Logic (STL) and propose a novel time-aware Reinforcement Learning (RL) framework that leverages the geometric properties of Spatiotemporal Tubes (STTs). While traditional analytical STT controllers often struggle to enforce input constraints, and existing RL approaches rely on memory-intensive state history, our method natively overcomes both limitations. By mapping the logical and temporal complexities of the full class of STL into time-varying geometric boundaries, we directly constrain the multidimensional system state without relying on scalar robustness metrics. Augmenting the state space with time, we train a time-aware Soft Actor-Critic (SAC) agent using a continuous, geometry-aware reward function that eliminates the need to explicitly evaluate complex logical semantics during execution. The proposed framework offers a history-free, computationally efficient approach to learn continuous control policies that ensure robust satisfaction of specifications while strictly adhering to system input constraints.
• 中文摘要 本文探讨了机器人系统的控制问题，包括在未知动力学和严格执行器限制下运行的非全源和欠驱动平台，以满足复杂的高级规格。我们使用信号时序逻辑（STL）表示这些高级规范，并提出了一种新颖的时间感知强化学习（RL）框架，利用时空管（STTs）的几何属性。虽然传统的分析STT控制器常难以强制执行输入约束，且现有强化学习方法依赖内存密集型状态历史，但我们的方法原生克服了这两方面的局限。通过将完整类STL的逻辑和时间复杂度映射为时间变化的几何边界，我们直接约束多维系统状态，而无需依赖标量鲁棒度度量。通过时间扩展状态空间，我们使用连续且几何感知的奖励函数训练一个时间感知的软演员-批判者（SAC）代理，消除了在执行时显式评估复杂逻辑语义的需求。该框架提供了一种无历史、计算高效的方法，用于学习连续控制策略，确保在严格遵守系统输入约束的同时，稳健满足规范。

Keyword: diffusion policy

There is no result