生成时间: 2026-09-18 20:49:03 (UTC+8); Arxiv 发布时间: 2026-09-18 20:00 EDT (2026-09-19 08:00 UTC+8)

今天共有 43 篇相关文章

Keyword: reinforcement learning

CovR: Coverage-Aware Hardware Verification via Reasoning-Guided Reinforcement Learning

CovR：通过推理引导强化学习实现覆盖感知硬件验证

• Authors: Manar Abdelatty, Maryam Nouh, Sherief Reda
• Subjects: Subjects: Hardware Architecture (cs.AR); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.19189
• Pdf link: https://arxiv.org/pdf/2609.19189
• Abstract Design verification remains one of the most resource-intensive stages of hardware development, often consuming up to 70% of the total design effort. While recent work has explored using Large Language Models (LLMs) to automate testbench generation, most existing approaches focus narrowly on functional correctness, overlooking the critical aspect of coverage quality. To bridge this gap, we present CovR, an agentic framework for automated testbench generation that combines self-reflection loops with simulation-based feedback to maximize coverage. Using this pipeline, we construct a large-scale dataset of 16,514 natural specification RTL reasoning testbench tuples with a strong teacher model, enabling coverage-aware supervision. Building on this, we propose a reinforcement learning (RL) framework tailored for coverage-driven testbench generation, leveraging tool-derived rewards from simulation and coverage feedback to optimize a student model. Experimental results show that the CovR finetuned model achieves 93.81% cov@10 on VerilogEval and RTLLM V2.0, and 87.76% cov@10 on CVDP, outperforming state-of-the-art approaches by 7.97% and 3.59%, respectively. Furthermore, deploying the finetuned model back into the agentic refinement pipeline further improves cov@10 to 94.27% on VerilogEval and RTLLM V2.0 and 91.39% on CVDP. Moreover, when integrated as a plug-in stimulus engine for full verification workflows, CovR improves coverage by 18.95% and mutation detection score by 1.19%, while revealing 4.46% undetected failures, highlighting the importance of optimizing for coverage in LLM-based hardware verification.
• 中文摘要 设计验证仍是硬件开发中资源最耗费的阶段之一，常占总设计工作的70%。虽然近期研究尝试使用大型语言模型（LLM）自动化测试平台生成，但大多数现有方法仅聚焦于功能正确性，忽视了覆盖质量的关键方面。为弥合这一差距，我们提出了CovR，一种自动生成测试平台的代理框架，结合自我反思循环与基于仿真的反馈以最大化覆盖率。利用该流程，我们构建了一个包含16,514个自然规范RTL推理测试平台元组的大规模数据集，并采用强教师模型，实现覆盖意识监督。基于此，我们提出了一个专为覆盖驱动测试平台生成量身定制的强化学习（RL）框架，利用仿真和覆盖反馈中工具衍生的奖励来优化学生模型。实验结果显示，CovR微调模型在VerilogEval和RTLLM V2.0上cov@10率为93.81%，在CVDP上cov@10为87.76%，分别优于最先进方法7.97%和3.59%。此外，将微调模型重新部署到agentic refinement流水线中，VerilogEval和RTLLM V2.0的cov@10提升至94.27%，CVDP上的91.39%。此外，作为插件刺激引擎集成以实现完整验证工作流，CovR覆盖率提升18.95%，突变检测得分提升1.19%，同时揭示4.46%未检测失败，凸显了基于LLM硬件验证中覆盖度优化的重要性。

AUDITPLAN: Commit, Then Answer for Auditable Safety Alignment

AUDITPLAN：承诺，然后负责可审计的安全对齐

• Authors: Sai Sri Pushpa Jampani, Kshitij Mishra, Asif Ekbal
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.19325
• Pdf link: https://arxiv.org/pdf/2609.19325
• Abstract Safety tuning pipelines judge only the final answer, which makes it difficult to distinguish robust refusal from two undesirable shortcuts: blanket refusal on benign requests and polished but unfaithful safety rationales that do not actually constrain the answer. We propose AUDITPLAN, a single-model plan-then-answer approach where the model first emits a compact structured safety plan and then answers conditioned on it. The plan records a threat label, intended action, and explicit constraints, enabling machine-checkable auditing while remaining hidden from users at deployment. We train this behavior with supervised fine-tuning followed by reinforcement learning with FAITHGATE, a reward-gating objective that grants answer reward only when the safety plan is correct. This discourages safe-looking but unfaithful behavior and promotes tighter plan-answer coupling. Across Qwen backbones, AUDITPLAN improves both robustness and auditability: on Qwen2.5-3B-Instruct, FAITHGATE reduces ASR from 24.0% to 11.6%, LSR from 1.0% to 0.36%, and over-refusal from 11.0% to 2.0%, outperforming answer-only RL, free-form explanation, and weighted-sum structured rewards. Similar trends hold for Qwen2.5-1.5B-Instruct. Larger-model confirmation runs on Qwen-3-4B-Instruct and Qwen2.5-7B-Instruct preserve the same trend suggesting that explicit internal commitments can make safety alignment more faithful, robust, and auditable.
• 中文摘要 安全调优流程只判断最终答案，这使得强健拒绝与两个不理想的捷径难以区分：对无害请求的全面拒绝和精心设计但不忠实的安全理由，这些理由实际上并未限制答案。我们提出了AUDITPLAN方法，这是一种单模型的计划-后答复方法，模型先发布紧凑的结构化安全计划，然后基于该方案给出有条件的回答。该计划记录威胁标签、预期行动和明确约束，支持机器可检查的审计，同时在部署时对用户保持隐藏。我们通过监督微调训练这种行为，随后用FAITHGATE进行强化学习，FAITHGATE是一种奖励门槛目标，只有在安全计划正确时才给予答案奖励。这抑制了看似安全但不忠实的行为，促进了更紧密的计划-答案耦合。在Qwen骨干网中，AUDITPLAN提升了鲁棒性和可审计性：在Qwen2.5-3B-Instruct上，FAITHGATE将ASR从24.0%降至11.6%，LSR从1.0%降至0.36%，过度拒绝从11.0%降至2.0%，优于仅回答式强化学习、自由形式解释和加权和结构化奖励。Qwen2.5-1.5B-Instruct也呈现类似趋势。Qwen-3-4B-Instruct和Qwen2.5-7B-Instruct上的大型模型确认运行保持相同趋势，表明明确的内部承诺可以使安全对齐更忠实、稳健且可审计。

Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation

通过选择性奖励刺激改善离线目标条件强化学习

• Authors: Jing Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.19414
• Pdf link: https://arxiv.org/pdf/2609.19414
• Abstract Goal-conditioned reinforcement learning aims to learn policies that reach specified goals, but remains challenging in offline settings with sparse rewards and long-horizon dependencies. In such settings, goal-completion information can be temporally distant from the early decisions that enable success, while offline value estimation introduces additional error. We study this issue from a reward-propagation perspective and show, in a stylized delayed-goal setting, how goal-directed value separation can become small relative to local estimation error. Motivated by this analysis, we propose Reward Stimulation Implicit Q-Learning (RSIQL), a simple non-hierarchical method that introduces additional reward signals at progress-making intermediate states in offline trajectories. RSIQL uses an auxiliary goal-conditioned value function to identify intermediate states estimated to make progress toward the goal and applies reward stimulation to provide less-delayed training supervision. Unlike hierarchical methods, RSIQL does not learn a separate high-level subgoal policy. Experiments on D4RL goal-reaching benchmarks and OGBench show that RSIQL improves over goal-conditioned IQL on average and achieves performance competitive with hierarchical offline goal-conditioned methods, while retaining a simple flat policy structure.
• 中文摘要 目标条件强化学习旨在学习能够实现特定目标的策略，但在奖励稀疏且依赖较长的离线环境中仍具挑战性。在此类环境中，目标完成信息可能与促成成功的早期决策相距甚远，而离线价值估计则引入额外误差。我们从奖励-传播视角研究该问题，展示了在风格化延迟目标设定下，目标导向价值分离如何相较于局部估计误差变得较小。基于此分析，我们提出了奖励刺激隐性Q-学习（RSIQL），这是一种简单的非层级方法，在离线轨迹中进展的中间状态引入额外奖励信号。RSIQL使用辅助目标条件值函数识别估计能朝目标推进的中间状态，并应用奖励刺激提供更小延迟的训练监督。与分层方法不同，RSIQL不学习独立的高级子目标策略。在D4RL达目标基准测试和OGBench上的实验显示，RSIQL平均优于目标条件的IQL，并实现与层级离线目标条件方法竞争的性能，同时保持简单的扁平策略结构。

Winning a Won Game: Strict Reach-Avoid-Stay Control Barrier Functions for High-Dimensional Black-Box Systems

赢得胜利：高维黑匣子系统的严格伸手-避开-停留控制屏障函数

• Authors: Donggeon David Oh, Duy P. Nguyen, Gongkai Yuan, Qingchen Li, Jaime Fernández Fisac, Haimin Hu
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.19449
• Pdf link: https://arxiv.org/pdf/2609.19449
• Abstract Robots must complete their tasks and maintain the achieved outcomes while avoiding safety failures at all times. Strict reach-avoid-stay (sRAS) formalizes this requirement: safely reaching a target and remaining there indefinitely after first entry. We propose an sRAS Q-control barrier function (CBF) safety filter for high-dimensional black-box systems under bounded uncertainty. Our construction combines a stay value encoding safe permanent residence in a target subset with a reach-avoid value encoding safe reachability of this subset while avoiding target states from which safe permanent residence cannot be guaranteed. We prove that these values jointly yield a valid robust discrete-time CBF and lift them to state-action Q-functions for runtime intervention. For exact values and under a measure-zero condition, our filter preserves sRAS feasibility from almost every winnable initial state and keeps the system safely within the target after first entry, against all admissible uncertainty realizations. We adopt reachability-based adversarial reinforcement learning for scalable value approximation using only black-box interactions. Notably, neither synthesis nor deployment of our filter requires known dynamics, affine structure, value derivatives, or hand-designed barriers. We validate our framework in quadruped gap jumping in simulation and hardware, where the robot crosses the gap, lands safely, and remains safe afterward. Simulated F1TENTH races further demonstrate safe overtaking and lead retention.
• 中文摘要 机器人必须完成任务并保持已取得的结果，同时始终避免安全故障。严格的到达-避免-停留（sRAS）形式化了这一要求：安全到达目标并在首次进入后无限期停留。我们提出了针对高维黑箱系统在有界不确定性下使用sRAS Q控制障碍函数（CBF）安全滤波器。我们的构造结合了目标子集中的停留值编码安全永久居留与达避的达避值编码该子集的安全可达性，同时避免无法保证安全永久居留的目标状态。我们证明这两个值共同产生有效的稳健离散时间CBF，并将其提升为状态-动作Q函数以进行运行时干预。对于精确值且在度量为零条件下，我们的滤波器几乎保留了几乎所有可获胜初始状态的sRAS可行性，并在首次进入后保持系统安全在目标内，尽管符合所有可接受的不确定性现实。我们采用基于可达性的对抗强化学习，仅使用黑盒交互实现可扩展的价值近似。值得注意的是，我们的滤波器合成和部署均不需要已知的动力学、仿射结构、值导数或手工设计的障碍。我们在模拟和硬件中验证了四足间隙跳跃的框架，机器人跨越间隙，安全着陆，之后保持安全。模拟F1TENTH赛进一步展示了安全超车和领先保持。

GLAMDRING: Gait Learning And Morphology co-Design via Reinforcement LearnING of CPGs

GLAMDRING：步态学习与形态学通过强化学习共同设计消费品

• Authors: Amogh Joshi, Kaushik Roy
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.19452
• Pdf link: https://arxiv.org/pdf/2609.19452
• Abstract Robots are moving out of the structured factory floor and into unstructured environments such as disaster sites, planetary surfaces, and agricultural fields, for which the right robot often does not yet exist. We present GLAMDRING, a framework that synthesizes the optimal robot for a locomotion task and, jointly, learns the controller that drives it. For the given specifications of forward-velocity bounds, a per-actuator power budget, an actuator library, and a payload requirement, GLAMDRING returns a matched quadruped morphology (link geometry and per-joint actuators) and a Hopf-oscillator Central Pattern Generator (CPG) gait policy. We rank feasible designs against a target design objective, viz., maximum speed, minimum Cost of Transport (CoT), or max Payload Margin. Because body and locomotion are coupled, the optimal morphology dictates how a robot is driven, while optimal gait depends on the physical body. We train a small number of CPG policies by reinforcement learning across the space of candidate morphologies, co-learning the gait with the underlying robot hardware. Link lengths and actuators are then resolved post-hoc from the policy's logged operating envelope, reducing synthesis cost to a small, fixed number of reinforcement-learning runs instead of one per candidate. Our experiments show three key findings: co-designing body and gait is necessary to satisfy locomotion constraints; actuator-envelope feasibility, rather than locomotion success alone, determines realizable payload capacity; and canonical animal gaits emerge naturally in most designs from morphology and constraints alone. A real-world demonstration further highlights the efficacy of our work.
• 中文摘要 机器人正从结构化的工厂车间转向非结构化环境，如灾难现场、行星表面和农田，而这些环境通常尚未有合适的机器人。我们提出了GLAMDRING框架，该框架综合了运行任务的最优机器人，并共同学习驱动该机器人的控制器。针对前向速度界限、每个执行器功率预算、执行器库和有效载荷需求，GLAMDRING返回匹配的四足形态（链节几何和每关节执行器）和Hopf振荡器中央模式发生器（CPG）步态策略。我们将可行设计与目标设计目标进行排名，即最大速度、最小运输成本（CoT）或最大有效载荷裕量。由于身体和运动耦合，最优形态决定机器人的驾驶方式，而最优步态则依赖于物理身体。我们通过在候选形态空间中强化学习训练少量CPG策略，并与底层机器人硬件共学步态。链路长度和执行器随后从策略记录的操作包络中事后解析，将合成成本降低为少量固定的强化学习运行，而非每个候选人一次。我们的实验显示了三个关键发现：共同设计身体和步态以满足运动约束是必要的;执行器-包络线的可行性，而非仅靠运动成功决定可实现的有效载荷能力;而大多数设计中，标准动物步态则仅凭形态学和约束自然产生。真实世界的演示进一步强调了我们工作的有效性。

Compositional Reasoning in Language Models under Reinforcement Learning Post-Training

强化学习后语言模型中的组合推理

• Authors: Yu He, Yingxi Li, Yifei Wang, Ellen Vitercik
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.19465
• Pdf link: https://arxiv.org/pdf/2609.19465
• Abstract Compositional reasoning is critical for real-world problem solving: since training data is necessarily limited, models must generalize by composing learned skills in new ways. While post-training methods such as reinforcement learning (RL) have substantially improved the reasoning abilities of language models (LMs), their effects on compositional reasoning remain less well understood. We propose a dependency-graph framework to formalize compositional reasoning, yielding three levels of compositionality with increasing complexity. Empirically, we instantiate this framework with data-structure tasks, which provide deterministic reward computation and clear compositional structure. We find a consistent decomposed-to-composed asymmetry: decomposed-skill training does not reliably transfer to composed tasks, whereas composed-task training transfers more readily back to decomposed tasks. We provide theoretical explanation for this asymmetry, and further evaluate compositional generalization under length extrapolation, structural distribution shift, and transfer to tasks requiring unseen skills. Finally, we present a pilot study on real-world tool-calling benchmarks, showing preliminary evidence that the decomposed-to-composed asymmetry can extend to practical settings.
• 中文摘要 组合推理对现实问题解决至关重要：由于训练数据必然有限，模型必须通过以新方式组合所学技能来泛化。尽管强化学习（RL）等后训练方法显著提升了语言模型（LM）的推理能力，但其对组合推理的影响仍不十分明了。我们提出了依赖图框架来形式化组合推理，产生三个复杂度递增的组合性层次。通过实证方式，我们将该框架实例化为数据结构任务，这些任务提供了确定性奖励计算和清晰的组合结构。我们发现了一致的分解到组合任务的不对称性：分解技能训练不可靠地转移到组合任务，而组合任务训练则更容易转回分解任务。我们为这种不对称性提供了理论解释，并进一步评估了在长度外推、结构分布偏移及对需要未知技能任务的迁移下的组成推广。最后，我们提出了一项基于现实世界工具调用基准的试点研究，初步证据表明分解到复合的不对称可以扩展到实际环境。

Agentic AI Networking for Heterogeneous Unmanned Aerial Systems in Low-Altitude Wireless Networks

低空无线网络中异构无人机系统的代理人工智能网络

• Authors: Nguyen Duc Minh Quang, Chang Liu, Shuangyang Li, Derrick Wing Kwan Ng
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.19538
• Pdf link: https://arxiv.org/pdf/2609.19538
• Abstract Low-altitude wireless networks (LAWNs) are emerging as a key infrastructure for heterogeneous unmanned aerial systems that support concurrent services within a shared three-dimensional airspace. Their coexistence creates strong coupling among mobility, connectivity, and shared network resources, while heterogeneous services impose distinct and time-varying requirements. These interactions naturally form a dynamic non-cooperative game in which both operating conditions and coordination objectives evolve over time. Conventional optimization and learning-based controllers typically rely on predefined objectives, limiting their ability to adapt autonomously to changing service requirements and resource priorities. To address this challenge, we propose a hierarchical hybrid large language model (LLM)- multi-agent reinforcement learning (MARL) architecture organized as a dual-loop structure. Specifically, an outer adaptation loop employs LLM-assisted game orchestration to interpret service requirements and operator intent, and reconfigure objectives and resource priorities, while an inner loop executes decentralized, parameter-conditioned MARL policies under the configured game. A logistics-monitoring case study illustrates how the proposed framework facilitates coordinated coexistence among heterogeneous services, adapting to evolving operating conditions without retraining the underlying MARL policies. Finally, we discuss key challenges and research directions toward scalable, trustworthy, and adaptive agentic LAWNs.
• 中文摘要 低空无线网络（LAWNs）正作为支持共享三维空域内并发服务的异构无人机系统的关键基础设施。它们的共存在移动性、连接性和共享网络资源之间形成了强烈耦合，而异构服务则带来了不同且随时间变化的需求。这些交互自然形成了一个动态的非合作游戏，操作条件和协调目标随时间演变。传统的优化和基于学习的控制器通常依赖预定义目标，限制了其自主适应服务需求和资源优先级变化的能力。为应对这一挑战，我们提出了一种分层混合大型语言模型（LLM）-多智能体强化学习（MARL）架构，组织为双循环结构。具体来说，外部适配环采用LLM辅助的游戏编排来解读服务需求和操作员意图，并重新配置目标和资源优先级，而内环则在配置好的博弈下执行去中心化、参数条件化的MARL策略。一个物流监控案例研究展示了所提框架如何促进异构服务之间的协调共存，适应不断演变的运营条件，而无需重新训练底层MARL策略。最后，我们讨论了面向可扩展、可信和自适应代理LAWNs的关键挑战和研究方向。

Recovering Aggressively Pruned Vision-Language-Action Models with Offline Hidden-State Distillation

通过离线隐藏状态蒸馏恢复被激进修剪的视觉-语言-行动模型

• Authors: Chiyoung Kim, Sanghyuk Roy Choi, Minhyeok Lee
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.19579
• Pdf link: https://arxiv.org/pdf/2609.19579
• Abstract Vision-language-action (VLA) models let robots follow language instructions, but their language backbones of several billion parameters are the main obstacle to running them on robot hardware. Structured pruning reduces that backbone, and removing 63% of it from OpenVLA-OFT drops LIBERO-Long success from 93.2% to 0.8%. A recent approach restores such a model with supervised fine-tuning followed by reinforcement learning, which needs online rollouts and hundreds of GPU-hours. We recover most of the lost success entirely offline. Width pruning narrows the blocks but keeps the residual stream at its original size, so teacher and student hidden states have the same shape and are matched directly, without a projector. Training against a cache built in one teacher pass lifts the 63%-reduced student to within 3.5 points of the teacher in about 8 GPU-hours. A sweep over nine ratios locates where the recovery objective starts to matter. Up to 45% reduction the two do not differ significantly on OpenVLA-OFT. Hidden-state distillation then adds +2.1 to +4.5 points there between 63% and 87%, and +9.4 to +22.1 points on CogACT from 63% onward. At 81% on CogACT, a tripled recovery budget narrows the distilled student's gap to the teacher to 3.9 points on average, while supervised recovery stays more than 20 points below. At matched compression, width pruning yields higher success and depth pruning lower latency. On a 6-DoF manipulator, the distilled student at 72% reduction reaches 77.5% success against 59.5% for supervised recovery, runs 2.23x faster on-board than the teacher, and uses 62% less memory.
• 中文摘要 视觉-语言-动作（VLA）模型允许机器人遵循语言指令，但其数十亿参数的语言骨干是机器人硬件运行的主要障碍。结构化剪枝减少了该骨干，移除OpenVLA-OFT中63%的骨干使LIBERO-Long成功率从93.2%降至0.8%。一种最新方法通过监督微调和强化学习恢复了此类模型，后者需要在线推广和数百GPU小时。我们完全离线恢复了大部分失去的成功。宽度剪枝缩小了区块范围，但保持残余流的原始大小，因此教师和学生隐藏状态形状相同，且能直接匹配，无需投影器。针对一个教师通行证内置的缓存进行训练，约8个GPU小时内，63%缩减后的学生积分提升到与教师3.5分以内。扫描九个比率可确定恢复目标开始重要的地方。在OpenVLA-OFT上，在最大45%的减少时，两者差异不显著。隐藏状态蒸馏在63%至87%之间增加+2.1至+4.5点，CogACT从63%起增加+9.4至+22.1点。在CogACT上，恢复预算增加三倍，使得蒸馏学生与教师的平均差距缩小至3.9分，而监督恢复则低于20分以上。在匹配压缩下，宽度剪枝成功率更高，深度剪枝延迟更低。在6深度操作器上，蒸馏学生在72%减少时成功率为77.5%，监督恢复为59.5%，在机上运行速度是教师的2.23倍，内存消耗减少62%。

Reach or Solve? Attributing Agentic RL Gains with Checkpoint Handoffs

伸手还是解决？通过检查点交接归因于能动强化学习的收益

• Authors: Xuan Liu, Jingbin Qian
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.19636
• Pdf link: https://arxiv.org/pdf/2609.19636
• Abstract Reinforcement learning now trains language-model agents that act over dozens of steps in live environments. The gains are large, and they are read as better decision-making. An agent in a closed loop writes its own inputs. Each observation follows from its own earlier actions, so the states it meets late in an episode are partly of its own making. An SFT checkpoint and an RL checkpoint are then scored from different states, even on identical tasks. Endpoint success mixes two changes: where the agent arrives, and what it does once it is there. Restricting the comparison to states both policies reach does not separate them. That restriction selects on an outcome, and in our data it flips the sign of the effect. We introduce checkpoint handoff, an evaluation protocol that clones a state one released checkpoint reached and hands it to another, with no retraining. Crossing a reacher role and a solver role over SFT and RL splits an endpoint gain into REACH and SOLVE. REACH is how often a policy arrives at a state the environment confirms is a fixed number of actions from success. SOLVE is how often it finishes from an identical cloned state. Across two benchmarks and two independently released pipelines, the reacher by solver interaction is positive in all five conditions. An RL history is worth more to an RL solver than the same history is to an SFT solver. On ALFWorld, RL improves both terms, and the SFT solver never succeeds where the RL solver fails. Independent REACH and SOLVE gaps predict the aggregate interaction. Handoff asks only that one checkpoint's history can be replayed under another, so long-horizon evaluation can report arrival and completion beside endpoint success.
• 中文摘要 强化学习现在训练语言模型代理，这些代理在实时环境中能完成数十步的行动。收益巨大，且被视为更好的决策方式。闭环中的代理会写入自己的输入。每个观察都源自其早期的行为，因此它在事件后期遇到的状态部分是其自身造成的。SFT检查点和强化学习检查点则从不同状态评分，即使是在相同任务上。端点成功结合了两种变化：代理到达地点和到达后所做的动作。将比较限制在两个策略都达到的状态并不会区分它们。该限制选择结果，在我们的数据中它反转了效果的符号。我们引入了检查点切换，这是一种评估协议，克隆一个已释放检查点达到的状态并交给另一个，无需重新训练。在SFT和RL上交叉使用reacher角色和求解者角色，将端点增益分为REACH和SOLVE。REACH是策略达到环境确认为固定动作数的状态的频率。SOLVE是从相同克隆状态结束的频率。在两个基准测试和两个独立发布的管道中，Reacher与求解器的交互在五个条件下均为正。RL历史对RL求解器的价值高于相同的历史对SFT求解器的价值。在ALFWorld中，RL改进了这两个术语，SFT求解器从未成功，而RL求解器失败。独立的REACH和SOLVE缺口预测总体交互。Handoff仅要求一个检查点的历史可在另一个检查点下重放，因此长期视野评估可以报告到达和完成，同时报告端点成功。

EmbodiedMind: Adaptive Data Curation and Prefix-Tree Reinforcement Learning for Efficient Embodied Intelligence

EmbodiedMind：自适应数据管理与前缀树强化学习，实现高效的具身智能

• Authors: Feifan Wang, Zongbing Zhang, Yu Zhang, Lingfeng Wang, Yurui Zhu, Jin Deng, Mingliang Zhang, Zhengguang Gao, Yongcheng Wang, Jin Xu, Ri Yang
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.19659
• Pdf link: https://arxiv.org/pdf/2609.19659
• Abstract Training embodied foundation models typically requires massive-scale datasets and extensive computational resources, yet often suffers from three critical limitations: (1) inefficient sample utilization due to low-informative samples; (2) imbalanced gradient contributions across heterogeneous tasks; and (3) severe credit assignment problem in long-horizon planning, where trajectory-level rewards indiscriminately penalize all tokens. To address these issues, we propose an efficient training paradigm that achieves state-of-the-art average performance through strategic data selection and hierarchical policy optimization. Our approach consists of three synergistic stages. First, Rejection Sampling-based Fine-Tuning (RSFT) filters out low-informative samples to establish robust behavioral priors while preventing distributional collapse. Second, Iterative Rejection GRPO (IR-GRPO) employs task-specific queues stratified by difficulty to keep datasets balanced across reinforcement learning iterations, coupled with a hybrid reward mechanism for precise cross-task feedback. Third, to enhance long-horizon task planning, we introduce Trie-GRPO, a novel reinforcement learning algorithm based on action prefix trees, which enables step-level advantage estimation. This resolves the credit assignment problem by isolating intermediate correct decisions from downstream errors, while effectively balancing exploration efficiency and depth compared to conventional search trees. As a result, EmbodiedMind achieves a state-of-the-art average performance of 70.02% across 18 benchmarks, and significantly outperforms other embodied foundation models in long-horizon task planning accuracy. Our project will be released for reproducibility.
• 中文摘要 训练具象基础模型通常需要大规模数据集和大量计算资源，但常常面临三个关键限制：（1）由于信息量低导致样本利用效率低下;（2）异构任务间梯度贡献不平衡;（3）长期规划中的严重信用分配问题，轨迹级奖励无差别惩罚所有代币。为解决这些问题，我们提出了一种高效的训练范式，通过战略性数据选择和层级策略优化实现最先进的平均性能。我们的方法包含三个协同阶段。首先，基于拒绝抽样的微调（RSFT）过滤掉信息量低的样本，建立稳健的行为先验，同时防止分布崩溃。其次，迭代拒绝GRPO（IR-GRPO）采用按难度分层的任务特定队列，保持强化学习迭代间数据集的平衡，并结合混合奖励机制以实现精确的跨任务反馈。第三，为了增强长期任务规划，我们引入了Trie-GRPO，一种基于动作前缀树的新型强化学习算法，实现了步级优势估计。该方法通过隔离中间正确决策与后续错误，解决了功劳分配问题，同时在探索效率和深度之间取得了优于传统搜索树的平衡。因此，EmbodiedMind在18个基准测试中实现了70.02%的顶尖平均性能，并在长期任务规划准确性上显著优于其他具象基础模型。我们的项目将发布以保证可重复性。

Towards High-DoF Dexterous Manipulation through VLA Post-Training

通过VLA后训练实现高景深灵巧操作

• Authors: Junlei Zhu, Shenzhe Yao, Chaogui Huang, Wenkai Zhu, Jingwei Peng, Guanqi He, Soren Schwertfeger, Jiahao Chen, Yide Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.19666
• Pdf link: https://arxiv.org/pdf/2609.19666
• Abstract Imitation-learned vision--language--action (VLA) foundation models acquire broad manipulation capabilities by scaling robot data across tasks and embodiments, but reliable deployment on a specific downstream task and hardware platform still requires post-training. Dexterous hands make this adaptation particularly difficult: their broad behavioural repertoire and high degree of freedom create a large and structured action space. Three obstacles are central: open-source VLAs do not natively provide an action interface for high-DoF hands; gesture mismatch during human-gated DAgger takeover creates command discontinuities and contaminates corrective trajectories; and reinforcement learning in the raw joint space is sample-inefficient. We present a unified four-step post-training pipeline comprising a learned temporal hand-action codec, supervised fine-tuning, DAgger, and real-world residual reinforcement learning. The codec adapts a pretrained VLA to absolute dexterous-hand commands. Buffered rollback, pose alignment, and smooth command blending enable continuous, task-relevant DAgger corrections, while latent residual RL confines exploration to coordinated hand motions captured by the codec. We evaluate the pipeline on five diverse real-world tasks spanning bimanual transfer, in-hand reorientation, and tool use. Within the reported post-training budgets, the resulting policies achieve 100\% success on every evaluated task over 20 trials per task. These results provide a practical path for adapting VLA foundation models to reliable real-world dexterous manipulation.
• 中文摘要 模仿学习的视觉-语言-动作（VLA）基础模型通过跨任务和具象扩展机器人数据获得广泛的操作能力，但在特定下游任务和硬件平台上的可靠部署仍需后期训练。灵巧的手使这种适应尤为困难：其广泛的行为库和高度自由度创造了一个庞大且结构化的动作空间。三个核心障碍是：开源VLA未能原生为高景深手提供动作接口;人门控DAgger接管时的手势不匹配造成指令不连续并污染纠正轨迹;原始关节空间中的强化学习样本效率低。我们呈现了一个统一的四步训练后流程，包括学习的时序手动作编解码器、监督式微调、DAgger和现实世界的残余强化学习。该编解码器将预训练的VLA适配为绝对灵巧手指令。缓冲回滚、姿态对齐和平滑指令混合使DAgger修正持续且符合任务相关，而潜在残余强化学习则限制探索至编解码器捕获的协调手部动作。我们评估了五个不同真实世界任务的流程，涵盖双手转移、手中重新定位和工具使用。在报告的训练后预算内，所有评估任务在20次试验中均实现100%成功率。这些结果为将VLA基础模型适应可靠且真实的灵巧操作提供了实用路径。

UniExo: Unified Multi-Skill Policies for Musculoskeletal Locomotion and Co-Adaptive Exoskeleton Control

UniExo：肌肉骨骼运动与共适应外骨骼控制的统一多技能政策

• Authors: Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.19690
• Pdf link: https://arxiv.org/pdf/2609.19690
• Abstract Daily locomotion encompasses diverse activities and frequent transitions between them, yet most exoskeleton controllers are designed for a single activity or a narrow set of related movements. Changes in activity therefore typically require explicit mode switching and separately tuned or retrained controllers. Simulation-based learning reduces the need for hardware-based tuning but generally retains this limitation. Here we present UniExo, a framework that first constructs a multi-skill musculoskeletal human policy and then jointly trains an exoskeleton control policy with it. Four single-skill imitation experts for walking, turning, running and backward walking are distilled into a single network structured by a skill latent and subsequently fine-tuned through reinforcement learning on transition sequences. The resultant unified human policy achieves a mean tracking success rate of 94.7% on unseen clips of the four skills and exhibits greater robustness to perturbations than its constituent experts. A single hip exoskeleton controller (UniExo) is initialized from hip moment prediction of the human policy and co-adapted with it through multi-agent reinforcement learning across the four skills. This co-adaptation shifts the timing of the assistance torque and raises the fraction of positive work delivered to the hip. When deployed on a custom hip exoskeleton, the controller generalizes across four treadmill speeds in six participants and assists one participant through a continuous route of all four skills and their transitions, without skill labels or explicit mode switching. UniExo thus provides a step towards replacing activity-specific controllers with unified, user-specific controllers that support diverse locomotor activities and the transitions between them.
• 中文摘要 日常运动涵盖了多样化的活动和它们之间的频繁转换，但大多数外骨骼控制器设计用于单一活动或狭窄的相关动作集合。因此，活动变化通常需要显式模式切换，并对控制器进行单独调优或重新训练。基于模拟的学习减少了基于硬件的调谐需求，但通常仍保留这一限制。这里介绍UniExo，一个先构建多技能肌肉骨骼人类策略，然后与之联合训练外骨骼控制策略的框架。四个单一技能模拟专家，分别负责行走、转弯、跑步和倒后行，被提炼成一个由潜在技能构建的单一网络，随后通过对过渡序列的强化学习进行微调。最终的统一人类策略在四项技能的未公开剪辑上实现了94.7%的平均追踪成功率，并且对干扰的韧性优于其组成专家。单一髋外骨骼控制器（UniExo）由人类策略的髋矩预测初始化，并通过多智能体强化学习在四项技能间共同适应。这种协同适应调整了辅助扭矩的时机，提高了传递给髋部的正向工作比例。当该控制器部署在定制的髋外骨骼上时，可以推广六名参与者的四种跑步机速度，并协助一名参与者连续贯穿四项技能及其转换路径，无需技能标签或显式模式切换。因此，UniExo为用统一的用户专用控制器取代活动专用控制器迈出了一步，支持多样化的运动活动及其间的转换。

Region-Level Policy Optimization for Fine-grained MLLM Perception

区域级政策优化以实现细粒度MLLM感知

• Authors: Yuheng Shi, Xiaohuan Pei, Minjing Dong, Chang Xu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.19745
• Pdf link: https://arxiv.org/pdf/2609.19745
• Abstract Fine-grained visual perception in MLLMs is commonly improved by raising the resolution, but the added visual tokens inflate vision-encoding and language-model prefilling costs. We show that the two operations underlying fine-grained perception, localizing the region of interest (RoI) and recognizing its content, have different resolution requirements. In a controlled diagnostic, localization tolerates roughly 3 to 4 times stronger token compression than recognition, which motivates localizing from a coarse view and concentrating resolution on the selected evidence. Decoding coordinates with the MLLM can be trained end-to-end from answers, but costs a full model pass per query and depends on grounding ability. A lightweight proposal network distilled from the model's attention is fast, but inherits the noise of its attention targets. The RoI from the proposal network reaches the answer through a discrete region choice, so its faithfulness to the answer cannot supervise the network. We therefore optimize the proposal network with region-level reinforcement learning, which we call Vision-RL2. It treats coherent regions as actions, and a frozen MLLM reader scores each one by how its removal changes the answer likelihood. Complementary subtractive and additive objectives suppress distracting proposals and recover missing evidence, updating only the predictor without region annotations, response sampling, or reasoning trajectories. The refined proposal further enables a sparse encoding that magnifies evidence and excludes background tokens. Across six fine-grained benchmarks and four MLLM backbones, Vision-RL2 improves accuracy over the base model at every token budget and surpasses its largest-budget accuracy with about 4 times fewer visual tokens. Code is available at this https URL .
• 中文摘要 MLLM中细粒度视觉感知通常通过提高分辨率来改善，但新增的视觉标记会增加视觉编码和语言模型预填充的成本。我们展示了细粒度感知背后的两种操作——关注区域（RoI）定位和内容识别——具有不同的分辨率要求。在受控诊断中，定位能容忍的标记压缩强度大约是识别的3到4倍，这促使从粗略视角进行局部化，并将分辨率集中在所选证据上。与MLLM的解码坐标可以从答案端到端训练，但每次查询需完成一次完整的模型通行，且依赖于接地能力。由模型注意力提炼出的轻量级提案网络速度快，但会继承其注意力目标的噪声。提案网络的投资回报通过离散区域选择达到答案，因此其对答案的忠实度无法监督网络。因此，我们通过区域级强化学习优化提案网络，称之为Vision-RL2。它将相干区域视为动作，冻结的MLLM阅读器根据移除对答案似然的变化对每个区域进行评分。互补的减法和加法目标抑制分散注意力的提案，恢复缺失的证据，仅更新预测变量，无需区域注释、响应抽样或推理轨迹。该优化方案进一步实现稀疏编码，放大证据并排除背景标记。通过六个细粒度基准和四个MLLM骨干，Vision-RL2在每个标记预算下提升了基础模型的准确性，并以约4倍的视觉标记数量超越最大预算准确率。代码可在此 https 访问 。

DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum

DeliveryGym：一个具备长期具身代理规划的强化学习环境，结合自适应课程

• Authors: Haoqiang Kang, Yiming Zhang, Yiyang Guo, Chuying Li, Jianzhi Shen, Tianruo Rose Xu, Xiaokang Ye, Lianhui Qin
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.19801
• Pdf link: https://arxiv.org/pdf/2609.19801
• Abstract Executable environments enable LLM agents to learn from the consequences of their actions. For embodied agents, those consequences extend beyond whether the current task succeeds: completing a delivery can consume the time, energy, or money needed for later work. Learning to plan therefore requires environments that preserve these dependencies and turn them into feedback across a complete trajectory. We introduce DeliveryGym, a 3D environment for evaluating and training agents on continuous courier shifts. It couples multimodal tool interaction with persistent world dynamics and computes trajectory rewards from simulator events, making the costs of an agent's decisions available for reinforcement learning (RL). The environment also adapts future training shifts to the policy's observed weaknesses while keeping evaluation fixed. Across six models and 13 city maps, evaluation exposes a gap between reliably executing assigned deliveries and choosing and sequencing work over a shift. On the fixed test suite, RL improves Qwen3-VL-4B's net income by 54.3%, showing that learning from complete shifts improves performance under these coupled constraints. Adapting the training environment improves test income by 16.5% over uniform sampling at the same rollout budget, indicating that which situations an agent practices also matters. DeliveryGym provides an executable setting for studying how agents learn to coordinate deliveries and preserve resources for later orders within an episode.
• 中文摘要 可执行环境使LLM代理能够从其行为的后果中学习。对于具身代理来说，这些后果不仅限于当前任务是否成功：完成一次交付可能消耗后续工作所需的时间、精力或金钱。因此，学习规划需要环境能够保留这些依赖关系，并将其转化为贯穿整个轨迹的反馈。我们介绍了DeliveryGym，一个用于评估和训练代理持续快递班次的三维环境。它将多模态工具交互与持续世界动态结合，并从模拟器事件中计算轨迹奖励，使代理决策的成本可供强化学习（RL）使用。该环境还会根据策略观察到的弱点调整未来的训练班次，同时保持评估固定。在六个模型和13个城市地图上，评估揭示了可靠执行分配交付与选择和排序工作之间的差距。在固定测试套件中，RL将Qwen3-VL-4B的净收益提升了54.3%，显示从完全班次中学习能提升在这些耦合约束下的表现。调整训练环境相比于同一部署预算下均匀采样，测试收入提升16.5%，表明代理所执行的情境也很重要。DeliveryGym提供了一个可执行的设置，用于研究代理如何学习协调交付并为后续订单保留资源。

Dual-Axis Policy Optimization for LLM Agents: Bayesian Feedback Attribution and Trajectory Mass Normalization

LLM代理的双轴策略优化：贝叶斯反馈归因与轨迹质量归一化

• Authors: Yingxuan Zhuang, Binhe Yu, Jingxiao Yang, Ruopei Sun, Ziting Li, Cheng Tan, Xuhong Zhang, Jianwei Yin, Jintao Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.19830
• Pdf link: https://arxiv.org/pdf/2609.19830
• Abstract Reinforcement learning for LLM agents involves two distinct optimization di- mensions: how environment feedback is exploited within a trajectory, and how complete trajectories are aggregated across a batch. We formulate these dimen- sions as Intra-Trajectory Feedback Attribution and Inter-Trajectory Objec- tive Aggregation, and introduce BATON (Bayesian Attribution and Trajectory Objective Normalization), a dual-axis policy optimization framework. BATON instantiates the first axis with Bayesian Feedback Attribution, which constructs a feedback-conditioned posterior over sampled actions, and the second with Trajec- tory Mass Normalization (TMN), which assigns equal optimization mass to com- plete trajectories. Experiments with GRPO and GiGPO on ALFWorld, WebShop, and SearchQA show that both axes provide independent gains and that their combi- nation consistently achieves the strongest overall performance across model scales.
• 中文摘要 LLM代理的强化学习涉及两种不同的优化维数：如何在轨迹中利用环境反馈，以及如何在整个批次中汇总完整轨迹。我们将这些维数定义为轨迹内反馈归因和轨迹间目标聚合，并引入BATON（贝叶斯归因与轨迹目标归一化），这是一种双轴策略优化框架。BATON用贝叶斯反馈归因实现第一个轴，构建一个反馈条件的采样后验;第二个轴则用轨迹质量归一化（TMN），为完整轨迹分配相等的优化质量。在ALFWorld、WebShop和SearchQA上对GRPO和GiGPO的实验显示，这两个轴都能独立获得收益，且其组合状态在模型尺度上始终实现最强的整体性能。

GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control

GR2PO：连续机器人控制的组相对返回策略优化

• Authors: Pengqin Wang, Qiming Zhang, Shaojie Shen, Jun Ma
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.19850
• Pdf link: https://arxiv.org/pdf/2609.19850
• Abstract Actor-critic architecture has been widely used in continuous robot control. However, they rely on learning a value network, introducing additional computational overhead during training. Moreover, policy learning may also be affected by the approximation error of value estimation. Critic-free group relative policy optimization methods provide a simpler training approach by removing the need for a critic. However, they fail to learn long-term action outcomes when directly applying immediate rewards to policy optimization in dense-reward environments. To address these problems, we propose Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control. GR2PO estimates the discounted returns from the parallelly collected trajectories, performs group normalization at each rollout time index, and uses relative advantages and clipped targets to update the policy. To evaluate the effectiveness of the proposed framework, we instantiate it on robot control simulation environments and deploy the model to a real-world edge device. The results show that GR2PO significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods. Furthermore, GR2PO demonstrates competitive training efficiency. Inference tests on NVIDIA Jetson TX2 demonstrate the feasibility of deploying the learned policies on edge platforms. Further ablation experiments analyze the effects of parallel group size, return estimation methods, and target clipping ratio on learning performance. To support follow-up research, we will make the complete code publicly available after the paper is accepted, including the framework implementation, experimental configuration, and training and evaluation scripts.
• 中文摘要 演员-批判者架构已被广泛应用于连续机器人控制。然而，它们依赖于学习价值网络，这在训练过程中增加了额外的计算开销。此外，策略学习还可能受到价值估计的近似误差的影响。无批判群相对策略优化方法通过消除对批判者的需要，提供了更简单的训练方法。然而，在高回报环境中直接将即时奖励应用于策略优化时，它们无法学习长期行动结果。为解决这些问题，我们提出了群相对回报策略优化（GR2PO），这是一种用于连续机器人控制的无批判强化学习框架。GR2PO估算并行收集轨迹的折现回报，在每个推广时间指数进行群体归一化，并利用相对优势和截断目标更新策略。为评估所提框架的有效性，我们将该框架实例化于机器人控制仿真环境中，并将模型部署到现实边缘设备。结果显示，GR2PO在无批评的即时奖励基线中表现显著优于，并且在与最先进的actor-critic方法中具有竞争力。此外，GR2PO还展现了具有竞争力的训练效率。在NVIDIA Jetson TX2上的推理测试展示了在边缘平台上部署所学策略的可行性。进一步的消融实验分析了并行组规模、返回估计方法和目标裁剪比对学习表现的影响。为支持后续研究，论文被接受后，我们将公开完整代码，包括框架实现、实验配置以及训练和评估脚本。

Learning Reliable Parking Policies via Offline Reinforcement Learning with Quantized Action Representations

通过带有量化行动表征的离线强化学习学习可靠的停车政策

• Authors: Zewei Yang, Zengqi Peng, Jun Ma
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.19894
• Pdf link: https://arxiv.org/pdf/2609.19894
• Abstract Parking is a routine yet safety-critical task for autonomous vehicles operating in urban environments. However, cluttered and weakly structured parking spaces, compounded by the interactive uncertainty from surrounding vehicles, hinder reliable maneuver generation. To address these challenges, we develop a waypoint-level offline reinforcement learning framework for interaction-aware autonomous parking. Specifically, a dedicated parking dataset is constructed from hierarchical expert rollouts with rotational waypoint augmentation, covering both non-interactive scenarios and interactive ones. The policy is then conditioned on a compact state representation, in which LiDAR-based obstacle features are adapted to the target pose via feature-wise linear modulation. A state-conditioned tokenizer further quantizes continuous waypoint sequences into discrete action tokens, over which conservative Q-learning is performed to suppress value overestimation on poorly supported actions. Extensive closed-loop experiments are conducted in the high-fidelity CARLA simulator. The proposed framework attains the highest parking success rate among all baselines and transfers reliably to unseen parking slots.
• 中文摘要 停车是城市环境中自动驾驶车辆的常规且安全关键任务。然而，杂乱且结构薄弱的停车空间，加上周围车辆的交互性不确定性，阻碍了可靠的机动生成。为应对这些挑战，我们开发了一个基于交互感的自主停车的路径点级离线强化学习框架。具体来说，由层级专家推送和旋转路径点增强构建专用停车数据集，涵盖非交互场景和交互场景。该策略基于紧凑状态表示，基于激光雷达的障碍物特征通过特征层线性调制调整至目标姿态。状态条件分词器进一步将连续路径点序列量化为离散动作标记，对其进行保守Q学习以抑制支持不足动作中的价值高估。在高精度CARLA模拟器中进行了大量闭环实验。该框架在所有基线中实现了最高的停车成功率，并且能够可靠地转移到看不见的停车位。

VERA: Reinforcement Learning for Dynamic Memory Scaling of HPC Workloads in Kubernetes

VERA：Kubernetes 中 HPC 工作负载动态内存扩展的强化学习

• Authors: Ade Pramono, Jie Ren, Ivy Peng
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.19936
• Pdf link: https://arxiv.org/pdf/2609.19936
• Abstract Memory over-provisioning results in resource underutilization when HPC workloads run on Kubernetes. The default Vertical Pod Autoscaler (VPA) cannot anticipate phase-driven memory spikes for first-run HPC jobs. In this work, we present a reinforcement learning (RL) recommender VERA that formulates vertical memory scaling as a Markov Decision Process and trains an agent on 3353 real Prometheus traces. Evaluated on a live Google Kubernetes Engine cluster using LAMMPS, graph analytics, in-memory analytics, and MLPerf 3D-UNet, the RL agent reclaims 31.6% of the available memory headroom and incurs at most one OOM event while VPA reclaims -7.9% over the same runs, raising memory provisioning, and its recommendation would have been insufficient to avoid OOM in 30 runs. The results demonstrate that an observation-driven RL recommender could outperform retrospective heuristics for dynamic memory scaling.
• 中文摘要 内存过载会导致在 Kubernetes 上运行 HPC 工作负载时资源利用率不足。默认的垂直 Pod 自动缩放器（VPA）无法预见首运行 HPC 作业的相位驱动内存尖峰。在本研究中，我们提出了一个强化学习（RL）推荐 VERA，将垂直内存缩放表述为马尔可夫决策过程，并训练代理使用 3353 条真实 Prometheus 跟踪。在实时的 Google Kubernetes Engine 集群上，使用 LAMMPS、图分析、内存内分析和 MLPerf 3D-UNet 评估，RL 代理回收了 31.6% 的可用内存余量，最多产生一次 OOM 事件，而 VPA 在相同运行中回收 -7.9%，提高了内存配置，其建议不足以避免 30 次运行时出现 OOM。结果表明，基于观察的强化学习推荐器在动态记忆尺度方面可能优于回顾性启发式方法。

EPIG-Tree: Compute-Optimal Branching for Gradient-Efficient Reinforcement Learning

EPIG树：用于梯度高效强化学习的计算最优分支

• Authors: Nikita Khomich, Leopold Hermansson, Ido Hakimi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.20004
• Pdf link: https://arxiv.org/pdf/2609.20004
• Abstract Reward-based reinforcement learning for language models, exemplified by Group Relative Policy Optimization (GRPO), collapses an entire stochastic trajectory into a single scalar reward. This is clean and scalable, but it explores and allocates reward inefficiently: a trajectory may contain many causal decisions, recovery attempts, and environment-randomness events, yet every token or action inherits one trajectory-level advantage. We study tree-based rollout construction as a compute-allocation problem for policy-gradient estimation. Our central claim is that branches should be placed not where the policy is merely uncertain, but where an additional branch most reduces uncertainty about the policy gradient per unit of compute. From a law-of-total-variance decomposition of the local policy-gradient random variable, we derive two allocation laws: new branches reduce decision uncertainty, while repeated suffix rollouts reduce continuation uncertainty. The resulting EPIG-Tree score allocates branches using the already computed rollouts. It estimates occupancy- and score-weighted value uncertainty, along with a suffix law $n_e \propto w_e \|\nabla_\theta \log \pi(a_e|h_e)\| \sigma_e / \sqrt{c_e}$. Empirically, EPIG reduces gradient MSE in cloned-state control, winning in all nine dense continuous-control environments of a 13-environment sweep and recovering the reference gradient direction near-perfectly, and it improves frozen-LLM gradient calibration relative to entropy branching. In online single-turn math, tree-local credit beats flat GRPO, while branch placement is secondary to token-level credit assignment. In online multi-turn Wordle, EPIG attains the highest final win rate (0.850), overtaking flat GRPO, which saturates early at 0.790, and entropy branching as training proceeds, confirming that the gradient-estimation advantage transfers to a stateful, large-action setting.
• 中文摘要 基于奖励的语言模型强化学习，以群体相对策略优化（Group Relative Policy Optimization，GRPO）为代表，将整个随机轨迹压缩为单一标量奖励。这种方式干净且可扩展，但探索和分配奖励效率低下：一条轨迹可能包含许多因果决策、恢复尝试和环境随机事件，但每个代币或动作都继承一个轨迹层级优势。我们将基于树的展开构建作为策略梯度估计的计算分配问题进行研究。我们的核心主张是，分支不应仅仅放置在策略不确定的地方，而应在一个额外分支最能最大限度地减少每单位计算策略梯度不确定性的地方。通过局部策略梯度随机变量的总方差定律分解，我们推导出两种分配定律：新分支减少决策不确定性，而重复后缀滚动减少延续不确定性。所得的EPIG树评分利用已计算的扩展分配分支。它估计占用率和分数加权价值不确定性，以及后缀定律$n_e \propto w_e \|nabla_\ \theta \log \pi（a_e|h_e）\|\sigma_e / \sqrt{c_e}$。经验上，EPIG在克隆态控制中降低了梯度MSE，在13环境扫描的九个密集连续控制环境中获胜，并几乎完美恢复参考梯度方向，同时提升了相对于熵分支的冻结LLM梯度校准。在线单回合数学中，树-局部信用优于单一GRPO，而分支放置次要于代币级信用分配。在线多回合Wordle中，EPIG获得最高最终胜率（0.850），超过了早期饱和于0.790的平坦GRPO，并随着训练出现熵分支，确认梯度估计优势能转移到有状态、大动作环境中。

MAGMA-GEN: Validated Recovery Supervision from Ambiguous Failures via Counterfactual Re-Execution

MAGMA-GEN：通过反事实重执行验证的模糊失败恢复监督

• Authors: Loan Bernat (LAAS-GEPETTO), Matthieu Grard, Ariane Herbulot (LAAS-RAP), Florent Lamiraux (LAAS-GEPETTO)
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.20056
• Pdf link: https://arxiv.org/pdf/2609.20056
• Abstract Hierarchical robotic systems executing long-horizon manipulation tasks must make high-level semantic decisions that orchestrate stochastic low-level skills. In this setting, failed rollouts are ambiguous: a poor downstream state may reflect an invalid high-level decision, partial observation, or a valid decision whose physical execution failed. Traditional supervised learning lacks data for such recovery states, while reinforcement learning struggles with sparse rewards and non-local credit assignment. We propose MAGMA-GEN, an on-policy data-generation pipeline that converts ambiguous failed rollouts into validated recovery supervision. MAGMA-GEN first uses a privileged coach to hypothesize an early decision-level error and propose localized correction or recovery actions. Because this diagnosis is fallible, candidates are retained only if re-execution from the same state under matched conditions improves downstream progress. This produces supervised examples from the agent's own failure distribution without per-step human demonstrations. Evaluated on interactive long-horizon manipulation tasks, MAGMA-GEN improves task success and recovery capabilities, against distillation and trajectory-repair baselines under evolving task constraints in both simulation and real-robot execution.
• 中文摘要 执行长视野操作任务的分层机器人系统必须做出高层语义决策，以协调随机的低层技能。在此环境中，失败的推广是模糊的：糟糕的下游状态可能反映无效的高层决策、部分观察，或物理执行失败的有效决策。传统的监督学习缺乏此类恢复状态的数据，而强化学习则在奖励稀疏和非本地学分分配方面苦恼。我们提出MAGMA-GEN策略数据生成流水线，将模糊失败的推广转化为验证的恢复监督。MAGMA-GEN首先使用特权教练假设早期决策层错误，并提出局部纠正或恢复措施。由于该诊断可能出错，只有在匹配条件下从同一状态重新执行能改善后续进展时，候选人才被保留。这基于智能体自身的失败分布生成监督示例，无需逐步人工演示。通过交互式长视野操作任务评估，MAGMA-GEN在模拟和真实机器人执行中，针对蒸馏和轨迹修复基线，提升任务成功率和恢复能力。

MATCH: Model-Aware Tool Learning with Curriculum Scheduling and Hierarchically Gated Rewards

匹配：模型感知工具学习，结合课程安排和分层奖励

• Authors: Shihao Liu, Hao Yin, Lijun Liu, Zhengzong Chen, Yuanyuan Zhao, Fei Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.20082
• Pdf link: https://arxiv.org/pdf/2609.20082
• Abstract Tool learning enables large language models (LLMs) to use external tools for tasks beyond parametric knowledge. Reinforcement learning can optimize tool-call behavior from feedback, but current methods still face two problems: fixed-threshold curricula can become misaligned with the policy's evolving capability boundary, and additive rewards can leak argument-level credit when the predicted tool is wrong. To address these problems, we propose MATCH, a closed-loop framework for model-aware tool learning with curriculum scheduling and hierarchically gated rewards. Model-Aware Curriculum Learning (MACL) maintains reward-derived sample difficulty that co-evolves with the policy, and each epoch selects samples near the current capability boundary together with a top-k pool of harder cases. Hierarchical Tool-call Gated Reward (HTGR) scores tool name, argument key, and argument value as a gated chain, granting credit at each level only when prerequisites hold. The same HTGR rewards drive both GRPO updates and MACL's difficulty refresh, closing the loop between policy optimization and sample scheduling. On API-Bank and BFCL V3, MATCH reaches 72.19% and 62.87% overall accuracy, outperforming the main supervised and RL-based baselines. Backbone experiments further show consistent improvements across four backbones from two model families.
• 中文摘要 工具学习使大型语言模型（LLMs）能够使用外部工具完成超出参数化知识的任务。强化学习可以从反馈中优化工具调用行为，但现有方法仍面临两个问题：固定阈值课程可能与策略不断演变的能力边界不匹配，加法奖励在预测工具错误时可能流失论证级的学分。为解决这些问题，我们提出了MATCH框架，这是一个闭环框架，用于模型感知工具学习，具备课程调度和层级门槛奖励。模型感知课程学习（MACL）保持与策略共同演化的奖励衍生样本难度，每个时代选择接近当前能力边界的样本，并结合一个顶K级更难案例池。层级工具调用门控奖励（HTGR）以门控链形式评分工具名称、参数键和参数值，只有在前提条件成立时才在各级授予信用。相同的HTGR奖励驱动GRPO更新和MACL难度刷新，实现策略优化与样本调度之间的闭环。在API-Bank和BFCL V3中，MATCH准确率分别达到72.19%和62.87%，优于主要监督基准和基于强化学习的基线。骨干实验进一步显示，来自两个模型族的四个骨干链均有持续的提升。

Safety-Critical Scenanrio Emerges from Initial Scene

安全关键的场景从初始场景中浮现

• Authors: Yin Wu, Jiarong Wei, Carl Esselborn, Shubham Phoolari, Ahmed Abouelazm, Daniel Slieter, J. Marius Zöllner
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.20103
• Pdf link: https://arxiv.org/pdf/2609.20103
• Abstract Safety-critical driving scenario generation has largely focused on manipulating the behavior of surrounding agents while starting from an initial scene from driving data. This assumption can limit the space of discoverable failures, since driving data can provide little opportunity for meaningful interaction. For example, in the Waymo Open Motion Dataset, 20.44% of recorded slices feature a stationary ego vehicle that never moves, and 30.39% of initial frames contain no nearby traffic participants within 10 meters. We instead study safety-critical scenario generation as an initialization problem: given agnostic black-box driving policies, we learn to generate realistic initial scenes that are more likely to evolve into critical interactions. We propose AdvScene, a conditional latent diffusion model that is trained in two stages. Starting from pretraining on naturalistic driving data, we post-train the adversarial-agent generation branch using reinforcement learning with feedback from closed-loop simulator rollouts. Conditioning on ego driving displacement prevents the ego from remaining static, and RL finetuning induces criticality directly with non-differentiable safety-critical metrics. Experiments on the Waymo dataset across 12 combinations of ego and traffic policies show that our AdvScene substantially increases the rate of ego-fault collision events and TTC<3s events.
• 中文摘要 安全关键驾驶场景生成主要侧重于从驾驶数据的初始场景开始时，操控周围代理的行为。这一假设限制了可发现故障的空间，因为驾驶数据几乎无法提供有意义的交互机会。例如，在Waymo开放运动数据集中，20.44%的录制切片中出现了静止且永不移动的自我车辆，30.39%的初始帧10米内没有交通参与者。我们转而将安全关键场景生成作为初始化问题进行研究：在中立黑箱驾驶策略下，我们学习生成更有可能演变成关键交互的真实初始场景。我们提出了AdvScene，一种条件潜在扩散模型，分两个阶段进行训练。从对自然驾驶数据的预训练开始，我们利用强化学习并结合闭环模拟器推送的反馈，对对抗性-代理生成分支进行后期训练。对自我驱动位移的条件化防止自我保持静止，强化学习微调直接通过不可微分的安全关键指标诱导临界性。在Waymo数据集上，12种自我与交通政策组合的实验显示，我们的AdvScene显著提高了自我故障碰撞事件和TTC<3s事件的发生率。

AnyViewDex: View-Invariant Dexterous Manipulation from RGB Observations

AnyViewDex：RGB观测中的视野不变灵活操作

• Authors: Soham Patil, Om Sanjay Gunjal, Sourabh Bhosale, Arhan Chavare, Ramandeep Singh Hora, Spandan Roy
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.20107
• Pdf link: https://arxiv.org/pdf/2609.20107
• Abstract Visuomotor policies for multi-fingered dexterous manipulation are highly sensitive to camera viewpoint shifts. To achieve view invariance, recent methods increasingly rely on explicit 3D modalities like RGB-D or point clouds, which can introduce hardware dependencies, calibration requirements, and vulnerability to sensor noise during real-world deployment. In this work, we show that view-invariant control can be achieved without explicit test-time 3D sensing by encoding geometric knowledge into the visual representation during simulation. We present AnyViewDex, an asymmetric training pipeline that combines multi-view contrastive alignment with privileged 3D geometric supervision. By regressing absolute 3D object coordinates during simulated training, this auxiliary objective provides a geometric grounding signal that mitigates the spatial collapse of the globally pooled contrastive embedding. At deployment, the policy operates zero-shot using only uncalibrated monocular RGB and proprioception. We validate this approach across both reinforcement learning and student-teacher distillation. In hardware evaluation on an xArm7 with a 16-DoF LEAP Hand, AnyViewDex reaches 76.7% grasping success across eight unseen objects and six uncalibrated viewpoints (480 trials; 2,400 across all ablation conditions), indicating that geometrically grounded monocular policies transfer zero-shot without test-time depth. Project Page: this https URL
• 中文摘要 多指灵巧操作的视觉运动策略对摄像机视点变化极为敏感。为实现视野不变，近期方法越来越依赖显式三维模态，如RGB-D或点云，这可能引入硬件依赖、校准要求以及在实际部署时对传感器噪声的脆弱性。本研究展示了，在模拟过程中将几何知识编码到视觉表示中，无需显式测试时三维感知即可实现视野不变控制。我们介绍AnyViewDex，一种非对称训练流水线，结合了多视角对比对齐与特权三维几何监督。通过在模拟训练中回归绝对三维物体坐标，这一辅助目标提供了几何接地信号，减轻了全球对比嵌入的空间崩溃。部署时，该策略仅使用未校准的单眼RGB和本体感觉进行零拍摄。我们在强化学习和师生提炼中验证了该方法。在配备16度LEAP手的xArm7硬件评估中，AnyViewDex在8个未见物体和6个未校准视点（480次试验;2400次所有消融条件下）中实现76.7%的抓取成功率，表明几何基础单眼策略可在无测试时间深度的情况下传输零样本。项目页面：此 https URL

Multi-Dimensional Prosody Judgment For Live Streaming Speech Synthesis

直播语音合成的多维韵律判断

• Authors: Zifan Guan, Longyu Lu, Junan Zhang, Zhizheng Wu, Meiguang Jin, Junfeng Ma
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.20124
• Pdf link: https://arxiv.org/pdf/2609.20124
• Abstract Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy which traditional MOS predictors fail to capture. While proprietary Large Language Models (LLMs) like Gemini can evaluate these aspects, they are too costly for massive inference and reinforcement learning feedback. To address this, we first introduce Live-ProsodyJudge (LPJ), a cost-effective pairwise evaluator distilled from Gemini into Qwen3-Omni. However, we identify a critical flaw in standard multi-dimensional evaluation: verdict coupling. The judge tends to lazily align all individual dimension scores with its overall preference, collapsing a rich multi-dimensional rubric into a single preference bit. To resolve this, we further propose Decoupled-Live-ProsodyJudge (D-LPJ). D-LPJ eliminates the overall verdict target to prevent blind following, masks uncertain pair-dimensions during Supervised Fine-Tuning(SFT), and introduces a novel span-local GRPO strategy that applies normalized advantages strictly to their corresponding rationale spans. Evaluated on highly curated human-annotated test sets, 10 sample balanced-order LPJ achieves higher point accuracy than a single Gemini call, while D-LPJ successfully produces independent,decoupled dimension judgments. Furthermore, in a Best-of-8 TTS candidate selection tournament, the LPJ-selected utterance falls within the human top-3 in 85.29% of high-confidence cases, demonstrating its efficacy for fine-grained TTS preference optimization.
• 中文摘要 评估实时流语音合成（TTS）需要评估细粒度、高度表现力的韵律，如情感、语调和能量，这些传统MOS预测器未能捕捉到。虽然专有的大型语言模型（LLM）如Gemini可以评估这些方面，但它们的成本过高，无法进行大规模的推理和强化学习反馈。为此，我们首先引入了Live-ProsodyJudge（LPJ），这是一种从Gemini提炼到Qwen3-Omni的成本效益高的两对评估器。然而，我们发现标准多维评估中的一个关键缺陷是：判决耦合。Judge倾向于懒惰地将所有个体维度评分与其整体偏好对齐，将丰富的多维评分标准压缩为单一偏好位。为解决这个问题，我们进一步提出了Decoupled-Live-ProsodyJudge（D-LPJ）。D-LPJ消除了整体判决目标以防止盲跟踪，掩盖了监督微调（SFT）期间不确定的配对维度，并引入了一种新颖的span-local GRPO策略，将归一化优势严格应用于对应的理性范围。在高度策划的人工注释测试集上评估时，10个样本平衡阶LPJ的点精度优于单个Gemini调用，而D-LPJ成功产生独立、解耦的维度判断。此外，在8局三胜TTS候选人选择比赛中，LPJ选出的话语在85.29%的高置信度案例中位列人类前三名，展示了其在细粒度TTS偏好优化中的有效性。

Robust Federated Q-Learning with Almost No Communication

几乎无交流的稳健联邦Q-学习

• Authors: Sreejeet Maity, Aritra Mitra
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.20174
• Pdf link: https://arxiv.org/pdf/2609.20174
• Abstract We consider a federated reinforcement learning setting involving $M$ agents, all of whom interact with a common Markov Decision Process (MDP). The agents exchange information via a central server to learn the optimal value function. Our goal is to understand to what extent one can hope for collaborative sample-complexity speedups in such a setting, when a small fraction of the agents are adversarial and can act arbitrarily. To that end, we propose Robust Fed-Q}, a federated Q-learning algorithm that blends ideas from both model-based and model-free RL, along with the median-of-means device from robust statistics. We prove that despite corruption, with high-probability, Robust Fed-Q (i) guarantees exact convergence to the optimal value function in the limit of infinite samples, and (ii) enjoys near-optimal finite-time rates that benefit from collaboration. In addition, our approach requires just $\tilde{O}(1)$ rounds of communication to achieve each of the above guarantees, a feature of independent interest in FL where communication is the major bottleneck.
• 中文摘要 我们考虑一个联邦强化学习环境，涉及$M$代理，所有代理都与共同的马尔可夫决策过程（MDP）交互。代理通过中央服务器交换信息以学习最优价值函数。我们的目标是理解在此类环境中，当少数代理具有对抗性且可以任意行动时，协作样本复杂度的提升在多大程度上。为此，我们提出了Robust Fed-Q}，一种结合基于模型和无模型强化学习的联邦Q学习算法，以及稳健统计学中的均值中位数装置。我们证明尽管存在数据损坏，但高概率，Robust Fed-Q（i）在无限样本极限下保证精确收敛到最优值函数，并且（ii）拥有近乎最优的有限时间收敛率，从协作中受益。此外，我们的方法只需$\tilde{O}（1）$轮次的通信即可实现上述保证，这是佛罗里达独立关注的特点，因为通信是主要瓶颈。

VLN on the Fly: An Onboard Vision-Language Navigation Stack for Aerial Robots

VLN即时运行：用于空中机器人的机载视觉语言导航协议栈

• Authors: Marco S. Tayar, Felipe Tommaselli, Gianluca Capezutto, Pedro Antonio Rabelo Saraiva, Pedro H. V. de Freitas, Lucas Kido, Guilherme Sonego, Ricardo V. Godoy, Marcelo Becker
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.20191
• Pdf link: https://arxiv.org/pdf/2609.20191
• Abstract Running vision-language navigation fully onboard an aerial robot is hard, since grounding, planning, and control must share limited compute and a single-stage error is difficult to isolate in flight. End-to-end aerial policies fuse these stages into one network, giving up the observability and safety checks a modular stack keeps available. We propose VLN on the Fly, an onboard stack that keeps grounding, planning, and control as separate, inspectable stages. A quantized VLM grounds an instruction to a coarse image cell, depth lifts it to a 3D goal, a fast B-spline planner returns a feasible trajectory, and a pretrained reinforcement learning policy tracks it to motor commands across quadrotors. Across 15 onboard flights over three everyday referents in a controlled indoor volume, the stack reaches the target in 13 of 15 trials with 5.72 cm mean goal error and 39.3% average GPU utilization. In 6 additional cluttered-environment trials, the stack tracks collision-free trajectories under onboard perception gating.
• 中文摘要 在空中机器人上完全运行视觉语言导航很困难，因为接地、规划和控制必须共享有限的计算，且单级误差难以在飞行中隔离。端到端的航拍策略将这些阶段融合为一个网络，放弃了模块化堆栈所保留的可观测性和安全检查。我们提出了VLN on the Fly，一种机载栈，将接地、规划和控制作为独立且可检查的阶段。量化VLM将指令接地到粗图像单元，深度使指令提升到三维目标，快速B样条规划器返回可行轨迹，预训练强化学习策略将其追踪到跨四旋翼的电机指令。在15次机载飞行中，使用三个日常参考物，室内受控体积，堆栈在15次试验中13次达到目标，平均目标误差为5.72厘米，平均GPU利用率为39.3%。在另外6项杂乱环境试验中，堆栈在机载感知门控下追踪无碰撞轨迹。

Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillation

通过双向行为事先蒸馏提升在线强化学习

• Authors: Gong Gao, Xiao Lai, Jiaji Shen, Ning Jia, Xianhui Liu, Weidong Zhao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.20268
• Pdf link: https://arxiv.org/pdf/2609.20268
• Abstract Online reinforcement learning (RL) algorithms frequently exhibit poor sample efficiency and unstable learning dynamics, stemming from systematic critic estimation errors that are exacerbated by greedy policy updates. Existing behavior-prior reinforcement learning methods attempt to alleviate this issue by relying on offline pre-training to learn behavior models from fixed datasets and using policy priors to constrain online policy updates. However, the limited quality of offline datasets often hinders the ability to provide high-value policies that can effectively guide policy updates. The absence of expert trajectories significantly impairs online policy learning, leading to low sample efficiency and suboptimal performance. To address these challenges, we depart from conventional behavior prior approaches and propose a Bidirectional Behavior Prior Distillation (B2PD) algorithm. B2PD leverages action-value priors to guide a conditional variational autoencoder (CVAE) in generating a high-value behavior support set. The resulting expert behavior priors are further distilled into the agent, effectively reducing inefficient exploration and enabling stable policy optimization, while establishing a bidirectional knowledge flow mechanism. Empirical evaluations on both state- and pixel-based tasks verify that B2PD substantially improves sample efficiency while maintaining stable policy optimization. More broadly, this work shows that enforcing high-quality behavioral support during online learning effectively mitigates critic-induced error amplification, enabling structured behavior priors to guide policy updates in a principled and sample-efficient manner.
• 中文摘要 在线强化学习（RL）算法常表现出样本效率低和学习动态不稳定，源于系统性批评者估计错误，且被贪婪的策略更新加剧。现有行为先验强化学习方法试图通过离线预训练从固定数据集学习行为模型，并利用策略先验限制在线策略更新来缓解这一问题。然而，离线数据集质量有限，常常阻碍提供有效指导策略更新的高价值策略。缺乏专家轨迹显著削弱在线策略学习，导致样本效率低且性能不理想。为应对这些挑战，我们提出一种双向行为先验（B2PD）算法。B2PD利用动作值先验引导条件变分自编码器（CVAE）生成高价值行为支持集。由此产生的专家行为先验进一步提炼到代理中，有效减少低效的探索，实现稳定的策略优化，同时建立双向知识流机制。基于状态和像素任务的实证评估验证了B2PD在保持稳定策略优化的同时显著提升样本效率。更广泛地，这项工作表明，在线学习期间强制执行高质量行为支持有效减轻批评者引发的错误放大，使结构化行为先验能够以原则性和样本效率的方式指导策略更新。

Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation

通过边界感知数据增强提升离线强化学习中的泛化性和鲁棒性

• Authors: Gong Gao, Weidong Zhao, Xianhui Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.20300
• Pdf link: https://arxiv.org/pdf/2609.20300
• Abstract Current offline reinforcement learning (ORL) algorithms tend to overfit the training dataset and exhibit poor in-distribution generalization and robustness performance when deployed to real environments, thus compromising their effectiveness. Existing methods typically enhance in-distribution generalization and robustness by leveraging regularization techniques widely used in computer vision. However, due to the high sensitivity of low-level physical signals to distributional shifts, these methods still suffer from notable limitations in in-distribution generalization and robustness, making it difficult to achieve stable performance in complex environments. To address this issue, we theoretically analyze the error bounds of the behavior policy and action-value function trained with random episode interpolation, revealing that the error scales positively correlated with the distance between states. Based on this insight, we propose a method called $\bf{B}$oundary-$\bf{A}$ware $\bf{D}$ata $\bf{A}$ugmentation (BADA), which leverages neighboring states to construct interpolation boundaries, enabling the generation of synthetic data that more faithfully preserves the original data distribution. We first conduct qualitative studies in a toy environment, showing that BADA generates mixed samples that preserve desirable policy smoothness while accurately reconstructing multimodal value distributions. Extensive experiments on limited offline datasets further demonstrate that BADA attains state-of-the-art performance across diverse benchmarks.
• 中文摘要 当前离线强化学习（ORL）算法往往会过度拟合训练数据集，且在部署到真实环境中时表现较差，导致其效果受损。现有方法通常通过利用计算机视觉中广泛使用的正则化技术来增强分布内泛化和鲁棒性。然而，由于低级别物理信号对分布变化的高度敏感性，这些方法在分布内泛化和鲁棒性方面仍存在明显限制，难以在复杂环境中实现稳定性能。为解决这一问题，我们理论上分析了随机插值训练的行为策略和动作值函数的误差边界，发现误差与状态间距离呈正相关。基于这一见解，我们提出了一种名为$\bf{B}$oundary-$\bf{A}$ware $\bf{D}$ata $\bf{A}$ugmentation（BADA）的方法，该方法利用邻近状态构建插值边界，从而生成更忠实保存原始数据分布的合成数据。我们首先在玩具环境中进行定性研究，显示BADA生成混合样本，既保持理想的策略平滑性，又准确重建多模态值分布。在有限的离线数据集上进行的大量实验进一步证明，BADA在多样化基准测试中达到了最先进的性能。

DDQN-MLP: An Explainable and Adversarially Robust DRL-Guided Adaptive Learning Framework for Ransomware Detection

DDQN-MLP：一个可解释且具对抗性强的DRL引导自适应学习框架，用于勒索软件检测

• Authors: Jannatul Ferdous, Rafiqul Islam, Arash Mahboubi, Md Zahidul Islam
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.20314
• Pdf link: https://arxiv.org/pdf/2609.20314
• Abstract Ransomware detection remains challenging because modern variants exhibit diverse, evasive, and partly benign-like behaviors that undermine fixed supervised learning objectives. This study proposes DDQN-MLP, a training-time deep reinforcement learning framework for behavioral ransomware detection using Windows 11 sandbox telemetry. A Double Deep Q-Network (DDQN) acts as a discrete adaptive sample-weighting controller by observing batch-level loss and prediction-confidence dynamics and assigning sample-importance weights to guide a lightweight Multilayer Perceptron (MLP). After training, the DDQN is discarded, leaving only the efficient MLP for deployment. The framework was evaluated using 5-fold stratified cross-validation on a balanced dataset of 2,000 executable profiles comprising 1,000 ransomware samples from 30 families and 1,000 benign samples. DDQN-MLP achieved 99.30% accuracy, an F1-score of 0.9930, and an ROC-AUC of 0.9991, outperforming conventional static weighting, focal-loss, and alternative DRL variants. Explainability was assessed using SHAP and LIME, together with a SHAP-gradient alignment diagnostic for evaluating consistency between feature attribution and model sensitivity. White-box adversarial testing across multiple perturbation levels further showed that adversarial training improved feature-space robustness without reducing clean-data accuracy. The results demonstrate that DDQN-MLP provides an accurate, explainable, robust, and computationally efficient framework for high-throughput ransomware detection.
• 中文摘要 勒索软件检测依然具有挑战性，因为现代变种表现出多样、规避性且部分良性的行为，削弱了固定监督学习目标。本研究提出了DDQN-MLP，一种基于训练时间的深度强化学习框架，用于利用Windows 11沙箱遥测进行行为勒索软件检测。双深度Q网络（DDQN）通过观察批次级丢失和预测-置信度动态，并赋予样本重要性权重，以指导轻量级多层感知器（MLP），作为离散自适应样本加权控制器。训练完成后，DDQN被丢弃，只留下高效的MLP供部署。该框架通过五重分层交叉验证对一个平衡数据集进行评估，该数据集包含2000个可执行配置文件，包含来自30个家族的1000个勒索软件样本和1000个良性样本。DDQN-MLP实现了99.30%的准确率，F1得分为0.9930，ROC-AUC为0.9991，优于传统的静态加权、焦损及其他DRL变体。可解释性通过SHAP和LIME以及SHAP梯度比对诊断进行评估，以评估特征归因与模型敏感度之间的一致性。多扰动级别的白盒对抗测试进一步表明，对抗训练提升了特征空间的鲁棒性，同时不降低纯净数据的准确性。结果表明，DDQN-MLP为高通量勒索软件检测提供了一个准确、可解释、稳健且计算高效的框架。

Learning Principal-Agent Contracts for Equitable Smallholder Carbon Farming under Moral Hazard and Adverse Selection

学习道德风险和逆向选择下公平小农碳农业的主要代理人合同

• Authors: Rishi Bharadwaj, Yadati Narahari
• Subjects: Subjects: Machine Learning (cs.LG); Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2609.20404
• Pdf link: https://arxiv.org/pdf/2609.20404
• Abstract Agricultural soils are a major untapped carbon sink. Carbon farming is emerging as a promising practice for tapping this potential. Smallholder farmers, who dominate agriculture across South Asia and sub-Saharan Africa, are key to scaling climate mitigation via carbon farming. It is ironic that real-world carbon programs largely fail to reach them. We study this important gap through the lens of contract design. An aggregator offers a single pooled contract to a heterogeneous population of smallholder farmers who have private adoption costs (adverse selection) and exert unobserved effort (moral hazard), with agronomic outcomes evolving over multiple seasons. We formulate this evolving contracting problem as a POMDP and use reinforcement learning to learn a dynamic profit-maximising contract. We analyse the performance of the aggregator under various conditions. We find that a profit-maximising aggregator does not merely inherit the exclusion of smallholders, it amplifies it. On large farms the aggregator realises 87.7% of achievable adoption, against only 8.2% on smallholdings. Per-hectare Measurement, Reporting and Verification (MRV) costs fall as farm size rises, and the aggregator's pooling contract compounds this gradient rather than offsetting it. A counterfactual that makes MRV costs purely area-proportional eliminates this disparity. Our results and simulation can guide contract and policy design that opens carbon income to smallholders while enabling agricultural soils to contribute to climate mitigation at scale.
• 中文摘要 农业土壤是未被充分利用的重要碳汇。碳农业正逐渐成为开发这一潜力的有前景实践。小农主主导南亚和撒哈拉以南非洲的农业，是通过碳农业扩大气候缓解的关键。具有讽刺意味的是，现实中的碳项目大多未能触及这些差距。我们通过合同设计的视角研究这一重要空白。聚合商向一群具有私人收养成本（逆向选择）且付出无人观察努力（道德风险）且农艺结果在多个季节中演变的小农户群体提供单一合并合同。我们将这一不断演变的合同问题提出为POMDP，并利用强化学习学习动态利润最大化合同。我们分析聚合商在各种条件下的表现。我们发现，利润最大化的聚合者不仅继承了小农户的排除，还放大了这一差异。在大型农场，聚合者实现了87.7%的可实现采用率，而小农场仅为8.2%。每公顷测量、报告和核查（MRV）成本随着农场规模增加而下降，聚合商的合并合同使这一梯度增加而非抵消。将MRV成本完全按面积比例计算的反事实因素消除了这一差异。我们的结果和模拟可以指导合同和政策设计，既开放碳收入给小农户，又使农业土壤能够大规模参与气候缓解。

The Bias of Nonlinear Two-Time-scale Stochastic Approximation under Constant Step-Sizes

在恒定步长下非线性两时间尺度随机近似的偏置

• Authors: Djamel Rassem Lamouri, Dorian Baudry, Nicolas Gast
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.20409
• Pdf link: https://arxiv.org/pdf/2609.20409
• Abstract Two-timescale stochastic approximation (TTSA) is a fundamental tool for analyzing coupled iterative algorithms in reinforcement learning, optimization, and stochastic control. However, finite-time guarantees for nonlinear two-timescale schemes remain difficult to obtain, especially under constant step-sizes. In this paper, we study nonlinear TTSA with step-sizes $\alpha\gg\beta$. Under standard stability, regularity, and Markovian noise assumptions, we upper bound the mean-squared error and the bias of both iterates around their limiting equilibria. Our bounds scale as $O(\alpha+\beta^2/\alpha^2)$, which we prove to be tight when $\beta\le\alpha^{3/2}$. The analysis separates the contributions of initial conditions, fast-timescale tracking error, Markovian dependence, and timescale coupling, thereby clarifying the origin of the $\beta^2/\alpha^2$ term. Our results reveal qualitative differences from the linear TTSA setting previously studied, showing that nonlinear dynamics introduce additional finite-time effects that are absent in the linear case.
• 中文摘要 两时间尺度随机近似（TTSA）是分析强化学习、优化和随机控制中耦合迭代算法的基本工具。然而，对于非线性二时间尺度方案，有限时间保证仍然难以获得，尤其是在步长恒定的情况下。本文研究步长为$\alpha\gg\beta$的非线性TTSA。在标准稳定性、正则性和马尔可夫噪声假设下，我们对均方误差进行了上界，两者的偏差绕其极限均衡进行迭代。我们的上界为$O（\alpha+\beta^2/\alpha^2）$，当$\beta\le\alpha^{3/2}$时，我们证明其紧密。分析分离了初始条件、快速时间尺度跟踪误差、马尔可夫依赖性和时间尺度耦合的贡献，从而澄清了$\β^2/\alpha^2$项的起源。我们的结果揭示了与之前研究的线性TTSA设置的质的不同，表明非线性动力学引入了线性情况下不存在的额外有限时间效应。

Visual Sim-to-Real Learning for Robotic Insertion under Geometric Variations: Application to Rebar Installation

在几何变分下机器人插入的可视化模拟到真实学习：在钢筋安装中的应用

• Authors: Tao Sun, Beining Han, Patrick Yin, Rui Xu, Harry He, Abhishek Gupta, Szymon Rusinkiewicz, Yi Shao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.20477
• Pdf link: https://arxiv.org/pdf/2609.20477
• Abstract Rebar insertion is among the most repetitive and physically demanding tasks on construction sites, and a contact-rich problem at 1.4 mm clearance. The parts, however, vary at two levels: a nominal design per structural member, and fabrication tolerance around each nominal design. Real-world data therefore has to be re-collected as designs and batches change. We present RebarSim, a visual sim-to-real system trained entirely in simulation. A privileged state-based teacher is trained with reinforcement learning over procedurally generated rebar geometries, then distilled into a multi-view student that maps raw RGB and proprioception directly to actions under extensive domain randomization. The student transfers to the real world zero-shot, seating rebars taken from a real factory production run in 91.3% of real-robot rollouts. Underlying that result, geometry diversity and pretraining both bring benefits. Training across a diverse set of nominal designs rather than one lifts the zero-shot success of both the teacher and the student on unseen designs, and the student policy outperforms a single-design specialist on that specialist's own design. A pretrained student then adapts to a new design with 4--6x fewer distillation samples than one trained from scratch. Visual sim-to-real transfer depends on appearance randomization and the DAgger mixture: removing either one sharply lowers success. Videos, code, and task assets are available at this https URL.
• 中文摘要 钢筋插入是建筑工地上最重复且体力要求极高的任务之一，且在1.4毫米间隙时接触性较大。然而，这些部件在两个层面上有所不同：每个结构构件的标称设计，以及围绕每个标称设计的制造公差。因此，随着设计和批次的变化，必须重新收集真实世界的数据。我们介绍RebarSim，一个完全由仿真训练的可视化模拟到现实系统。一位特权基于状态的教师接受对过程生成钢筋几何的强化学习培训，然后被提炼成多视角学生，将原始RGB和本体感觉直接映射到经过广泛领域随机化的动作。学生在91.3%的真实机器人推广中，将零射点、座位钢筋从真实工厂生产中迁移到。在这一结果基础上，几何多样性和预训练都带来了好处。在多样的名义设计中进行培训，而非单一设计，提升了师生在未见设计上的零样本成功率，而学生政策在该专家自身设计上表现优于单一设计专家。预培训学生随后适应新设计，蒸馏样本数量比从零训练的学生少4至6倍。视觉模拟到现实的传输依赖于外观随机性和DAgger混合：去除其中任何一种都会大幅降低成功率。视频、代码和任务资源可在此 https URL 访问。

Mitigating Retaliatory Algorithmic Collusion in Repeated Games

缓解重复游戏中报复性算法串通的问题

• Authors: Karthik Sivachandran, Rohan Paleja
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.20548
• Pdf link: https://arxiv.org/pdf/2609.20548
• Abstract Reinforcement learning agents trained to maximize their own reward in repeated interactions can converge to supra-competitive outcomes resembling explicit collusion, without communication or shared design. Existing mitigation approaches are largely tied to specific economic settings, like two-sided platforms and auctions, leaving open how to design interventions for general repeated games. We address this gap by formalizing the connection between empirical observations from prior work on Q-learning collusion and classical theory of Simple Penal Codes (SPCs). We show any non-trivial SPC induces a quantifiable conditional dependence in agents' policies, detectable via the total variation distance between an agent's action distributions across cooperation and defection histories. Building on this connection, we propose CURB (Collusion Unwinding via Reward shaping and Belief injection), a reward-shaping framework that penalizes this Total Variation (TV) distance signal during Q-learning and is guaranteed to convert any SPC fixed point of the dynamics into a trivial one, thus precluding collusive equilibria sustained by punishment threats. Empirically, CURB substantially reduces collusion by Q-learning agents in both Bertrand and Cournot Competition Repeated Games. We further demonstrate that CURB extends to deep Q-network agents in Bertrand competition, suggesting the mechanism generalizes beyond tabular Q-learning.
• 中文摘要 训练以最大化自身奖励的强化学习代理，可以在无沟通或共享设计的情况下收敛到类似显性共谋的超竞争结果。现有的缓解方法主要依赖于特定的经济环境，如双边平台和拍卖，因此设计通用重复博弈的干预措施仍有待地。我们通过形式化先前关于Q学习共谋的实证观察与简单刑罚码（SPCs）经典理论之间的联系来弥补这一空白。我们展示了任何非平凡的SPC都会诱导代理策略中可量化的条件依赖性，这种依赖性通过代理在合作与背叛历史中行动分布的总变异距离来检测。基于这一联系，我们提出了CURB（通过奖励塑造和信念注入的共谋解开），这是一个奖励塑造框架，在Q学习过程中惩罚该全变异（TV）距离信号，并保证将动态中的任意SPC不动点转换为平凡点，从而排除惩罚威胁所维持的共谋均衡。从实证角度看，CURB在贝特朗和库诺竞争重复博弈中显著减少了Q学习代理的串通。我们进一步证明CURB扩展到贝特朗竞争中的深度Q网络代理，表明该机制已超越表格Q学习。

Learning Slope-Adaptive Whole-Body Locomotion for Humanoid Robots in Roofing Construction

学习屋顶施工中人形机器人的坡度自适应全身运动

• Authors: Songyang Liu, Shuai Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.20558
• Pdf link: https://arxiv.org/pdf/2609.20558
• Abstract Roofing requires workers to coordinate locomotion, balance, and work-related body motions on pitched surfaces, creating a challenging application for humanoid robots. Directly retargeted human demonstrations, however, may preserve motion appearance while placing the robot's feet or hands incorrectly relative to the roof. This study presents a task-semantic scene-grounded framework for learning roofer-style whole-body motions on a Unitree G1. Human demonstrations are captured using a tracking system and retargeted to the robot, while a metric roof model supplies the spatial reference unavailable from the tracking system. A trajectory-level optimization grounds inferred support contacts and annotated work relations to the roof, and execution-aware reinforcement learning encourages the resulting policy to preserve these relations under dynamic tracking errors. The framework is evaluated through a multi-motion tracking study, a roof-pitch coverage matrix, a five-way nailgun ablation, cross-task experiments on hammering and lateral pushing, and comparisons with pure reinforcement learning and zero-shot teleoperation. Our method enables the robot to satisfy support, work-clearance, and nonpenetration criteria across all evaluated seeds. Across nailgun, hammering, and pushing, it achieves work-clearance errors between 0.256 and 0.531 cm and 3/3 successful evaluations per task. Physical experiments reproduce uphill walking, nailgun, hammering, and bending motions with mean base-frame motion errors below 80 mm. These findings establish scene-grounded human motion learning as a promising basis for construction-oriented humanoid motion primitives.
• 中文摘要 屋顶施工要求工人协调在斜面上的运动、平衡和与工作相关的身体运动，这为类人机器人带来了挑战性。然而，直接重新定向的人类演示可能在机器人双脚错误放置于屋顶时保持运动外观。本研究提出了一个基于任务语义场景的框架，用于在Unitree G1上学习屋顶工人式的全身运动。人类演示通过跟踪系统捕捉并重新定位到机器人，而度量屋顶模型则提供跟踪系统无法提供的空间参考。轨迹级优化基于推断的支持接触和与屋顶的注释工作关系，执行感知强化学习鼓励策略在动态跟踪误差下保持这些关系。该框架通过多运动跟踪研究、屋顶坡度覆盖矩阵、五向钉枪消融、锤击和横向推力的跨任务实验，以及与纯强化学习和零发子遥控操作的比较进行评估。我们的方法使机器人能够满足所有评估种子的支撑、功功间隙和非穿透性标准。在钉枪、锤击和推搡中，其工件间隙误差在0.256至0.531厘米之间，且每任务成功评估率为3/3。物理实验重现了上坡行走、钉枪、锤击和弯曲运动，其平均基准帧运动误差低于80毫米。这些发现确立了基于场景的人体运动学习作为建筑导向类人运动原件的有前景基础。

Reasoning Quality Matters: Combating Reasoning Collapse in LLM-based Embedding Learning

推理质量重要：应对基于LLM的嵌入式学习推理崩溃

• Authors: Zihan Gong, Xiaohan Ye, Jiangchao Yao, Jinsong Lan, Xiaoyong Zhu, Xu Chen
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.20563
• Pdf link: https://arxiv.org/pdf/2609.20563
• Abstract Large Language Models (LLMs) have recently shown strong potential for producing context-rich text embeddings for retrieval. Most existing methods either treat embedding learning as passive feature extraction or exploit LLM reasoning through instruction following for better embedding optimization. However, specialization toward embedding objectives can suppress useful reasoning generation or produce retrieval-irrelevant text. We refer to these two forms of degradation as reasoning collapse. To address this issue, we propose CoFree (Collapse-Free Reasoning Embedding), a two-stage framework that progressively integrates LLM reasoning into query and document embedding optimization while preserving reasoning quality. At the first stage, CoFree applies reference-guided supervised fine-tuning to restore the reasoning ability and retain representational strength of the foundation embedding model. At the second stage, we introduce dual rewards, an embedding-oriented reward and a reasoning-oriented reward, to guarantee fine-grained reasoning of the relevance toward the embedding goal in reinforcement learning. This endpoint-coupled optimization transforms embedding learning from static alignment into a high-quality reasoning-guided search process for retrieval. Extensive experiments demonstrate the effectiveness of CoFree, with CoFree-4B achieving an average absolute improvement of 2.8 nDCG@10 points over Qwen3-Embedding-4B across 22 datasets from MTEB and BRIGHT. Online experiments in a real-world retrieval system further show consistent gains. Code, RTED, and model checkpoints will be made publicly available.
• 中文摘要 大型语言模型（LLM）最近展现出生成上下文丰富文本嵌入用于检索的强大潜力。大多数现有方法要么将嵌入学习视为被动特征提取，要么通过指令跟踪利用LLM推理以优化嵌入。然而，专注于嵌入目标可能会抑制有用的推理生成或产生无关检索的文本。我们将这两种退化形式称为推理崩溃。为解决这个问题，我们提出了CoFree（无崩溃推理嵌入）的两阶段框架，逐步将LLM推理整合进查询和文档嵌入优化，同时保持推理质量。第一阶段，CoFree采用引用引导的监督微调，恢复推理能力并保持基础嵌入模型的表征强度。第二阶段引入双重奖励：面向嵌入的奖励和以推理为导向的奖励，以保证对嵌入目标相关性的细致推理。这种端点耦合优化将静态对齐的嵌入学习转变为高质量的推理引导检索过程。大量实验证明了CoFree的有效性，CoFree-4B在MTEB和BRIGHT的22个数据集中平均比Qwen3-Embedding-4B绝对提升2.8 nDCG@10点。在真实世界检索系统中的在线实验进一步显示出持续的提升。代码、RTED和模型检查点将公开。

UniPolicy: Unified Objective-Specific Policies for Generative Search Advertising

UniPolicy：生成式搜索广告的统一目标特定政策

• Authors: Kun Yao, Yuhang Zhou, Yichi Zhang, Zeliang Tong, Shengri Xue, Haitao Wang, Siyu Lu, Qianlong Xie, Xingxing Wang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.20630
• Pdf link: https://arxiv.org/pdf/2609.20630
• Abstract Search advertising connects user intent with commercial content and plays a critical role in platform monetization. Recent systems typically align pretrained generative models with a single business reward, such as eCPM, or use naive reward fusion for preliminary multi-objective alignment. However, an ideal search advertising system must jointly account for heterogeneous objectives, including relevance, click propensity, and commercial value, to balance user experience and business value while mitigating globally suboptimal performance caused by gradient competition. We propose UniPolicy, an objective-aware multi-policy alignment framework. UniPolicy combines objective-specific prefix tokens, sparse MoE-LoRA routing, and objective-specific residual FFNs to hierarchically decouple parameters within a shared backbone, providing differentiated parameter and policy-expression spaces for different business objectives. It further constructs pairwise preferences from multi-stage behavioral feedback, supplementing the relative preference information in exposed-but-unclicked samples and strengthening the relative advantage of clicked candidates in the generation distribution. At inference, UniPolicy supports parallel, business-customizable multi-policy beam search, flexibly allocating candidate quotas across objectives under a fixed retrieval budget. Large-scale offline experiments show that UniPolicy delivers balanced improvements across multiple metrics while preserving retrieval quality, outperforming single-objective reinforcement learning and naive reward-fusion baselines. In a 7-day online A/B test on a real search advertising system, UniPolicy improves CTR by 0.71%, RPS by 1.58%, and advertising revenue by 1.32%, while maintaining stable serving latency.
• 中文摘要 搜索广告将用户意图与商业内容连接起来，并在平台变现中发挥关键作用。近期系统通常将预训练生成模型与单一业务奖励（如eCPM）对齐，或使用朴素奖励融合进行初步多目标对齐。然而，理想的搜索广告系统必须共同考虑异质目标，包括相关性、点击倾向和商业价值，以平衡用户体验与业务价值，同时缓解梯度竞争导致的全球次优性能。我们提出UniPolicy，一种目标感知的多策略对齐框架。UniPolicy 结合了目标专属前缀令牌、稀疏的MoE-LoRA路由和目标专属残余FFN，在共享骨干内层级解耦参数，为不同业务目标提供差异化的参数和策略表达空间。它进一步构建了多阶段行为反馈的两两偏好，补充了暴露但未点击样本中的相对偏好信息，并强化了被点击候选者在生成分布中的相对优势。推断时，UniPolicy 支持并行、可企业定制的多策略束搜索，在固定检索预算下灵活分配候选配额。大规模离线实验表明，UniPolicy 在多个指标上实现平衡改进，同时保持检索质量，优于单一目标强化学习和简单的奖励融合基线。在真实搜索广告系统的 7 天在线 A/B 测试中，UniPolicy 点击率提升了 0.71%，RPS 提升了 1.58%，广告收入提升了 1.32%，同时保持了稳定的服务延迟。

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

不要掩盖环境：观察监督改变了强化学习下代理的探索方式

• Authors: Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.20715
• Pdf link: https://arxiv.org/pdf/2609.20715
• Abstract Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.
• 中文摘要 代理轨迹记录代理的行为及其后续发生。然而，标准监督微调（SFT）仅对代理自导的动作标记施加损失，使用环境观察作为上下文，而非预测目标。我们探讨这一惯例是否为后续强化学习提供了最佳初始化。我们引入了ActObs，它也监督每个轨迹中已有的观察标记。虽然部署的代理从不生成观测值，但学习预测它们鼓励策略在不添加数据、参数、序列标记或前向传递的情况下建模动作后果。这些方法在SFT后表现类似，但在GRPO之后出现分歧。在Qwen3-4B上，ActObs的GRPO在每次评估采样预算下都达到了比Terminal-Bench 2.0上单动作对应物更高的pass@k。在Qwen3-8B上，它以牺牲部分pass@1的可靠性换取更高的pass@k（pass@16时+3.4pp），并解决更多独特的任务。优势还体现在辅助多语种（4B pass@1+4.2pp）的跨域代码编辑中，这些任务在SFT和RL期间是看不到的。ActObs在RL期间保留更多熵，同时需要更少的策略移动，使最终策略更接近其SFT初始化。我们的分析将这一差异追溯到SFT：动作和观测梯度迅速变得正交，而仅动作训练则留下较大的观测梯度，降低基线模型以下的环境预测能力。联合监督防止了这种单方面的专精，保持了结果预测，并为下游探索准备策略。

MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving

MILER：非结构化自动驾驶中模拟到现实强化学习的语义中级表示

• Authors: Thomas Steinecker, Denis Trescher, Alexander Bienemann, Thorsten Luettel, Mirko Maehlisch
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.20747
• Pdf link: https://arxiv.org/pdf/2609.20747
• Abstract Reinforcement learning constitutes a promising approach owing to its potential for superhuman performance and self-learned policies. However, its application to real-world autonomous driving remains scarce, particularly in unstructured environments, because of the challenges associated with sim-to-real transfer for unstructured environments. In this work, we present MILER, an end-to-end policy framework with zero-shot sim-to-real transfer. During offline training, we employ a custom semantic mid-level representation (MLR) simulator and train the policy network using reinforcement learning, with its control outputs applied directly to a bicycle model. During deployment on the real vehicle, camera and LiDAR data are processed by BEVFusion to generate a semantic bird's-eye-view representation consistent with that of the MLR simulator. The actions generated by the policy network are not applied directly to the real vehicle. Instead, we employ a trajectory-alignment strategy that enables zero-shot sim-to-real transfer of both perception and control. We extensively evaluate the proposed framework on a diverse test track comprising numerous challenges, including various obstacles, hairpin curves, velocities of up to 33.6 km/h, and off-road sections. In total, we drove 17.3 km with two different vehicles on a 3.0 km test track without human intervention, thereby demonstrating the effectiveness of our approach. Furthermore, the entire software stack runs on a Jetson AGX Orin.
• 中文摘要 强化学习因其超人性能和自学策略的潜力，被认为是一种有前景的方法。然而，由于模拟到现实的非结构化环境传输存在挑战，其在现实自动驾驶中的应用仍然稀少，尤其是在非结构化环境中。本研究提出了MILER，一个端到端的策略框架，支持零样本模拟到现实传输。在离线训练期间，我们使用自定义语义中级表示（MLR）模拟器，并利用强化学习训练策略网络，其控制输出直接应用于自行车模型。在实际车辆部署过程中，摄像头和激光雷达数据由BEVFusion处理，生成与MLR模拟器一致的语义鸟瞰图表示。策略网络生成的动作不会直接应用于真实车辆。相反，我们采用轨迹对齐策略，实现感知与控制的零拍摄模拟到现实传输。我们在包含多项挑战的多样测试赛道上对所提框架进行了全面评估，包括各种障碍、发夹弯、最高33.6公里/小时的速度以及越野路段。我们共驾驶两辆不同车辆在3.0公里的测试赛道上行驶了17.3公里，无需人工干预，从而验证了我们方法的有效性。此外，整个软件栈运行在Jetson AGX Orin上。

OPTED: On-Policy Fine-Tuning for End-to-End Driving using a Render-Free Teacher

被选：使用无渲染教师进行政策内端到端驾驶微调

• Authors: Damiano Da Col, Maximilian Igl, Peter Karkus, Kashyap Chitta, Boris Ivanovic, Marco Pavone, Konrad Schindler, Christos Sakaridis
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.20756
• Pdf link: https://arxiv.org/pdf/2609.20756
• Abstract As scaling pre-training data alone yields diminishing returns, post-training is becoming increasingly important across physical AI domains such as autonomous driving. End-to-end driving policies are pre-trained in open loop with behavior cloning on human demonstrations. However, compounding errors during closed-loop deployment can take the vehicle outside the training data distribution, increasing the risk of safety-critical incidents. Closed-loop post-training can mitigate this risk but requires costly simulation for sensor-based policies. We propose OPTED (on-policy fine-tuning for end-to-end driving) which decouples reinforcement learning from the post-training of the end-to-end policy: a privileged teacher is trained using RL on vectorized inputs (HD-map and bounding boxes). This teacher then provides supervision to the pre-trained student during closed-loop post-training. We apply OPTED to two camera-based models, TransFuser and VaVAM, and fine-tune them in AlpaSim, using neural reconstructions (3DGS) of real driving logs. Driving scores increase by factors of 1.6$\times$ and 9.5$\times$, respectively. In controlled experiments OPTED matches closed-loop performance with approximately three orders of magnitude fewer simulator interactions than direct RL post-training, while staying closer to the human prior. Project page: this https URL
• 中文摘要 随着仅仅扩展预训练数据带来递减收益，后训练在自动驾驶等物理人工智能领域变得越来越重要。端到端驾驶策略在开环中预训练，行为克隆对人类演示进行。然而，闭环部署过程中错误叠加可能导致车辆超出训练数据分布范围，增加安全关键事件的风险。闭环后训练可以降低这一风险，但需要对基于传感器的策略进行昂贵的模拟。我们提出了OPTED（端到端驾驶策略微调），将强化学习与端到端策略的后期训练脱钩：特权教师使用强化学习对向量化输入（HD-map和边界框）进行培训。该教师随后在闭环后培训期间对预培训学生进行监督。我们将OPTED应用于两个基于摄像头的模型TransFuser和VaVAM，并在AlpaSim中通过真实驾驶日志的神经重建（3DGS）进行微调。驾驶得分分别提升了1.6$\times$和9.5$\times$。在对照实验中，OPTED在模拟器互动比直接强化学习后少约三个数量级，且更接近人类先验，与闭环性能相当。项目页面：此链接

RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning

退休OPD：能动强化学习的自我退役策略提炼

• Authors: Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.20784
• Pdf link: https://arxiv.org/pdf/2609.20784
• Abstract Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.
• 中文摘要 接受强化学习（RL）训练的多回合代理每条轨迹获得单一标量奖励，这激励自我策略提炼（OPD）从拥有特权任务技能的自学者那里提供密集的代币级监督，让无技能学生内化这些技能。然而，这一方案被代理任务中的两个发现所推翻：仅凭特权信息并不总能让教师可靠，且教师监督的益处取决于阶段。因此，我们提出了RetireOPD（自我退休政策提炼），先用环境奖励优化解耦的技能条件教师，然后与强化学习和OPD共同训练无技能学生。退休OPD不遵循预设的蒸馏计划，而是采用适应性退休：学生在差异停止缩小且达到教师成功率的目标分数后自行放弃教师，之后仅以强化学习进行培训。在Qwen2.5模型中，从1.5B到7B，RetireOPD将ALFWorld在强化学习基线上的成功率提升14.1%至18.8%，WebShop准确率提升11.8%至19.0%，并在各方面超越自身技能条件教师。

Score Centering Stabilizes Off-policy Reinforcement Learning

评分中心稳定了非策略强化学习

• Authors: Martin Marek, Max Ryabinin
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.20807
• Pdf link: https://arxiv.org/pdf/2609.20807
• Abstract Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.
• 中文摘要 大型语言模型的强化学习（RL）对训练引擎与推理引擎之间的细微差异极为敏感，通常称为训练-推理不匹配（TIM）。然而，完全消除TIM不切实际，因为这将严重影响推广效率。本文表明，在TIM下RL的不稳定性主要由漂移引起：训练引擎与推理引擎之间持续存在的偏倚，随着每个训练步骤不断积累。我们推导出一个加法“分数中心化”修正项，通过抵消漂移稳定TIM下的RL。当训练模型参数从0.6B到30B时，仅靠分数中心化就能匹配或优于基于量化重要性抽样的方法，且随着不匹配的加剧，差距会扩大。由于修正是可加性的，评分中心化还与重要性抽样结合——在我们的陈旧实验中，其组合优于纯重要性抽样基线。

Keyword: diffusion policy

ULOHA: An Underwater Bimanual Robot System for Robot Learning

ULOHA：一种用于机器人学习的水下双手动机器人系统

• Authors: Masato Kobayashi, Takeru Tsunoori
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.19200
• Pdf link: https://arxiv.org/pdf/2609.19200
• Abstract Underwater visuomotor policy learning has focused primarily on single manipulators, while bimanual imitation learning has been studied largely in air. We present ULOHA, an underwater bimanual robot learning platform that combines custom-designed leader--follower hardware with software extensions to LeRobot, integrating teleoperation, multi-view sensing, demonstration collection, policy training, and autonomous deployment. Real-robot experiments demonstrate a range of coordinated underwater bimanual behaviors, including inter-arm transfer, shared-object manipulation, and buoyancy-driven interception. We evaluate ACT, Diffusion Policy, and the vision--language--action model SmolVLA on the platform. We investigate how learning methods and execution strategies developed for manipulation in air perform underwater, examining bubble disturbances, buoyancy-driven object motion, action-execution horizons, and real-time chunking. A separate single-arm study examines policy transfer between air and water and shows that demonstrations spanning both media support execution in both under the tested conditions. ULOHA provides a unified experimental platform for studying underwater bimanual robot learning under the coupled perceptual and physical effects of underwater environments. Additional material: this https URL
• 中文摘要 水下视觉运动策略学习主要聚焦于单机械臂，而双手模拟学习则主要在空中进行研究。我们介绍ULOHA，一款水下双手机器人学习平台，结合了定制的领导者-跟随者硬件与LeRobot的软件扩展，集成了远程操作、多视角感知、演示收集、策略训练和自主部署。真实机器人实验展示了一系列协调的水下双手行为，包括臂间转移、共享物体操作和浮力驱动拦截。我们评估了平台上的ACT、扩散政策以及视觉-语言-动作模型SmolVLA。我们研究了为空气操控开发的学习方法和执行策略在水下的表现，包括气泡扰动、浮力驱动物体运动、动作-执行视野和实时分块。另一项单臂研究探讨空气与水中之间的政策转移，展示了跨越媒体在两者测试条件下的演示。ULOHA提供了一个统一的实验平台，用于研究水下双手机器人在水下感知与物理耦合效应下的学习。附加材料：此 https URL

Learning Foresight without Explicit Trajectories for 3D Diffusion Policies

学习3D扩散策略中没有显式轨迹的前瞻性

• Authors: Zhongbo Zhang, Zaibin Zhang, Yifan Wang, Changbo Yan, Lijun Wang, Huchuan Lu
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.20669
• Pdf link: https://arxiv.org/pdf/2609.20669
• Abstract 3D diffusion policies are strong at generating geometrically grounded actions from current observations, but successful manipulation requires not only knowing what motion is feasible now, but also anticipating where the interaction is heading. Existing policies largely leave such foresight to emerge implicitly from action learning. We introduce Movement Trend Guidance, a simple but effective way to provide this foresight without introducing an explicit plan. From a short observation history, the policy learns a compact latent representation of interaction evolution. During training, sparse future gripper states supervise this representation; at inference, only the latent is retained as future-oriented conditioning alongside the current observation. The latent provides global conditioning for action generation, while an additional gated FiLM branch is used only at the UNet bottleneck. Despite adding only 3.52% more parameters to DP3, our method preserves the original dense-action and receding-horizon formulation and consistently improves upon DP3 across RoboTwin2.0, LIBERO-40, and DexArt. It reaches 62.8% vs. 56.1% in 50-task RoboTwin2.0 mixed training, 71.93% vs. 37.08% on LIBERO-40, and 72.0% vs. 49.0% on five real-robot tasks. These results show that a diffusion policy can benefit substantially from knowing where an interaction is heading, without being told exactly where to move.
• 中文摘要 三维扩散策略在从当前观测中生成几何基础动作方面表现优异，但成功的操作不仅需要知道当前可行的运动，还需预判交互的走向。现有策略大多将此类前瞻性隐含地从行动学习中显现。我们引入了运动趋势指导，这是一种简单但有效的方法，无需引入显式计划即可提供这种前瞻性。通过简短的观测历史，策略学习出交互演化的紧凑潜在表示。在训练过程中，稀疏的未来抓握者状态监督该表示;在推断时，只有潜在因素作为当前观测的面向未来条件保留。潜在变量为动作生成提供全局条件，而额外的门控FiLM分支仅在UNet瓶颈处使用。尽管只增加了3.52%的参数，我们的方法保留了原始的密集动作和后退视界表述，并在RoboTwin2.0、LIBERO-40和DexArt上持续改进DP3。在50任务RoboTwin2.0混合训练中，该方法达到62.8%对56.1%;在LIBERO-40中为71.93%对37.08%，在五个真实机器人任务中为72.0%对49.0%。这些结果表明，扩散策略可以通过知道交互的走向而显著受益，而无需明确指示具体移动方向。