生成时间: 2026-10-15 21:11:48 (UTC+8); Arxiv 发布时间: 2026-10-15 20:00 EDT (2026-10-16 08:00 UTC+8)

今天共有 62 篇相关文章

Keyword: reinforcement learning

Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data

用于高效后期训练和仅答案数据验证的逆推理模型

• Authors: Eric O. Korman
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.18878
• Pdf link: https://arxiv.org/pdf/2610.18878
• Abstract Post-training reasoning LLMs often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt-answer tuples and lacks the groundtruth chain-of-thought (CoT) sequence. On the other hand, reinforcement learning (with verifiable rewards, for example) can operate without CoT data but it is much more compute-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non-zero policy gradient. In this work we explore a middleground, a self-distillation technique that uses answer-only data, treating CoT as latent variables. This relies on training a partner ``inverse reasoning LLM" that generates the original's reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, with all generation happening offline and up front, making the approach particularly suited to continual post-training. Using Qwen3-0.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts and, on Countdown, outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio (with smaller gains on GSM8K); rejection sampling by construction has no signal on problems the model never solves on its own. Continued GRPO on the fresh data retains an edge at hard pass@1 but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness. When used for best-of-$n$ selection at test time, it raises selection accuracy on Countdown from 0.753 (majority voting) to 0.969.
• 中文摘要 训练后推理LLM通常需要在昂贵的监督和计算效率低下之间做出权衡：监督式微调通常需要推理痕迹，这在数据效率较低，因为在许多应用中，只有即时回答元组，缺乏Groundtruth Chain-of-thought（CoT）序列。另一方面，强化学习（例如带有可验证奖励）可以在没有CoT数据的情况下运行，但由于每个优化步骤需要多次自回归补全，且这些补全过程都不保证能获得非零策略梯度，因此计算效率较低得多。本研究探讨了一种中间地带，即使用仅有答案数据的自蒸馏技术，将CoT视为潜在变量。这依赖于训练伙伴“逆向推理LLM”，生成基于答案的原始推理痕迹。逆模型训练一次后，在输入的仅答案数据批次中重复使用，所有生成均在离线且前期完成，使该方法特别适合持续的后期训练。利用Qwen3-0.6B在倒计时和GSM8K任务中，我们展示了在GRPO训练饱和后，逆模型的单轮离线蒸馏在硬提示上大约翻倍pass@1，在倒计时中，在每个混合比值下，拒绝采样自蒸馏的表现优于硬尾指标（GSM8K的收益较小）;拒绝采样通过构造对模型自身无法解决的问题没有信号。对新数据进行持续GRPO在硬pass@1保持优势，但在蒸馏pass@8和后pass@32，同时在每个步骤重新生成部署。额外好处是，逆模型还有第二个应用，作为准确性验证工具。在测试时用于$n美元最佳选拔时，它能将倒计时中的选择准确率从0.753（多数票）提升到0.969。

PMO-Dock: Benchmarking Docking, Specificity, and Generalization in Molecular Optimization

PMO-Dock：分子优化中的基准对接、特异性与泛化

• Authors: Gor Simonyan, Tatevik Abrahamyan, Narek Abelyan, Tigran Fahradyan, Hrant Khachatrian
• Subjects: Subjects: Machine Learning (cs.LG); Quantitative Methods (q-bio.QM)
• Arxiv link: https://arxiv.org/abs/2610.19011
• Pdf link: https://arxiv.org/pdf/2610.19011
• Abstract The Practical Molecular Optimization (PMO) benchmark standardized evaluation in molecular optimization, but it is built on simple property-based oracles that do not capture structure-based drug design. As the field has moved to docking-based objectives, evaluation practice has become unstandardized, and different methods report results on different docking tasks, often without a shared protocol for fair comparison or sample-efficiency constraints. To address this, we introduce Practical Molecular Optimization for Docking (PMO-Dock), a benchmark and protocol for docking-based optimization consisting of 25 tasks covering hit generation, lead optimization, and a new specificity task requiring strong on-target binding while penalizing off-target interactions. The protocol separates model development from final evaluation: validation tasks are used for hyperparameter tuning, and test tasks are held out for final evaluation and are not used during development. Evaluation on the test tasks is done under a strict oracle budget to reflect realistic drug-discovery settings. We benchmark four diverse high-performing methods spanning different optimization paradigms, Saturn (reinforcement learning), GenMol (discrete diffusion), Genetic-guided GFlowNet, and ChemLlama (LLM-based evolutionary search). Our results show no universally best method across tasks, additional analyses of hyperparameter transfer (cross-task and cross-category) and sensitivity show that transfer is often limited and varies by method, motivating method-dependent selection strategies. This benchmark provides a concrete standard for evaluating generalization to unseen tasks and sample efficiency under a limited test-time oracle budget.
• 中文摘要 实用分子优化（PMO）基准测试标准化了分子优化的评估，但基于简单的基于性质的预言机，未能涵盖基于结构的药物设计。随着领域转向基于对接目标，评估实践变得不标准化，不同方法报告不同对接任务的结果，往往缺乏共享的公平比较或样本效率约束协议。为此，我们引入了实用分子对接优化（PMO-Dock），这是一个基于对接优化的基准测试和协议，包含25个任务，涵盖命中生成、导向优化以及一项需要强靶点结合并惩罚非靶点相互作用的新特异性任务。该协议将模型开发与最终评估分开：验证任务用于超参数调优，测试任务则留待最终评估，开发期间不使用。测试任务的评估在严格的预言机预算下进行，以反映现实的药物发现环境。我们对四种跨越不同优化范式的高效能方法进行了基准测试：Saturn（强化学习）、GenMol（离散扩散）、遗传引导GFlowNet和CheMllama（基于LLM的进化搜索）。我们的结果显示，没有跨任务的普遍最佳方法，对超参数转移（跨任务和跨类别）及敏感性进行额外分析显示，转移往往有限且因方法而异，促使采用方法依赖的选择策略。该基准测试为评估对未见任务的泛化和样本效率在有限测试时间预言机预算下提供了具体标准。

What Information Buys in Active Inference

主动推理中信息能买到什么

• Authors: Melih Kandemir, Mustafa Mert Çelikok
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.19046
• Pdf link: https://arxiv.org/pdf/2610.19046
• Abstract Information-theoretic exploration is of primary interest in methodological studies of reinforcement learning (RL) and computational cognitive neuroscience, while the established RL theory builds regret guarantees on confidence sets. This situation leaves open the conditions under which the information-driven exploration found in nature is more advantageous than the exploration of the established RL theory. We put the state-of-the-art methods of these two paradigms, Active Inference (AIF) and Bayes-UCBVI, on an equal footing: both act through the same state abstraction of a tabular MDP with $S$ states, and their greedy forms differ only in the exploration term. We find that AIF's information gain decays too fast to serve as a confidence width, locking greedy agents that explore by it out of the optimal action. Calibrating the information gain by a square root yields calibrated AIF, which attains regret of order $\sqrt T$ for an exact abstraction, but no better guarantee than Bayes-UCBVI for a fixed reward. When an adversary chooses rewards from the agent's data, Bayes-UCBVI's value-specific widths lock out, whereas calibrated AIF pays a factor $\sqrt S$, which no goal-memoryless agent avoids on some MDP whose start states the adversary also chooses. Calibrated information gain thus acts as a reward-agnostic $\ell_1$ confidence set, preferable to value-specific widths when rewards are chosen from the agent's data, not when they are fixed.
• 中文摘要 信息理论探索在强化学习（RL）和计算认知神经科学的方法论研究中尤为重要，而既有的强化学习理论则基于置信集构建遗憾保证。这种情况为自然界中基于信息驱动的探索比探索既有的强化学习理论更有利的条件提供了开放空间。我们将这两种范式——主动推理（AIF）和贝叶斯-UCBVI——的先进方法置于同等地位：它们都通过具有$S$状态的表格MDP的同一状态抽象来作用，它们的贪婪形式仅在探索项上有所不同。我们发现AIF的信息获得衰减过快，无法作为置信宽度，这使贪婪的智能体无法采取最优行动。对信息获得进行平方根校准得到校准AIF，对于精确抽象，后悔阶为$\sqrt T$，但对于固定奖励，其保证不比贝叶斯-UCBVI更有利。当对手从代理数据中选择奖励时，贝叶斯-UCBVI的价值特定宽度会被锁定，而校准后的AIF支付的是因子$\sqrt S$，而无目标记忆的智能体在起始状态对手也选择的MDP上避免了这一点。校准信息获得因此作为一个与奖励无关的$\ell_1$置信集合，当奖励是从代理数据中选择时，比价值特定宽度更优，而非固定奖励。

Backpropagating Verifier Rewards Through Continuous Flow Language Models

通过连续流语言模型反向传播验证者奖励

• Authors: Xu Han, Jiajing Hu, Li-Ping Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19080
• Pdf link: https://arxiv.org/pdf/2610.19080
• Abstract Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose \method{}, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, \method{} reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.
• 中文摘要 强化学习（RL）显著提升了自回归和离散扩散语言模型的推理能力。产生连续符号嵌入的语言模型正作为强有力的替代方案出现，但其强化学习后训练仍缺乏研究。我们提出了 \method{}，一种连续流语言模型的后训练框架。该模型在多步中对符号嵌入进行去噪，并仅在最后一步将其解码为离散符号。为获得路径梯度，这种梯度通过连续动作反向传播且方差远低于 REINFORCE，我们引入了一个评估者，该模型估计中间符号嵌入的价值，绕过不可微译码步骤。批判者还必须跟踪策略变化时的价值函数;为此，我们设计了一个能从每个符号表示中预测值并改进策略优化的批判者。在四项推理任务中，\method{} 通过强化学习后可靠地改进了连续流语言模型，并且优于同类基线方法。

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

GroundGUI：通过高质量数据和蒸发到强化学习实现的GUI基础化

• Authors: Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.19100
• Pdf link: https://arxiv.org/pdf/2610.19100
• Abstract Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and datasets are available at this https URL.
• 中文摘要 构建稳健的图形界面代理依赖多模态视觉基础：能够精确地将自然语言意图映射到可操作的屏幕组件。尽管桌面、网页和移动端基准充足，针对不同环境的高精度资源却出人意料地稀少。为弥合这一差距，我们介绍了屏幕指向基准数据集（ScreenRef），这是一个全面的图形界面基础数据集，由图形界面截图、自然语言引用表达式及框标注或点标注组成。ScreenRef涵盖桌面、网页和移动领域，包含58.6万张截图和超过63.3万个引用表达实例，涵盖文本识别、视觉匹配、空间理解和拒绝场景等任务。为提升模型的基础性能，我们提出了蒸馏到强化学习（D2RL）方法，以解决监督微调的泛化有限和强化学习中的稀疏奖励问题。利用ScreenRef，我们通过顺序SFT和D2RL训练Qwen3-VL-8B-Instruct，生成GroundGUI-8B。GroundGUI-8B在ScreenSpot-V2上实现了95.2%，在ScreenSpot-Pro上达到63.9%，在OSWorld-G上达到65.4%，在MMBench-GUI L2上为86.8%，在UI-Vision上达到38.3%。在8B级模型中，它的性能显著优于基于开源数据集的模型，性能与基于闭源数据训练的模型相当甚至更优。代码、模型和数据集可在该 https 网址获取。

Contextual Credit Policy Optimization

情境信贷政策优化

• Authors: Huiwen Han, Ziyao Zeng, Zihao Zhu, Sisong Bei, Hao Wang, Hongyu Xu, Yan Han, Degui Zhi, Zhiwen Fan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19103
• Pdf link: https://arxiv.org/pdf/2610.19103
• Abstract Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. To address these two challenges, we propose Contextual Credit Policy Optimization (CCPO). For the first, CCPO weights observation-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step's baseline. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation. A step is thus credited for the progress it makes, not only for its trajectory's eventual outcome. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic. Across two Qwen2.5-Instruct backbones (1.5B and 7B), CCPO achieves up to 97.4% success on ALFWorld and 79.4% on WebShop and outperforms GRPO by up to 21.6 and 18.5 points. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals.
• 中文摘要 基于群体的强化学习使大型语言模型代理能够从长期任务中的稀疏结果奖励中学习。最新方法通过比较跨轨迹的观察匹配经验来获得阶级功劳。然而，在不同情境中可能出现相同的观察，匹配的同伴对某一决策的信息量并不相同。此外，有利的终极结果可能掩盖成功轨迹中的冗余动作或绕行。为应对这两个挑战，我们提出了情境信用政策优化（CCPO）。对于第一种，CCPO通过上下文相似度加权来自其他轨迹的观察匹配同伴，该相似性基于冻结的LLM表示和累计轨迹统计计算得出，使相似情境的同伴能对步骤的基线贡献更大。第二种情况中，CCPO将历史信用（将实现回报与加权同行基线进行比较）与未来信用（衡量短期实现延续期间情境结果价值的变化）配对。因此，一步不仅因其发展轨迹的最终结果而被认可，还因其取得的进展而被认可。这两种信用都是从已收集的培训推广中事后计算的，使CCPO能够在没有辅助价值批评者的情况下应对这些挑战。在两个Qwen2.5-Ininstruction骨干链（1.5B和7B）中，CCPO在ALFWorld上达到了最高97.4%的成功率，在WebShop上达到了79.4%，并且比GRPO高出最多21.6点和18.5个百分点。消融结果显示，情境加权优于统一加权，历史和未来加权则作为互补的学习信号。

Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models

迈向以数据为中心的多模态大型语言模型空间推理理解

• Authors: Mahdieh Mallahnezhad, Masoud Jafaripour, Mohamadali Tofigh, Sheila Schoepp, Osmar R. Zaiane
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.19108
• Pdf link: https://arxiv.org/pdf/2610.19108
• Abstract Multimodal Large Language Models (MLLMs) have been widely applied in embodied agents and robotics for vision tasks. However, MLLMs suffer from limitations in spatial reasoning and perception. Post-training techniques have shown promise for improving these capabilities; yet, there are limited data-driven surveys of such approaches. In this survey, we consider approaches to fine-tuning pretrained MLLMs for spatial reasoning tasks, ranging from supervised to reinforcement learning, with an emphasis on the datasets used. We investigate themes of dataset construction, data modality and annotation, and evaluation of the resulting models. Based on these themes, we categorize research into three groups: dataset design, dataset construction and evaluation. We find four main methods for constructing fixed spatial annotations: rules and templates, LLM prompting, procedural generation, and human-in-the-loop. We also observe substantial variation in evaluation approaches, including different scoring methods, tolerance metrics, and benchmark types. Finally, we identify opportunities for future research in unified dynamic annotation, efficient spatial reasoning, standardization of evaluation, expanded spatial range, and human-centered spatial interactions.
• 中文摘要 多模态大型语言模型（MLLMs）已被广泛应用于具身智能体和视觉任务机器人中。然而，MLLM在空间推理和感知方面存在局限性。后训练技术显示出提升这些能力的潜力;然而，关于此类方法的数据驱动调查仍然有限。本综述探讨了对预训练MLM进行空间推理任务微调的方法，涵盖监督式到强化学习，重点关注所用数据集。我们探讨了数据集构建、数据模态与注释以及模型评估等主题。基于这些主题，我们将研究分为三类：数据集设计、数据集构建和评估。我们发现了构建固定空间注释的四种主要方法：规则与模板、LLM提示、程序生成和人机循环。我们还观察到评估方法存在显著差异，包括不同的评分方法、容忍度指标和基准类型。最后，我们发现未来在统一动态注释、高效空间推理、评估标准化、扩展空间范围和以人为本的空间交互等领域有研究机会。

PairIT: Autoregressive Transformers for Low-Data Molecule Optimization

PairIT：用于低数据分子优化的自回归变换器

• Authors: Hannah Lawrence, Bodhi P. Vani, Ji Won Park, Natasa Tagasovska, Samuel Don Stanton, Andrew Martin Watkins, Michael Maser, Ewa M. Nowara
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Chemical Physics (physics.chem-ph)
• Arxiv link: https://arxiv.org/abs/2610.19123
• Pdf link: https://arxiv.org/pdf/2610.19123
• Abstract Generating novel molecules with desired properties is a central challenge in drug discovery. Although foundational work focused on de novo generation, practical use cases require a more complex setting: conditioning on an initial seed molecule, optimizing multiple properties, and operating in data-sparse regimes. We introduce PairIT, a flexible training framework for autoregressive transformers over SMILES sequences that enables seeded generation by learning from pairs of structurally similar molecules. By leveraging matched pairs, the model learns to generate local, chemically meaningful edits that improve a property of the seed molecule. Prompt tokens extend this framework by specifying the target property, desired direction, and approximate magnitude of change. Multiple prompt tokens can be concatenated to enable multi-property optimization at inference time. For properties where pair supervision alone is insufficient, reinforcement learning post-training can further improve optimization, providing a tunable trade-off between proximity to the seed and the optimization objective. PairIT naturally supports pretraining on a large dataset and fine-tuning on a smaller dataset, which is critical for performance in data-sparse regimes. Surprisingly, we observe diminishing returns to the scale of the pretraining dataset. We demonstrate empirically on MoleculeNet and Therapeutics Data Commons datasets that, with appropriate paired-data curation, a standard transformer model of modest size (68M parameters) can achieve robust seeded molecular optimization in low-data regimes.
• 中文摘要 生成具有期望性质的新分子是药物发现中的核心挑战。虽然基础工作聚焦于新生成，但实际应用场景需要更复杂的环境：对初始种子分子进行条件化、优化多重性质以及在数据稀疏环境中运行。我们介绍了PairIT，一个灵活的SMILES序列自回归变换器训练框架，通过从结构相似分子对中学习实现种子生成。通过利用匹配对，模型学习生成局部且化学意义深的编辑，从而改善种子分子的某一性质。提示令牌通过指定目标性质、期望方向和近似变化幅度来扩展该框架。多个提示令牌可以串接，在推理时实现多属性优化。对于仅靠配对监督不足的属性，训练后强化学习可以进一步优化，提供种子接近度与优化目标之间的可调节权衡。PairIT自然支持大数据集的预训练和较小数据集的微调，这对数据稀疏环境下的性能至关重要。令人惊讶的是，我们观察到预训练数据集规模的收益递减。我们在MoleculeNet和Therapeutics Data Commons数据集上实证证明，通过适当的配对数据管理，一个规模适中的标准变换器模型（68M参数）可以在低数据环境中实现稳健的种子分子优化。

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

揭开政策提炼的神秘面纱：教师优势在哪里？

• Authors: Yiping Li, Fan Zhao, Zhibo Hou, Zhiyu An, Wan Du
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19145
• Pdf link: https://arxiv.org/pdf/2610.19145
• Abstract On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.
• 中文摘要 策略提炼（OPD）越来越被研究为强化学习的廉价替代方案，但它仍是一种依赖教师监督的知识提炼（KD）形式。我们从KD的角度重新审视OPD，探讨教师在学生生成轨迹中的优势所在。师生分歧时的受控代币翻转直接衡量教师纠正带来的准确性。单代币翻转在选定失败轨迹上带来适度的平均收益，而重复翻转则将准确率从28.4%提升至74.5%，接近教师的76.9%。随着翻转次数增加，提升率会持续累积，但在类似规模的计数中，基于质量和基层的选择比随机翻转高出15-18个百分点。OPD后选择性干预仍有用，但更高的argmax一致性并不总是能在模型对中持续提升准确率。我们倡导培训计划，既能学习更有价值的教师纠正，又能保持分布稳定性。受控翻转和叉子分辨率分析通过测量学生获得的教师偏好以及进一步纠正可恢复的标准，评估OPD的准确性。

Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models

扩散语言模型中的成果导向提炼与强化学习

• Authors: Wentao Lu, Tianyu Zhu, Jesse Clark
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19271
• Pdf link: https://arxiv.org/pdf/2610.19271
• Abstract On-policy distillation provides dense supervision on the student's own samples, but its targets still come from the teacher. Verifier reinforcement learning (RL) instead uses an outcome signal, but it provides no direct constraint on capabilities outside the verifier. Combining the two is difficult for diffusion language models (dLLMs), because bidirectional denoising makes the policy likelihoods required by ratio-based RL unreliable. We develop a likelihood-free pipeline for a mixture-of-experts dLLM. Each rollout group is routed by verifier outcome. Mixed groups use ratio-free, group-centred REINFORCE. If every student rollout fails but a teacher succeeds, the model receives dense KL supervision on the student's denoising canvases. All other groups are skipped. A KL term anchored to the distilled checkpoint limits changes outside the verifier reward. The tool-calling teacher succeeds on only 18.6% of the training prompts. At 24 denoising steps, distillation exceeds the teacher by 6.77 percentage points, and routing improves the distilled checkpoint by a further 18.31 points. Routing lowers reasoning-trace accuracy by 0.71 points but raises direct-answer accuracy on the same items by 13.21 points. The anchored arm restores trace accuracy. Its tool-calling score differs by -0.16 points, with a confidence interval that includes zero, although this comparison also removes a small fallback distillation route. A domain-conditional recipe retains a 16.37-point tool-calling gain; a second seed gives 18.23 points and reproduces both preservation results, but the math result is seed-dependent. We pre-registered every arm and evaluated each comparison on paired items in a single sweep on one machine.
• 中文摘要 策略提炼对学生自身样本提供密集监督，但其目标仍来自教师。验证器强化学习（RL）则使用结果信号，但不直接限制验证者外部的能力。对于扩散语言模型（dLLM）来说，两者结合较为困难，因为双向去噪使得基于比率的RL所需的策略似然度不可靠。我们为专家混合dLLM开发了无似然流水线。每个推广组由验证者结果路由。混合组使用无比率、以群体为中心的REINFORCE。如果每个学生推广失败但教师成功，模型将在学生的去噪画布上获得密集的Kle-Learning监督。其他所有组均被跳过。锚定于蒸馏检查点的KL项限制了验证者奖励之外的变化。工具调用教师仅成功完成18.6%的训练提示。在24个去噪步骤中，蒸馏比教师高出6.77个百分点，路由则使蒸馏检查点提升18.31个百分点。路由使推理追踪准确率降低0.71点，但相同题目直接回答准确率提升13.21个点。锚定臂恢复追踪准确度。其工具调用得分相差-0.16个百分点，置信区间为零，尽管此比较也去除了一小段回溯蒸馏路径。域条件配方保留16.37分的工具调用增益;第二个种子获得18.23分，并重现两个保存结果，但数学结果依赖种子。我们预先注册了所有臂，并在一台机器上对配对项目进行了一次扫描。

Verifier-Grounded Agentic RL for Provably Correct Code

验证器基础代理强化学习（Verifier-grounded agentic RL）用于可证明正确代码

• Authors: George Ma, Yuhao Zhang, Bing He, Zeren Shui, Ferhat Erata, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19298
• Pdf link: https://arxiv.org/pdf/2610.19298
• Abstract Reinforcement learning for code usually rewards passing tests, which a model can do while remaining wrong on the inputs the tests do not cover. A program verifier instead decides whether a program satisfies a formal specification for all inputs. Language models are almost always paired with verifiers at inference time only, and turning a verifier into a reward raises two difficulties. It reports only acceptance or rejection, which gives no gradient when every attempt fails, and a policy can cheat it, most simply by telling it to assume what the proof was supposed to establish. We build an agentic environment and reward that address both. The reward reads the verifier's diagnostics rather than its verdict, scoring a candidate by the weighted fraction of proof obligations it discharges, extended with hidden probes from a reference proof's annotations. An integrity audit outranks the verifier, rejecting any candidate that alters the specification or assumes what it was meant to prove. The model works as an agent against a live verifier over many turns, with a separately trained proof specialist as a frozen collaborator. Across Qwen3 models from 1.7B to 14B and a 35B Qwen3.6 mixture-of-experts model, held-out verification on code generation rises from 66.1% to 93.5% on DafnyBench and from 56.3% to 100% on SWE-Proof. This reward leads every other training signal we compare, including a binary pass-or-fail reward trained under otherwise identical settings, with the widest margin on code generation. When the agent must also deliver the repository patch, graded by each project's withheld tests, the fraction of held-out issues it resolves rises from 59.6% to 72.5%. Evaluated unchanged on three verification suites withheld from training, the checkpoints improve on all of them, including one whose language and verifier are absent from training.
• 中文摘要 代码的强化学习通常奖励通过测试，模型可以在测试未覆盖的输入上保持错误的情况下完成测试。程序验证器则决定程序是否满足所有输入的形式规范。语言模型几乎总是在推理时与验证器配对，将验证器变成奖励会带来两个难题。它只报告接受或拒绝，且每次尝试失败时不提供梯度，策略可以作弊，最简单是让它假设证明应建立的内容。我们构建一个代理环境，并奖励这两者。奖励读取验证者的诊断而非裁决，通过加权比例对候选人进行评分，并通过引用证明注释中的隐藏探针扩展。完整性审计优先于验证者，拒绝任何修改规范或假设其本意证明内容的候选者。该模型作为代理对抗活验证者，多个回合内由单独训练的证明专家作为固定协作者。在1.7B至14B的Qwen3模型和35B的Qwen3.6专家混合模型中，DafnyBench的代码生成延迟验证率从66.1%升至93.5%，SWE-Proof则从56.3%升至100%。这一奖励领先我们对比的所有其他训练信号，包括在相同条件下训练的二元通过或失败奖励，代码生成优势最大。当代理还必须交付仓库补丁，并根据每个项目的保留测试评分时，其解决的未解决问题比例从59.6%上升到72.5%。在培训中未被保留的三个验证套件的评估结果显示，检查点在所有测试套件上均有所改进，包括其中一个缺少语言和验证器的检查点。

MMKG-R1: Constructing Multimodal Knowledge Graphs via Reinforcement Learning

MMKG-R1：通过强化学习构建多模态知识图谱

• Authors: Zongyu Wu, Yilong Wang, Xiaochen Wang, Minhua Lin, Yuan Zhong, Fenglong Ma, Xiang Zhang, Suhang Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.19341
• Pdf link: https://arxiv.org/pdf/2610.19341
• Abstract Multimodal knowledge graphs (MMKGs) organize information across different modalities to support multimodal Graph Retrieval-Augmented Generation (RAG), which aims to improve the performance of multimodal large language models (MLLMs). The effectiveness of these multimodal Graph RAG methods largely depends on the quality of the constructed MMKGs. However, most existing methods construct MMKGs without explicit guidance on graph quality, while reinforcement learning (RL)-based methods that rely on final-answer correctness may provide only indirect feedback on MMKG quality. To address these limitations, we propose MMKG-R1, a novel MMKG-quality-driven RL framework for training an MLLM as a multimodal knowledge constructor. To be specific, MMKG-R1 introduces multiple quality-aware rewards to directly evaluate both the quality of information extracted from individual knowledge sources and the quality of the assembled MMKG. These rewards are then used to guide the RL training. Experiments across multiple datasets show that MMKG-R1 constructs MMKGs that improve downstream question-answering accuracy.
• 中文摘要 多模态知识图谱（MMKGs）组织跨不同模态的信息，支持多模图检索增强生成（RAG），旨在提升多模态大型语言模型（MLLM）的性能。这些多模态图RAG方法的有效性很大程度上取决于构建的MMKG的质量。然而，大多数现有方法构建MMKG时没有明确的图质量指导，而依赖最终答案正确性的强化学习（RL）方法可能仅对MMKG质量提供间接反馈。为解决这些局限性，我们提出了MMKG-R1，一种新型MMKG质量驱动的RL框架，用于将MLLM训练为多模态知识构建器。具体来说，MMKG-R1引入了多种质量感知奖励，直接评估从单个知识源提取的信息质量和组装后的MMKG质量。这些奖励随后用于指导强化学习训练。跨多个数据集的实验表明，MMKG-R1构建了提升后续问答准确性的MMKGs。

Beyond Corrected Actions: Training LLM Agents on Textual Critiques

超越纠正行为：培训LLM代理进行文本批评

• Authors: Thomas Zeng, Zelei Cheng, Mingyang Zhou, Sambit Sahu, William Campbell, Genta Indra Winata
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.19367
• Pdf link: https://arxiv.org/pdf/2610.19367
• Abstract Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains $\tau^2$-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.
• 中文摘要 大型语言模型（LLM）越来越多地被用于需要通过长时间操作序列与用户和环境交互的多回合代理任务。改进LLM代理的一种常见方法是使用更强大的专家模型来提供训练监督，无论是通过行为克隆专家轨迹，还是让专家纠正由较弱学生模型生成的轨迹中的动作。然而，专家级LLM不仅能提供更丰富的监督：它们可以直接生成文本批评，解释为何特定动作不正确。我们研究了对此类反馈的培训是否有利于LLM代理，并提出了批判增强代理微调（CAFT）框架，该框架能够对学生生成的轨迹引发文本专家反馈。对于每个错误的动作，专家都会生成一个纠正动作和一个解释学生错误的文本批评——然后我们将其作为微调的训练目标。在来自Nemotron 3和Gemma 4家族的四个学生模型中，针对agentic domain（$\tau^2$-Bench Airline+Retail）和WebShop，向动作级监督添加批评平均比仅动作蒸馏提升8%的性能。此外，CAFT检查点还能比专家轨迹蒸馏更好地为强化学习提供温暖起点。我们的结果表明，文本批评可以加强对多回合LLM代理的监督。

Learning What to Keep: Transfer-Aware Personal Skill Learning from User Feedback

学习保留什么：转移意识型个人技能 从用户反馈中学习

• Authors: Li Wang, Chen Zhang, Yirong Yang, Xingyu Chen, Jinkun Zhao, Tianhao Peng, Wenjun Wu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.19380
• Pdf link: https://arxiv.org/pdf/2610.19380
• Abstract User feedback enables LLMs to refine reusable personalized skills, but often mixes persistent preferences with request-specific corrections. Indiscriminately retaining both risks negative transfer, motivating selective retention at the level of individual requirements. We frame this challenge as selective transfer over atomic rules and propose SkillGrain, a framework for learning reusable personalized skills from user feedback. SkillGrain decomposes feedback into independently actionable rules, predicts their transferability, and retains only those suitable for future requests. To learn these decisions, we combine rubric-guided skill rewards with fine-grained credit assignment that separately evaluates rule content and retention decisions. We further jointly optimize skill induction and skill-conditioned generation through reinforcement learning, ensuring that learned skills support downstream personalization. We construct a benchmark spanning five personalized writing tasks to evaluate cross-request generalization. Experiments across three base models in both in-domain and out-of-domain settings show that SkillGrain achieves the best overall average performance among competing methods.
• 中文摘要 用户反馈使大型语言模型能够精炼可重复使用的个性化技能，但通常将持续偏好与请求特定修正混合在一起。无差别地保留这两种风险会导致负向转移，从而在个体需求层面激励选择性保留。我们将这一挑战框架为原子规则的选择性转移，并提出了SkillGrain框架，一个通过用户反馈学习可重复使用的个性化技能的框架。SkillGrain将反馈分解为独立可操作的规则，预测其可转移性，并只保留适合未来请求的反馈。为学习这些决策，我们将以评分标准为导向的技能奖励与细粒度的学分分配结合，分别评估规则内容和保留决策。我们进一步通过强化学习共同优化技能归纳和技能条件生成，确保所学技能支持下游个性化。我们构建了一个涵盖五个个性化写作任务的基准，以评估跨请求的泛化。在域内和域外的三个基模型实验显示，SkillGrain在竞争方法中实现了最佳的整体平均性能。

ForkGRPO: Efficient Exploration and Fine-Grained Credit Assignment for Diffusion Language Models

ForkGRPO：扩散语言模型的高效探索与细粒度的信用分配

• Authors: Zhaoyan Jia, Chengwei Zhou, Gourav Datta, Massoud Pedram
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.19408
• Pdf link: https://arxiv.org/pdf/2610.19408
• Abstract Diffusion large language models (dLLMs) have advanced rapidly, yet strengthening their reasoning through reinforcement learning (RL) remains an open challenge. Any-order generation can suffer from entropy degradation, which narrows the reasoning paths available for exploration. Recent work mitigates this issue by applying standard GRPO with left-to-right generation. However, independently generating every rollout from the input prompt can leave alternative continuations from intermediate states underexplored, while trajectory-level advantages assign uniform credit to all tokens without distinguishing the contributions of individual reasoning segments. We propose ForkGRPO, an RL framework that couples efficient exploration with fine-grained segment-level credit assignment. It samples multiple continuations from high-entropy intermediate states along a reference trajectory, prioritizing textual boundaries and reusing shared prefixes. The continuations provide complementary learning signals: within-state group-relative advantages guide which continuations to reinforce, while Monte Carlo state-value differences indicate which reasoning segments improve expected outcomes. Among the evaluated architecture-preserving RL methods, ForkGRPO achieves the best performance on three of four benchmarks, while improving over the directly comparable JustGRPO by up to 1.2 percentage points and using up to 27.8% fewer generated tokens. ForkGRPO-Fast further achieves a $2.49\times$ training wall-clock speedup with negligible accuracy loss.
• 中文摘要 扩散大型语言模型（dLLMs）发展迅速，但通过强化学习（RL）强化推理仍是一个开放挑战。任意序生成可能受到熵退化的影响，从而缩小了可探索的推理路径。近期工作通过应用带有左至右生成的标准GRPO来缓解这一问题。然而，独立从输入提示生成每个展开可能会使中间状态的替代延续未被充分探索，而轨迹层优势则能为所有令牌均分配统一的信用，而不区分单个推理片段的贡献。我们提出了ForkGRPO框架，这是一种将高效探索与细粒度段级信用分配相结合的强化学习框架。它沿参考轨迹采样高熵中间态的多个延续，优先考虑文本边界并重复使用共享前缀。续写提供了互补的学习信号：州内组相对优势指导哪些延续应强化，而蒙特卡洛状态值差异则指示哪些推理片段能改善预期结果。在评估的保持架构的强化学习方法中，ForkGRPO在四个基准测试中有三个表现最佳，且比直接对应的JustGRPO提升最多1.2个百分点，且生成代币数量减少多达27.8%。ForkGRPO-Fast还实现了2.49美元/倍的训练时钟加速，且准确率损失极小。

AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

AgentSafeEvo：LLM代理安全的对抗性共进化

• Authors: Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Cheng Jin, Chaochao Lu, Qiaosheng Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.19422
• Pdf link: https://arxiv.org/pdf/2610.19422
• Abstract Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.
• 中文摘要 大型语言模型（LLM）代理通过与外部工具的多回合交互完成复杂任务。然而，嵌入用户查询、工具描述或环境观察中的恶意内容可能操纵其决策，诱导不安全的行为。对抗共进化已成为加强LLM安全性的有效方法，但现有工作主要聚焦于网络任务中的语言级安全性或间接提示注入。本文介绍了AgentSafeEvo，这是一个基于攻击者-防御者博弈、涵盖多任务和攻击类型的共进化强化学习框架。攻击者学习构建对抗性代理任务，揭示防御者工具使用中的弱点，而防御者则学习在这些攻击下安全完成任务。我们进一步引入可学习性引导任务权重，利用任务内防御者奖励的变异性调整每个任务对攻击者和防御者更新的贡献。关于代理安全和通用能力基准的广泛实验表明，我们的方法在保持通用代理能力的同时，提升了对抗的鲁棒性。

Comprehensive Comparisons of Reinforcement Learning Algorithms in Autonomous Driving

自动驾驶强化学习算法的综合比较

• Authors: Dechen Xie, Mushuang Liu
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.19429
• Pdf link: https://arxiv.org/pdf/2610.19429
• Abstract This paper presents a comprehensive comparison of four representative reinforcement learning (RL) methods in autonomous vehicle (AV) decision-making, including: Fitted Value Iteration (FVI), Rollout Classification Policy Iteration (RCPI), Deep Q-Network (DQN), and a discrete-action adaptation of Deep Deterministic Policy Gradient (D-DDPG). The four methods are applied to the same AV decision-making problem at an unsignalized intersection and are compared in terms of optimality and safety, training stability, and training and decision cost. RCPI is the closest to a reference solution and has the most stable training process. However, it is the least data-efficient. D-DDPG reaches the second-smallest gap to a reference solution with the highest data efficiency, but has the least training stability. DQN is the most stable model-free method but has the highest collision rate. FVI achieves a lower collision rate than DQN, but is less data-efficient and less stable. Based on these results, we provide suggestions for selecting a method according to whether a model is available, the budget for training, and whether stable training is required.
• 中文摘要 本文全面比较了自动驾驶汽车（AV）决策中四种代表性的强化学习（RL）方法，包括：拟合值迭代（FVI）、展开分类策略迭代（RCPI）、深度Q网络（DQN）以及深确定性策略梯度（D-DDPG）的离散动作适应。这四种方法应用于同一无信号交叉口的自动驾驶决策问题，并在最优性、安全性、训练稳定性以及训练和决策成本方面进行比较。RCPI是最接近参考解的，训练过程最稳定。然而，它是数据效率最低的。D-DDPG在数据效率最高的参考解中差距第二小，但训练稳定性最低。DQN是最稳定的无模型方法，但碰撞率最高。FVI的碰撞率低于DQN，但数据效率较低且稳定性较低。基于这些结果，我们根据模型是否可用、训练预算以及是否需要稳定训练，提出了方法选择建议。

MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces

MIPS：在大型结果空间中缓解强化学习模式崩溃

• Authors: Armaan Sandhu, Abhijeet Sinha, Dianbo Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.19457
• Pdf link: https://arxiv.org/pdf/2610.19457
• Abstract Expected-reward maximization in reinforcement learning concentrates probability mass on a few outcomes, even in tasks that require many distinct, high-quality solutions, a failure known as mode collapse. Inverse Probability Scaling (IPS) counters this in group-based policy gradients such as GRPO by dividing each reward by the outcome's probability, estimated from its frequency within the sampled group. When the outcome space is large relative to the group, outcomes rarely repeat and these estimates become uninformative. Since an outcome can be constructed along many trajectories, its probability is a sum over all of them, and computing it directly is intractable. We propose Marginal IPS (MIPS), which performs marginal evaluation by combining the sampled trajectory with a learned model of the trajectories that lead to each outcome. On Hypergrids, where the exact sampling distribution is computable, MIPS stays near the reward-distribution target as the outcome space grows relative to the rollout group, while frequency-based IPS degrades. In molecular synthesis and phylogenetic inference, MIPS sustains diverse, high-quality sampling where GRPO and frequency-based IPS collapse, showing that MIPS serves as a reliable solution to mode collapse in RL.
• 中文摘要 强化学习中的期望奖励最大化将概率质量集中于少数结果，即使在需要多个不同且高质量解的任务中，这种失败被称为模式崩溃。逆概率尺度（IPS）通过将每个奖励除以结果概率，基于抽样组的频率估计，从而抵消了这一点，如GRPO。当结果空间相对于组较大时，结果很少重复，这些估计值变得无益。由于结果可以沿多个轨迹构建，其概率是所有轨迹的总和，直接计算概率较为困难。我们提出了边际IPS（MIPS），通过将抽样轨迹与学习到的导致每个结果的轨迹模型结合，进行边际评估。在超网格中，精确采样分布可计算，MIPS保持在奖励分布目标附近，结果空间相对于推广组增长，而基于频率的IPS则会退化。在分子合成和系统发育推断中，MIPS能够维持多样且高质量的采样，而GRPO和基于频率的IPS会崩溃，表明MIPS是强化学习模式崩溃的可靠解决方案。

VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning

VLRLA：通过上下文强化学习对视觉-语言-行动模型的部署时间适应

• Authors: Diyuan Shi, Yiqi Tang, Hongyin Zhang, You Wu, Donglin Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.19517
• Pdf link: https://arxiv.org/pdf/2610.19517
• Abstract Vision-language-action (VLA) models connect visual perception and language understanding with robot control, providing a foundation for instruction-conditioned manipulation. However, conventional pretrained VLA policies lack a mechanism for gradual adaptation through accumulated interaction experience during deployment. We therefore propose VLRLA, a general framework that equips existing VLA models with \textit{in-context reinforcement learning} capabilities for deployment-time adaptation. Our approach augments a VLA model with a reinforcement learning module that learns from historical observations, actions, rewards, and termination signals. During deployment, the policy gradually refines its action generation and progressively improves task success while keeping model parameters fixed. Our approach is also designed for general applicability and compatibility with diverse learning history construction strategies and model training pipelines. Finally, through extensive experiments, we systematically investigate the learning properties of VLRLA in simulation and demonstrate its ability to adapt to unseen scenarios during real-world deployment.
• 中文摘要 视觉-语言-动作（VLA）模型将视觉感知和语言理解与机器人控制连接起来，为指令条件操作奠定基础。然而，传统的预训练VLA策略缺乏通过部署期间积累交互经验逐步适应的机制。因此，我们提出了VLRLA，这是一个通用框架，为现有VLA模型配备部署时间适配的\textit{上下文强化学习}能力。我们的方法通过强化学习模块增强VLA模型，从历史观察、动作、奖励和终止信号中学习。部署过程中，策略逐步优化动作生成，逐步提升任务成功率，同时保持模型参数固定。我们的方法还设计为通用适用性，并兼容多样化的学习历史构建策略和模型训练流水线。最后，通过大量实验，我们系统地研究VLRLA在模拟中的学习特性，并展示了其在实际部署中适应未知场景的能力。

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

在可验证的指导下诊断和纠正奖励黑客行为

• Authors: Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.19525
• Pdf link: https://arxiv.org/pdf/2610.19525
• Abstract Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.
• 中文摘要 可验证的指令跟随基准强调输出约束的满足度，但可能忽视响应是否实质性地回答底层请求，造成评估盲区。我们展示了前沿模型对于内容较少的响应，可以获得全部认可。使用与训练奖励相同的验证者标准，可能会诱发奖励黑客行为，因为公开发布的检查点在强化学习后表现出更高的约束满意度，但响应质量较低，且奖励可验证。为应对这种奖励黑客问题，我们引入了CALIF一种轻量级方法，通过校准基于似然的奖励来增强验证者奖励，以保持响应质量。CALIF仅基于请求评估响应可能性，使用现有的冻结参考策略，无需额外的评判或学习奖励模型。在推理和非推理模式的1.7B至8B模型中，CALIF提升了IFBench的约束满足度，同时保持响应质量接近或高于RL前水平。在3个基于评分标准的指令跟随基准中，45个模型-指标比较中，CALIF表现优于相应基础策略，展示了超越训练验证器的提升。

Retrieve What Helps, Only When It Helps: Agentic Reinforcement Learning for Selective Retrieval-Augmented Machine Translation

只在有帮助时检索有效的内容：选择性检索增强机器翻译的代理强化学习

• Authors: Shijun Li, Yang Liu, Najmeh Sadoughi, Dongmei Wang, Xinjie Li, Zhu Liu, Rohith Mysore Vijaya Kumar, Yuzong Liu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.19528
• Pdf link: https://arxiv.org/pdf/2610.19528
• Abstract Retrieving external knowledge can improve large language model (LLM) translation by providing supplementary information such as idiom interpretations and terminology. However, existing retrieval-augmented machine translation systems typically apply predefined retrieval strategies uniformly across all inputs. We argue that translation operates in a partial-evidence regime: LLM's parametric knowledge suffices for many inputs, while external evidence benefits a subset and usually supports rather than directly provides the desired translation. This creates two challenges: deciding when to retrieve and what evidence to acquire. We introduce Selective Retrieval-Augmented Translation (SRAT), an agentic reinforcement learning framework that learns when to translate directly or retrieve, what to retrieve, and how to integrate these decisions by optimizing final translation quality. To align evidence retrieval with downstream translation quality rather than simple semantic similarity, we further develop joint Plackett--Luce optimization and translation-quality contrastive training for retriever-training. Experiments on idiom, knowledge-intensive, and document-level translation with offline or online retrieval show that SRAT consistently outperforms never- and always-retrieve baselines, and quality-aligned retriever training provides further gains when similarity-based retrieval is insufficient. The results demonstrate the benefit of treating retrieval as a learnable translation action rather than fixed preprocessing.
• 中文摘要 检索外部知识可以通过提供习语解释和术语等补充信息来提升大型语言模型（LLM）翻译。然而，现有的检索增强机器翻译系统通常在所有输入中统一应用预定义的检索策略。我们认为翻译在部分证据体系下运作：LLM的参数化知识足以满足许多输入，而外部证据则有利于部分部分，通常支持而非直接提供所需的翻译。这带来了两个挑战：决定何时检索以及获取哪些证据。我们介绍了选择性检索增强翻译（SRAT），这是一种代理强化学习框架，学习何时直接翻译或检索什么，以及如何通过优化最终翻译质量整合这些决策。为了使证据检索与下游翻译质量而非单纯语义相似性保持一致，我们进一步开发了联合Plackett-Luce优化和翻译质量对比训练用于检索器训练。在惯用语、知识密集型和文档级翻译中，离线或在线检索的实验表明，SRAT始终优于无检索和始终检索基线，质量对齐检索训练在基于相似度的检索不足时提供了进一步提升。结果表明，将检索视为可学习的翻译动作而非固定的预处理具有优势。

PolyMorph: Learning Locomotion Policy for Cross-Embodiment Adaptation

PolyMorph：学习跨身体适应的运动策略

• Authors: Hyunsik Oh, Seunghun Jeon, Donghoon Youm, Jungwoo Hur, Hyeongjun Kim, Jeong Hyun Lee, Moonkyu Jung, Jemin Hwangbo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.19615
• Pdf link: https://arxiv.org/pdf/2610.19615
• Abstract Reinforcement learning has enabled robust locomotion for legged robots, but learned policies are typically tied to a specific robot morphology and require additional engineering when the number of legs, actuators, or mechanical properties change. We present PolyMorph, a morphology-conditioned framework that learns a single locomotion policy for heterogeneous legged robots. PolyMorph converts each robot's URDF into joint-link modules and combines a recurrent encoder, graph-based Transformer, and per-joint decoder to generate actions across variable leg counts and joint configurations. To improve training stability, PolyMorph uses morphology-normalized rewards, adaptive robot sampling, and URDF-derived PD-gain initialization. We evaluate PolyMorph in simulation across bipedal, tripedal, and quadrupedal robots, including unseen morphology variants and platforms, and deploy the policy on real quadruped platforms operated in multiple configurations. The results show that a single PolyMorph policy generalizes across diverse robot morphologies without per-robot policy retraining.
• 中文摘要 强化学习使有腿机器人实现了稳健的运动，但学习到的策略通常与特定的机器人形态相关联，当腿数、执行器或机械性能变化时，需要额外的工程设计。我们提出了PolyMorph，一个形态条件框架，用于学习异构腿机器人的单一运动策略。PolyMorph将每个机器人的URDF转换为关节链路模块，并结合循环编码器、基于图的变换器和每关节解码器，以生成跨可变腿数和关节配置的动作。为提升训练稳定性，PolyMorph采用形态归一化奖励、自适应机器人采样和URDF衍生的PD增益初始化。我们在双足、三足和四足机器人的模拟中评估PolyMorph，包括未见的形态变体和平台，并在多重配置下运行的真实四足平台上部署该策略。结果表明，单一的PolyMorph策略可以在不同的机器人形态上推广，而无需每个机器人的策略重新训练。

E2DRL: End-to-End Differentiable Reinforcement Learning for 3D Obstacle Avoidance in Quadruped Robots

E2DRL：四足机器人中端到端可微分强化学习，用于三维障碍避让

• Authors: Cheng Fan, Bingyang Zhou, Gang Yang, Zixuan Zhuang, Aoqian Zhang, Pengzhi Yang, Xin Liu, Chunzheng Wang, Yan Rui Tan, Fan Shi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.19627
• Pdf link: https://arxiv.org/pdf/2610.19627
• Abstract Differentiable reinforcement learning shows strong promise for robot policy learning, but its application to legged locomotion has remained limited to blind walking or simulated 2D tasks. Extending it to end-to-end visual obstacle avoidance is challenging because hard collision constraints and early termination upon contact corrupt first-order gradients, trapping optimizers in local minima. We present E2DRL, which addresses this with a soft differentiable obstacle-interaction formulation: penetration penalties across multiple body keypoints with velocity-conditioned stop-gradient shaping to provide the first-order training signals for 3D obstacle avoidance. To realize a deployable perception-control system, a privileged teacher trained via Soft Analytic Policy Optimization (SAPO) in a differentiable simulator is distilled into a recurrent depth-based student via DAgger, then blended with a base locomotion policy using a terrain-aware gating module for deployment. In simulation, E2DRL achieves higher sample efficiency than PPO and demonstrates superior velocity-tracking performance and 3D obstacle-avoidance capability compared with both PPO and Duality-based DCBF NMPC. We present the first sim-to-real validation of end-to-end differentiable RL for 3D obstacle avoidance on a Unitree Go2 in static and dynamic environments.
• 中文摘要 可微强化学习在机器人策略学习中展现出巨大潜力，但其在腿部移动中的应用仍限于盲行或模拟二维任务。将其扩展到端到端的视觉障碍物规避具有挑战性，因为硬碰撞约束和接触后提前终止会破坏一阶梯度，使优化器被困在局部极小值。我们提出了E2DRL，通过软可微障碍相互作用表述解决这个问题：通过速度条件的停止梯度整形，跨多个物体关键点施加穿透惩罚，以提供三维障碍物规避的一阶训练信号。为了实现可部署的感知控制系统，通过软分析策略优化（SAPO）培训的特权教师通过可微分模拟器提炼为重复深度学生，然后利用地形感知门控模块与基础移动策略混合部署。在仿真中，E2DRL比PPO获得更高的采样效率，且在速度跟踪性能和三维障碍避让能力上优于PPO和基于对偶的DCBF NMPC。我们首次在Unitree Go2上模拟到实际验证了端到端可微化强化学习，用于静态和动态环境下的三维障碍避让。

Model What Matters: Decision-Guided Selective World Modeling for Language Agents

建模重要因素：语言代理的决策引导选择性世界建模

• Authors: Maolin Ran, Minda Chen, Di Huang, Wei Xiong, Fei Huang, Weiwen Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19642
• Pdf link: https://arxiv.org/pdf/2610.19642
• Abstract Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.
• 中文摘要 强化学习为大型语言模型驱动的智能体提供了优化任务目标的直接机制，但任务奖励通常是稀疏的轨迹级信号，仅评估任务成功，而非直接监督动作条件环境转变，导致模型对行动后果和环境动态的明确理解有限。近期工作引入了通过政策与世界建模共训练的辅助监督，但通常将完整的环境观测统一视为世界建模目标，忽视观察内容决策相关性的变化。我们提出了DGS-WM（决策引导选择性世界建模），这是一种面向语言代理的决策引导政策-世界建模共训练方法。DGS-WM采用决策引导观察选择组件，根据其对后续政策响应的支持，过滤具有决策价值的观察单元作为世界建模目标;以及一个信心自适应加权组件，根据模型对所选目标的预测置信度自适应调整监督强度。这些组件共同构成一个决策感知辅助的世界建模损失，并与同一政策模型上的强化学习损失共同优化。在涵盖单跳/多跳问答和交互式网页任务的多个基准测试中，DGS-WM在不同模型尺度和强化学习算法下实现了持续的改进，展现出比全观测世界建模方法更稳定、更高效的性能。

Function-Space Temporal Self-Distillation for Policy Optimization

功能空间时间自蒸馏以优化策略

• Authors: Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.19687
• Pdf link: https://arxiv.org/pdf/2610.19687
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了语言模型中的推理能力，但标量结果对反应轨迹提供了粗略的监督。自我蒸馏通过比较无辅助策略与反馈知情自学者，提供更密集的代币级指导，但教师会随着学生演进，创造不稳定的目标并激励额外的正则化。我们介绍功能空间策略优化（FSPO），这是一个单一策略框架，结合了当前特权适应与函数空间时间整合。FSPO将稀疏的历史下一标记分布存储为固定时间目标，提供时间分离，而无需维持持久教师模型。它还引入了特权分歧纠正，明确抑制局部高估的行为，增强自生成证据反复失效时的参考监督，以及质量和覆盖感知的时间记忆，保留已验证的行为并提升特权演示。在推理和工具使用基准中，FSPO在5小时训练预算内实现最高平均准确率，较最强基线平均提升约4%，同时将Olmo3-7B-Instruct的峰值内存从约70GB降至56 GB。

C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning

C3：通过信心引导信用修正重新思考过程监督，在多回合强化学习中

• Authors: Ai Jian, Haoran Pan, Eryu Guo, Haojin Yang, Jinghan He, Xiaoyun Zhang, Jiangbo Pei, Peiyu Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.19695
• Pdf link: https://arxiv.org/pdf/2610.19695
• Abstract Large language models are increasingly trained as multi-turn agents that interact with external tools. However, outcome-only reinforcement learning can produce a form of misguided reinforcement: even on successful trajectories, undesirable local behaviors may receive positive token-level advantages and be favored by the policy update. To address this issue, we propose Confidence-Guided Credit Correction (C$^3$), a plug-in method for local credit correction. C$^3$ identifies spans corresponding to undesirable behaviors and selectively corrects positive credit after advantage estimation, with the correction strength adaptively controlled by the behavior policy's confidence in the corresponding behavior, while leaving the task reward and original credit estimation unchanged. Across multiple multi-turn benchmarks, C$^3$ improves performance under both {critic-based and critic-free reinforcement learning methods}. Injecting the same behavioral signals as turn-level or outcome rewards does not consistently improve performance. Our results show that local credit correction provides an effective form of process supervision for outcome-driven RL.
• 中文摘要 大型语言模型越来越多地被训练为多回合代理，能够与外部工具交互。然而，仅结果强化学习可能产生一种误导性的强化：即使在成功的轨迹上，不良的局部行为也可能获得正向的代币级优势，并被策略更新所青睐。为解决这个问题，我们提出了信心引导信用修正（C$^3$），一种用于局部信用纠正的插件方法。C$^3$识别对应不良行为的范围，并在优势估计后选择性纠正正向信用，纠正强度由行为策略对相应行为的信心自适应控制，同时任务奖励和原始信用估计保持不变。在多个多回合基准测试中，C$^3$在{基于批评和无批评的强化学习方法}下均提升了性能。注入与回合级或结果奖励相同的行为信号并不能持续提升绩效。我们的结果表明，局部信用修正为结果驱动的强化学习提供了一种有效的过程监督形式。

Soloist: A Generative Model for Blind Voice Separation in the Wild

独唱者：一种用于野外盲声分离的生成模型

• Authors: Chen Yang, ChuFan Yu, Botian Jiang, Jingqi Chen, Wenxuan Wang, Yitian Gong, Zhaoye Fei, Xipeng Qiu
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.19716
• Pdf link: https://arxiv.org/pdf/2610.19716
• Abstract Real-world voice separation must recover individual voices from overlapping speech, environmental noise, and music, yet isolated source recordings are rarely available for supervision. We introduce Soloist, a generative model for blind voice separation that autoregressively generates individual voice streams as audio-codec sequences conditioned on the mixture, without requiring a target-speaker query or a predefined voice count. We propose a two-stage training paradigm to learn from synthetic mixtures and adapt to real recordings. The first stage combines supervised separation on synthetic mixtures with isolated source targets and an auxiliary task that predicts speaker timelines from real recordings, learning who speaks when from natural acoustics. The second stage applies reinforcement learning to real recordings, with rewards supplied by a model judge that compares candidate separations based on voice purity and content preservation without isolated source targets. Across diverse scenarios, Soloist recovers clean speech from multi-speaker overlap and noisy recordings, and extracts singing vocals from music accompaniment while maintaining relatively high pitch accuracy and intelligibility. On real-world recordings with unknown voice counts, Soloist also separates voices well. Finally, we integrate Soloist into a full-duplex conversation processing pipeline that assembles separated outputs into continuous speaker tracks, demonstrating its practical utility in real conversations.
• 中文摘要 现实中的语音分离必须从重叠的语音、环境噪音和音乐中恢复单个声音，但孤立的源录音很少可供监督使用。我们介绍了Soloist，一种生成式盲声分离模型，它能自回归生成单个语音流，作为基于混合条件的音频编解码器序列，无需目标-说话者查询或预定义的语音计数。我们提出一个两阶段训练范式，旨在从合成混合中学习并适应真实录音。第一阶段结合了合成混合的监督分离、隔离的源目标，以及辅助任务预测说话者与真实录音的时间线，从自然声学中学习谁何时说话。第二阶段将强化学习应用于真实录音，奖励由模型评审提供，基于声音纯度和内容保存比较候选分离，且无孤立源目标。在各种场景下，Soloist能够从多扬声器重叠和嘈杂录音中恢复干净的语音，并从音乐伴奏中提取歌唱人声，同时保持相对较高的音高准确性和清晰度。在真实录音中，语音数量未知，Soloist也能很好地分离声音。最后，我们将Soloist集成到一个全双工会话处理流程中，将分离输出组装成连续的扬声器轨道，展示了其在真实对话中的实用性。

Reinforcement Learning via Advantage Guidance

通过优势指导进行强化学习

• Authors: Zhiyuan Zhou, Sergey Levine
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19783
• Pdf link: https://arxiv.org/pdf/2610.19783
• Abstract Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by the quality of the demonstrated behavior. In this paper, we ask whether we can perform policy optimization with something as simple and stable as supervised learning but as performant as RL. Unfortunately, naively training the policy with supervised learning and conditioning on future rewards or advantages typically underperform dedicated RL methods. We introduce a new policy extraction method, Advantage Guidance, that retains the simple supervised learning structure with advantage conditioning while outperforming prior RL methods through effective conditional training and contrastive diffusion guidance during sampling. We show that Advantage Guidance can significantly improve a model's ability to respond to conditional advantage signals, and the contrastive guidance during sampling can use the learned advantage-conditioned policy to target the KL-regularized optimal policy. Empirically, we find that Advantage Guidance significantly outperforms prior state-of-the-art RL methods on 26 long-horizon, sparse reward tasks across datasets with varying degrees of optimality.
• 中文摘要 强化学习（RL）有望学习出能够超越数据显示的策略。然而，在实际操作中，这类系统存在不稳定性和超参数敏感性的问题。相比之下，监督学习提供了一个稳定且可扩展的训练目标，但从根本上受限于所展示行为的质量。本文探讨，是否能用像监督学习这样简单稳定、性能同样高的强化学习来进行策略优化。遗憾的是，天真地用监督学习训练策略并对未来奖励或优势进行条件化，通常表现不如专门的强化学习方法。我们引入一种新的策略提取方法——优势指导，它保留了简单的监督学习结构和优势条件，同时通过有效的条件训练和对比扩散引导在抽样中表现优于以往的强化学习方法。我们证明优势指导能显著提升模型对条件优势信号的响应能力，而抽样中的对比指导则利用学习到的优势条件策略来针对KL正则化的最优策略。实证显示，优势指导在26个不同最优度数据集上的长视野稀疏奖励任务中，显著优于以往最先进强化学习方法。

More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

更多方法，更好的代理？代理强化学习中自我蒸馏的现实检验

• Authors: Linhao Zhong, Bozhen Fang, Xuehao Wang, Duochao Shi, Zekai Luo, Guangming Yao, Yi Yuan, Hao Chen, Jingdong Chen, Chunhua Shen
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19789
• Pdf link: https://arxiv.org/pdf/2610.19789
• Abstract Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: this https URL
• 中文摘要 智能体强化学习中的自我蒸馏产生了越来越多的方法。然而，现有评估常常重复使用不同实验条件下已发表的基线分数，缺乏对同时代方法的直接比较，且对训练稳定性和计算成本关注有限。这些局限性掩盖了报告收益的来源以及竞争方法的实际优缺点。昂贵的代理训练进一步激励了一个共同的比较基础。为弥补这些空白，我们在ALFWorld、ScienceWorld、Search-QA和WebShop上比较了十种代表性方法与GRPO的结合，涵盖三种模型规模。我们的研究包括132次训练运行，累计12,124.4小时的H800 GPU小时，针对预训练主干的8个推广种子和全部11种训练方法进行了反复评估。除了最终任务表现外，我们还考察了训练进度、稳定性和计算成本测量。我们的评估得出了三个关键发现。首先，相较GRPO的提升范围有限，没有任何方法能在不同环境中持续优于GRPO。其次，强劲的最终表现可以掩盖不稳定的训练。第三，额外的培训成本并不总是转化为更好的表现。我们建立这一基准，旨在支持公平、全面的比较，并引导未来研究实现可靠性和成本效益的代理培训改进。项目页面：此 https URL

Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards

Explorer-Zero：提升长期任务的自我游戏体验，并提供可分解且可验证的奖励

• Authors: Tianyu Liu, Davide Belli, Bence Major, Arash Behboodi, Tribhuvanesh Orekondy
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19868
• Pdf link: https://arxiv.org/pdf/2610.19868
• Abstract Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification, both expensive to construct for stateful environments. Self-play offers an alternative, but must generate achievable, compositional tasks and provide feedback beyond binary outcomes. We propose Explorer-Zero, a self-play framework that jointly generates tasks and fine-grained, verifiable rewards through environment interaction. A learned proposer explores the environment, constructs tasks, and records intermediate and final states as verification targets. These checkpoints reward partial completion, while alternating proposer--solver training adapts the generated curriculum. We show theoretically that subtask rewards reduce collapse among unsuccessful rollouts when they complete different numbers of subtasks. Without human-annotated training tasks, Explorer-Zero outperforms the evaluated self-play baselines across ALFWorld, ScienceWorld, and AppWorld; on AppWorld, our 30B agent reaches 59.67% task success, approaching 62.65% for RL on human-annotated tasks. Further analyses support task-relevant composition and benefits on complex tasks, while self-play initialization improves subsequent human-data RL across all three benchmarks.
• 中文摘要 通过强化学习训练LLM代理进行长期互动任务，通常依赖人工注释任务和可靠验证，这两者在有状态环境中构建成本较高。自我游戏提供了另一种选择，但必须生成可实现的组合任务，并提供超越二元结果的反馈。我们提出了Explorer-Zero，一种自我游戏框架，通过环境交互共同生成任务和细粒度、可验证的奖励。学习型提案者探索环境，构建任务，并记录中间和最终状态作为验证目标。这些检查点奖励部分完成，而交替提案者-求解者训练则调整生成的课程。我们理论上证明，子任务奖励减少了失败推广中完成不同子任务数量时的崩溃。在没有人工注释训练任务的情况下，Explorer-Zero在ALFWorld、ScienceWorld和AppWorld上表现优于评估中的自我游戏基线;在AppWorld上，我们的30B代理任务成功率为59.67%，在人工注释任务中，强化学习接近62.65%。进一步分析支持复杂任务的相关组合和益处，而自我游戏初始化则提升了后续人类数据强化学习在三大基准测试中。

BluePixel: Toward Native 4K Agentic Image Editing

BluePixel：迈向原生4K代理图像编辑

• Authors: Shanyan Guan, Shiyu Li, Xin Li, Zhizhou Chen, Yunzhe Xu, Yuehao Liu, Ruixue Shen, Shiju Zhang, Min Jin, Qiang Liu, Junbo Liu, Penggang Shi, Sheng Sun, Xiao Wang, Yong Zhang, Zhongang Qi, Wei Li, Chao Ma, Ying Tai, Yanhao Ge
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.19882
• Pdf link: https://arxiv.org/pdf/2610.19882
• Abstract Instruction-driven image editing has reached native 4K, yet how such editors are trained remains undisclosed, and pretraining one from scratch is beyond most teams. The task itself compounds the difficulty: a single end-to-end editor cannot make precise parametric adjustments, match specialized experts, or interpret the vague instructions users actually write. We argue that 4K editing is a system problem rather than a single-model problem, and present BluePixel, which couples a native-4K editing model with an agent harness; the complete training recipe is disclosed in this paper. The model makes 4K training practical: it turns a 1K-pretrained editor into a native-4K one through progressive VAE adaptation that keeps 4K training as affordable as 1K, absorbs a long tail of vertical editing skills through continual learning, and refines the aesthetic-fidelity trade-off with reinforcement learning. Distillation via source-anchored target projection (SATP) then compresses 50 denoising steps to 4, completing a 4K edit in 8.7 s on a single Kunlunxin P800 XPU while preserving task success rate and background consistency. The harness, BluePixel Agent, makes editing controllable: it decomposes user instructions into specialist and general operations, verifies every result with the same reward model that guided training, and retries within bounded attempts. On VINS-4KEval++ (3,305 native-4K cases), BluePixel ranks second among seven models, behind only GPT Image 2 in non-edited-region consistency.
• 中文摘要 指令驱动图像编辑已实现原生4K，但此类编辑者如何训练仍未公开，且从零开始预训练对大多数团队来说是难以理解的。这一任务本身加剧了难度：单一端到端编辑器无法做出精确的参数调整，匹配专业专家，也无法解读用户实际编写的模糊指令。我们认为4K编辑是系统问题，而非单一模型问题，并介绍了BluePixel，它将原生4K编辑模型与代理框架结合;本文披露了完整的训练方案。该模型使4K训练变得实用：通过渐进的VAE适应，将1K预训练编辑器转变为原生4K编辑器，使4K训练与1K同等经济，通过持续学习吸收大量垂直编辑技能，并通过强化学习优化美学与真实度的权衡。通过源锚定目标投影（SATP）进行蒸馏，将50个去噪步骤压缩为4步，在单个昆仑鑫P800 XPU上完成4K编辑，时间为8.7秒，同时保持任务成功率和背景一致性。该工具BluePixel Agent使编辑变得可控：它将用户指令分解为专业和通用操作，使用指导训练的同一奖励模型验证每个结果，并在有限制的尝试内重试。在VINS-4KEval++（3,305个原生4K案例）中，BluePixel在七个模型中排名第二，仅次于GPT Image 2，在非编辑区域一致性方面。

Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extraction

通过漂移进行占用建模：一步生成实现直接策略提取

• Authors: Robin Uhrich, Marco Bagatella, Núria Armengol Urpí, Pavel Kolev, Georg Martius
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19883
• Pdf link: https://arxiv.org/pdf/2610.19883
• Abstract The growing interest in behavior foundation models, combined with recent advancements in generative modeling, has led to an appealing recipe for reinforcement learning from unlabeled data: pre-train a generative model to model occupancies, i.e., discounted state visitations, on reward-free trajectories, such that a policy can be easily extracted once a small reward-labeled dataset becomes available. We decompose this recipe along three axes: (i) the degree to which the multi-modal structure of occupancies must be captured, (ii) the class of generative model, and (iii) the method used for policy extraction. Along each axis, we consider existing choices, propose simpler alternatives, and validate each decision experimentally. This analysis culminates in a novel, streamlined method that leverages drifting, a modern one-step generative sampler, to model the occupancy. We show that this enables policy extraction via direct differentiation through the occupancy model, and bypasses the need for critic distillation. Overall, we observe that this nimble setup leads to improvements in performance on standard benchmarks.
• 中文摘要 对行为基础模型日益增长的兴趣，加上生成建模的最新进展，催生了一种有吸引力的无标签数据强化学习方案：预训练生成模型以建模无奖励轨迹上的占用，即折扣状态访问，以便在获得小型奖励标记数据集后轻松提取策略。我们将该方案沿三个轴分解：（i）多模态占用结构的捕捉程度，（ii）生成模型类别，（iii）策略提取方法。在每个轴上，我们考虑现有选择，提出更简单的替代方案，并通过实验验证每个决策。该分析最终形成一种新颖且简化的方法，利用漂移这一现代一步生成采样器来建模占用情况。我们证明，这使得通过占用率模型的直接区分实现政策提取，绕过了批评者提炼的需求。总体来看，我们观察到这种灵活的配置带来了标准基准测试性能的提升。

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

并非所有标准都重要：基于评分标准的LLM评估中的重要性感知标准加权

• Authors: Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao, Chunyuan Zheng, Hao Wang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.19915
• Pdf link: https://arxiv.org/pdf/2610.19915
• Abstract Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.
• 中文摘要 基于评分标准的评估通过汇总显式标准的验证结果来评估大型语言模型（LLM）的响应，但并非所有标准的重要性都相同。然而，现有方法通常以统一权重聚总验证结果，这使得次要和重叠的标准与关键标准权重相同，从而产生误导性的评分。虽然有些方法促使LLM直接分配权重，但直接加权需要对所有标准进行整体判断，而LLM难以可靠地做出这些判断。为解决这些局限性，我们提出了重要性感知标准权重（ICW），该方法通过LLM进行的两两重要性比较推导出标准权重。具体来说，ICW将所有标准的整体判断转化为一系列成对重要性比较，检查并修正不一致的比较，并从比较矩阵的主特征向量中推导权重。大量实验表明，ICW始终优于统一加权和竞争加权方法，并在下游强化学习中转化为更优的策略模型。

MolHop: Scaffold-Hopping Molecular Design with Agentic Reinforcement Learning

MolHop：带有代理强化学习的支架跳跃分子设计

• Authors: Yanyi Su, Chenyan Wang, Yulei Gong, Yeyun Chen, Xiaohui Yang, Jun Cheng
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.19920
• Pdf link: https://arxiv.org/pdf/2610.19920
• Abstract Molecular design is often formulated as an optimization problem over existing molecular spaces. However, molecular optimization and the discovery of structurally distinct replacement molecules represent two fundamentally different search problems. LLM-based molecular design agents typically perform the former, conducting local refinement within the neighborhood of existing structures to improve target properties through structural modifications. In contrast, the latter requires exploration of a broader chemical space to discover molecules with entirely different scaffolds while preserving the functional characteristics of reference molecules. We propose MolHop, an agentic framework for cross-scaffold molecular discovery that enables models to navigate and edit within a pre-organized chemical space. MolHop organizes approximately 1.6 million molecular scaffolds from commercial molecules into a coarse-to-fine hierarchical action space. The agent selects new exploration regions through scaffold navigation actions and adjusts candidate properties through molecular editing. The framework maintains historical trajectory memory to support long-horizon search processes. On drug molecule and commercial compound tasks, connecting Grok-4.6 and DeepSeek-Flash-V4.1 to the MolHop framework substantially improves their cross-scaffold joint success rates. We develop MolHop-9B through agentic reinforcement learning, and MolHop-9B outperforms generative baselines including GPT-5.6-sol, Claude-5-Opus, and Gemini-3.1-Pro, while generating candidate molecules with larger scaffold differences from the reference molecules. These results demonstrate that effective molecular discovery beyond the traditional optimization paradigm requires structured chemical and action spaces, enabling agents to explore new structural regions rather than merely refine existing molecules.
• 中文摘要 分子设计通常被表述为对现有分子空间的优化问题。然而，分子优化和结构上不同替代分子的发现代表了两种根本不同的搜索问题。基于LLM的分子设计代理通常执行前者，在现有结构附近进行局部细化，通过结构改造改善靶点性质。相比之下，后者需要探索更广泛的化学空间，以发现具有完全不同支架的分子，同时保持参考分子的功能特性。我们提出了MolHop，一种用于跨支架分子发现的代理框架，使模型能够在预设的化学空间中导航和编辑。MolHop将约160万个商业分子支架组织成粗细层级作用空间。该代理通过支架导航动作选择新的探索区域，并通过分子编辑调整候选性质。该框架保持历史轨迹记忆，支持长视野搜索过程。在药物分子和商业化合物任务中，将Grok-4.6和DeepSeek-Flash-V4.1连接到MolHop框架，显著提升了其跨支架联合成功率。我们通过代理强化学习开发MolHop-9B，MolHop-9B优于生成基线，包括GPT-5.6-sol、Claude-5-Opus和Gemini-3.1-Pro，同时生成的候选分子与参考分子有更大的支架差异。这些结果表明，超越传统优化范式的有效分子发现需要结构化的化学和作用空间，使代理能够探索新的结构区域，而不仅仅是精炼现有分子。

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

SAM3RL：通过强化学习实现视觉物体追踪的记忆控制

• Authors: Tomáš Čížek, Matej Straka, Illia Volkov, Matěj Kripner, Ervin Macić, Jiri Matas, Klara Janouskova, Martin Schmid
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.19931
• Pdf link: https://arxiv.org/pdf/2610.19931
• Abstract Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.
• 中文摘要 Segment Anything Model 3（SAM 3）是图像和视频分割的基础模型，奠定了视觉对象追踪的技术水平。其追踪器继承自SAM 2，将过去帧的信息存储在内存库中，通过简单的先进先出规则管理，使复杂视频实现时间一致性。多种近期方法通过手工设计的内存更新规则改进SAM 2，每种规则针对特定故障模式，如干扰、遮挡或物体运动。然而，这些收益仅部分转移到更强大的SAM 3骨干网上。我们提出一种更通用的数据驱动方法，将内存控制视为顺序决策任务。SAM3RL是SAM 3追踪器的扩展，配备轻量级内存控制器，通过强化学习训练，决定哪些帧进入内存库以最大化跟踪性能。SAM3RL在12个跟踪和视频对象分割基准测试中10个实现了新的技术水平，在无运行时开销的情况下超越所有手工规则。同样的方法应用于SAM 2，使SAM2RL在12个基准测试中有9个优于手工规则，显示了我们方法跨模型世代的通用性。

Provable Transformer Context Compression for In-Context Reinforcement Learning

可证明的变换器上下文压缩用于上下文强化学习

• Authors: Yajie Zhao, Yingjun Mo, Zhiyong Wang, Bo Tang, Chao Xu, Fang Kong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20004
• Pdf link: https://arxiv.org/pdf/2610.20004
• Abstract Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context sizes that grow with the number of sampled transitions, leading to increasing memory costs in large-scale problems. This raises a fundamental scalability question: how much context is intrinsically necessary to carry out such learning? We answer this question for fixed-policy TD evaluation. We show that, with a shared linear decoder, the minimum context dimension required to exactly reproduce the first $L$ TD iterates is the Krylov dimension induced by the sampled transitions. When the available context dimension $m$ falls below this threshold, exact reproduction is no longer possible. In this regime, we characterize the optimal approximation error and show that it decays as $\Theta(\gamma^m\max{1,1/(m(1-\gamma))})$ where $\gamma$ is the discounting factor. Moving from representation complexity to actual memory cost, we characterize the optimal retained memory for exact continuous representations as $\Theta(nk_L)$ and show experimentally that on a sampled task whose exact rank is full but whose numerical rank is small, a context accurate to $10^{-2}$ is $88.7\%$ smaller than the sparse sufficient statistics of TD. Finally, we establish the learnability of the shared executor through pretraining by proving global convergence of an auxiliary objective over effective attention parameters under suitable task diversity. Experiments across diverse TD evaluation problems validate parameter recovery and the predicted context--accuracy tradeoff.
• 中文摘要 变换器可以在不更新参数的情况下进行上下文强化学习。然而，现有的理论构造通常要求上下文大小随着采样转移数量增加而增加，导致大规模问题中的内存成本增加。这引发了一个根本性的可扩展性问题：执行此类学习本质上需要多少上下文？我们将回答固定策略TD评估的问题。我们证明，使用共享线性解码器，精确重现TD的第一个$L$迭代所需的最小上下文维度是由采样转移诱导的Krylov维度。当可用的上下文维度$m$低于该阈值时，精确重现就无法实现。在此范围内，我们刻画了最优近似误差，并证明其衰减为$\Theta（\gamma^m\max{1,1/（m（1-\gamma））}）$，其中$\gamma$为折现因子。从表示复杂度转向实际内存成本，我们将精确连续表示的最优保留内存刻画为$\Theta（nk_L）$，并通过实验证明，在一个精确秩满但数值排名较小的抽样任务中，精确到$10^{-2}$的上下文比TD的稀疏充分统计量小$88.7\%$。最后，我们通过预训练证明辅助目标在适当任务多样性下对有效注意力参数的全局收敛性，来确定共享执行者的可学习性。在各种TD评估问题上的实验验证了参数恢复和预测的上下文-准确性权衡。

Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL

用双曲目标条件强化学习，超越了1000欧几里得层

• Authors: Timo Klein, Thomas Lang, Andrii Shkabrii, Lukas Miklautz, Yllka Velaj, Sebastian Tschiatschek
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.20009
• Pdf link: https://arxiv.org/pdf/2610.20009
• Abstract Scaling self-supervised contrastive reinforcement learning (CRL) to 1000-layer Euclidean neural networks has enabled state-of-the-art goal-reaching performance, yet wall-clock time scales linearly with depth. We find that changing the geometry achieves stronger performance with far shallower networks. This matters in complex navigation environments, where walls can make route distance deviate substantially from Euclidean distance. In a U-shaped maze, we show that the longer the route, the more it deviates from Euclidean distance. Hyperbolic CRL addresses this mismatch by replacing Euclidean distances in CRL's contrastive objective with geodesic distances in hyperbolic space. On hard navigation tasks, inductive bias beats scale: depth 64 hyperbolic agents outperform 1024-layer-deep Euclidean networks while training more than $12 \times$ faster and using about $13 \times$ fewer parameters. We observe that hyperbolic agents behave qualitatively better in hard humanoid mazes, where they follow corridors to reach goals, whereas Euclidean agents try straight-line shortcuts over walls. Our results show that matching the representation geometry to the task can substitute for substantial network depth. Code is at this https URL .
• 中文摘要 将自监督对比强化学习（CRL）扩展到1000层欧几里得神经网络，实现了最先进的目标达标性能，但壁钟时间随深度线性递增。我们发现，改变几何体能在更浅的网络中获得更强的性能。这在复杂的导航环境中尤为重要，因为墙壁可能使路线距离与欧几里得距离有显著偏离。在U形迷宫中，我们展示了路线越长，偏离欧几里得距离的越大。双曲CRL通过用双曲空间中的测地线距离替代CRL对比物镜中的欧几里得距离来解决这一不匹配。在困难导航任务中，归纳偏置胜过规模：深度64的双曲代理表现优于10²4层深的欧几里得网络，同时训练速度快12美元，参数数量减少约13倍。我们观察到双曲代理在硬质类人迷宫中表现优异，它们沿着走廊达成目标，而欧几里得代理则尝试直线捷径越墙。我们的结果表明，匹配表示几何与任务可以替代显著的网络深度。代码位于此 https URL 。

On the Convergence of Semi-Gradient TD: A Mean Field Perspective

关于半梯度TD的收敛：均场视角

• Authors: Haruki Settai, Tadashi Kozuno, Shinji Ito
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20014
• Pdf link: https://arxiv.org/pdf/2610.20014
• Abstract Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability. We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target. Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution. We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.
• 中文摘要 半梯度时间差分学习是强化学习的核心算法组成部分，但结合函数近似的自助法通常会诱导非保守的向量场，可能导致不稳定性。我们分析了平均场区内的两层神经网络，并识别出一种简单的稳定机制，其中熵引起的松弛主导移动引导目标的灵敏度。在此条件下，我们证明了指数收敛到唯一不动点，并量化其相对于非正则化贝尔曼解的偏置。我们还展示了同一机制通过软Q学习推广到熵正则化的非策略控制。

When Pairwise Preferences Cycle: A Reliability-Anchored Preference Walk for Video GRPO

当成对偏好循环：视频GRPO的可靠性锚定偏好游走

• Authors: Xinrui Lin, Fan Li, Shumin Wang, Dehong Kong, Yanyong Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.20089
• Pdf link: https://arxiv.org/pdf/2610.20089
• Abstract Pairwise preference feedback has become a widely adopted supervision signal for reinforcement learning in video generation models, as pairwise comparisons provide a more reliable alternative than pointwise scalar ratings for evaluating the quality of generated videos. Specifically, we define a preference graph for each GRPO group under a pairwise evaluator, where nodes represent sampled rollouts and edges encode pairwise preferences. Auditing 6,976 on-policy comparison graphs, we find that cyclic preferences are prevalent and persist throughout training. Naive pairwise reward aggregation based on average win rates discards the underlying comparison structures and treats each rollout independently, which may introduce unreliable advantage estimates for certain rollouts and consequently affect reinforcement learning optimization. Therefore, we propose the \emph{Reliability-Anchored Preference Walk} (\ours{}), a graph-aware advantage aggregation method that strengthens the global preference signal from win rates with reliable relational evidence from cyclic preference graphs through a reliability-guided preference walk. Under matched evaluator, comparison protocol, group size, prompt schedule, and query budget, \ours{} requires no additional evaluator queries beyond the observed comparisons within each rollout group and improves over Pairwise-GRPO across controlled comparisons, transfer settings, and blind human evaluations.
• 中文摘要 成对偏好反馈已成为视频生成模型中强化学习中广泛采用的监督信号，因为成对比较比点对标量评分提供了更可靠的替代方式来评估生成视频质量。具体来说，我们在成对评估器下为每个GRPO组定义了偏好图，节点表示采样的展开，边编码成对偏好。审计了6,976个策略上的比较图，我们发现循环偏好普遍存在，并且在整个训练过程中持续存在。基于平均胜率的朴素成对奖励聚合会丢弃底层比较结构，独立处理每个推广，这可能为某些推广引入不可靠的优势估计，从而影响强化学习优化。因此，我们提出了\emph{Reliability-Anchored Preference Walk}（\ours{}），这是一种基于图的优势聚合方法，通过可靠引导偏好行走，通过循环偏好图的可靠关系证据，强化了胜率的全局偏好信号。在匹配评估器、比较协议、组规模、提示时间表和查询预算下，\ours{}在每个推广组内观察到的比较外无需额外评估者查询，且在受控比较、转移设置和盲测人类评估中优于Pairwise-GRPO。

RM-OPD: On-Policy Distillation for Generative Reward Models

RM-OPD：生成奖励模型的政策提炼

• Authors: Qinglin Su, Licheng Pan, Zhen Fang, Jiakang Wang, Qingnan Ren, Yunsheng Lu, Yan Gao, Zhixuan Chu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.20174
• Pdf link: https://arxiv.org/pdf/2610.20174
• Abstract Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.
• 中文摘要 基于混合领域偏好数据的强化学习（Mix-RL）已成为训练通用生成奖励模型（GRMs）的主流方法。然而，Mix-RL存在两个关键瓶颈：结果奖励引起的奖励稀疏性，以及因联合优化异构能力而产生的跨域干扰，这两者共同导致能力获取不平衡和奖励黑客现象普遍存在。受多域能力集成中策略提炼（OPD）成功的启发，我们引入了RM-OPD（通过策略提炼的奖励建模），这是一个两阶段框架，将专业获取与能力集成区分开来。首先，我们通过单域GRPO微调培养领域专门教师模型，使每位专家能够单独达到其性能上限;然后，我们将这些教师整合为一个学生模型，通过（i）路由策略提炼（routed on-policy distillation），对学生生成的批评提供密集的签名代币级监督;（ii）领域贡献重权，将代币指导重新加权于目标领域份额，防止特定领域主导优化;以及（iii）可选条件上下文蒸馏，通过领域特定评估标准为教师设定条件，以加强代币级监督。在多个RM基准和两个基础模型中，RM-OPD始终优于普通Mix-RL及其他强基线，实现整体优越表现，并展现出新兴的“超越教师”效应。这些结果确立了RM-OPD作为构建通用GRM的有前景范式。

Noise-Aware Lombard Speech Synthesis via Flow-GRPO for Intelligibility Optimization

通过Flow-GRPO进行噪声感知伦巴第语音合成，实现可理解性优化

• Authors: Zhaoyang Wang, Dingding Yao, Chengzhong Wang, Junfeng Li
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.20181
• Pdf link: https://arxiv.org/pdf/2610.20181
• Abstract Voice-based conversational agents are increasingly deployed in real-world settings, where environmental noise often degrades the interactive experience. Therefore, robust acoustic adaptability is essential to maintain natural and intelligible human-computer communication. However, most existing systems remain acoustically insensitive and fail to emulate the human Lombard effect, in which speakers involuntarily adjust their vocal production to preserve intelligibility in the presence of noise. As a result, their communicative reliability degrades under adverse acoustic conditions. To remain intelligible in noisy environments, conversational agents require a mechanism to emulate noise-induced adaptive behavior during synthesis. We propose NIL-TTS, a noise-induced Lombard TTS framework. We integrate a feature-wise linear modulation (FiLM) mechanism into a flow-matching backbone for dynamic vocal adaptation. A two-stage training paradigm is utilized: initial supervised fine-tuning on paired Lombard corpora, followed by Flow-GRPO post-training to optimize for intelligibility. Evaluations confirm that NIL-TTS autonomously adjusts its output in response to varying noise conditions, exhibiting key correlates of the human Lombard effect. Driven by these adaptive modulations, the synthesized speech achieves significant intelligibility improvements that scale proportionally with noise intensity. Ultimately, this work presents a flow-matching and reinforcement learning-based advancement within the adaptive Lombard speech synthesis paradigm for challenging real-world scenarios.
• 中文摘要 基于语音的对话代理越来越多地被应用于现实环境中，环境噪声常常会降低交互体验。因此，强大的声学适应性对于维持自然且可理解的人机通信至关重要。然而，大多数现有系统仍然对声学不敏感，未能模拟人类伦巴第效应，即说话者在噪声存在下不自愿地调整发声以保持可理解性。因此，在恶劣的声学条件下，其沟通可靠性会下降。为了在嘈杂环境中保持可理解性，会话代理需要一种机制来模拟合成过程中噪声引起的适应行为。我们提出了NIL-TTS，一种噪声诱导的伦巴第TTS框架。我们将一个按特征方向的线性调制（FiLM）机制集成到一个动态人声适应的匹配主干网中。采用两阶段训练范式：对配对伦巴第语料库进行初步监督微调，随后进行Flow-GRPO后训练以优化可理解性。评估确认NIL-TTS能够自主调整输出以响应不同的噪声条件，展现出人类伦巴第效应的关键相关性。在这些自适应调制的驱动下，合成语音实现了显著的可理解性提升，且其可理解度与噪声强度成正比。最终，这项工作为适应伦巴第语音合成范式提供了基于流匹配和强化学习的进展，以应对具有挑战性的现实场景。

Value-Guided Repulsion for Offline Reinforcement Learning

价值引导排斥用于离线强化学习

• Authors: Yixue Hao, Jiahao Zhang, Jiesong Lian, Xufei Guo, Xumeng Guo, Wenhui Zhang, Long Hu, Rui Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20190
• Pdf link: https://arxiv.org/pdf/2610.20190
• Abstract Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.
• 中文摘要 漂移模型已成为离线强化学习（RL）中策略设计的高效且稳定的范式。然而，标准漂移策略仅根据生成动作之间的几何关系决定排斥，因此无法通过期望值区分它们。为解决这一限制，我们引入了价值引导排斥Q-学习（VRQL），将动作值估计整合到粒子交互场中。对于每个状态，VRQL结合几何核距离与归一化Q值尺度。该机制赋予较低值粒子更大的排斥影响，同时严格保留对行为行为的吸引力。因此，该价值引导交互场将行为正则化与策略改进对齐，而无需增加辅助行为模型阶段。在50个OGBench任务配置和6个D4RL AntMaze配置中，VRQL在既有基线中表现出极高的竞争力，表明价值引导粒子交互是离线强化学习的有效方法。

A Multiarmed Bandit Approach for LTE-U/Wi-Fi Coexistence in a Multicell Scenario

多基站场景中LTE-U/Wi-Fi共存的多臂盗贼方法

• Authors: Iago Diógenes do Rego, José M. de Castro Neto, Sildolfo F. G. Neto, Pedro M. de Santana, Vicente A. de Sousa Jr., Dario Vieira, Augusto Venâncio Neto
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.20222
• Pdf link: https://arxiv.org/pdf/2610.20222
• Abstract Recent studies and literature reviews have shown promising results for 3GPP system solutions in unlicensed bands when coexisting with Wi-Fi, either by using the duty cycle (DC) approach or licensed-assisted access (LAA). However, it is widely known that general performance in these coexistence scenarios is dependent on traffic and how the duty cycle is adjusted. Most DC solutions configure their parameters statically, which can result in performance losses when the scenario experiences changes on the offered data. In our previous works, we demonstrated that reinforcement learning (RL) techniques can be used to adjust DC parameters. We showed that a Q-learning (QL) solution that adapts the LTE DC ratio to the transmitted data rate can maximize the Wi-Fi/LTE-Unlicensed (LTE-U) aggregated throughput. In this paper, we extend our previous solution by implementing a simpler and more efficient algorithm based on multiarmed bandit (MAB) theory. We evaluate its performance and compare it with the previous one in different traffic scenarios. The results demonstrate that our new solution offers improved balance in throughput, providing similar results for LTE and Wi-Fi, while still showing a substantial system gain. Moreover, in one of the scenarios, our solution outperforms the previous approach by 6% in system throughput. In terms of user throughput, it achieves more than 100% gain for the users at the 10th percentile of performance, while the old solution only achieves a 10% gain.
• 中文摘要 近期研究和文献综述显示，当3GPP系统在无授权频段与Wi-Fi共存时，通过占空比（DC）方法或许可辅助接入（LAA）取得有希望的效果。然而，众所周知，这些共存场景中的总体性能依赖于流量及占空比的调整方式。大多数DC解决方案静态配置参数，当场景发生数据变化时，可能导致性能损失。在我们之前的工作中，我们展示了强化学习（RL）技术可用于调整DC参数。我们展示了将LTE DC比率调整为传输数据率的Q-learning（QL）解决方案，可以最大化Wi-Fi/LTE-Unlicensed（LTE-U）的总吞吐量。本文通过基于多臂强盗（MAB）理论实现更简单高效的算法，扩展了之前的解决方案。我们评估其性能，并在不同流量场景下与前述算法进行比较。结果显示，我们的新方案在吞吐量上提供了更好的平衡，在LTE和Wi-Fi上表现相似，同时仍显示出显著的系统收益。此外，在其中一个场景中，我们的方案在系统吞吐量方面比之前的方法高出6%。在用户吞吐量方面，它为性能第10百分位的用户带来了超过100%的提升，而旧方案仅提升了10%。

Design Space Exploration of Reinforcement Learning Configurations for Post-Legalization Placement

设计后合法化后强化学习配置的空间探索

• Authors: Foteini Oikonomou, Maria Pantazi-Kypraiou, George Goudroumanis, Pavlos Stoikos, Athanasios Tziouvaras, Christos Anagnostopoulos, Georgios Karakonstantis, George Floros
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20224
• Pdf link: https://arxiv.org/pdf/2610.20224
• Abstract Graph Neural Networks (GNNs) and Reinforcement Learning (RL) have recently been explored for VLSI placement optimization, but their effectiveness for post-legalization placement refinement remains unclear. This paper presents a systematic design space exploration of a GNN-Proximal Policy Optimization (GNN-PPO) framework across three ISPD2015 benchmarks. We evaluate 18 configurations covering reward functions, GNN architectures, action spaces, candidate-set sizes, and training strategies. We analyse the results we obtain to identify pitfalls and to assess the prospects and opportunities of the RL methods for post-legalization placement. On top of that we propose a design classification approach to guide the design decisions of RL models.
• 中文摘要 图神经网络（GNN）和强化学习（RL）最近被探索用于VLSI安置优化，但它们在合法化后安置优化中的有效性尚不明确。本文提出了一个跨ISPD2015基准测试的GNN-近端策略优化（GNN-PPO）框架系统性的设计空间探索。我们评估了18种配置，涵盖奖励函数、GNN架构、动作空间、候选集大小和训练策略。我们分析所得结果，识别陷阱，并评估强化学习方法在合法化后安置中的前景和机遇。此外，我们还提出了一种设计分类方法，以指导RL模型的设计决策。

Multi-Cell LTE-U/Wi-Fi Coexistence Evaluation Using a Reinforcement Learning Framework

使用强化学习框架进行多单元LTE-U/Wi-Fi共存评估

• Authors: José M. de C. Neto, Sildolfo F. G. Neto, Pedro M. de Santana, Vicente A. de Sousa Jr
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.20230
• Pdf link: https://arxiv.org/pdf/2610.20230
• Abstract Cellular broadband Internet of Things (IoT) applications are expected to keep growing year-by-year, generating demands from high throughput services. Since some of these applications are deployed over licensed mobile networks, as long term evolution (LTE), one already common problem is faced: the scarcity of licensed spectrum to cope with the increasing demand for data rate. The LTE-Unlicensed (LTE-U) forum, aiming to tackle this problem, proposed LTE-U to operate in the 5 GHz unlicensed spectrum. However, Wi-Fi is already the consolidated technology operating in this portion of the spectrum, besides the fact that new technologies for unlicensed band need mechanisms to promote fair coexistence with the legacy ones. In this work, we extend the literature by analyzing a multi-cell LTE-U/Wi-Fi coexistence scenario, with a high interference profile and data rates targeting a cellular broadband IoT deployment. Then, we propose a centralized, coordinated reinforcement learning framework to improve LTE-U/Wi-Fi aggregate data rates. The added value of the proposed solution is assessed by a ns-3 simulator, showing improvements not only in the overall system data rate but also in average user data rate, even with the high interference of a multi-cell environment.
• 中文摘要 蜂窝宽带物联网（IoT）应用预计将逐年增长，带来高吞吐量服务的需求。由于部分应用部署在持牌移动网络上，作为长期演进（LTE），一个已存在的问题是：持证频谱的稀缺，以应对日益增长的数据速率需求。LTE-Unlicensed（LTE-U）论坛旨在解决这一问题，提议将LTE-U应用于5 GHz无执照频段。然而，Wi-Fi已是该频段内的整合技术，此外，无执照频段的新技术需要机制促进与传统技术的公平共存。本研究扩展文献，分析多小区LTE-U/Wi-Fi共存场景，该场景高干扰特征和数据速率针对蜂窝宽带物联网部署。随后，我们提出了一个集中、协调的强化学习框架，以提升LTE-U/Wi-Fi汇总数据速率。通过ns-3模拟器评估该方案的附加价值，显示出不仅整体系统数据速率的提升，也提升了平均用户数据速率，即使多细胞环境干扰较大。

Adversarial Reinforcement Learning for Robust Planetary Landing

强健行星着陆的对抗强化学习

• Authors: Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio, Hodei Urrutxua, Víctor Rodríguez-Fernández
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.20252
• Pdf link: https://arxiv.org/pdf/2610.20252
• Abstract Deep reinforcement learning (DRL) policies for planetary landing and other aerospace guidance, navigation, and control tasks are commonly trained under randomized disturbances to improve robustness. However, typical stochastic perturbation sampling assumes independent variations, whereas real mission disturbances often exhibit spatial and temporal correlation and cannot be reproduced through independent random draws. This work investigates adversarial reinforcement learning to discover challenging off-nominal conditions. A secondary agent generates physically constrained perturbations that degrade the performance of a landing controller, forming an asymmetric sequential zero-sum game. Training proceeds in three stages: baseline controller learning, robustness enhancement with stochastic perturbations, and an iterative adversarial phase in which the controller and disturbance generator are alternately optimized. Preliminary experiments in a three-degree-of-freedom landing environment show that adversarial disturbances reveal vulnerabilities missed by conventional randomization and provide early indications that, with a more extensive training campaign, the approach can yield improved robustness and stronger generalization under previously unseen perturbation regimes.
• 中文摘要 行星着陆及其他航空航天导航和控制任务的深度强化学习（DRL）策略通常在随机扰动下训练以提高鲁棒性。然而，典型的随机扰动采样假设存在独立变化，而真实任务扰动往往表现出空间和时间相关性，无法通过独立随机抽取重现。本研究研究对抗强化学习，以发现具有挑战性的非标值条件。次级智能体生成物理约束扰动，降低着陆控制器的性能，形成非对称顺序零和博弈。训练分为三个阶段：基线控制器学习、随机扰动增强鲁棒性，以及迭代对抗阶段，在该阶段交替优化控制器和扰动生成器。三自由度着陆环境下的初步实验表明，对抗性扰动揭示了传统随机化忽略的脆弱性，并早期表明，通过更广泛的训练活动，该方法可以在此前未见的扰动环境中实现更强的鲁棒性和更强的泛化。

Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields

采用Receding-Horizon Safety Critic和可微分动作盾牌的安全强化学习

• Authors: Siwei Ju, Jan Peters, Oleg Arenz
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.20255
• Pdf link: https://arxiv.org/pdf/2610.20255
• Abstract Despite the Markov assumption, safety failures in robot reinforcement learning rarely occur instantaneously; they typically emerge over several steps as the policy enters unfamiliar states under disturbances and domain shift. We address this limitation by learning a receding-horizon safety critic that predicts whether a state will remain non-terminating over the next receding horizon, enabling anticipatory detection of impending locomotion failures. The critic is trained from rollout-derived horizon labels with a composite objective that combines horizon classification, control-barrier-function-inspired temporal consistency, and smoothness regularization, and we aggregate an ensemble of critics with a conservative Conditional Value at Risk estimate to account for uncertainty. Building on the learned critic, we derive a closed-form, differentiable CBF-style safety filter that minimally modifies the policy action to improve predicted safety, and integrate it into PPO with an anticipation loss to reduce policy-filter mismatch during training. In simulated quadruped locomotion, our method improves robustness under action noise, external perturbations, and terrain shifts, reducing catastrophic failures while maintaining competitive returns.
• 中文摘要 尽管有马尔可夫假设，机器人强化学习中的安全性失效很少瞬间发生;它们通常在策略进入陌生状态时，在扰动和领域转移下逐步出现。我们通过学习一个远征视界安全批评者来解决这一限制，该批评者预测状态在下一个远征视界内是否会保持不终止状态，从而实现对即将到来的运动失效的预判性检测。该批评者基于滚出衍生的视界标签训练，目标物结合了视界分类、控制障碍函数启发的时间一致性和平滑正则化，我们汇总一组批评者，并以保守的“风险条件值”估计来考虑不确定性。基于已学批评者，我们推导出一种闭式、可微分的CBF式安全滤波器，对策略动作进行最小化修改以提升预测安全性，并将其集成到PPO中，并带有预期损失，以减少训练期间的策略滤波不匹配。在模拟四足行走中，我们的方法提升了在动作噪声、外部扰动和地形变化下的鲁棒性，减少灾难性失效，同时保持竞争力。

TANGO: RNA Topology and Geometry Co-Design

TANGO：RNA拓扑与几何的共同设计

• Authors: Tianmeng Hu, Biao Luo, Ke Li
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.20277
• Pdf link: https://arxiv.org/pdf/2610.20277
• Abstract Designing RNA sequences that satisfy target tertiary-structure objectives is an important problem in RNA engineering, with broad applications in synthetic biology and therapeutics. Existing 3D RNA design methods have made substantial progress, but they largely treat the target fold as a geometry-only objective. This is incomplete for RNA, whose functional folds are specified not only by tertiary geometry but also by secondary base-pairing topology. We therefore formulate RNA inverse design as a topology-and-geometry co-design problem and propose TANGO, a multi-objective multi-agent reinforcement learning framework for this task. TANGO adopts a staged curriculum: it first learns a topology-aware policy from secondary-structure targets, then transfers this policy to tertiary-structure targets to train a general co-design model, and finally applies frontier-guided fine-tuning to refine target-specific trade-offs among secondary-structure fidelity, tertiary-structure fidelity, and sequence diversity. Experiments show that TANGO generates diverse and novel designs, improving tertiary- and secondary-structure performance by 18.3% and 52.7%, respectively, over the strongest baseline.
• 中文摘要 设计满足目标三级结构目标的RNA序列是RNA工程中的重要问题，广泛应用于合成生物学和治疗学。现有的3D RNA设计方法取得了显著进展，但它们大多将目标折叠视为仅几何目标。这对RNA来说是不完整的，因为RNA的功能折叠不仅由三级几何指定，还由次级碱基配对拓扑决定。因此，我们将RNA逆设计表述为一个拓扑与几何的共设计问题，并提出了TANGO，一种多目标多智能体强化学习框架。TANGO采用分阶段课程：首先从次级结构目标学习拓扑感知策略，然后将该策略转移到三级结构目标以训练通用共设计模型，最后应用前沿引导微调以优化二级结构保真度、三级结构保真度和序列多样性之间的目标特定权衡。实验显示，TANGO生成多样且新颖的设计，三级和二级结构性能分别提升18.3%和52.7%，在最强基线条件下表现显著。

Non-asymptotic Convergence of Average-reward Q-learning with Options

与选项的平均奖励Q学习非渐近收敛

• Authors: Kintan Saha, Ahana Deb, Anders Jonsson, Debabrota Basu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20347
• Pdf link: https://arxiv.org/pdf/2610.20347
• Abstract We study the finite-sample convergence of Q-learning algorithms for the average-reward formulation of Reinforcement Learning (RL) with options. While options allow temporarily extended and hierarchical actions, the average-reward formulation incorporates continual tasks beyond discounted and episodic RL. We establish the first $\tilde{O}(1/\text{iterations})$ convergence guarantees, in the mean-square sense, for average-reward option Q-learning algorithms in both the inter- and intra-option settings. First, we prove that both inter- and intra-option Bellman operators satisfy multi-step span contractions under unichain assumptions. Then, we propose synchronous estimators of these multi-step option Bellman operators leading to two variants of option Q-learning. Finally, we extend the stochastic approximation framework to prove convergence rates of these two algorithms under the minimal assumptions used in the literature.
• 中文摘要 我们研究了Q学习算法在强化学习（RL）与期权平均奖励表述中的有限样本收敛性。虽然期权允许暂时扩展和层级动作，而平均奖励表述则包含了超越贴现和情节式RL的连续任务。我们建立了第一个$\tilde{O}（1/\text{iterations}）$收敛保证，适用于平均奖励选项Q学习算法在选项间和选项内的学习算法。首先，我们证明了在单链假设下，选项间和选项内的Bellman算子都满足多步跨度收缩。随后，我们提出了这些多步Bellman算子的同步估计，导致选项Q学习的两种变体。最后，我们扩展了随机近似框架，以证明文献中使用的最小假设下这两种算法的收敛率。

Training Language Models to Fail Gracefully

训练语言模型以优雅地失败

• Authors: Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh, Fahim Tajwar, Abitha Thankaraj, Ruslan Salakhutdinov, Haiwen Feng, Yuda Song, Andrea Zanette
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.20401
• Pdf link: https://arxiv.org/pdf/2610.20401
• Abstract Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.
• 中文摘要 强化学习使语言模型更擅长解决任务，但同样的机制也奖励猜测。当成功是获得奖励的唯一途径时，选择安全的退选总是比冒险尝试更糟糕。当错误的行为可能破坏数据库或发送损坏代码时，这种激励可能代价高昂。因此，我们提出训练模型优雅地失败：尽可能保持成功行为，同时学习在行动不可靠时执行安全的退回策略。然而，天真地使用带有中间奖励的策略梯度来进行退回行为，由于优化动态不利，是脆弱的。我们用GracefulRL来解决这一挑战，它重塑了优化格局，促进优雅失败。重要的是，这种行为直接源自结果级奖励信号，没有注释说明何时退回，也没有额外的推理时间机制如验证器。在各种任务和尺度上，我们训练模型在行动不可靠时选择性地回退任务解决能力。更广泛地说，我们的结果表明，强化学习不仅能激励模型成功，还能推理其能力的极限，并在判断成功无法实现时改变方向。

Hyperspherical Contrastive Representations Unlock Planning and Exploration

超球对比表示解锁了规划与探索的门槛

• Authors: Jonghae Park, Sungil Son, Jonghun Shin, Junseo Lee, H. Jin Kim, Benjamin Eysenbach
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.20409
• Pdf link: https://arxiv.org/pdf/2610.20409
• Abstract We develop a self-supervised reinforcement learning algorithm that learns to build towers of up to eight cubes from scratch, without prior data, rewards, demonstrations, or pretrained models. Our method, Contrastive Planning via Spherical Linear Interpolation (C-Slerp), uses contrastive representations as reusable building blocks and repeatedly composes them to infer a sequence of intermediate waypoints. Under our hyperspherical parameterization, the learned representations allow intermediate waypoints to be inferred in closed form through spherical linear interpolation, without training a separate high-level policy or performing graph search. Importantly, the representation geometry also tells us how to adjust the number of waypoints based on the difficulty of each goal-reaching problem. Experiments show that our method succeeds on long-horizon navigation and robotic manipulation tasks, including settings where strong baselines fail to achieve a single success. Ablations further show that using a fixed number of waypoints substantially degrades performance, highlighting the importance of adapting the plan to task difficulty.
• 中文摘要 我们开发了一种自监督强化学习算法，能够从零开始构建最多八个立方体的塔，无需先前数据、奖励、演示或预训练模型。我们的方法“球面线性插值对比规划”（C-Slerp）将对比表示作为可重复使用的构建单元，反复组合以推断中间路径点序列。在我们的超球面参数化下，学习到的表示允许通过球面线性插值以封闭形式推断中间路径点，无需训练单独的高级策略或进行图搜索。重要的是，表示几何还告诉我们如何根据每个目标达成问题的难度调整路径点数量。实验表明，我们的方法在长视野导航和机器人操作任务中取得成功，包括强基线未能实现单一成功的环境。消融进一步表明，使用固定数量的航点会显著降低性能，凸显了根据任务难度调整计划的重要性。

When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?

Momentum在何时有助于带可验证奖励的强化学习？

• Authors: Kaifei Wang, Zikang Shan, Liwei Wang, Yinyu Ye, Han Zhong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.20413
• Pdf link: https://arxiv.org/pdf/2610.20413
• Abstract Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient $\beta$ controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing $\beta$ within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate $\eta$ decreases, the expected gain with fixed $\beta$ vanishes faster than these fluctuations, whereas increasing $\beta$ to keep $\eta/(1-\beta)$ within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing $\beta_1$ improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.
• 中文摘要 动量是加速凸优化的经典工具，也是现代大型语言模型训练的标准组成部分。然而，近期实证研究质疑了第一时刻动量在可验证奖励强化学习（RLVR）中的实用性。我们通过RLVR的带偏置修正指数移动平均（EMA）动量的bandit表述来研究这个问题，其中EMA系数$\beta$控制过去梯度权重衰减的速度。当精确评估策略梯度时，我们证明在防止超跃的范围内增加$\beta$严格加速优化。当从采样响应估算梯度时，正的期望增益可能被采样波动所掩盖。我们表明，随着学习率 $\eta$ 下降，固定 $\beta$ 的期望增益比这些波动为零更快，而增加 $\beta$ 以保持 $\eta/（1-\beta）$ 在安全区间，则保持一个最终占主导的非零增益。Bandit 模拟定量验证了这些预测。使用 SGD 配合 EMA 和 AdamW 在 Qwen3-4B-Base 上的 RLVR 实验支持了我们在实际训练后大型语言模型中选择更大系数的原则。在 Qwen2.5-32B 搭配 AdamW 的基础上，仅仅增加 $\beta_1$ 即可使平均验证准确率比传统选择提升近 $2 百分点，展示了该原理在更大模型尺度下的有效性。

A Practical Framework for Incremental Testing and Technology Maturation of Spacecraft Autonomy

航天器自主性渐进测试与技术成熟的实用框架

• Authors: Kerianne Hobbs, Ben Bycroft, Pamela Wood, Mathew W. Bissonnette, M.J. Durst, Kimberly Sover, Alexandra Mangel, Gabriel Behrendt, Upneet Singh, Juan Rodriguez, Lukas Dillingham, Zeve Akerling, Nicholas Oune, Daniel Thomlinson, Evan N. Sperber, Denise Burrell, Phaedrus Leeds, Brian Hardy, Nathaniel Hamilton, Raul Santos, Arturo de la Barcena, Kyle Dunlap
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.20433
• Pdf link: https://arxiv.org/pdf/2610.20433
• Abstract This paper addresses the "trust gap" in deploying AI-enabled spacecraft autonomy through an integrated verification framework combining offline ground testing with online flight monitoring. The offline phase progresses systematically through simulation studies, processor-in-the-loop (PIL), hardware-in-the-loop (HIL), and robotic emulation testing, culminating in flight validation. The online phase employs AI watchdog systems for performance monitoring and safety guardrails for real-time constraint enforcement. Key contributions include: (1) systematic organization of verification techniques emphasizing closed-loop behavioral testing at each phase, (2) integration of AI watchdogs with runtime assurance, and (3) demonstration of the framework through a reinforcement learning-based attitude control case study. This approach integrates mature technologies to provide a viable pathway for safe deployment of AI-enabled autonomy in high-consequence space operations while addressing simulation-to-reality transfer challenges.
• 中文摘要 本文通过结合离线地面测试与在线飞行监控的集成验证框架，解决了部署AI驱动航天器自主能力的“信任差距”问题。离线阶段系统性地通过仿真研究、处理器在环（PIL）、硬件在环（HIL）和机器人仿真测试，最终以飞行验证为高潮。在线阶段采用AI看门狗系统进行性能监控，并设置安全护栏以实现实时约束执行。主要贡献包括：（1）系统化组织验证技术，强调每个阶段的闭环行为测试，（2）将AI看门狗与运行时保证集成，（3）通过基于强化学习的姿态控制案例研究演示该框架。该方法整合成熟技术，为高影响空间操作中安全部署AI自主性提供了可行路径，同时解决模拟到现实转移的挑战。

Average-Reward Policy Identification from a Single Continuing Trajectory

单一持续轨迹中的平均奖励保单识别

• Authors: Tongxi Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.20486
• Pdf link: https://arxiv.org/pdf/2610.20486
• Abstract In reinforcement learning, an agent must explore an unknown environment to learn how to make good decisions. Unlike methods that can freely sample any state--action pair, an agent learning from a single trajectory must spend additional time navigating between states. It is therefore natural to expect that learning from a single trajectory requires more experience. In this paper, we show that this additional cost is not always necessary. For average-reward reinforcement learning, we develop an algorithm that learns where to collect information while ensuring that the required samples can be obtained efficiently. We establish an upper bound on the number of interactions needed to identify a near-optimal policy. Furthermore, we prove a matching information theoretic lower bound, showing that, for a broad class of environments and at sufficiently high accuracy, learning from a single trajectory is no harder in the worst case than learning with unrestricted access to state action samples. Our results reveal that the additional complexity cost of navigation can be avoided by adapting information gathering requirements to how the environment provides information.
• 中文摘要 在强化学习中，智能体必须探索未知环境，学习如何做出良好决策。与可以自由采样任意状态-动作对的方法不同，从单一轨迹学习的智能体必须花费更多时间在状态间导航。因此，自然而然地预期从单一轨迹学习需要更多经验。本文表明，这种额外成本并非总是必要的。对于平均奖励强化学习，我们开发了一种算法，学习信息采集地点，同时确保高效获得所需样本。我们建立了识别近优策略所需交互次数的上界。此外，我们证明了一个匹配的信息理论下界，表明对于广泛类别且准确性足够高的环境，从单一轨迹学习在最坏情况下并不比无限制访问状态动作样本学习更难。我们的结果表明，通过调整信息收集需求以适应环境提供信息的方式，可以避免导航带来的额外复杂性成本。

Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling

事后诸葛亮的RLVR：无需群体抽样的单次推广代币信用

• Authors: Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, Min Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20497
• Pdf link: https://arxiv.org/pdf/2610.20497
• Abstract Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过检查任务结果训练语言模型，但为单个推理标记或行动分配终端奖励仍然困难。群体相对方法如群体相对策略优化（Group Relative Policy Optimization，GRPO）在每个提示中抽样多次尝试，但当所有尝试结果相同时，其基于奖励的学习信号就会丢失。我们提出了事后视角特权RLVR（HP-RLVR），这是一种单次展开方法，将已完成轨迹的未来转化为代币级更新权重。生成策略快照根据每个符合条件的代币根据其因果历史和包含实现的未来和验证者反馈的事后视角上下文进行评分。两种观点之间的归一化Jensen-Shannon 分歧会根据历史基线加权奖励，成功和失败尝试均有界的代币积分，且无参考解或学到批评者。在Qwen3-8B配对运行中，采用共享100美元步进计划，HP-RLVR在数学、代码生成和ALFWorld上训练速度比GRPO快1.21美元、1.46倍和2.30倍次，每个提示使用了八分之一的推广次数。整体准确率相当，HP-RLVR在三种规模的数学上领先，GRPO在代码和ALFWorld上领先。在ALFWorld中，GRPO的梯度范数记录为零，4B为43.2美元（4B）和49.0%$（8B），而HP-RLVR为零。

PrismaCoder: Pushing the Boundaries of Open Multimodal Code Intelligence through Decomposed Rubric Synthesis

PrismaCoder：通过分解评分规包综合推动开放多模态代码智能的边界

• Authors: Qiushi Sun, Jingyang Gong, Yang Liu, Qiaosheng Chen, Zhangyue Yin, Zixian Huang, Hanxu Hu, Jing Xiong, Fangzhi Xu, Haochen Tan, Haoli Bai, Lifeng Shang, Xiaoguang Li, Ben Kao, Qipeng Guo, Lingpeng Kong, Jiahui Gao
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.20538
• Pdf link: https://arxiv.org/pdf/2610.20538
• Abstract Multimodal code intelligence connects the symbolic world of programs with the visual world their execution creates. Interest in this capability is burgeoning, as models are increasingly asked to produce charts, webpages, animations, and semi-structured documents. Recently, proprietary models have pushed far ahead, producing interactive artifacts even from a text description or a screenshot alone. Yet their data and training remain closed, and open progress is held back by the quantity and quality of available data and by the pipelines and methods for producing it. Quality is the harder of the two, since a sample must be correct as a program, well built, and above all visually right once rendered. No existing signal covers all three. Execution often returns just a pass or fail, code similarity does not map to the visual output, and task-specific verifiers do not generalize. To push open progress further, we start from the data side with PrismaForge, a hierarchical data pipeline centered on a synthesized rubric that decomposes these factors into checkable criteria. The same rubric serves as both filter and reward: it curates the supervised corpus, then drives reinforcement learning on visual-coding tasks. Leveraging PrismaForge for data synthesis and filtering, we build PrismaCoder-2M, the most diverse open dataset for multimodal code intelligence and the largest to date. On the model side, we train PrismaCoder-9B and PrismaCoder-35B on this data and release both. They lead the open state of the art by a wide margin and rival leading proprietary systems. Our analyses further show that the proposed rubric-based pipeline improves multimodal coding performance across model backbones. We release our code and data at this https URL.
• 中文摘要 多模态代码智能将程序的符号世界与其执行所创造的视觉世界连接起来。随着模型越来越多地被要求生成图表、网页、动画和半结构化文档，这一能力的兴趣日益增长。近年来，专有模型取得了很大进步，仅凭文本描述或截图就能产生交互式的伪影。然而，它们的数据和训练仍然封闭，开放的进展受限于可用数据的数量和质量，以及生成数据的管道和方法。质量是两者中更难的，因为样本必须作为程序正确、构建良好，最重要的是渲染后视觉上正确。没有现有信号能覆盖这三者。执行通常仅返回通过或失败，代码相似度不映射到视觉输出，任务特定的验证器也不会进行泛化。为了进一步推进开放进展，我们从数据端开始使用 PrismaForge，这是一个以综合评分标准为中心的分层数据流水线，将这些因素分解为可检验的标准。同一评分标准既是过滤器，也是奖励：它策划监督语料库，然后推动可视化编码任务的强化学习。利用 PrismaForge 进行数据综合和过滤，我们构建了 PrismaCoder-2M，这是多模态代码智能领域最多样化的开放数据集，也是迄今为止最大的。在模型端，我们基于这些数据训练 PrismaCoder-9B 和 PrismaCoder-35B，并发布两者。它们在开放技术中遥遥领先，且可与领先的专有系统竞争。我们的分析进一步表明，基于评分标准的流水线提升了模型骨干上的多模态编码性能。我们将代码和数据发布至此 https URL。

Player-1: Learning to Play from Synthetic Experience

玩家1：从合成经验中学习游戏

• Authors: Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng, Honghao Chen, Ying Shan, Mike Zheng Shou, Hwee Tou Ng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.20584
• Pdf link: https://arxiv.org/pdf/2610.20584
• Abstract Computer-use agents have demonstrated strong capabilities in interpreting screens and executing keyboard and mouse actions. However, games, with their visually rich worlds and diverse mechanics, remain challenging and offer a demanding next step toward general digital control. Current game-agent learning pipelines often rely on large-scale demonstrations with limited coverage of dynamic game states, while scalable environments for interactive training remain difficult to obtain. We introduce Player-1, a post-training framework for learning game control beyond demonstrations through synthetic experience. We extend the computer-use action space with relative pointer motion, sustained inputs, and composite actions. For supervised initialization, we fine-tune computer-use models on 6,044 curated gameplay trajectories that pair recorded actions with task instructions and visual reasoning. We construct Player-1-Gym, sixteen synthetic 2D and 3D environments designed around control demands shared across games. Programmatically verifiable rewards and randomized observations and control dynamics support multi-environment reinforcement learning with PPO. The resulting Player-1-9B and Player-1-35B-A3B improve MineDojo-300 success over their SFT policies by 27.8 and 17.7 percentage points, respectively. Player-1-9B reaches 38.33% success compared with 12.63% for the strongest evaluated baseline, with further gains over SFT on ViZDoom-10 and GameWorld. Further analyses show that broader synthetic environment coverage improves transfer, while the use of composite actions that combine multiple keyboard and mouse inputs increases during RL. We demonstrate that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite.
• 中文摘要 计算机代理已展现出强大的屏幕解读能力和执行键盘和鼠标操作的能力。然而，游戏凭借其视觉丰富的世界和多样的机制，依然具有挑战性，并为迈向通用数字控制提供了艰难的下一步。当前的游戏代理学习流程通常依赖于对动态游戏状态有限的大规模演示，而可扩展的交互式训练环境仍然难以获得。我们引入了Player-1，这是一个超越演示的后培训框架，通过合成体验学习游戏控制。我们通过相对指针运动、持续输入和复合动作扩展了计算机使用的动作空间。对于监督初始化，我们对6044个精心策划的游戏轨迹进行微调计算机使用模型，将录制的动作与任务指令和视觉推理结合起来。我们构建了Player-1-Gym，这是16个合成的2D和3D环境，围绕游戏间共享的控制需求设计。可程序验证的奖励以及随机观察和控制动态支持PPO的多环境强化学习。由此产生的Player-1-9B和玩家1-35B-A3B在SFT策略上的成功率分别提升了27.8个百分点和17.7个百分点。玩家1-9B的成功率达到38.33%，而最强评估基线的成功率为12.63%，并且在ViZDoom-10和GameWorld上相比SFT也有进一步提升。进一步分析显示，更广泛的合成环境覆盖改善了迁移，同时结合多键鼠输入的复合动作在强化学习中有所增加。我们证明，从合成经验中学习能显著提升监督初始化之外的游戏控制，且其提升可推广到强化学习训练套件之外。

Co-Evolving Acting and Forecasting for Language Agents

语言代理的共演化作用与预测

• Authors: Haofei Yu, Yixing Chen, Yining Zhao, Xuyan Ye, Yuxiang Liu, Zhigang Hua, Yan Xie, Shuang Yang, Jiaxuan You
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.20597
• Pdf link: https://arxiv.org/pdf/2610.20597
• Abstract Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose \name, which adds a third kind of multi-step action targets. On successful on-policy trajectories, \name trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $\tau^2$-bench, and AppWorld), \name (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at this https URL.
• 中文摘要 语言代理的训练因长期推广的高昂成本而成为瓶颈，而标准的智能体强化学习（RL）仅从终端标量奖励中学习，导致中间步骤的监督基本未被使用。以往工作通过估算每步成功概率的价值目标或符合环境响应的观察目标，利用了这种监督。这两者都难以构建良好：（1）价值目标来自轨迹中每一步的同一单一标量信号，（2）观察目标，如网页和终端日志，时间长且噪声大，仅能领先一步。我们提出了\name，它增加了第三种多步动作目标。在成功的策略中轨迹中，\name训练策略骨干，生成跟踪每个历史的动作序列。每个动作在预测前一历史时也会被监督，形成一个时间系综，共享骨干在训练过程中吸收，从而使行动和预测能够协同演化。该策略仍然一步步进行推断，且无额外开销。在五个基准测试（ALFWorld、ScienceWorld、WebShop、$\tau^2$-bench 和 AppWorld）中，\name （1） 在模拟环境中通过预测辅助目标实现了比GRPO和RL基线更强的汇总性能，同时提升了两个现实应用的整体性能;（2） 在ScienceWorld上的采样效率分别比ECHO和GRPO高出1.9美元和2.0倍倍;（3） 通过单一策略骨干实现这些提升，无需额外模型、单独预训练或推理开销。代码库可在该 https URL 访问。

Modeling and Understanding Neural Weights of Humanoid Policies with Diffusion

模拟与理解人形政策的神经权重与扩散

• Authors: Huixuan Cheng, Bike Zhang, Shankar Sastry, Koushil Sreenath
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.20606
• Pdf link: https://arxiv.org/pdf/2610.20606
• Abstract Recent advances in robot learning have driven rapid progress in humanoid control, enabling increasingly agile behaviors in the real world. However, humanoid control networks are typically obtained through conventional training procedures, while comparatively little attention has been given to directly modeling the distribution of their learned weights and generating functional networks in weight space. In this work, we investigate whether high-performing humanoid control networks can be generatively modeled directly through their weights. We adopt a latent diffusion framework that couples a linear autoencoder with a Diffusion Transformer. The autoencoder compresses actor network parameters into a compact latent space, and the Diffusion Transformer models the distribution of these latents. Trained on networks collected from reinforcement learning for the Unitree G1 robot, our approach generates deployable networks that perform various motions in the real world, including walking, dancing, and spin kick. Beyond generation, we further study the learned weight space to better understand the structure captured by these humanoid control networks. We identify a structural principle in the first-layer weights that consistently reflects the robot's kinematic prior. Notably, this kinematic prior emerges under both Reinforcement Learning (RL) and Imitation Learning (IL) training, suggesting that it is not specific to a particular optimization process but may instead reflect common knowledge associated with humanoid embodiment. Together, these results demonstrate the feasibility of generative modeling in humanoid network weight space, while providing initial insights into the information encoded by their learned weights. Additional results are available on our project website: this https URL
• 中文摘要 机器人学习的最新进展推动了类人生物控制的快速进展，使现实世界中的行为更加灵活。然而，类人生物控制网络通常通过传统训练程序获得，而对直接建模其学习权重分布和在权重空间中生成功能网络的关注相对较少。在本研究中，我们探讨了高性能类人生物控制网络是否可以通过其权重直接生成建模。我们采用了一种潜在扩散框架，将线性自编码器与扩散变换器耦合。自编码器将演员网络参数压缩到紧凑的潜在空间，扩散变换器则建模这些潜在变量的分布。我们的方法基于Unitree G1机器人强化学习收集的网络训练，生成可部署的网络，在现实世界中执行多种动作，包括行走、舞蹈和旋转踢腿。除了生成之外，我们还进一步研究了学习权重空间，以更好地理解这些类人生物控制网络所捕捉的结构。我们在第一层权重中识别出一个结构原则，该原则始终反映机器人的运动学先验。值得注意的是，这一运动学先验在强化学习（RL）和模仿学习（IL）训练下均有出现，表明它并非特定优化过程的特有，而是反映了与类人生物具象化相关的共同知识。这些结果共同展示了在类人网络权重空间中生成建模的可行性，同时为其学习权重编码的信息提供了初步见解。更多结果可在我们的项目网站获取：此 https URL

Keyword: diffusion policy

Collective Learning for Unified Robotic Manipulation via Predictive Dynamics Adaptation

通过预测动力学适应实现统一机器人操作的集体学习

• Authors: Shaocheng Liu, Yuan Meng, Zhenshan Bing, Alois Knoll
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.19141
• Pdf link: https://arxiv.org/pdf/2610.19141
• Abstract Learning a collective policy for state-based Cartesian manipulation across diverse robotic embodiments presents a fundamental challenge due to embodiment-dependent transition dynamics. Collective learning aggregates multi-robot experience to discover shared task structure, yet naive aggregation can average incompatible action modes when different embodiments realize the same Cartesian command through different transition behavior. This leads to strategy ambiguity and performance collapse on dynamically outlying embodiments. To address this, we propose PACE, a two-stage collective learning framework that separates shared task-space intent from embodiment-dependent transition dynamics. The first stage employs a representation layer pre-trained on offline trajectories to extract task-level context from collective data. The second stage introduces a decision layer equipped with a predictive dynamics adapter. By forcing the latent state to predict future transitions, this adapter acts as a latent regularizer that grounds the policy in the transition behavior of the executing robot while preserving real-time inference without online planning. Evaluations across simulated embodiments demonstrate that our method prevents performance collapse on distinct robots. It achieves over 93\% overall success and 89\% on dynamically out-of-distribution zero-shot simulated robots, outperforming the Transformer behavior cloning (82\%) and diffusion policy baselines (66\%). Physical KUKA deployment further validates the feasibility of the full state-based pipeline under zero-shot sim-to-real transfer.
• 中文摘要 学习基于状态的笛卡尔操作集体策略，因具身依赖的过渡动态而面临根本挑战。集体学习汇总多机器人经验以发现共享任务结构，但当不同实体通过不同的过渡行为实现同一笛卡尔指令时，朴素聚合可能导致不兼容的行动模式。这导致战略模糊和动态外围实体的性能崩溃。为此，我们提出了PACE两阶段集体学习框架，将共享任务空间意图与具象依赖的过渡动态分离。第一阶段使用预训练离线轨迹的表示层，从集体数据中提取任务层上下文。第二阶段引入配备预测动力学适配器的决策层。通过强制潜态预测未来转变，该适配器作为潜在正则化器，将策略基于执行机器人的转换行为，同时保持实时推断，无需在线规划。跨模拟实例的评估表明，我们的方法防止了不同机器人的性能崩溃。其整体成功率超过93%，在动态非分布零样本模拟机器人中成功率为89%，优于Transformer行为克隆（82%）和扩散策略基线（66%）。物理KUKA部署进一步验证了零样本模拟到实传输下全状态流水线的可行性。

Back on Track: Deployment-time Policy Improvement via World Model Planning

重回正轨：通过世界模型规划实现部署时间政策改进

• Authors: Colton Crosby, Hao Wang, Mac Schwager, Somil Bansal
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.19579
• Pdf link: https://arxiv.org/pdf/2610.19579
• Abstract State-of-the-art visuomotor policies are increasingly capable of executing complex tasks, yet can fail during deployment in ways they cannot predict or recover from on their own. Improving their performance and robustness typically requires collecting more data, retraining, or modifying the policy architecture, all of which are time- and resource-intensive in practice. We instead propose an offline-trained monitoring-and-intervention framework that improves a policy during deployment using only black-box query access. From a limited set of labeled on-policy rollouts, we learn a policy-conditioned liveness value that measures the nominal policy's progress toward task completion; we additionally train a compact latent world model using these rollouts together with play data. At deployment, the liveness value continuously monitors execution. When it detects a loss of progress, a short-horizon planner searches the world model for corrective actions using the same value as its planning objective, after which control returns to the nominal policy. We evaluate our method on the LIBERO-Goal task suite and on a Franka Panda manipulator, using the vision-language-action model $\pi_{0.5}$ and a diffusion policy as nominal policies, and demonstrate significant improvements in task success rates, while supporting real-time control. Additional results and videos are available on the project website (this https URL).
• 中文摘要 最先进的视觉运动策略越来越能执行复杂任务，但在部署过程中可能以无法自行预测或恢复的方式失败。提升其性能和鲁棒性通常需要收集更多数据、重新训练或修改策略架构，这些过程在实际中都耗时且耗费资源。我们提出一个离线训练的监控与干预框架，仅通过黑箱查询访问在部署期间改进策略。从有限的带标签策略启动中，我们学习到一个策略条件活体值，衡量名义策略在任务完成上的进展;此外，我们利用这些部署和游戏数据训练一个紧凑的潜在世界模型。部署时，活体值持续监控执行。当检测到进展丢失时，短视野规划器会用与规划目标相同的值搜索世界模型中的纠正措施，之后控制权返回名义策略。我们在LIBERO-Goal任务套件和Franka Panda操作器上评估了我们的方法，采用视觉-语言-行动模型$\pi_{0.5}$和扩散策略作为名义策略，展示了任务成功率的显著提升，同时支持实时控制。更多结果和视频可在项目网站（https URL）获取。

Physical Time as a Missing Dimension of Robot Policy Memory Dynamics

物理时间作为机器人政策记忆动力学中缺失的维度

• Authors: Yushen Liang, Yue Peng, Baosheng Jin, Tianluo Zhang, Xinyu Zhang, Shuyi Zhou, Zhuoran Chen, Xinqi Liu, Shenji Wan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.20524
• Pdf link: https://arxiv.org/pdf/2610.20524
• Abstract Existing robot policy memories typically evolve with observation sequences, implicitly treating each update as the same amount of physical time. Yet identical observation histories can span different durations and require different actions, exposing physical time as a missing dimension of policy memory. We introduce Temporal Flow Policy (TFP), which learns memory dynamics to retain past events and evolve the memory state over measured time intervals. Observations and prior memory determine the content and time scales of each transition; elapsed time determines how far the state advances. TFP conditions diffusion and flow-matching policies through this memory state and is trained with their native imitation objectives. We evaluate TFP with Diffusion Policy and $\pi_{0.5}$ through controlled comparisons of memory designs. These experiments show that successful timing depends on both measured intervals and retained history. TFP achieves higher average success than the recurrent and state-space memories we equip with the same intervals and than a variant that feeds the interval to a learned gate. Memory readouts recover the time since an event and motion information after occlusion. Changing the supplied intervals systematically delays or advances actions, linking time inputs to behavior. Real-robot tasks requiring timing or memory, together with LIBERO evaluations, demonstrate benefits in physical and general manipulation. These results suggest viewing robot memory not as stored observation history, but as a state that retains past events and evolves with physical time.
• 中文摘要 现有的机器人策略记忆通常随着观察序列演进，隐含将每次更新视为相同的物理时间。然而，相同的观察历史可能跨越不同持续时间，并需要不同的操作，暴露出物理时间作为策略记忆缺失的维度。我们引入时间流策略（TFP），它学习记忆动态以保留过去事件，并在测量时间区间内演进记忆状态。观察和先前记忆决定每个过渡的内容和时间尺度;经过时间决定状态推进的距离。TFP条件扩散和流量匹配策略通过该记忆状态进行训练，并以其原生模仿目标进行训练。我们通过对内存设计的受控比较，评估TFP与扩散策略和$\pi_{0.5}$。这些实验表明，成功时序取决于测量区间和保留历史。TFP的平均成功率高于我们配备的相同间隔的循环记忆和状态空间记忆，也高于将该间隔输入学习门的变体。记忆读出恢复事件发生后的时间和遮挡后的运动信息。改变提供的时间间隔可以系统地延迟或推进动作，将时间输入与行为联系起来。需要时间或记忆的真实机器人任务，结合LIBERO评估，在物理和一般操作方面显示出优势。这些结果表明，机器人记忆不应被视为存储的观察历史，而是一种保留过去事件并随物理时间演变的状态。