生成时间: 2026-10-16 21:05:15 (UTC+8); Arxiv 发布时间: 2026-10-16 20:00 EDT (2026-10-17 08:00 UTC+8)

今天共有 59 篇相关文章

Keyword: reinforcement learning

Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

答案相同，换个新盒子：RLVR域外收益有多少来自答案格式？

• Authors: Zongye Lyu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.20639
• Pdf link: https://arxiv.org/pdf/2610.20639
• Abstract Reinforcement learning with verifiable rewards (RLVR) is credited with gains beyond the domain its rewards check, such as higher MMLU accuracy after training on mathematics. Such accuracies are usually read out of the model's text by an answer extractor. A model that learns to write its answer where the extractor looks can therefore score higher without answering better. We measure how much of the gain reported by extractors copied from public code arises this way. We generate outputs from a base model and its RL model and score them twice: with those extractors, and with one extractor of our own that reads a stated answer whether it is in the answer box \boxed{} or in the text. On four released 7B SimpleRL base-RL pairs, 82% of the out-of-domain multiple-choice gain that these extractors report disappears under ours: it is a gain in answer format, not in answers. The base models already state most of their answers in text, and RL moves them into the box, where these extractors look first. Independent LLM readers, in place of our extractor, put the figure at 77%. With a prompt that itself asks for the box, our extractor puts it at 79%. On a 32B pair trained with another recipe, almost none of the reported gain is format gain: its base model boxes its answers when asked, and its RL model rarely does when not. A reported gain is partly a property of the extractor that read it, and re-scoring stored outputs with a second extractor shows how much, without re-running RL.
• 中文摘要 带可验证奖励的强化学习（RLVR）被认可为其奖励检定域之外的收益，例如数学训练后MMLU准确率提高。这些准确性通常由答案提取器从模型文本中读取。一个学会写出答案的模型，可以获得更高分数，但答案不会更好。我们通过这种方式衡量提取器从公共代码复制报告的增益有多少。我们从基础模型及其强化学习模型生成输出，并对其进行两次评分：用这些提取器，以及我们自己的一个提取器，该提取器读取了答案，无论答案框中是否在 \boxed{} 或文本中。在四个发布的7B SimpleRL base-RL对中，这些提取器报告的域外选择题增益中有82%在我们的版本中消失：这是答案格式上的收益，而非答案。基础模型已经在文本中陈述了大部分答案，强化学习会把它们放进框里，提取器先看那里。独立的LLM读者代替我们的提取器，将数据定为77%。而提取器则提示了该框，结果是79%。在用另一个配方训练的32B对上，几乎没有报告的增益是格式增益：其基础模型在被问及时会框定答案，且当未被要求时，强化学习模型很少会。报告的增益部分是读取它的提取器特性，用第二个提取器重新评分存储的输出显示了多少，无需重跑强化学习。

ReasonUni: Reasoning-Augmented Reconstruction Advances Unified Multimodal Models

ReasonUni：推理增强重建推动统一多模态模型的发展

• Authors: Feize Wu, Zhiyuan Yan, Yuhuan Hong, Yuwei Niu, Zongjian Li, Zheyuan Liu, Bin Lin, Jinbo Yan, Jie Qin, Xudong Mao, Limeng Qiao, Guanglu Wan, Li Yuan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20781
• Pdf link: https://arxiv.org/pdf/2610.20781
• Abstract Image-to-text (I2T) understanding and text-to-image (T2I) generation are dual tasks in unified multimodal models (UMMs). Reconstructive post-training couples these tasks, but existing reconstruction-based methods remain perception-centric and provide limited supervision for reasoning about structure, intent, and transformation. We propose ReasonUni, a reasoning-augmented reconstructive post-training framework that extends this coupling beyond perception. Given an original image and an editing instruction, the UMM first reasons about what and how to edit and describes the target state, then generates the corresponding target image conditioned on this understanding output. To jointly optimize both modules, we introduce TreeGRPO, a tree-structured reinforcement learning method that samples multiple reasoning paths and candidate images per path, then aggregates visual rewards to update both modules under a unified objective. Experiments across multiple benchmarks demonstrate consistent gains in both understanding and generation, showing that reasoning-augmented reconstruction yields genuine mutual gains for UMMs.
• 中文摘要 图像到文本（I2T）理解和文本到图像（T2I）生成是统一多模态模型（UMM）中的双重任务。重建式后期训练将这些任务结合起来，但现有基于重建的方法仍以感知为中心，对结构、意图和转换的推理提供有限的监督。我们提出了ReasonUni，一种推理增强的重建后训练框架，将这种耦合扩展到感知之外。给定原始图像和编辑指令后，UMM首先推理编辑内容和方式并描述目标状态，然后基于该理解输出生成相应的目标图像。为共同优化这两个模块，我们引入了TreeGRPO，一种树结构强化学习方法，该方法采样多条推理路径和候选图像，然后汇总视觉奖励以在统一目标下更新两个模块。跨多个基准测试的实验显示，理解和生成方面均有持续提升，表明推理增强重建为UMM带来了真正的互惠收益。

EvolvingBrain: An Adaptive Skill System for Multi-modal Agents Self-Evolution in Open Worlds

EvolvingBrain：多模态代理在开放世界中自我进化的自适应技能系统

• Authors: Runxuan Liu, Jingqi Shi, Baoxin Wang, Honghong Zhao, Haoyuan Xu, Yueqing Sun, Jiafeng Liang, Tao He, Dayong Wu, Qi Gu, Shijin Wang, Ming Liu, Bing Qin
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20786
• Pdf link: https://arxiv.org/pdf/2610.20786
• Abstract Multi-modal agents have made significant progress in interacting with open worlds. However, their ability to self-evolve remains limited. Supervised fine-tuning enables imitation but not active experience organization, while outcome-based reinforcement learning promotes exploration yet underuses intermediate trajectory information. Therefore, we propose EvolvingBrain, which uses an adaptive skill system as the brain for agent self-evolution, consisting of two following components. Stratified Skill Self-Evolution: extracts Procedural Patterns and Problematic Patterns from model-specific trajectories, summarizes experience by generating skills, and evolves through multiple interactions with the open world, ultimately forming a Stratified Skill Architecture consisting of Domain, Plan, and Action/Trap Skills, which provides the agent with a multi-granularity knowledge repository and a novel evolution approach. State-Aware Adaptive Execution: summarizes long-horizon interactions into Structured Abstractive Memory and uses it to guide the State-Aware Skill Fusion Strategy, enabling the agent to utilize knowledge according to its current state. Experiments on VAB-Minecraft and DiscoveryWorld demonstrate that EvolvingBrain improves success rates on open-world tasks and exhibits generalization capabilities. Our code is provided in the supplementary materials and will be publicly released in the camera-ready version.
• 中文摘要 多模态智能体在与开放世界交互方面取得了显著进展。然而，它们自我演化的能力仍然有限。监督式微调支持模仿但无法主动组织经验，而基于结果的强化学习促进探索，但减少中间轨迹信息的使用。因此，我们提出了EvolvingBrain，它使用自适应技能系统作为智能体自我进化的大脑，由以下两个组成部分组成。分层技能自我进化：从模型特定轨迹中提取程序模式和问题模式，通过生成技能总结经验，并通过与开放世界的多次交互进化，最终形成由领域、计划和行动/陷阱技能组成的分层技能架构，为智能体提供多粒度的知识库和一种新颖的进化方法。状态感知自适应执行：将长期相互作用总结为结构化抽象记忆，并用以指导状态感知技能融合策略，使智能体能够根据当前状态利用知识。在VAB-Minecraft和DiscoveryWorld上的实验表明，EvolvingBrain能提高开放世界任务的成功率并展现泛化能力。我们的代码已提供于补充材料中，并将以相机准备好版本公开发布。

Blame Errant Words, Not the Canvas: Defect-Aware Spatial Credit Assignment for Visual Text Generation

怪错词语，而非画布：缺陷感知空间署名分配用于视觉文本生成

• Authors: Sujie Hu, Huaisong Zhang, Pujing Yang, Qian Wang, Xiaomin Li, Xinhang Leng, Chen Li, Jing Lyu, Xiu Li, Ling Pan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.20794
• Pdf link: https://arxiv.org/pdf/2610.20794
• Abstract Recent reinforcement learning approaches have improved visual text generation, yet generated images still suffer from incorrect wording and malformed glyphs. We identify two coupled limitations: recognition-based rewards often overlook visible glyph defects, while whole-image scalar feedback obscures the uneven quality of individual text regions. These limitations motivate our central principle: $\textit{Blame errant words, not the canvas}$, to effectively preserve local quality distinctions. First, we introduce $\mathbf{TextDART}$, a defect-aware region-targeted reward model that localizes individual text regions and independently evaluates their semantic correctness and glyph quality, providing fine-grained, region-precise feedback. Building on TextDART, we propose $\mathbf{SCANNER}$, which turns these regional judgments into spatially differentiated policy feedback. It normalizes rewards for each text target across sampled images and casts the resulting regional advantages onto the corresponding latent tokens. Extensive experiments on existing benchmarks and our newly established $\textbf{PolyText-Bench}$ demonstrate that our approach achieves state-of-the-art performance in text accuracy and glyph fidelity. Project page: this https URL
• 中文摘要 近期强化学习方法改进了视觉文本生成，但生成图像仍存在措辞错误和字形畸形的问题。我们识别出两个耦合局限：基于识别的奖励常忽略可见的字形缺陷，而全图标量反馈则掩盖了单个文本区域质量的不均。这些限制驱动了我们的核心原则：$\textit{责怪错误词语，而非画布}$，以有效保持局部质量差异。首先，我们引入$\mathbf{TextDART}$，一种基于缺陷感知的区域目标奖励模型，该模型定位单个文本区域，独立评估其语义正确性和字形质量，提供细粒度、区域精确的反馈。基于TextDART，我们提出$\mathbf{SCANNER}$，将这些区域判断转化为空间差异化的政策反馈。它对采样图像中每个文本目标的奖励进行规范化，并将由此产生的区域优势投射到相应的潜在代币上。对现有基准测试和我们新建立的 $\textbf{PolyText-Bench}$ 的广泛实验表明，我们的方法在文本准确性和字形忠实度方面实现了最先进的性能。项目页面：此 https URL

Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards

针对非策略强化学习的Gibbs-Gap策略优化，且可验证奖励

• Authors: Xuanchang Zhang, Tong Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.20814
• Pdf link: https://arxiv.org/pdf/2610.20814
• Abstract Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights. We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of-$K$ reward, minus terms for the raw finite-group loss, regularization, and finite training data. We characterize reference refresh through exact Gibbs-target composition and a finite-data final-policy reward bound under local reference control and tokenwise behavior mismatch. This refresh is empirically important: in an ablation, fixed-reference GGPO scores $32.96$ on the five-benchmark problem-weighted average, versus $46.58$ with the default refresh schedule. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training-inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows.
• 中文摘要 带有可验证奖励的强化学习通常会根据不同且可能已过时的行为策略生成的反应来训练语言模型。通过学习者与行为重要性权重纠正这种不匹配，可能会在长响应中引入较高的方差。我们引入了Gibbs-Gap策略优化（GGPO），该方法使同一提示的响应之间KL调整奖励的对数均值-经验对比最小化，且不使用学习者与行为的重要性权重。我们从KL正则化策略优化中推导出GGPO：在策略下，其总体损失正好是KL正则化最优性差距，在$K适当覆盖下，学习者的单次响应期望奖励至少是行为策略的最佳奖励，减去原始有限组损失、正则化和有限训练数据的项。我们通过精确的Gibbs目标组合和有限数据最终策略奖励在局部参考控制和token行为不匹配下界定来描述参考刷新。这种刷新在实证上很重要：在消融中，固定引用GGPO在五个基准问题加权平均值中得分为32.96美元，而默认刷新计划为46.58美元。数学推理实验进一步表明，GGPO在训练-推断不匹配和固定离线数据下优于GRPO和AGRO，同时在受控策略陈旧性增加时保持性能。

Race to Recovery: Designing Advanced Safety Systems for Autonomous Driving via Learning Autonomous Racing

复苏竞赛：通过学习自动驾驶设计先进安全系统

• Authors: Teodor Ilie, Kleber Cabral, Sidney Givigi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.20873
• Pdf link: https://arxiv.org/pdf/2610.20873
• Abstract Vehicles that lose control at the handling limit through icing, hydroplaning, or an abrupt evasive maneuver fall outside the operating envelope of production safety systems such as electronic stability control, which focus on preventing instability rather than recovering from it. Expert drivers regain authority in these conditions by drifting. Existing autonomous recovery approaches are validated on idealized scenarios, define "recovered" states empirically and inconsistently, or rely on offline-planned trajectories rather than acting online. We address these gaps by first training a racing policy with reinforcement learning that learns drifting as a time-optimal maneuver, then adapting it to recovery through continued training with a minimum-time reward. We define recovery relative to a phase-plane-grounded stable equilibrium and introduce Gym-Khana for controller training and evaluation. Across 294 initial loss-of-control states, the transferred policy improves recovery by 186.8% over a Stanley path tracker and outperforms both a nonlinear model predictive controller and a recovery-only policy trained from scratch, while also recovering faster on states the baseline can already handle. An ablation shows that racing pretraining, rather than broader initial-state exposure alone, drives this gain. Deployed on a physical 1:10-scale F1TENTH vehicle using system identification, actuator-lag modelling, and domain randomization, the transferred policy outperforms Stanley on every tested loss-of-control state, including recovering 50% of the time from a hard state the baseline never recovers from.
• 中文摘要 因结冰、水滑或突然规避动作导致操控极限失控的车辆，超出了电子稳定控制等生产安全系统的工作范围，后者专注于预防不稳定而非恢复。专家驾驶员通过漂移在这些条件下重新获得权威。现有的自动恢复方法在理想场景下得到验证，经验且不一致地定义“恢复”状态，或依赖离线规划轨迹而非在线行动。我们通过先通过强化学习训练赛车策略，学习漂移作为时间最优动作，然后通过持续训练以最小时间奖励调整以适应恢复，弥补这些空白。我们将相对于相位面接地稳定平衡定义恢复，并引入Gym-Khana用于控制训练和评估。在294个初始失控状态中，转移策略比Stanley路径追踪器提升了186.8%的恢复率，并且优于从零训练的非线性模型预测控制器和仅恢复策略，同时在基线已能处理的状态恢复速度更快。消融显示，竞速预训练而非仅仅是更广泛的初始状态暴露，推动了这一收益。该策略部署在物理1：10比例的F1TENTH车辆上，采用系统识别、执行器-滞后建模和域随机化技术，转移策略在所有测试失控状态上都优于Stanley，包括从基线永远无法恢复的硬状态恢复率为50%。

Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL

错误的投票，正确的梯度：为什么多数票奖励会追踪可验证的强化学习

• Authors: Dylan P. Wilson, Chengsong Huang, Jiaxin Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.20898
• Pdf link: https://arxiv.org/pdf/2610.20898
• Abstract Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards. This is puzzling from the usual prompt-level view, in which majority vote only returns a model's modal answer and cannot add information about correctness. We argue that the relevant unit is the batch-level update. Because parameters are shared, a prompt's own gradient accounts for less than 1\% of the change in that prompt's output at practical batch sizes, so a wrong label matters only to the extent that label errors share a direction across prompts. We decompose the majority-vote update into shrinkage and misdirection relative to the gold update, and show that misdirection vanishes with batch size whenever errors are incoherent across prompts, however often the labels are wrong. Measuring per-response gradients on five models, we find that majority-vote errors are as incoherent as a random-reward null, that the majority-vote update agrees with the gold update as closely as two independent gold updates agree with each other, and that its noise-corrected cosine with the gold update is 1 within error. Randomly flipping up to 30\% of gold labels leaves the update direction intact, while the same fraction flipped by a shared rule destroys it, and rewards that favor a kind of response on every prompt, such as a learned reward model, fall measurably short. The equivalence holds at every checkpoint we examine but is local: gold-reward training still ends ahead on held-out majority accuracy. Error coherence, not error rate, determines when self-generated labels can stand in for verification.
• 中文摘要 使用自我生成标签的强化学习，例如对模型自身样本的多数投票，通常能通过可验证的奖励恢复大部分训练收益。这与通常的提示层面观点不同，后者多数票仅返回模型的模态答案，无法添加正确性信息。我们认为相关单元是批次级更新。由于参数共享，提示本身的梯度在实际批处理下对提示输出变化的贡献不到1/%，因此错误标签仅在标签错误在提示之间有共同方向时才重要。我们将多数投票更新分解为相对于黄金更新的缩小和误导，并证明当提示之间错误不一致时，误导随批次大小而消失，无论标签多么错误。通过五个模型的每次响应梯度测量，我们发现多数票错误的不连贯性如随机奖励空判，多数票更新与黄金更新的一致性与两个独立的金币更新一致的程度相同，其与黄金更新的噪声修正余弦误差为1。随机翻转最多30%的黄金标签会保持更新方向不变，而相同比例的重复则会破坏更新方向，且在每个提示上都偏好某种反应的奖励（如学习奖励模型）明显不足。在我们检查的每个检查点，这种等价性成立但仅限于局部：黄金奖励训练仍以保留的多数准确率结束。错误一致性，而非错误率决定何时可以由自生成标签替代验证。

Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation

错误指定回归的符号投注：方差适应性与最优近似

• Authors: Yinan Li, Han Han, Kwang-Sung Jun
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.20903
• Pdf link: https://arxiv.org/pdf/2610.20903
• Abstract We study bounded regression under model misspecification, evaluating a function $f$ by its $L_1(P_X)$ distance to the conditional mean $\eta(x)=\mathbb{E}[Y\mid X=x]$, rather than the standard $L_2(P_X)$. This $L_1$ metric is natural for downstream decision problems such as bandits and reinforcement learning. We analyze the min--max estimator of Baraud & Maillard (2025), which we call \emph{sign-betting regression}. Our main results show that sign-betting regression achieves high-probability $L_1$ oracle inequalities under misspecification. Under realizability, our bound adapts to the average conditional variance and recovers the variance-adaptive rate of Li et al. (2026). We further characterize the optimal universal approximation factor for proper learning by showing that sign betting achieves factor 3, while no proper learner can satisfy a distribution-free, high-probability $L_1$ oracle inequality with a universal approximation factor strictly below 3. We extend the upper-bound analysis beyond finite classes and complement the theory with experiments in synthetic and real-world model-selection tasks.
• 中文摘要 我们研究模型错误指定下的有界回归，通过函数$f$的$L_1（P_X）$距离条件均值$\eta（x）=\mathbb{E}[Y\mid X=x]$来评估，而非标准的$L_2（P_X）$。这种$L_1$度量对于下游决策问题如强盗和强化学习来说是自然的。我们分析了Baraud & Maillard（2025）的最小最大估计量，称之为\emph{符号投注回归}。我们的主要结果表明，符号投注回归在错误指定下实现了高概率$L_1$的预言机不等式。在可实现性下，我们的界限适应平均条件方差，并恢复了Li等人（2026）的方差适应率。我们进一步刻画了正确学习的最优普遍近似因子，证明符号投注可达3因子，而没有任何正学习者能满足一个分布自由、概率高概率$L_1$的预言不等式且普遍近似因子严格低于3。我们将上界分析扩展到有限类之外，并通过综合和现实模型选择任务中的实验补充理论。

MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

MAI-Gym：一个面向移动和计算机环境的GUI代理评估和强化学习的开放一体化平台

• Authors: Panrong Tong, Quyu Kong, Pengxiang Li, Tianyu Xia, Hanzhang Zhou, Lei Wang, Chenglin Cai, Long Li, Long Chen, Jianan Zhang, Zhengwen Zeng, Xu Zhang, Gongjie Zhang, Yue Wang, Steven Hoi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20939
• Pdf link: https://arxiv.org/pdf/2610.20939
• Abstract Online reinforcement learning (RL) with verifiable rewards has driven rapid progress in mathematics, coding, and tool use, but scaling this success to GUI agents remains challenging due to the high cost of interactive environments and the scarcity of realistic training tasks. We introduce MAI-Gym, an open all-in-one platform that unifies GUI agent evaluation, data collection, and online RL training across mobile, computer, and web environments. Through a shared rollout pipeline, MAI-Gym ensures consistent agent behavior across evaluation and training, supports ten established benchmarks, and scales to 20,000 parallel environment instances. Beyond scalable infrastructure, MAI-Gym enables automated synthesis of diverse GUI applications and challenging tasks, including resettable mocked apps and coherent cross-app worlds with execution-validated task verifiers. Using MAI-Gym, we train MAI-Gym-9B, a unified 9B agent for mobile and computer use, achieving state-of-the-art performance among open GUI agents under 10B parameters: 78.4% on MobileWorld and 76.2% on OSWorld-Verified. Importantly, RL in a mocked train-ticketing app transfers to its real-world counterpart, improving task success on a phys- ical phone from 28.8% to 48.1%. We release the framework, synthesis skills, mocked apps, 2,000 verified tasks, and the model checkpoint to support repro- ducible research and further advances in scalable GUI agent training.
• 中文摘要 带有可验证奖励的在线强化学习（RL）推动了数学、编程和工具使用的快速进展，但由于互动环境成本高昂且真实训练任务稀缺，将这一成功推广到图形界面代理仍具挑战。我们介绍MAI-Gym，一个开放的一体化平台，统一了移动、计算机和网络环境中的图形界面代理评估、数据收集和在线强化学习。通过共享的推广流程，MAI-Gym确保代理在评估和培训中的一致行为，支持十个既定基准，并可扩展至2万个并行环境实例。除了可扩展基础设施外，MAI-Gym还支持多种图形界面应用和复杂任务的自动综合，包括可重置的模拟应用和带有执行验证任务验证器的连贯跨应用世界。利用MAI-Gym，我们训练MAI-Gym-9B，这是一款面向移动和计算机的统一9B代理，在10B参数下，在开放GUI代理中实现了最先进的性能：MobileWorld为78.4%，在OSWorld-Verified为76.2%。重要的是，模拟火车票务应用中的强化学习可转移到现实世界，使物理手机上的任务成功率从28.8%提升至48.1%。我们发布框架、综合技能、模拟应用、2000个已验证任务和模型检查点，以支持可重复性研究和可扩展GUI代理培训的进一步进展。

SOMA: Stack-of-Objects Manipulation using a Quadruped Robot

SOMA：利用四足机器人进行物体堆叠操作

• Authors: Stergios E. Bachoumas, Panagiotis Artemiadis
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20968
• Pdf link: https://arxiv.org/pdf/2610.20968
• Abstract Executing heavy-duty physical manipulation in unpredictable environments remains a major challenge for quadrupedal robots lacking dedicated manipulators. This paper presents SOMA, a hierarchical reinforcement learning framework for whole-body non-prehensile manipulation of multi-object stacks using a quadruped robot. Unlike traditional methods restricted to two-dimensional planar pushing, SOMA integrates explicit robot base-height control, unlocking full three-dimensional pushing strategies. This capability enables the quadruped to manipulate complex payload stacks as a unified system while preventing relative sliding or tipping of upper layers. The architecture factorizes control into a high-level manipulation policy using a Simplified Structured State Space Sequence (S5) encoder to reason over long-horizon temporal history and predict bounded target tracking commands via Beta distributions, and a low-level locomotion policy that dynamically adapts leg stance configurations. In extensive simulation benchmarks, SOMA achieves a $90.3\%$ success rate for single-object reposing and up to $90.1\%$ for dynamic two-box stacks under stringent error margins ($0.05\text{ m}$ position error, $1^\circ$ orientation error). Finally, we validate zero-shot sim-to-real transfer on a Unitree Go1 quadruped robot across diverse payload masses and geometries, demonstrating active base-height adjustments to maintain traction and balance during heavy pushing tasks.
• 中文摘要 在不可预测环境中执行高强度物理操作仍是缺乏专用机械臂的四足机器人面临的重大挑战。本文介绍了SOMA，一种用于四足机器人实现多对象堆栈全体非抓握操作的分层强化学习框架。与传统仅限于二维平面推力的方法不同，SOMA集成了显式机器人基准高度控制，解锁了完整的三维推力策略。这一能力使四足机器人能够作为一个统一系统操控复杂的有效载荷堆栈，同时防止上层的相对滑动或倾斜。该架构将控制分解为高级操作策略，使用简化结构化状态空间序列（S5）编码器推理长视野时间历史，并通过Beta分布预测有界目标跟踪指令，同时采用动态调整腿部站姿配置的低级运动策略。在广泛的模拟基准测试中，SOMA在单对象定位中实现了90.3美元/%%%的成功率，在严格误差范围内（$0.05\text{ m}$位置误差，$1^\circ$方向误差）下动态双盒堆栈成功率最高达90.1美元。最后，我们验证了Unitree Go1四足机器人在不同有效载荷质量和几何条件下的零发射模拟到实物传输，演示了主动基准调整以在重载荷推力任务中保持牵引和平衡。

How to Train Your Little Dragon: Teaching a Small Language Model to Construct and Refine GPU Optimization Scripts

如何训练你的小龙：教一个小型语言模型构建和优化GPU优化脚本

• Authors: Annabelle Sujun Tang, Wei Tang, Songyao Jiang, Tianhui Li, Shengzhe Xu, Jiin Woo, Kaan Ozkara, Youngsuk Park, Zhen Jia, Hadi Esmaeilzadeh
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.20980
• Pdf link: https://arxiv.org/pdf/2610.20980
• Abstract Compiler transformations interact, making GPU kernel optimization a search for combinations that preserve the computation while improving performance. We introduce HOW TO TRAIN YOUR LITTLE DRAGON, a training method that teaches a small language model both to construct complete optimization scripts and to improve existing ones. Instead of ranking candidates or generating kernel code, our model generates optimization scripts. We devise a compact, lossless representation whose deterministic decoding reconstructs the exact compiler trace, preserving its transformation parameters and dependencies. We use a compiler framework TVM MetaSchedule to build training data on 54,116 distinct workloads, and construct 108,594 examples in two training sets: 60,236 teach the model to generate an optimized script from an unoptimized kernel, while 48,358 teach it to produce a better script given the kernel and a somewhat optimized script. Supervised fine-tuning teaches the model to construct and improve optimization scripts from these search-derived examples. We then add reinforcement learning with a shaped reward that combines correctness with measured performance, awarding a latency-based bonus only to numerically correct kernels. We apply this training method to a model with 8 billion parameters. After supervised fine-tuning and reinforcement learning, the model achieves per-attempt correctness of 91.27% for construction and 92.01% for refinement, while reinforcement learning produces faster kernels than supervised training on 88 of 102 construction tasks and 145 of 170 refinement prompts. With only 8B parameters, our LITTLE DRAGON achieves higher per-attempt correctness and better kernel performance than six frontier LLMs: GPT-6 Astra, Opus 5, Kimi-K3, and GLM-5.3 among them, each with hundreds of billions to trillions of parameters, for which task-specific training is unavailable or expensive.
• 中文摘要 编译器转换相互作用，使GPU内核优化成为一种既保持计算又提升性能的组合。我们引入了“如何训练你的小龙”训练方法，该训练方法既教一个小型语言模型构建完整的优化脚本，也用于改进现有脚本。我们的模型不对候选对象进行排名或生成内核代码，而是生成优化脚本。我们设计了一个紧凑、无损的表示法，其确定性解码能够精确重建编译器轨迹，同时保持其变换参数和依赖关系。我们使用编译器框架TVM MetaSchedule在54,116个不同工作负载上构建训练数据，并在两个训练集中构建了108,594个示例：其中60,236个教模型从未优化的内核生成优化脚本，48,358个教模型在内核和较为优化的脚本下生成更好的脚本。监督微调教模型从这些搜索衍生的例子中构建和改进优化脚本。随后，我们加入带有形状奖励的强化学习，该奖励结合了正确性与测量性能，仅对数值正确核授予基于延迟的加成。我们将该训练方法应用于一个拥有80亿参数的模型。经过监督微调和强化学习后，模型在构建中每次尝试正确度为91.27%，精细中92.01%;而强化学习在102个构建任务中的88个和170个精炼提示中的145个中产生比监督训练更快的核。仅有8B参数，我们的LITTLE DRAGON在每次尝试中实现了更高的正确性和更优的核心性能，优于六个前沿大型语言模型：包括GPT-6 Astra、Opus 5、Kimi-K3和GLM-5.3，这些大型语言模型拥有数千亿到万亿参数，而这些模型任务专属训练既不可用，也成本高昂。

Deep Reinforcement Learning for HPC Job Scheduling: A Statistical Evaluation

HPC作业调度的深度强化学习：统计评估

• Authors: Justin M. Cheney, Clement N. Nyirenda
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21008
• Pdf link: https://arxiv.org/pdf/2610.21008
• Abstract Deep Reinforcement Learning (DRL) offers learned job selection for High-Performance Computing (HPC), but performance against heuristics alone does not establish value beyond action masking. Six DRL treatments are compared: Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), and Deep Q-Network (DQN), each with and without masking. Three heuristics and a seed-paired masked uniform random control provide baselines across ten seeds on two real Slurm (Simple Linux Utility for Resource Management) workload traces, one homogeneous and one heterogeneous, under a common non-parametric framework. All masked treatments complete evaluation, whereas unmasked treatments are truncated. On the homogeneous trace, no masked DRL treatment significantly improves average waiting time or average and maximum bounded slowdown over the control. On the heterogeneous trace, MaskablePPO improves all three metrics relative to the control. Its average waiting time is equivalent to that of the three heuristics within a 10% margin, and its maximum bounded slowdown is 22% lower than that of Last-Come, First-Served (LCFS). MaskablePPO has the leading average waiting-time rank among complete treatments on both traces, but Nemenyi comparisons detect no significant difference from MaskableA2C. Among masked variants, MaskablePPO has the lowest recorded mean training time, while MaskableDQN has the lowest mean decision latency. These findings distinguish completion from baseline-specific scheduling gains and support including a masked random control when assessing the value of learned selection.
• 中文摘要 深度强化学习（DRL）为高性能计算（HPC）提供了学习中的作业选择，但仅凭启发式表现无法确定除动作掩蔽之外的价值。比较了六种 DRL 处理方法：近端策略优化（PPO）、优势演员-批判者（A2C）和深度 Q 网络（DQN），均带有掩蔽和无遮蔽。三种启发式和一种种子配对的掩蔽均匀随机控制在两个真实 Slurm（简易 Linux 资源管理工具）工作负载跟踪中，在共同非参数框架下，提供了十个种子的基线，分别为同质和异构。所有掩蔽处理均完成评估，而无掩蔽处理则被截断。在同质迹中，没有掩蔽 DRL 处理显著改善对照组的平均等待时间或平均与最大有界减速。在异质追踪中，MaskablePPO相对于对照组提升了这三项指标。其平均等待时间相当于三种启发式方法在10%范围内的等待时间，最大有界减速比先到先得（LCFS）低22%。MaskablePPO在两种痕迹中均拥有完整处理中领先的平均等待时间排名，但Nemenyi比较未发现与MaskableA2C有显著差异。在掩蔽变体中，MaskablePPO的平均训练时间最低，而MaskableDQN的平均决策延迟最低。这些发现区分了完成度与基线特定调度的增益，以及在评估学习选择价值时包含掩蔽随机对照的支持。

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

通过对抗元提示演化实现递归奖励改进

• Authors: Wei Dai, Qianchu Liu, Guanghui Qin, Xing Han, Tristan Naumann, Paul Pu Liang, Sheng Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.21040
• Pdf link: https://arxiv.org/pdf/2610.21040
• Abstract Recursive Self-Improvement (RSI) has gained wide attention for its ability to work with limited data. In a proposer-solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where every answer can be checked automatically. Open-ended expert tasks, like clinical report generations, have no answer key, which makes simple question-answer generation not viable. Here the proposer must also write a rubric that lists the content an answer should include and the errors it should avoid. However, the unique challenge behind model-written rubric is the comprehensiveness of it. For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the learning process. To solve this problem, we introduce RRI, Recursive Reward Improvement, which uses a reward hacking adversary to supervise the reward. Before the solver trains on a generated task, an adversary writes an answer that games its rubric while being as useless as possible, the same model writes an honest answer without seeing the rubric, and up to two repair steps revise the rubric to make the honest answer win. The exploits found also rewrite how future tasks and rubrics are generated. On HealthBench Professional, RRI improves the adjusted score over the Generation Only self-improvement baseline by 10.9 points at 9B and 12.4 at 27B, and by 10.3 points without any external model. In particular, the gains are largest on difficult and writing tasks. We show that the 27B model trained on the HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench without seeing a task from either. We further analyze the exploitable points of model-generated question-rubric pairs to facilitate future research on open-ended self-play.
• 中文摘要 递归自我改进（RSI）因其能够处理有限数据而广受关注。在提案者-求解者循环中，提案者模型编写训练任务，求解者模型通过强化学习学习解决这些任务。这种循环在数学和编码领域取得了成功，每个答案都可以自动检查。开放式专家任务，如临床报告生成，没有答案键，这使得简单的问答生成不可行。在此情况下，提案者还必须编写评分标准，列出答案应包含的内容及应避免的错误。然而，模型编写评分标准的独特挑战在于其全面性。对于朴素生成的评分标准，求解者往往利用其特点，命名预期主题却没有实质内容，这破坏了学习过程。为解决这个问题，我们引入了RRI（递归奖励改进），它利用奖励黑客对手来监督奖励。在解算器训练生成任务之前，对手会写出一个在尽可能无用的情况下操控评分标准的答案，同一模型在不查看评分标准的情况下写出诚实答案，最多两个修复步骤修改评分标准，使诚实答案获胜。发现的漏洞还重写了未来任务和评分标准的生成方式。在HealthBench Professional上，RRI在仅生成自我提升基线上9B提升了10.9分，27B时提高了12.4分，且在没有外部模型的情况下提高了10.3分。特别是在困难任务和写作任务上，提升幅度最大。我们展示了，在HealthBench Professional描述下训练的27B模型在PRBench Hard上提升了6.6分，在ProfBench上提升了8.0分，且未从任一描述中均有任务。我们还进一步分析了模型生成的问题-评分标准对的可利用点，以促进未来开放式自我游戏的研究。

Manifold Generative Annealing for Contact Constraint-aware Impedance Control with Reinforcement Learning Prior

流形生成退火用于接触约束感知阻抗控制，并有强化学习先验

• Authors: Zhilin He, Guoquan Qi, Changliu Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.21140
• Pdf link: https://arxiv.org/pdf/2610.21140
• Abstract Contact-rich robotic tasks require coordinated motion and impedance under uncertain interaction dynamics and physical constraints. While recent methods on learned policies provide useful interaction strategies, they may fail when contact conditions differ from those encountered during training. Model-based generative methods show promising receding-horizon trajectories optimization results, but often lack structured behavioral guidance and refinement that jointly accounts for constraint geometry and actual contact responses. To this end, we propose Manifold Generative Annealing (MGA), a joint model-free and model-based generative framework for motion--impedance optimization. MGA first organizes pretrained reinforcement learning policies into structured sequence proposals, retaining task-specific interaction strategies without committing to their execution. It then evaluates these proposals alongside exploratory samples through model-based rollouts for refinement or separate execution selection. Crucially, realization-aware manifold refinement aligns updates from the refinement bank with both task constraints and locally achievable physical responses, allowing learned experience to guide online adaptation under changing contact conditions. Experiments cover surface scanning, peg insertion, and humanoid pushing, with improved safety success rate of impedance control under nominal and sensing-shift conditions.
• 中文摘要 接触丰富的机器人任务需要在不确定的交互动力学和物理约束下协调运动和阻抗。虽然最新的学习策略方法提供了有用的交互策略，但当接触条件与训练时不同时，这些方法可能会失效。基于模型的生成方法显示出有前景的后退视界轨迹优化效果，但通常缺乏结构化的行为指导和细化，无法共同考虑约束几何和实际接触响应。为此，我们提出了流形生成退火（MGA），这是一种联合无模型和基于模型的生成式运动阻抗优化框架。MGA首先将预训练强化学习策略组织为结构化序列提案，保留任务特定的交互策略而不承诺执行。然后通过基于模型的展开进行分析样本，进行细化或单独执行选择。关键是，实现感知型流形细化将精细库的更新与任务约束和局部可实现的物理响应对齐，使学习经验指导在线适应在变化接触条件下的应用。实验涵盖表面扫描、插销和类人推力，在标称和感应移位条件下提升阻抗控制的安全成功率。

Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

评判评委：评分标准的信心与分歧作为奖励

• Authors: Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, Yutong Bai
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.21179
• Pdf link: https://arxiv.org/pdf/2610.21179
• Abstract Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.
• 中文摘要 评分标准即奖励（RaR）通过使用LLM评判对结构化评分，将强化学习扩展到不可验证领域。此前的研究主要集中在改进评分标准或强化单一评审，较少关注RaR训练中不同单一或多评委的使用方式。我们首先证明，仅凭评审身份，在RaR医学和RaR科学上训练Qwen3-4B-Base时，分数在五个前沿和开放权重评审中显著分布。基于此，我们提出了RaRE（评分标准作为奖励与集合）的新型评审群议策略，具有两种机制：软李克特奖励，利用评审对数大概率加权，赋予更有信心的评委权重;以及排除高分歧推翻的“丢弃”掩膜。我们发现这一设计至关重要：虽然判官评分的天真平均无法超越最佳单一评判，但RaRE在HealthBench和GPQA Diamond上表现优于所有单评判RaR变体，主题分析显示其继承了其成员评审的互补优势。我们的发现确立了评审合奏作为提升RaR的一个有力轴。

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

评分标准判断缺乏不确定性：LLM评估中的共形权重

• Authors: Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen, Yichen Sun, Yuan Lu, Xiaoxi Li, Hao Wang, Zhixuan Chu, Kui Ren
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21214
• Pdf link: https://arxiv.org/pdf/2610.21214
• Abstract Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty signal from a fixed pool of paraphrased judgment templates and converts it into a per-criterion weight using split-conformal $p$-values with provable coverage guarantees. Extensive experiments demonstrate that URW consistently improves pairwise preference accuracy and delivers stronger downstream RL alignment.
• 中文摘要 基于评分标准的评估被广泛用于评估大型语言模型（LLM）输出，通过将质量分解为多个标准，并为每个标准分配LLM验证者的二元判断。然而，判断确定性在不同标准间存在显著差异。现有方法统一将这些判断汇总为最终评分，隐含地将每个标准视为同等确定，忽视了这种异质性。因此，不确定的判断仍被纳入奖励估计中，未加区分，可能导致下游强化学习（RL）产生误导性的训练信号。为解决这一限制，我们提出了不确定性感知评分标准权重（URW）框架，这是一种无训练框架，通过共形校准量化标准级判断不确定性。具体来说，URW从固定的改写判断模板池中引发软不确定性信号，并利用具有可证明覆盖率保证的分裂共形$p$值将其转换为每标准权重。大量实验表明，URW持续提升两两偏好准确率，并实现更强的下游强化学习对齐。

What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination

我们应该去除哪些？评估RLVR数据去污的可靠性和成本

• Authors: Esther Xin
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.21229
• Pdf link: https://arxiv.org/pdf/2610.21229
• Abstract Reinforcement learning with verifiable rewards (RLVR) math corpora are routinely screened for benchmark overlap, but the reliability and data-loss costs of cleaning policies remain underexamined. We audit sixteen open RLVR math corpora comprising 1,673,514 corpus-specific training items against six benchmarks containing 1,949 questions. We evaluate eight policies based on exact, lexical, and semantic matching using 594 independently and blindly dual-reviewed training-benchmark pairs. Calibration and held-out sets have disjoint benchmark questions and training-example families. A separate review of all 83 candidate matches for 80 sampled training items measures item-level false removal. At cosine threshold 0.80, 52 of 59 sampled removals (88%) were confirmed false removals; this is not a demonstrated population rate. No semantic threshold in our preregistered grid (0.80-0.98) meets the 90% conservative-precision target on calibration. The existing judge-adjudicated policy achieves 98.1% resolved precision and 98.2% candidate-frame recall on held-out corpora. It has zero confirmed false removals among six sampled removals, with one unresolved case; this does not establish zero risk. Within the evaluated candidate frame, this policy offers a more favorable precision-recall balance than the semantic-threshold alternatives tested. Recall does not measure contamination outside the candidate frame. We release reviewed labels, per-policy decisions, and coverage accounting to support evaluation of cleaning reliability and cost rather than overlap counts alone.
• 中文摘要 带可验证奖励的强化学习（RLVR）数学语料库常规筛查基准重叠，但清理策略的可靠性和数据丢失成本仍不足。我们审计了16个开放的RLVR数学语料库，包含1,673,514个语料库专用训练项目，对比6个基准测试，共1,949个问题。我们利用594对独立且盲目双重审查的训练-基准对，基于精确、词汇和语义匹配评估了8个策略。校准集和保留集的基准问题和训练示例族不相交。对80个抽样训练题目的83个候选匹配进行单独综述，测量了题目级错误移除情况。在余弦阈值0.80时，59个抽样删除中有52个（88%）确认为错误移除;这并非已证明的总体比例。我们的预注册网格中没有任何语义阈值（0.80-0.98）在校准时达到90%保守精度目标。现有由法官裁定的政策在保留语料库上实现了98.1%的解析精度和98.2%的候选帧回忆率。在六个抽样删除中，没有确认的错误移除，只有一个未解决案例;这并不意味着风险为零。在评估候选框架内，该政策比测试的语义阈值替代方案提供了更有利的精确度与召回平衡。召回不测量候选框架外的污染情况。我们发布经过审查的标签、逐保决策和覆盖核算，以支持清洁可靠性和成本的评估，而非仅仅是重叠计数。

Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning

注意力头的排序与划分，用于稀疏和持续强化学习的微调

• Authors: Ruizhe Shi, Zaiwen Yang, Qiubo Wang, Banghua Zhu, Maryam Fazel
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.21336
• Pdf link: https://arxiv.org/pdf/2610.21336
• Abstract We study sparse and continual reinforcement learning fine-tuning of language models through attention-head ranking and partitioning. For sparse learning, we rank attention heads by gradient norm to identify a small subset of heads that can be trained effectively. For continual learning, we partition heads across tasks to reduce cross-task interference and preserve previously learned capabilities. Theoretically, we relate gradient norm to training effectiveness, and further characterize why sparse, separated head updates can remain stable. Empirically, across math, instruction-following, and coding tasks, tuning higher-ranked heads consistently outperforms tuning lower-ranked ones while often matching denser baselines, and head partitioning improves retention while preserving adaptation to new tasks.
• 中文摘要 我们研究稀疏和持续强化学习通过注意力头排序和划分对语言模型的微调。对于稀疏学习，我们通过梯度范数对注意力头进行排序，以识别可有效训练的一小部分头。对于持续学习，我们将头分到不同任务，以减少跨任务干扰并保留先前学到的能力。理论上，我们将梯度范数与训练效果联系起来，进一步表征为何稀疏、分离的头更新能够保持稳定。实证地，在数学、指令跟随和编码任务中，调优高排名头始终优于低调头，且常常匹配更密集的基线，头部划分则提升记忆力，同时保持对新任务的适应性。

Honesty is the Best Policy ... or is it? Demand Management for Probabilistic Service Options

诚实是最佳政策......还是说不是？概率服务选项的需求管理

• Authors: Daniela Sailer, Robert Klein, Claudius Steinhardt
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.21351
• Pdf link: https://arxiv.org/pdf/2610.21351
• Abstract To enable well-informed customer choices and redistribute demand across service options, providers can display probabilities regarding predicted service levels at the time of booking, effectively offering probabilistic service options. In many service settings, the underlying truthful probabilities endogenously depend on the system state, rather than being static product attributes controlled by the provider. While customers rely on this information when making decisions, they cannot readily verify its accuracy. Consequently, providers may be tempted to manipulate the displayed probabilities to guide customers' choices in their favor, creating a novel lever for demand management. In this work, we model the resulting problem as a sequential decision process, define truthful probabilities recursively, and introduce transparency constraints that limit their distortion. Because the exact computation of truthful probabilities is intractable for realistic settings, we develop an adapted reinforcement learning algorithm to approximate them. Furthermore, we propose a sampling-based mixed-integer program to optimize the information communicated by the provider in each state. Our framework enables providers to gauge the benefits of dynamically disclosing probabilistic information at the time of booking and sheds light on incentives for manipulation. Our computational study is motivated by out-of-home delivery, i.e., the delivery to parcel lockers or shops, and reveals a trade-off between individual-level transparency and aggregate customer utilities.
• 中文摘要 为了实现客户明智选择并将需求重新分配到各服务选项之间，服务提供者可以在预订时显示预测服务水平的概率，从而有效提供概率服务选项。在许多服务环境中，底层的真实概率内生依赖于系统状态，而非由服务提供者控制的静态产品属性。虽然客户在决策时依赖这些信息，但他们无法轻易验证其准确性。因此，提供者可能会倾向于操控显示的概率，以引导客户的选择，从而创造一种新的需求管理杠杆。在本研究中，我们将由此产生的问题建模为一个顺序决策过程，递归定义真实概率，并引入限制其扭曲的透明度约束。由于真实概率的精确计算在现实环境中难以处理，我们开发了一种适应性的强化学习算法来近似这些概率。此外，我们提出了基于抽样的混合整数项目，以优化各州提供者传达的信息。我们的框架使提供者能够评估在预订时动态披露概率信息的效益，并揭示操控的激励机制。我们的计算研究以外送为驱动，即送达包裹柜或商店，揭示了个人层面透明度与整体客户效用之间的权衡。

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

感知还是推理？诊断和解决VLM中的强化学习瓶颈

• Authors: Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.21380
• Pdf link: https://arxiv.org/pdf/2610.21380
• Abstract Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty is that visual task failures can arise from different sources: the same benchmark may be limited by perception for one model and by reasoning for another. We introduce Perception and Reasoning Attribution (PARA), which measures the remaining perceptual and reasoning headroom of each model--benchmark pair by comparing how many current errors are recovered with more accurate visual information versus a stronger reasoner. Across 4 vision-language models and 17 multimodal benchmarks, PARA broadly agrees with the expected characteristics of existing benchmarks while revealing substantial model-specific variation. More importantly, it shows that RL gains become smaller as perceptual headroom increases, but larger when reasoning explains more of the recoverable errors, suggesting that current RL exploits reasoning headroom more effectively than perceptual headroom. Motivated by this gap, we analyze how existing RL objectives affect visual perception and find that reward- and token-level signals influence visual attention only indirectly. We therefore propose GRAD, a group-relative attention distillation objective that uses the best rollout in each group to directly guide the visual attention of weaker rollouts. GRAD consistently improves performance across two model scales and seven benchmarks, and further complements existing RL methods.
• 中文摘要 强化学习为语言模型推理带来了显著进步，但在视觉任务上的改进仍然不够一致。一个难点是视觉任务失败可能来自不同来源：同一基准可能受限于某一模型的感知和推理。我们引入了感知与推理归因（PARA），通过比较更准确的视觉信息与更强推理者相比，衡量每对模型-基准对剩余的感知和推理余量。在4个视觉语言模型和17个多模态基准中，PARA大致符合现有基准的预期特性，同时揭示了显著的模型特异性差异。更重要的是，它显示强化学习的收益随着感知余量增加而变小，但当推理解释更多可恢复错误时，收益会更大，表明当前强化学习比感知余量更有效地利用了推理余量。基于这一差距，我们分析现有强化学习目标如何影响视觉感知，发现奖励级和代币级信号仅间接影响视觉注意力。因此，我们提出了GRAD，一种群体相对注意力蒸馏目标，利用每组中最佳的推广直接引导较弱的推广的视觉注意力。GRAD在两个模型尺度和七个基准测试中持续提升表现，进一步补充现有强化学习方法。

Learning to Reason with Selective Internal Feedback

学习选择性内在反馈推理

• Authors: Xun Lu, Shumeng Yang, Xinrui Wang, Songjun Tu, Linjing Li, Daniel Dajun Zeng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21392
• Pdf link: https://arxiv.org/pdf/2610.21392
• Abstract Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates. SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support. Sample and group level filtering determines which responses contribute to policy optimization. Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B. Further analyses show better retention of accuracy during continued optimization, while test-time training experiments demonstrate sustained improvements with as few as four rollouts per prompt.
• 中文摘要 基于内部反馈的强化学习可以在没有真实答案或外部验证者的情况下改进语言模型。然而，信心和正确性在训练过程中可能出现分歧，持续优化平均自确定性可能会提升信心，同时降低推理准确性。我们介绍SIF，一种强化学习方法，结合了比较奖励构建与选择性策略更新。SIF比较多个窗口和节点的排名代币级自确定性档案，只有当两个投票层级都提供足够支持时才给予正向奖励。样本和组级过滤确定哪些响应有助于策略优化。在12个数学和科学推理基准测试中对三条骨干的实验显示，SIF的最终检查点的平均准确率高于Intuitor和Vigor的最佳验证检查点，较Qwen2.5-Math-1.5B的Intuitor提升4.71个百分点。进一步分析显示，在持续优化过程中准确度保持更好，测试时训练实验也显示出持续提升，每个提示只需四次展开即可实现。

V-SHARE: Provable Convergence Guarantees for Advantage-based Algorithms

V-SHARE：基于优势算法的可证明收敛保证

• Authors: Kintan Saha, Hamed Masoudi, Xinyi Sheng, Dominik Baumann, Joni Pajarinen, Arsenii Mustafin
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21448
• Pdf link: https://arxiv.org/pdf/2610.21448
• Abstract In reinforcement learning, advantage-based value-learning methods, unlike standard Q-learning, share information across actions by decomposing the action-value function into a state-dependent value component and action-specific advantages. While this mechanism has proven effective in both tabular and deep reinforcement learning, its theoretical understanding remains incomplete. We introduce V-SHARE, an advantage-based value-learning algorithm that combines cross-action value sharing with an importance-weighted Bellman-residual correction. To our knowledge, we provide the first complete theoretical characterization of an advantage-based algorithm, including both asymptotic convergence and non-asymptotic finite-sample guarantees. We identify a regime in which V-SHARE mitigates the rare-action bottleneck of asynchronous Q-learning and uncover a contraction--variance tradeoff between stronger deterministic contraction and increased importance-weighting noise. Motivated by this tradeoff, we develop decreasing residual-preconditioning schedules that combine faster early contraction with lower asymptotic noise. We further develop a two-timescale analysis that removes the single-timescale residual-weight threshold while preserving convergence guarantees. Experiments on discrete Markov decision processes (MDPs) and deep reinforcement-learning benchmarks corroborate our results.
• 中文摘要 在强化学习中，基于优势的价值学习方法不同于标准的Q学习，通过将动作价值函数分解为状态依赖的价值成分和动作特定优势，实现跨行动共享信息。尽管该机制在表格和深度强化学习中均有效，但其理论理解尚不完整。我们介绍了V-SHARE，一种基于优势的价值学习算法，结合了跨动作价值共享与重要性加权的贝尔曼残差修正。据我们所知，我们首次提供了基于优势算法的完整理论特征描述，包括渐近收敛和非渐近有限样本保证。我们识别出V-SHARE缓解异步Q学习罕见动作瓶颈的模式，并揭示了更强确定性收敛与重要性加权噪声增加之间的收缩-方差权衡。基于这一权衡，我们开发了递减残差预条件计划，结合了更快的早期收缩和更低的渐近噪声。我们进一步开发了一种二时间尺度分析，去除单一时间尺度残差权重阈值，同时保持收敛保证。离散马尔可夫决策过程（MDP）和深度强化学习基准测试的实验支持了我们的结果。

Incentivizing Tool-Use Reinforcement Learning for Post-Arrival Emergency Care Prediction

激励工具使用强化学习以实现抵达后急救预测

• Authors: Pengfei Hu, Chenwei Wu, Qiao Jin, Jun Deng, Zhiyong Lu, Liyue Shen, Yue Ning
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21449
• Pdf link: https://arxiv.org/pdf/2610.21449
• Abstract Large language models (LLMs) can reason over both structured measurements and free-text records, which makes them attractive for emergency department (ED) prediction. However, most existing LLMs still fall short of conventional ML methods like trees on EHR-based prediction tasks. Our controlled analysis traces this gap to two weaknesses: LLMs do not select informative features, and they misjudge what numerical values imply for risk even when given the right features. We propose TIR-EC, a tool-integrated reinforcement learning framework that addresses both. TIR-EC exposes EHR feature groups as tools the LLM can call. It trains the LLM with rewards from a gradient-boosted tree teacher, which provides a reference set of informative tools and a graded risk target. TIR-EC also distills task-specific clinical skills into the model weights through a reverse curriculum that gradually removes the skills from the prompt during RL process. Evaluation on three clinical prediction tasks shows that our 4B agent achieves the highest macro F1 of all compared models, outperforms conventional ML baselines, and is competitive with much larger frontier LLMs. Further analysis shows that TIR-EC selects more useful evidence with fewer tool calls and combines measurements more effectively.
• 中文摘要 大型语言模型（LLM）能够对结构化测量和自由文本记录进行推理，这使其在急诊科（ED）预测中具有吸引力。然而，大多数现有LLM仍不及传统的机器学习方法，如基于EHR的预测任务中的树状结构。我们的受控分析将这一差距追溯到两个弱点：LLMs不选择信息性特征，且即使赋予了正确的特征，它们也会误判数值对风险的含义。我们提出了TIR-EC，一个工具集成的强化学习框架，涵盖了这两者。TIR-EC将EHR特征组暴露为LLM可调用的工具。它通过梯度增强树状结构教师的奖励来训练LLM，提供一套参考信息工具和分级的风险目标。TIR-EC还通过逆向课程将任务特定的临床技能提炼进模型权重，逐步在强化学习过程中从提示中剔除这些技能。对三项临床预测任务的评估显示，我们的4B代理在所有对比模型中实现了最高的宏观F1，优于传统机器学习基线，并与更大型的前沿大型语言模型竞争。进一步分析显示，TIR-EC以更少的工具调用选择了更有价值的证据，并更有效地组合了测量数据。

QREF: Improving DeepSearch Agents by Aligning Search with Reasoning via Query Refinement and Evidence Filtering

QREF：通过查询细化和证据过滤，使搜索与推理对齐，提升深度搜索代理

• Authors: Yujia Chen, Yuhui Wang, Junzhe Wang, Shihan Dou, Di Liang, Yujiong Shen, Yuhao Zhou, Zhiheng Xi, Ming Zhang, Senjie Jin, Jiayi Chen, Dingwei Zhu, Yajie Yang, Yuan Hua, Chenyao Lu, Tao Gui, Qi Zhang, Xuanjing Huang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21512
• Pdf link: https://arxiv.org/pdf/2610.21512
• Abstract DeepSearch agents achieve strong performance on complex information-seeking tasks through iterative search and reasoning. However, they often issue intent-misaligned, under-constrained, or redundant search queries, resulting in poor evidence coverage and accumulating retrieval noise. To mitigate these limitations, we propose QREF, a plug-and-play Query Refiner and Evidence Filter. Specifically, the Query Refiner rewrites search queries according to the current reasoning intent while ensuring complementary coverage across each search batch. The Evidence Filter retains evidence relevant to both the current search batch and the reasoning objective, reducing distracting retrieval results. We initialize the Query Refiner with supervised fine-tuning and optimize it through reinforcement learning based on interactions with a real-world web search environment, reducing ineffective queries. Comprehensive experiments demonstrate that our approach improves the performance of DeepSearch agents across four challenging benchmarks, achieving average relative Pass@1 improvements of 33.55% on WebSailor and 11.04% on TongyiDeepResearch without modifying architectures or retraining their reasoning policies. Compared with the original DeepSearch backbone without QREF, the RL-trained refiner reduces the fraction of query batches that lack discriminative retrieval anchors by 95%. Compared with the SFT-trained refiner, it also lowers the filter-based ineffective query ratio.
• 中文摘要 DeepSearch代理通过迭代搜索和推理在复杂的信息寻求任务中表现出色。然而，它们常常发出意图错位、限制不足或冗余的搜索查询，导致证据覆盖度差和检索噪声累积。为缓解这些限制，我们提出了QREF，即插即用的查询精炼器和证据过滤器。具体来说，查询优化器根据当前推理意图重写搜索查询，同时确保每个搜索批次的覆盖性互补。证据过滤器保留与当前搜索批次和推理目标相关的证据，减少分散注意力的检索结果。我们通过监督微调初始化查询精炼器，并通过基于与现实世界网络搜索环境交互的强化学习进行优化，减少无效查询。综合实验表明，我们的方法在四个具有挑战性的基准测试中提升了DeepSearch代理的性能，在WebSailor上实现了33.55%的相对Pass@1提升，在TongyiDeepResearch上实现了11.04%的平均提升，无需修改架构或重新训练推理策略。与无QREF的原始DeepSearch骨干网相比，强化学习训练的优化器将缺乏判别性检索锚点的查询批次比例降低了95%。与SFT训练的优化器相比，它还降低了基于过滤器的无效查询比率。

Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability

部分可观测性下受限强化学习的有限时间保证

• Authors: Valentina Zaccaria, Philip Jordan, Tingting Ni, Gian Antonio Susto, Maryam Kamgarpour
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.21548
• Pdf link: https://arxiv.org/pdf/2610.21548
• Abstract We study reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent maximizes reward subject to a cost constraint using only incomplete information about the underlying state. The lack of state observability may require optimal policies to depend on arbitrarily long observation-action histories, making planning and learning challenging. We leverage recent advances in finite-memory approximations of POMDPs and represent finite history windows as states of a constrained MDP. We then develop a primal--dual algorithm that provides the first non-asymptotic guarantees that jointly bound average reward suboptimality and constraint violation in the original CPOMDP. A key challenge is that this finite-window approximation affects both reward values and the set of feasible policies. Under a filter-stability assumption, we derive bounds relating optimality and feasibility in the finite-state model to those in the original CPOMDP, with approximation errors that decay geometrically with the window length. Experiments in partially observable gridworlds demonstrate the effectiveness of our approach.
• 中文摘要 我们研究受限部分可观测马尔可夫决策过程（CPOMDPs）中的强化学习，该过程中，智能体在成本约束下仅使用不完整的底层状态信息最大化奖励。状态可观测性缺失可能需要最优策略依赖任意长的观察-动作历史，使得规划和学习具有挑战性。我们利用POMDP有限内存近似的最新进展，将有限历史窗口表示为受限MDP的状态。随后，我们开发了原始-对偶算法，提供了首个非渐近保证，共同约束了原始CPOMDP中平均奖励的次优性和约束违规。一个关键挑战是，这种有限窗口近似影响奖励值和可行策略集合。在滤波器稳定性假设下，我们推导有限状态模型中最优性和可行性的界限与原始CPOMDP的界限，近似误差随窗口长度几何衰减。在部分可观测网格世界中的实验展示了我们方法的有效性。

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

航点-OPD：通过门控策略提纯转移强化语言引发的政策转移

• Authors: Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.21601
• Pdf link: https://arxiv.org/pdf/2610.21601
• Abstract Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.
• 中文摘要 策略转移蒸馏将教师强化学习（RL）前后日志概率的差异作为学生生成轨迹的监督。然而，这一终点策略转变无法揭示局部行动偏好在教师培训期间是否一致演变。同一端点转移可能由早期和晚期转变产生，这些转变相互强化或部分抵消。我们研究早期和晚期教师政策转变的一致性是否能指导选择性转移。我们引入了Waypoint引导的On-Policy 蒸馏（Waypoint-OPD），该方法使用同一RL运行中的中间教师检查点，比较学生访问前缀下相对行动偏好的早期和晚期转变。我们根据早晚转移的方向一致对终点转移进行门控，当它们指向相反方向时则加以抑制。与仅终端转移相比，我们的方法只需额外一次冻结教师评分。利用经过强化学习训练的Qwen3-4B教师，我们以Qwen3-8B、Qwen3-14B和Qwen3-30B-A3B作为数学和代码推理的学生模型来评估我们的方法。在所有六个领域——学生环境中，Waypoint-OPD在比较的蒸馏方法中获得了最高的领域平均得分。对Qwen3-8B学生的消融显示，移除或调整门槛会降低数学推理表现。这些结果支持在学生培训期间，利用早期和晚期教师政策转变的方向一致来加权终点转移。

ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training

ReFlow：以回放驱动的解耦，实现高效的强化学习后训练

• Authors: Yukun Tian, Junwei Su, Haofeng Xu, Lansong Diao, Zhengping Qian, Weibo Zhao, Wenchao Zhou, Ruiting Zhou, Chuan Wu
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2610.21640
• Pdf link: https://arxiv.org/pdf/2610.21640
• Abstract Reinforcement learning (RL) has been a critical paradigm for improving large language model (LLM) reasoning and agentic capabilities, but its efficiency remains a major bottleneck. Existing RL systems commonly rely on asynchronous execution to overlap rollout generation with policy optimization for enhanced efficiency. However, this approach faces inherent limitations: mild policy staleness offers only limited training efficiency gains, whereas aggressive asynchrony can degrade model quality and yields diminishing efficiency gains. The dynamic nature of LLM RL and multi-turn agentic workloads further challenges RL system design, as resource demands shift over time, making static resource allocation and generation strategy inefficient. This paper presents ReFlow, a system-algorithm co-designed RL framework for efficient LLM post-training. Its key insight is that, unlike existing RL frameworks which consume each trajectory only once, experience replay of recent trajectories enables effective sample reuse, thereby amortizing the high cost of rollout generation without incurring excessive policy staleness. To fully exploit this opportunity, ReFlow adopts a decoupled and adaptive architecture with three key designs: a dual replay data cycle supports efficient experience reuse while balancing training throughput and optimization stability; an elastic resource pool enables dynamic resource placement to accommodate evolving RL workloads; an adaptive scheduler coordinates generation strategies and resource allocation to improve resource utilization and boost generation throughput. Experiments across representative single-turn reasoning and multi-turn agentic RL tasks show that ReFlow improves training throughput by up to $2.06\times$ over state-of-the-art synchronous and asynchronous RL systems, while maintaining comparable or better model performance.
• 中文摘要 强化学习（RL）一直是提升大型语言模型（LLM）推理和代理能力的关键范式，但其效率仍是主要瓶颈。现有的强化学习系统通常依赖异步执行来将推出生成与策略优化重叠，以提升效率。然而，这种方法存在固有局限：轻度策略陈旧只能有限提升训练效率，而激进异步则可能降低模型质量并递减效率提升。LLM RL的动态特性和多回合代理工作负载进一步挑战了RL系统设计，因为资源需求随时间变化，使静态资源分配和生成策略效率降低。本文介绍了ReFlow，一个由系统算法联合设计的高效LLM后训练框架。其关键见解是，与现有仅消耗每个轨迹一次的强化学习框架不同，近期轨迹的经验重放使得样本的重用成为可能，从而在不导致过度策略陈旧的情况下，摊销了高昂的展开生成成本。为充分利用这一机会，ReFlow采用了解耦且自适应的架构，具有三大设计：双重回放数据周期支持高效的经验重用，同时平衡训练吞吐量和优化稳定性;弹性资源池支持动态资源配置以适应不断演变的强化学习工作负载;自适应调度器协调生成策略和资源分配，以提升资源利用率和生成吞吐量。在代表性的单回合推理和多回合智能强化学习任务中的实验显示，ReFlow在保持可比或更好模型性能的同时，可将训练吞吐量提升高达2.06美元/倍数。

Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence

通过成对令牌对比和自适应轨迹置信度增强多模态RLVR

• Authors: Yiran Zhang, Hanbo Huang, Jielan Li, Zhen Zhao, Shiyu Liang, Hongxia Hao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.21671
• Pdf link: https://arxiv.org/pdf/2610.21671
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but group-relative policy optimization broadcasts a single sequence-level advantage to every token of a response, although errors usually localize to a few decisive steps. Recent self-distilled variants refine credit with a teacher--student likelihood gap. However, this signal cannot tell whether it relies on the image or on a language shortcut, and is applied uniformly even when the teacher is uninformative. To address this, we propose Pairwise token contrast and Adaptive trajectory ConfidencE (PACE). Pairwise token contrast scores each response under four teacher views, crossing a correct or incorrect answer hint with the image present or removed, and compares the two hints for every token, which cancels shifts shared by both hints and reveals whether the token's support relies on the image. Adaptive trajectory confidence measures how well the teacher predicts each response given the correct hint and the image, normalizes this score within the batch, and uses it to set how far the token contrast redistributes credit within that response. PACE achieves the best average accuracy on three backbones and ranks first on all evaluated multimodal reasoning benchmarks with Qwen3-VL-8B-Instruct. Experiments show that both components matter: replacing pairwise token contrast with a single-view signal underperforms our method, while adaptive trajectory confidence adds further improvement.
• 中文摘要 带有可验证奖励的强化学习（RLVR）提升了多模态推理能力，但群体相对策略优化则对每个反应符号广播单一序列层级优势，尽管错误通常局限于少数决定性步骤。最近的自我提炼变体通过教师-学生似然差距细化了信用。然而，该信号无法判断它是依赖图像还是语言捷径，即使教师信息不多，也同样统一地应用。为此，我们提出了成对令牌对比和自适应轨迹自信E（PACE）。成对令牌对比对每个回答在四个教师视图下评分，将正确或错误的提示与图像存在或移除的提示交叉，并比较每个标记的两个提示，从而抵消双方共享的偏移，揭示代币的支持是否依赖于图像。自适应轨迹置信度衡量教师在正确提示和图像下预测每个回答的准确性，在批次中对该分数进行归一化，并用以设定符号对比度在该回答中重新分配学分的程度。PACE在三个骨干链上实现了最佳的平均准确率，并在所有评估的多模态推理基准测试中与Qwen3-VL-8B-Instruct一起排名第一。实验显示这两个组成部分都很重要：用单视角信号替代成对符号对比度表现不如我们的方法，而自适应轨迹置信度则进一步提升了效果。

Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning

路由器漂移控制：减少专家混合强化学习中策略外的降级

• Authors: Liangyao Jin, Zhixin Wang, Liming Liu, Mingjun Zhang, Zhengzhou Zhu, Fangcheng Fu, Jinlong Hou, Yuan Cheng, Xiaohe Hu, Ping Zhang, Siyuan Feng
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.21682
• Pdf link: https://arxiv.org/pdf/2610.21682
• Abstract Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous this http URL resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout--training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.
• 中文摘要 强化学习（RL）可能因训练-推断不匹配、小批量更新和异步而出现偏策略。这种HTTP URL导致的不匹配在专家混合（MoE）模型中尤其有害，因为部署——训练差异可能改变路由决策并导致路由器漂移。部署路由重放（R3）通过在训练中重用部署阶段的专家选择来缓解这种不匹配。我们的分析显示，路由重放引入梯度偏见，因为它会丢失训练路由器选择的专家的贡献，而部署路由器则未参与。为此，我们提出了路由器漂移控制（RDC），它利用部署与训练之间的路由不一致作为策略外的陈旧信号，并选择性地拒绝高漂移响应。序列级过滤避免了重放引入的梯度偏置，同时保留了本地专家选择和相关的学习信号，以保留响应。实验表明，随着政策陈旧性增加，RDC在异步强化学习中获得更大收益，且效果依然有效。在异步训练中，RDC结合截断重要性抽样（TIS）使AIME 2024和AIME 2025的平均峰值验证准确率比TIS提高了5.5个百分点，比R3提高了4.5个百分点。

Uni-PNP: Unified Prehensile and Non-prehensile Policy Learning for Generalizable Object Pose Reaching

Uni-PNP：统一的可抓握与非抓握策略学习，用于可推广对象姿态达标

• Authors: Wenxuan Zhu, Yuze Wang, Jiangran Lyu, Jiayi Chen, Yixin Zheng, Mi Yan, Li Yi, Zhizheng Zhang, He Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.21735
• Pdf link: https://arxiv.org/pdf/2610.21735
• Abstract Object manipulation often requires combining non-prehensile actions with grasping. However, existing approaches typically develop these capabilities separately and integrate them by sequencing distinct primitives through prescribed intermediate objectives, limiting the coordination of contact motions and grasping. Their integration also remains sensitive to object and gripper geometry and depends on particular environmental supports, complicating real-world deployment. We present Uni-PNP, a unified policy that jointly learns prehensile and non-prehensile manipulation for object pose reaching across diverse objects and grippers. To achieve this, we first pretrain a geometry encoder to predict contact-induced object motion using a contact-dynamics dataset constructed from simulated interactions between diverse objects and grippers. We then train the policy in a two-stage reinforcement learning curriculum from graspable objects to ungraspable peers, encouraging the discovery of non-prehensile interactions and their integration with grasping. Experiments show 76.39% success on hundreds of seen objects with grippers, and 57.91% on unseen peers, outperforming the strongest pose-reaching baseline by 20.52 and 17.79 percentage points, respectively. We also analyze emergent behaviors, including nonprehensile and prehensile repositioning, grasp acquisition that requires grasping after tipping, and table-assisted reorientation while grasping, and transfer to the real-world scenario. Project page available at this http URL.
• 中文摘要 物体操作通常需要将非抓取动作与抓取结合起来。然而，现有方法通常分别开发这些能力，并通过规定的中间目标对不同原语进行序列整合，限制了接触运动和抓取的协调。它们的整合也对物体和抓握器几何结构敏感，并依赖特定的环境支撑，增加了实际部署的复杂性。我们提出了Uni-PNP，一种统一策略，联合学习跨越不同物体和抓握器的物体姿态操作。为此，我们首先预训练一个几何编码器，利用由不同物体与抓手之间模拟交互构建的接触动力学数据集预测接触诱导的物体运动。随后，我们将该策略训练为两阶段的强化学习课程，从可抓取对象到不可抓取的同伴，鼓励发现非抓握互动并将其与抓取整合。实验显示，数百个有抓握物的物体成功率为76.39%，在未被看见的同伴中有57.91%，分别比最强姿势达到基线高出20.52和17.79个百分点。我们还分析了涌现行为，包括非抓握和抓握重新定位、倾斜后需要抓握的抓取习得、抓握时的桌子辅助重新定向，以及向现实场景的转移。项目页面可在此 http 网址访问。

DeciCAD: Harness-Grounded Learning of Action Policies for CAD Agents

DeciCAD：基于行动政策的基于驱动学习，适用于CAD代理

• Authors: Yufeng Peng, Yixin Zhang, Ziyi Liu, Zhikun Xu, Jiarong Yang, Sijie Wang, Wenjie Feng, Qiyu Kang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.21755
• Pdf link: https://arxiv.org/pdf/2610.21755
• Abstract Large Language Model agents operate through closed-loop action policies whose decision space is defined by an agent harness. We investigate how this operating space shapes agent behavior and policy learning in computer-aided design (CAD), a professional domain characterized by persistent state, geometric constraints, and long-horizon procedures. We instantiate this perspective in DeciCAD, a domain-native harness that exposes semantic CAD actions, decision-oriented observations, environment feedback, and persistent state transitions. Building on this harness, we develop a learning pipeline that extracts causal decision supervision from interaction trajectories and improves the policy through supervised fine-tuning (SFT) and geometry-grounded online reinforcement learning (RL). Across four CAD benchmarks, DeciCAD achieves the best F1 and intersection over union (IoU) while reducing model-output tokens by 45--88% relative to a general-purpose coding harness. Under matched supervision, its SFT policy improves F1 from 0.6883 to 0.7437 and outperforms the coding policy; subsequent online RL further raises geometry availability from 66.5% to 73.5% and F1 to 0.7880. These results demonstrate that the decision space induced by a harness materially affects both agent execution and policy learning in CAD.
• 中文摘要 大型语言模型代理通过闭环动作策略工作，其决策空间由代理框架定义。我们研究该操作空间如何塑造计算机辅助设计（CAD）中的代理行为和策略学习，CAD是一个以持久状态、几何约束和长视野程序为特征的专业领域。我们将这一视角体现在DeciCAD中，DeciCAD是一种领域原生工具，揭示语义CAD动作、决策导向观察、环境反馈和持久状态转换。基于该工具，我们开发了一套学习流程，从交互轨迹中提取因果决策监督，并通过监督微调（SFT）和基于几何的在线强化学习（RL）改进策略。在四个CAD基准测试中，DeciCAD实现了最佳的F1和交叉于并集（IoU），同时相较于通用编码框架将模型输出令牌减少了45%至88%。在匹配监督下，其SFT策略将F1从0.6883提升至0.7437，并优于编码策略;后续在线强化学习进一步将几何可用性从66.5%提升至73.5%，F1提升至0.7880。这些结果表明，由约束诱导的决策空间对代理执行和策略学习在CAD中均有实质性影响。

Q($γ_H$): Entropy-Guided Horizon Mixing in Multi-Discount Q-Learning

问（$γ_H$）：熵引导视界与多折Q学习的混合

• Authors: Arwa Albelaihi, Abdulrahman Altahhan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.21764
• Pdf link: https://arxiv.org/pdf/2610.21764
• Abstract A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($\gamma_H$), a multi-horizon Q-learning framework that maintains multiple discount-specific heads and adaptively combines them at control time using an entropy-derived confidence signal. The resulting state-dependent mixing weights shift control from shorter- to longer-horizon heads as policy confidence increases, enabling adaptive horizon mixing without modifying individual Bellman updates. Across classical control and deep reinforcement-learning benchmarks, Q($\gamma_H$) shows its strongest improvements on delayed-credit tasks relative to strong fixed-discount baselines. By decoupling horizon adaptation from the underlying learning updates, Q($\gamma_H$) provides a lightweight framework for adaptive planning-horizon control in reinforcement learning, avoiding the need for manual discount-factor selection while remaining compatible with standard value-learning methods.
• 中文摘要 Q学习中的固定折扣因子在训练过程中强制执行单一规划视野，尽管早期探索和长期学分分配的要求不同。我们提出了Q（$\gamma_H$），这是一个多视野的Q学习框架，维护多个折扣专属的头，并在控制时利用熵导出的置信信号自适应地组合它们。随着政策置信度的增加，状态依赖的混合权重将控制从短视野转向更长的视野头，实现自适应视野混合而不修改单个贝尔曼更新。在经典控制和深度强化学习基准测试中，Q（$\gamma_H$）相较于强固定折扣基线在延迟学分任务中表现最强。通过将视界适应与底层学习更新解耦，Q（$\gamma_H$） 提供了一个轻量级的自适应规划视野控制强化学习框架，避免了手动选择折扣因子的必要，同时保持与标准价值学习方法的兼容性。

Learning to Think and Explore in 3D World

学习在三维世界中思考和探索

• Authors: Yijiang Li, Bingyang Wang, Nuno Vasconcelos
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.21783
• Pdf link: https://arxiv.org/pdf/2610.21783
• Abstract We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.
• 中文摘要 我们研究视觉语言模型（VLM）如何学习思考并积极探索三维空间。我们不向模型提供多视图图像、视频或三维表示，而是保留VLM的原生视觉编码，即一次一张图像，就像人类一样，并将3D表示仅视为模型通过动作（如向左移动）和观察（渲染图像）互动的环境。每次部署都从初始第一人称视角和一个视觉问题开始。模型随后可以迭代推理、导航并重新定位视角，最终确定最终答案。经过初步轨迹监督的微调后，我们应用强化学习（RL），奖励主要由最终答案正确性驱动。关键是，强化学习不直接监督模型应访问哪些视角或应采取哪些探索行动。相反，模型必须仅通过交互提升性能的程度来学习有效的探索策略。在六个三维基准测试中，我们的模型ACT3D平均准确率为53.2%，其基础Qwen模型提升了22.0%，超过了专有的Gemini 3.6 Flash。ACT3D还在使用50-70%的输入帧数的情况下，表现优于现有开源3D方法，并能推广到看不见的领域，平均比监督初始化提升5.9%。进一步分析显示，自适应、问题条件探索。在初始视图可回答的问题中，性能提升从基础模型的0.6%提升到需要广泛探索的问题的16.3%。ACT3D搜索遮挡目标，避免不必要的探索，并获得互补的空间比较视角。这些发现表明，基于结果的强化学习不仅使VLM能够通过视觉证据进行推理，还能学习何时以及如何获得这些证据。

Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents

通过优化确定性等效物实现可证明高效的风险敏感离线强化学习

• Authors: Hanzhen Wang, Boxiang Lyu, Zhongxiang Dai, Siyang Gao, Dake Zhang, Shuang Qiu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.21870
• Pdf link: https://arxiv.org/pdf/2610.21870
• Abstract Risk-sensitive reinforcement learning (RL) addresses decision-making under uncertainty and adverse outcomes. Learning risk-sensitive policies from fixed datasets is important when additional interaction with the environment is costly. Existing work on risk-sensitive RL focuses on online settings or specific risk measures, leaving a general theoretical understanding of its offline setting underdeveloped. In this work, we investigate efficient offline risk-sensitive RL using the optimized certainty equivalent (OCE), a general class of risk measures. We consider two formulations: dynamic-OCE RL, which applies OCE recursively at each step, and static-OCE RL, which applies a single OCE to the cumulative rewards. Under the tabular setting, we develop pessimistic value-iteration algorithms for both formulations and establish tight sublinear suboptimality bounds, explicitly capturing how risk sensitivity, encoded by utility functions, affects learning difficulty. Moreover, we establish minimax suboptimality lower bounds to characterize the problem's fundamental limits. To handle large state spaces, we further develop pessimistic least-squares value-iteration algorithms that incorporate the general OCE risk criterion into linear function approximation. We establish utility-dependent sublinear guarantees with explicit dependence on the feature dimension $d$. To our knowledge, we provide the first offline algorithms along with the first upper and lower suboptimality bounds for both dynamic- and static-OCE RL.
• 中文摘要 风险敏感强化学习（RL）在不确定性和不利结果下进行决策。当与环境的额外交互代价高时，从固定数据集学习风险敏感策略尤为重要。现有关于风险敏感强化学习的研究聚焦在线环境或特定风险度量，导致其离线环境的理论理解尚不充分。本研究中，我们利用优化确定性等价（OCE）——一类风险度量的通用方法，研究高效的离线风险敏感强化学习。我们考虑了两种表述：动态OCE RL，递归地在每步应用OCE，以及静态OCE RL，对累积奖励应用单一OCE。在表格框架下，我们为两种表述开发了悲观的价值迭代算法，并建立了严格的亚线性亚最优界限，明确捕捉由效用函数编码的风险敏感性如何影响学习难度。此外，我们建立了极小极大亚最优性下界，以刻画问题的基本极限。为处理大型状态空间，我们进一步开发了将一般OCE风险准则纳入线性函数近似的悲观最小二乘值迭代算法。我们建立了效用依赖的子线性保证，明确依赖特征维数$d$。据我们所知，我们提供了动态和静态OCE强化学习的首批离线算法以及首个上下次最优界限。

The Tilted Sampling Problem: Optimism, Posterior Sampling, and Logarithmic Regret

倾斜抽样问题：乐观主义、后验抽样与对数遗憾

• Authors: Mohammad Pedramfar, Siamak Ravanbakhsh
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.21908
• Pdf link: https://arxiv.org/pdf/2610.21908
• Abstract Many modern learning systems are used not to select a single best action, but to sample diverse high-quality outputs. Examples include entropy-regularized reinforcement learning, RLHF, and reward-guided generative modeling. Motivated by these settings, we study tilted sampling: an online learning problem in which the goal is to sample from the Gibbs distribution induced by an unknown reward function. We define sampling regret as the temperature-normalized cumulative KL divergence between the learner's distribution and the Gibbs target, corresponding to the suboptimality gap in entropy-regularized reward maximization. Our framework generalizes recent sampling-based and KL-regularized formulations in bandits and reinforcement learning. Our main contribution is a general regret framework that captures the different uncertainty structure of tilted sampling and classical regret minimization. While standard regret scales as $\widetilde{O}(\sqrt{T})$ due to first-order uncertainty, sampling regret is governed by squared confidence widths, yielding logarithmic-in-$T$ regret in standard settings at fixed temperature. We develop sampling analogues of UCB and Thompson sampling (UCS and TS$^2$) and provide unified regret decompositions that explain and extend recent logarithmic guarantees, including the first Bayesian sampling-regret decomposition based on new bounds for posterior-random Gibbs distributions. Our results show that learning to sample from a soft-optimal distribution is statistically distinct from learning to identify a hard optimum.
• 中文摘要 许多现代学习系统不用于选择单一最佳动作，而是用来采样多样且高质量的输出。例如熵正则化强化学习、RLHF和奖励引导生成建模。受这些环境激励，我们研究倾斜抽样：一种在线学习问题，目标是从未知奖励函数诱导的吉布斯分布中抽样。我们将抽样遗憾定义为学习者分布与吉布斯目标之间的温度归一累积KL散度，对应熵正则化奖励最大化的次优差距。我们的框架推广了近期基于抽样和基层正则化的强化学习表述。我们的主要贡献是一个通用遗憾框架，捕捉倾斜抽样和经典遗憾最小化的不同不确定性结构。由于一阶不确定性，标准后悔的比例为$\widetilde{O}（\sqrt{T}）$，而抽样后悔则受置信宽度的平方控制，在固定温度下标准环境下后悔为对数级的后悔$T$。我们开发了UCB和Thompson抽样的抽样类比（UCS和TS$^2$），并提供了统一的遗憾分解，解释并扩展了近期的对数保证，包括基于后验随机Gibbs分布新界限的首个贝叶斯抽样-遗憾分解。我们的结果表明，从软最优分布学习采样与学习识别硬最优是不同的。

Learning Rewards in Strategic Games

战略游戏中的奖励学习

• Authors: Antoine Bergerault, Cyrille Kone, Giorgia Ramponi
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.21912
• Pdf link: https://arxiv.org/pdf/2610.21912
• Abstract Understanding the objectives that drive agents' behavior is crucial in strategic environments, where multiple agents interact and adapt to one another. This challenge arises in domains such as cybersecurity, autonomous driving, and online markets, where agents' underlying incentives are typically unknown and must be inferred from their observed behavior. Inverse reinforcement learning (IRL) provides a natural framework for this problem, but extending IRL to strategic multi-agent settings introduces a fundamental challenge: a reward that explains observed behavior does not necessarily induce optimal strategies when used for decision-making. In this work, we study this problem through the lens of \emph{strategic identifiability}, and show that learning from a fixed policy pair can be insufficient to strategically identify rewards, independent of the number of expert samples $K$. Motivated by this, we introduce an active IRL algorithm for two-player repeated Stackelberg games, in which the leader adaptively selects policies to elicit informative responses about the follower's reward. Combining maximum-likelihood estimation with experimental design, we derive finite-sample guarantees on the Stackelberg sub-optimality induced by the learned reward, matching up to polylogarithmic factors a new minimax learning rate of $K^{-1/2}$. Finally, we evaluate our approach in a realistic anti-poaching setting, demonstrating the benefits of active exploration for reward recovery and downstream strategic performance.
• 中文摘要 理解驱动智能体行为的目标在多个智能体相互作用和适应的战略环境中至关重要。这一挑战出现在网络安全、自动驾驶和在线市场等领域，这些领域中智能体的潜在激励通常未知，必须通过观察到的行为推断。逆强化学习（IRL）为该问题提供了自然框架，但将IRL扩展到战略性多智能体环境则带来了根本性挑战：解释观察行为的奖励在决策中使用时不一定能诱导出最优策略。本研究通过\emph{战略可识别性}的视角研究该问题，表明从固定策略对中学习不足以战略性识别奖励，这与专家样本数量$K$无关。基于此，我们引入了一种主动的现实现实算法，用于两人重复的斯塔克尔伯格游戏，领导者自适应地选择策略，以引发关于追随者奖励的信息性反应。结合最大似然估计与实验设计，我们对由学习奖励诱导的斯塔克尔伯格次最优性推导出有限样本保证，匹配多对数因子的新极大学习率$K^{-1/2}$。最后，我们在现实的反偷猎环境中评估我们的方法，展示了主动探索对奖励回收和下游战略绩效的益处。

WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents

WEAVE：将现实代码演变编织进依赖结构化任务，用于基准测试和训练代码代理

• Authors: Weihao Li, Zihang Xu, Zhongqiu Wang, Songwei Yu, Wenqiang Zhu, Yuechen Luo, Jingxuan Xu, Yanan Wu, Haoyang Huang, Ken Deng
• Subjects: Subjects: Software Engineering (cs.SE); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.22039
• Pdf link: https://arxiv.org/pdf/2610.22039
• Abstract Real-world software engineering rarely consists of isolated changes. Features, migrations, and systematic fixes often unfold across multiple pull requests (PRs), requiring coordinated modifications across a repository. Yet most executable repository-level benchmarks retain a single issue or PR as the task unit, while existing multi-PR settings rely on coarse aggregation and structure-agnostic evaluation. This mismatch obscures dependencies among changes and limits progress measurement and learning signals for long-horizon code agents. We introduce WEAVE, a framework that transforms verified PR histories into executable, dependency-structured tasks. WEAVE leverages repository-calibrated typed change loci to organize related PRs into bounded DAGs and instantiate each as a unified executable task. We further introduce Graph Credit, a deterministic, test-grounded metric that measures dependency-consistent partial progress and supports dense, verifiable rewards for reinforcement learning. Experiments on task construction, agent evaluation, supervised fine-tuning, and reinforcement learning show that WEAVE yields coherent executable tasks, enables more faithful progress measurement than flat scoring, and provides effective supervision and verifiable rewards for training code agents.
• 中文摘要 现实中的软件工程很少包含孤立的变更。功能、迁移和系统性修复通常在多个拉取请求（PR）中展开，需要在整个仓库间协调修改。然而，大多数可执行的仓库级基准测试仍保留单一问题或PR作为任务单元，而现有的多PR设置依赖粗略聚合和结构无关性评估。这种不匹配掩盖了变更之间的依赖关系，限制了长视野代码代理的进度测量和学习信号。我们引入了WEAVE，一个将验证PR历史转换为可执行的依赖结构任务的框架。WEAV利用仓库校准的类型变更位置，将相关PR组织为有界DAG，并将每个PR实例化为统一的可执行任务。我们还进一步介绍了图学分（Graph Credit），这是一种确定性、基于测试的指标，用于衡量依赖一致性的部分进展，并支持密集且可验证的强化学习奖励。任务构建、代理评估、监督微调和强化学习的实验表明，WEAVE能够产生连贯的可执行任务，比单一评分更忠实地衡量进展，并为训练代码代理提供有效的监督和可验证的奖励。

Progress Reward Model for Reinforcement Learning via Large Language Models

通过大型语言模型进行强化学习的进步奖励模型

• Authors: Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen, Yuheng Pan, Mohan Zhang, Yue Deng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22056
• Pdf link: https://arxiv.org/pdf/2610.22056
• Abstract Traditional reinforcement learning (RL) algorithms face significant limitations in handling long-term tasks with sparse rewards. Recent advancements have leveraged large language models (LLMs) to enhance RL by utilizing their world knowledge for task planning and reward generation. However, planning-based approaches often depend on pre-defined skill libraries and fail to optimize low-level control policies, while reward-based methods require extensive human feedback or exhaustive searching due to the complexity of tasks. In this paper, we propose the Progress Reward Model for RL (PRM4RL), a novel framework that integrates task planning and dense reward to enhance RL. For high-level planning, a complex task is decomposed into a series of simple manageable subtasks, with a subtask-oriented, fine-grained progress function designed to monitor task execution progress. For low-level reward generation, inspired by potential-based reward shaping, we use the progress function to construct a Progress Reward Model (PRM), providing theoretically grounded optimality and convergence guarantees, thereby enabling effective policy optimization. Experimental results on robotics control tasks demonstrate that our approach outperforms both LLM-based planning and reward methods, achieving state-of-the-art performance. Code is available at this https URL.
• 中文摘要 传统的强化学习（RL）算法在处理奖励稀疏的长期任务时面临显著局限。近期进展利用大型语言模型（LLMs）通过利用其世界知识进行任务规划和奖励生成，增强强化学习。然而，基于计划的方法常依赖预设技能库，未能优化低层控制策略;而基于奖励的方法则因任务复杂性，需要大量人工反馈或详尽搜索。本文提出了强化学习进展奖励模型（PRM4RL），这是一个整合任务规划与密集奖励以增强强化学习的新框架。对于高层规划，复杂任务被分解为一系列简单且可管理的子任务，并配备一个面向子任务、细粒度的进度函数，用于监控任务执行进展。在受基于潜力的奖励塑造启发的低级奖励生成中，我们利用进度函数构建进步奖励模型（PRM），提供理论基础的最优性和收敛保证，从而实现有效的策略优化。机器人控制任务的实验结果表明，我们的方法优于基于LLM的规划和奖励方法，实现了最先进的性能。代码可在此 https 网址获取。

TIPS: A Tiny Inference-time Policy for Scaling in Diffusion Language Models

提示：扩散语言模型中一个微小的推理时间策略，用于扩展扩展

• Authors: Ya-Shi Zhang, Jian Tang, Pierre-André Noël, Oleksiy Ostapenko
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22099
• Pdf link: https://arxiv.org/pdf/2610.22099
• Abstract Masked diffusion language models expose an inference-time choice: which positions become visible after each denoising pass. Many existing decoding methods use hand-designed rules or require visibility to grow monotonically. We present a $\textbf{T}$iny $\textbf{I}$nference-time $\textbf{P}$olicy for $\textbf{S}$caling ($\textbf{TIPS}$), an approximately $600$K-parameter controller that learns both unmasking and remasking while keeping the underlying language model frozen. TIPS uses visibility-aware confidence features and a shallow transformer to select which tokens should be visible, while a fixed schedule controls the expected visible fraction. We train the controller through supervised warmup followed by reinforcement learning from terminal correctness rewards. On GSM8K and MATH-500 with LLaDA-8B-Instruct, the full-canvas and two-block TIPS variants achieve $58.46\%$ and $57.97\%$ micro-averaged accuracy at a budget of $32$ decoder evaluations, exceeding the strongest baseline in our main comparison by $12.1$ and $11.6$ percentage points, respectively. Both variants are trained at a budget of $32$ evaluations and remain competitive at higher budgets without retraining. These results demonstrate that learning when to reveal and reconsider tokens can improve the accuracy-compute trade-off without updating the decoder backbone.
• 中文摘要 掩盖扩散语言模型揭示了一个推理时间选择：每次去噪后哪些位置变得可见。许多现有解码方法使用手工设计的规则，或要求可见性单调增长。我们为$\textbf{$iny S}$caling（$\textbf{TIPS}$$nference时间的$\textbf{P}$olicy给出：$\textbf{S}，这是一个约600$K参数的控制器，在保持底层语言模型冻结的同时学习解除和重新掩蔽。TIPS使用可见性感知的置信度功能和浅变换器来选择哪些标记应被看到，同时固定时间表控制预期可见比例。我们通过监督预热训练控制器，随后通过终端正确性奖励进行强化学习。在GSM8K和MATH-500搭配LLaDA-8B-Instruct时，全画布和双块TIPS变体在32美元的解码器评估预算下实现了58.46美元和57.97%%美元的微平均精度，分别比我们主要比较中最强基线高出12.1美元和11.6美元百分点。这两个变体的评估预算均为32美元，且在更高预算下无需重新训练即可保持竞争力。这些结果表明，学习何时揭示和重新考虑令牌可以改善准确率与计算权衡，而无需更新解码器骨干。

Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld

规范博弈与清洁网格世界中的受限强化学习缓解

• Authors: Hilda Adwubi Osei
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22152
• Pdf link: https://arxiv.org/pdf/2610.22152
• Abstract Reinforcement learning agents optimize exactly the reward signal they are given, which can diverge sharply from an intended true objective when that signal is only a proxy for it; this failure mode is known as specification gaming. Existing evidence for this failure mode is largely observational, with comparatively few demonstrations pairing a controlled, deliberately constructed proxy/true gap with a mitigation that carries a verifiable guarantee rather than only an empirical improvement. We construct a minimal gridworld in which a cleaning-robot agent can either genuinely remove dirt or conceal it from a cheap sensor, and show that a standard deep Q-network trained only on the sensor's proxy reward converges to a policy that games it completely, reaching near-maximal proxy reward while never achieving the true objective. We then introduce a constrained reinforcement learning fix, penalizing the specific behaviors through which the agent exploits the gap, and show that it eliminates the gaming behavior from the final policy while substantially recovering true-objective performance. We further derive and empirically verify an exact accounting identity relating the proxy reward, true reward, and our cost signal for this environment, showing that the cost signal measures the proxy/true gap exactly rather than merely tracking it, and yields a deterministic guarantee under a hard-constrained variant of the method.
• 中文摘要 强化学习代理精确优化它们所获得的奖励信号，当该信号仅是其代理时，奖励信号可能与预期目标有明显偏离;这种失败模式被称为规格博弈。现有证据主要支持这种失败模式，较少有演示将受控、刻意构建的代理/真实差距与带有可验证保证而非仅仅经验改进的缓解措施结合起来。我们构建了一个最小网格世界，其中清洁机器人智能体可以真正清除污垢或将其隐藏于廉价传感器之外，并证明仅基于传感器代理奖励训练的标准深度Q网络收敛到完全操纵该机制的策略，达到近乎最大代理奖励但永远无法实现真实目标。随后引入受限强化学习修复，惩罚代理利用差距的具体行为，证明该方法消除了最终策略中的博弈行为，同时大幅恢复真实客观性能。我们进一步推导并实证了一个精确的会计同一性，将代理奖励、真实回报与成本信号联系起来，证明成本信号精确测量代理/真实差距，并在硬约束变体下给出确定性保证。

Mosaic: Multi-Session Agentic Reinforcement Learning

马赛克：多会话智能强化学习

• Authors: Yilong Xu, Minrui Xu, Xinyu Zhou, Xiang Long, Haochen Tan, Zhijiang Guo
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22174
• Pdf link: https://arxiv.org/pdf/2610.22174
• Abstract Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.
• 中文摘要 大多数现有的代理强化学习方法假设单会话轨迹，即交互在一个连续累积的上下文中进行。然而，上下文压缩和子代理分叉等代理技术可能产生跨越多个会话的轨迹，导致单会话训练假设与推理时间执行之间存在不匹配。因此，我们引入了多会话代理强化学习，这是一种更通用的环境，支持轨迹内任意数量的会话。在该框架下，广泛采用的方法是向所有会话广播轨迹级优势，但这并不区分不同会话的贡献，也无法指导轨迹内会话的分配。为解决这个问题，我们提出了Mosaic，一种多会话代理强化学习方法，具有无批评的层级优化。在轨迹层面，它将会话使用信号纳入优势估计中。在会话层面，它根据估计的边际贡献来塑造每个会话的优势。在 Qwen3.5-4B 的 BrowseComp 实验显示，Mosaic 在上下文压缩和子代理分叉场景中均优于广播，且平均使用更少会话数。进一步分析显示，我们的方法在会话效用和分配方面取得了改进，我们认为这是多会话代理强化学习未来进展的有前景目标。

Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

统一愿景语言模型的技能解耦政策专家提炼

• Authors: Yuqi Xiong, Chunyi Peng, Zhipeng Xu, Qing Yang, Zhenghao Liu, Zulong Chen, Yu Gu, Ge Yu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22217
• Pdf link: https://arxiv.org/pdf/2610.22217
• Abstract Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual evidence, and better preservation of specialized expertise in the unified student. All code and data are available at this https URL.
• 中文摘要 强化学习（RL）可以在视觉语言模型（VLM）中引发专业视觉能力，但联合优化异构目标常常引入跨能力梯度冲突。培训专家独立缓解这种干扰，但直接的奖励优化可能使专业知识与奖励黑客偏差纠缠在一起。本文提出了技能解耦策略上专家提炼（SCOPED），通过奖励引导反思培训特定能力专家并提取显性技能。在策略上提炼过程中，这些技能引导专家监督，同时对统一学生不可用，鼓励其在推理时内化程序知识，而无显性技能。在八个多模态基准测试中，SCOPED持续优于骨干和混合能力强化学习，展示了技能解耦政策内专家提炼在整合独立获得能力方面的有效性。进一步分析揭示了联合能力优化过程中的干扰，并表明技能增强促进了更一致的专家轨迹，更加关注任务相关视觉证据，并更好地保留统一学生的专业技能。所有代码和数据均可在此 https 网址获取。

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

MMSearch-OPD：简单难学，适合长视野多模态搜索代理

• Authors: Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.22240
• Pdf link: https://arxiv.org/pdf/2610.22240
• Abstract Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.
• 中文摘要 多模态深度搜索正从短查询转向持续获取证据的长期调查。然而，训练此类代理仍受限于有限的复杂训练问题和不足的过程反馈。我们引入了MMSearch-OPD，结合可控数据细化与联合强化学习（RL）和策略提纯（OPD）。我们的Refine流程针对现有简单多模态问题推导出的推理链进行有针对性扩展。参数化转换控制搜索深度和跨模态证据需求，同时重复使用现有图像-问题对。在培训中，答案奖励和轻量级行为惩罚指导强化学习，而奖励差距则调节教师对不完美学生轨迹的代币级指导。蒸馏系数呈指数衰减，随着培训进行，教师依赖度下降。实验显示数据精细带来的益处和SFT后联合训练的进一步提升，表明其长视界多模态深度搜索能力更强。

Baselines and Score Centering in Policy Gradient Estimation

政策梯度估算中的基线与评分中心

• Authors: Shuo Han, Ziqing Hu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22263
• Pdf link: https://arxiv.org/pdf/2610.22263
• Abstract Reinforcement learning has become a widely used approach to post-training large language models. In a typical implementation, learning is off-policy: The policy generating the responses differs from the policy being trained because of training-inference mismatch or policy staleness in asynchronous training. This discrepancy is commonly addressed by importance sampling, whose ratio is usually regularized in practice through truncation, masking, or clipping. Under regularized importance sampling, the importance-weighted policy gradient is no longer invariant to constant shifts in the reward. This invariance is recovered by using the covariance between the reward and the importance-weighted score as the estimand, which coincides with the policy gradient when the ratio is not regularized. This note develops this covariance perspective for both sequence-level and token-level importance sampling. The covariance perspective reveals the roles of the group-mean baseline and score centering in policy gradient estimation. Both perform centering in covariance estimation, but they differ in the mean being subtracted: The group-mean baseline uses a sample mean, whereas score centering uses a population mean. The two lead to distinct unbiased covariance estimators with different variances. Whether both forms of centering should be used simultaneously depends on whether importance sampling is applied at the sequence level or at the token level. With sequence-level importance sampling, score centering adds no benefit once the group-mean baseline is used; with token-level importance sampling, using the group-mean baseline without score centering generally introduces bias when the importance ratio is regularized.
• 中文摘要 强化学习已成为大型语言模型后训练中广泛使用的一种方法。在典型的实现中，学习是非策略的：生成响应的策略与被训练的策略不同，原因是异步训练中存在训练-推理不匹配或策略陈旧。这种差异通常通过重要性抽样来解决，其比率在实践中通常通过截断、掩蔽或裁剪来正则化。在正则化重要性抽样下，重要性加权策略梯度不再对奖励的恒定变化保持不变。这种不变性通过使用奖励与重要性加权分数之间的协方差作为刺激数来恢复，该共方差与未正则化比值时的策略梯度一致。本注释对序列级和token级重要性抽样都提出了协方差视角。协方差视角揭示了组均基线和分数中心化在策略梯度估计中的作用。两者在协方差估计中都执行中心化，但它们在减去均值上有所不同：组均值基线使用样本均值，而分数中心化使用总体均值。两者产生不同的无偏协方差估计，但方差不同。是否应同时使用两种中心化方式取决于重要性抽样是在序列层面应用还是代币层面。在序列级重要性抽样中，一旦使用组均值基线，分数中心化不会带来额外好处;而在代币级重要性抽样中，使用组均值基线而不使用分数中心化，当重要性比正则化时通常会引入偏差。

SCOPE: Semantic Bounded Credit Optimization for Prompt Engineering in Text-to-image Generation

范围：文本到图像生成中提示工程的语义有界信用优化

• Authors: Yufei Jiao, Jinfa Huang, Zeyue Xue, Tianyu Xie, Yuexiao Ma, Rongrong Ji, Xiawu Zheng, Jiebo Luo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.22264
• Pdf link: https://arxiv.org/pdf/2610.22264
• Abstract Reinforcement learning (RL) enables language models to improve text-to-image (T2I) generation by rewriting prompts for a frozen image generator. Existing methods typically apply a single signal derived from the image to individual subword tokens, although objects, counts, attributes, and relations form coherent semantic spans. We propose SCOPE, an objective based on Group Relative Policy Optimization (GRPO) that aligns update weighting with this structure. The rewriter jointly generates an enhanced prompt and its semantic partition. SCOPE pools shifts in policy log probabilities within each span to derive a shared coefficient for its token losses while retaining a single advantage for each completion. It requires neither separate span rewards nor additional image renders. Across Qwen-Image, SD3, and FLUX.1-dev, SCOPE achieves average relative gains over vanilla GRPO of 9.48% on GeneVal and 7.62% on T2I-CompBench. Ablation studies show that both semantic boundaries and pooling within spans contribute to these gains, supporting semantic structure as a practical inductive bias for prompt engineering with RL.
• 中文摘要 强化学习（RL）使语言模型通过重写冻结图像生成器的提示，提升文本到图像（T2I）生成。现有方法通常将图像中单个信号应用于单个子词标记，尽管对象、计数、属性和关系构成连贯的语义张广。我们提出基于群相对策略优化（GRPO）的目标SCOPE，使更新权重与该结构保持一致。重写器共同生成增强版提示及其语义划分。SCOPE池在每个区间内调整策略日志概率，推导其令牌丢失的共享系数，同时为每次完成保留单一优势。它不需要单独的区间奖励或额外的图像渲染。在Qwen-Image、SD3和FLUX.1-dev中，SCOPE在GeneVal上相比原版GRPO实现了9.48%的平均相对提升，在T2I-CompBench上为7.62%。消融研究显示，语义边界和跨度内的池化都对这些提升有贡献，支持语义结构作为强化学习快速工程的实用归纳偏见。

Minimizing Age of Incorrect Information in Pull-Based Remote Estimation of a Markov Source

最小化基于拉取的马尔可夫源远程估计中错误信息的年龄

• Authors: Ismail Cosandal, Sennur Ulukus, Nail Akar
• Subjects: Subjects: Information Theory (cs.IT); Networking and Internet Architecture (cs.NI); Probability (math.PR)
• Arxiv link: https://arxiv.org/abs/2610.22265
• Pdf link: https://arxiv.org/pdf/2610.22265
• Abstract In this paper, we consider a discrete-time pull-based remote estimation system involving an information source and a remote monitor where the information source is a finite-state irreducible discrete-time Markov chain (DTMC) process. In this system, the remote monitor sends a pull request towards the source which responds by sampling its process so as to transmit the sampled values towards the monitor through a delay channel, encapsulated in information packets. We consider i) zero delay (ZD) and ii) unit delay (UD) channels, where the packet transmission times are zero and one slot, respectively. For remote estimation, we use the maximum a posteriori (MAP) estimator, and for quantifying its performance, we use the mean age of incorrect information (MAoII) which is a recently proposed freshness and mismatch metric that penalizes an incorrect estimation along with its duration. Our goal is to obtain pull policies so as to minimize MAoII under a long-run sampling rate constraint. For the ZD model, we propose a semi-Markov decision process (SMDP) formulation of the problem, which allows us to find the optimum observation-dependent pull policy. On the other hand, the UD channel prevents the monitor from possessing perfect information on the actual AoII process at any given time, and the SMDP formulation of the ZD channel cannot readily be used for the UD model. For the system with the UD channel, we first provide a numerical algorithm to calculate the MAoII and the average sampling rate for a given observation-dependent pull policy. For the optimization of the underlying remote estimation system, we use the joint distribution of the age and source processes to be obtained from the history of all observations, termed as belief. Subsequently, we propose a belief-dependent policy obtained via the belief-MDP formulation and a constrained deep reinforcement learning (CDRL) algorithm.
• 中文摘要 本文考虑了一个离散时间拉取的远程估计系统，涉及信息源和远程监控器，其中信息源是一个有限状态不可约离散时间马尔可夫链（DTMC）过程。在该系统中，远程监控器向源发送拉取请求，源通过采样过程将采样值通过延迟通道向监测器发送，该通道封装在信息包中。我们考虑i）零延迟（ZD）和ii）单位延迟（UD）通道，其中数据包传输时间分别为零和一个时隙。对于远程估计，我们使用最大后验估计器（MAP）;在量化其性能时，使用错误信息的平均年龄（MAoII），这是最近提出的一种新颖性和不匹配度量，错误估计会惩罚错误估计及其持续时间。我们的目标是在长期采样率约束下获得拉取策略，以最小化MAoII。对于ZD模型，我们提出了半马尔可夫决策过程（SMDP）问题的表述，使我们能够找到最优的观测依赖拉取策略。另一方面，UD通道阻止监测器在任何时刻拥有对实际AoII过程的完美信息，ZD通道的SMDP表述无法轻易用于UD模型。对于带有UD通道的系统，我们首先提供一个数值算法，用于计算给定观测依赖拉取策略的MAoII和平均采样率。对于底层远程估计系统的优化，我们使用从所有观测历史中获得的年龄过程和来源过程的联合分布，称为信念。随后，我们提出了通过信念-MDP表述和受限深度强化学习（CDRL）算法获得的信念依赖策略。

X-RIC: An Agentic RAN-Intelligent Controller Benchmark for Enhancing Reinforcement Learning

X-RIC：增强强化学习的智能RAN智能控制器基准测试

• Authors: Adhwaa Al-Chaab, Ayman Younis, Dario Pompili
• Subjects: Subjects: Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR)
• Arxiv link: https://arxiv.org/abs/2610.22269
• Pdf link: https://arxiv.org/pdf/2610.22269
• Abstract The O-RAN ALLIANCE introduces architectural flexibility through disaggregation and AI-driven control. While the RAN Intelligent Controller (RIC) enables centralized optimization, current AI approaches lack reasoning capabilities and explainability. We propose eXplainable RIC (X-RIC), a novel framework integrating agentic AI into O-RAN architectures. X-RIC combines reactive Reinforcement Learning (RL) with strategic agentic reasoning through: (i) automated Behavioral Cloning (BC) using domain heuristics for sample-efficient training, (ii) a Telecom Language Model (TLM), a transformer-based reasoning module providing contextual abstraction and interpretability, and (iii) closed-loop co-adaptation within the Near-RT RIC control loop enabling continuous policy refinement. To ensure deployment readiness, we introduce BARX (Benchmarking Agentic Reasoning for RAN), a domain-specific evaluation framework assessing agentic control across latency, compliance, and determinism metrics. We validate X-RIC on SLA-aware network slicing in simulation and O-RAN-compliant testbed deployments. X-RIC achieves 89% operational cost reduction while maintaining SLA compliance, converging 75% faster with 50% fewer training steps and 46% lower variance.
• 中文摘要 O-RAN联盟通过拆分和AI驱动控制引入了架构灵活性。虽然RAN智能控制器（RIC）实现了集中优化，但当前AI方法缺乏推理能力和可解释性。我们提出了可解释的RIC（X-RIC），这是一个将代理AI整合进O-RAN架构的新框架。X-RIC结合了反应强化学习（RL）与战略智能体推理，方式包括：（i）利用领域启发式的自动化行为克隆（BC）进行样本高效训练，（ii）电信语言模型（TLM），基于变换器的推理模块，提供上下文抽象和可解释性，以及（iii）近RT RIC控制循环内的闭环共适，实现持续策略的细化。为确保部署准备，我们引入了BARX（Benchmarking Agentic Reasoning for RAN），这是一个领域特定的评估框架，评估跨延迟、合规性和确定性指标的代理控制。我们在模拟和O-RAN合规测试平台部署中验证了X-RIC在SLA感知网络切片上的应用。X-RIC在保持SLA合规的同时实现了89%的运营成本降低，收敛速度加快75%，训练步骤减少50%，方差降低46%。

Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries

图书管理员代理：训练大型语言模型策划预见未来查询的长上下文工作区

• Authors: Anikait Singh, Teresa Zhang, Yoonho Lee, Roshen Sanjay Nair, Aviral Kumar, Chelsea Finn
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.22273
• Pdf link: https://arxiv.org/pdf/2610.22273
• Abstract Reasoning over long document contexts remains a challenge for large language models (LLMs). Standard retrieval methods are inherently reactive, bottlenecking agents with complex, cross-document reasoning at inference time, while precomputed structures like knowledge graphs often fall short because they rely on rigid heuristics. Instead, can we allow agents to perform anticipatory directed curation: learning to proactively reorganize unstructured corpora prior to inference to efficiently serve diverse future queries. We build on a key asymmetry in modern agentic harnesses (e.g., Claude Code): while LLM agents are adept at navigating existing filesystems to solve an immediate task, they are rarely trained to construct reusable workspaces for future use. To instantiate this capability, we introduce Librarian Agents, a framework in which a builder agent transforms raw documents into a hierarchical filesystem of synthetic files and reference links, which forms a workspace tailored for a downstream answering agent. We formulate workspace construction as a Markov decision process and optimize the builder policy via reinforcement learning to directly maximize downstream answering accuracy. Empirically, Librarian Agents outperforms the strongest retrieval and precomputation baselines by 33 percentage points on BrowseComp-Plus, amortizes upfront construction costs across repeated queries, and generalizes zero-shot to unseen domains on BRIGHT with a 31-point average accuracy improvement.
• 中文摘要 在大型语言模型（LLM）中，在长文档上下文中的推理仍然是个挑战。标准检索方法本质上是反应性的，导致推理时的跨文档推理复杂，导致代理陷入瓶颈，而预计算的结构如知识图谱往往因依赖僵化启发式而不足。相反，我们能否允许代理执行预见定向策展：学会在推理前主动重组非结构语料库，以高效服务多样化的未来查询。我们基于现代代理工具（如Claude代码）的关键不对称性：虽然LLM代理擅长在现有文件系统中导航以解决即时任务，但很少接受训练去构建可重用的工作区以备未来使用。为实现此功能，我们引入了Librarian Agents框架，构建代理将原始文档转换为由合成文件和参考链接组成的层级文件系统，形成为下游答复代理量身定制的工作区。我们将工作区构建构建为马尔可夫决策过程，并通过强化学习优化构建策略，直接最大化下游答复的准确性。实证上，Librarian Agents在BrowseComp-Plus上比最强的检索和预计算基线高出33个百分点，将前期构建成本分摊到重复查询，并将零样本推广至BRIGHT上未见域，平均准确率提升31分。

Routing Policy Optimization for Online Diffusion Reinforcement Learning

在线扩散强化学习的路由策略优化

• Authors: Qiuhong Shen, Yiding Yang, Yuxin Zhang, Xinwei Huang, Minxuan Lin, Bo Liu, Hao Kang, Haotian Yang, Yizhi Wang, Chongyang Ma, Xinchao Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.22282
• Pdf link: https://arxiv.org/pdf/2610.22282
• Abstract Reinforcement learning can align visual generators with reward signals beyond their pretraining objectives, but existing score-function methods typically introduce stochasticity by perturbing the reverse denoising trajectory, leading to costly rollout-based optimization. We instead ask whether stochastic exploration can be introduced inside the generator while preserving its original sampler. Routing Policy Optimization (RPO) answers this question by turning attention routing into a stochastic exploration action. At a few selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention. The sampled route perturbs the velocity of that transition, and a Gaussian likelihood around the dense velocity scores the perturbation with a score-function objective. RPO further uses a controlled two-rollout estimator: two rollouts share the same prompt, initial latent, routed denoising levels, and dense anchor layers, differing only in their sampled attention routes. Their reward difference provides an advantage signal for the routed transitions. Because the likelihood differentiates the dense velocity itself, the learned updates remain effective after routing is removed. At inference, RPO restores dense attention and retains the original deterministic sampler. Experiments on text-to-image and text-to-video generation show that RPO improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods. Please see our project page at this https URL for visual results.
• 中文摘要 强化学习可以将视觉生成器与超出预训练目标的奖励信号对齐，但现有的评分函数方法通常通过扰动逆去噪轨迹引入随机性，导致成本高昂的基于展开的优化。我们转而探讨是否可以在生成器内部引入随机探索，同时保留其原始采样器。路由策略优化（RPO）通过将注意力路由转化为随机探索动作来回答这个问题。在少数选定的去噪级别，每个查询块保留自己的块，并根据生成器的原生注意力质量或保持密集注意力的比例采样更多关键块。采样路径扰动该转移的速度，围绕稠密速度的高斯似然会用评分函数目标对扰动进行评分。RPO进一步使用受控的双展开估计器：两个部署共享相同的提示、初始潜在、路由去噪水平和密集锚点层，仅在采样注意力路径上有所不同。它们的奖励差为路由转移提供了优势信号。由于概率本身会区分稠密速度，学习到的更新在移除路由后依然有效。推断时，RPO恢复了密集注意力并保留原始确定性采样器。文本到图像和文本到视频生成的实验表明，RPO在提升生成质量的同时，训练效率几乎是Flow-GRPO风格方法的4倍。请参阅我们的项目页面，链接链接获取可视化结果。

Learning Rational Strategies for Temporal Objectives in Multi-Agent Systems

学习多智能体系统中时间目标的理性策略

• Authors: Omar Adalat, Edwin Hamel-De le Court, Francesco Belardinelli
• Subjects: Subjects: Machine Learning (cs.LG); Logic in Computer Science (cs.LO); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.22287
• Pdf link: https://arxiv.org/pdf/2610.22287
• Abstract We study rational synthesis for stochastic multi-agent systems with unknown dynamics, where each agent is equipped with its own probabilistic temporal objective specified using Probabilistic Linear Temporal Logic on Finite Traces ($\mathsf{PLTL}_f}$). We impose solution concepts (e.g., Nash equilibrium, subgame-perfect equilibrium) as constraints on the joint strategy profile, using payoffs derived from each agent's probability of satisfying its temporal objective. In particular, we formulate per-agent rational synthesis as finding a strategy profile -- if one exists -- such that (i) the designated agent's probabilistic temporal constraint is satisfied, and (ii) the profile represents a stable solution under a chosen solution concept. We give conditional convergence guarantees for tabular Nash-Q learning, certify satisfaction with simultaneous finite-sample confidence bounds, and assess strategic stability with approximate best-response checking. Our approach operates in a stochastic \emph{Multi-Agent Reinforcement Learning} (MARL) environment, where model-free learning algorithms attempt to attain extremal satisfaction probabilities for temporal objectives automatically compiled into reward automata. Our approach is empirically validated using 8 general-sum concurrent stochastic games.
• 中文摘要 我们研究随机多智能体系统中未知动态的理性综合，其中每个智能体配备其自己的概率时间目标，使用有限迹上的概率线性时间逻辑（$\mathsf{PLTL}_f}$）指定。我们将解决方案概念（例如纳什均衡、亚博弈完美均衡）作为联合策略组合的约束，使用每个代理满足其时间目标的概率得出的收益。特别地，我们将每个代理理性综合表述为找到一个策略组合——如果存在的话——使得（i）指定代理的概率时间约束得到满足，且（ii）该组合在所选解概念下表示稳定解。我们为表格纳什-Q学习提供条件收敛保证，认证同时有限样本置信区间的满意度，并通过近似最佳响应检查评估战略稳定性。我们的方法运行在随机\emph{多代理强化学习}（MARL）环境中，无模型学习算法尝试获得时间目标的极端满意概率，这些目标自动编译成奖励自动机。我们的方法通过8个通用和并发随机博弈进行了实证验证。

EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search

EliteReplay：通过质量多样性搜索在强化学习后培训中保留多样化解决方案

• Authors: Hanane Nour Moussa, Doeun Lee, Sachin Kumar
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22290
• Pdf link: https://arxiv.org/pdf/2610.22290
• Abstract Reinforcement learning has become a common post-training technique to improve the accuracy of LLMs on verifiable tasks. However, such training can narrow the set of solutions the model can produce. Several approaches have attempted to address this behavior-sharpening effect, but they often optimize metrics that do not reflect the diversity of correct solutions. Ideally, we want a model that can produce multiple good solutions, not only a single reliable one. We propose EliteReplay, a post-training recipe that pairs GRPO with an "elite" archive built by quality-diversity search. Before training, we use the frozen base model to collect solutions that span distinct approaches; during training, GRPO is combined with a replay loss on this archive. The result is a post-trained model that improves accuracy while preserving more of the base model's solution diversity. On multiple text-to-SQL tasks, EliteReplay achieves a higher pass@128 than all baselines, producing up to 77% more distinct correct strategies than the strongest baselines. It preserves most of the base model's correct strategies, outperforming existing methods in strategy retention. This diversity translates into practical benefits on ambiguous questions, where EliteReplay covers both interpretations nearly twice as often as the strongest baseline on AmbiQT.
• 中文摘要 强化学习已成为一种常见的训练后技术，用于提高LLM在可验证任务上的准确性。然而，这种训练可能会缩小模型能产出的解集。有多种方法试图解决这种行为锐化效应，但它们往往优化的指标并未反映正确解的多样性。理想情况下，我们希望模型能够产生多个优质解，而不仅仅是一个可靠的解。我们提出了EliteReplay，这是一种训练后方案，将GRPO与通过质量多样性搜索构建的“精英”档案结合起来。在训练前，我们使用冻结的基础模型收集跨越不同方法的解;在训练过程中，GRPO与该档案中的重放丢失结合起来。结果是一个后训练模型，既提高了准确性，又保留了基础模型的解多样性。在多项文本转SQL任务中，EliteReplay实现的pass@128高于所有基线，产生多达77%的明显正确策略，比最强基线多出77%。它保留了大部分基础模型的正确策略，在策略保持率上优于现有方法。这种多样性转化为在模糊问题上的实际优势，EliteReplay涵盖两种解释的频率几乎是AmbiQT上最强基线的两倍。

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

迈向持续学习：通过亚空间调谐缓解遗忘

• Authors: Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon, Devvrit Khatri
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22293
• Pdf link: https://arxiv.org/pdf/2610.22293
• Abstract Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99\% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior strengthens with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing $2$--$4\times$ less KL drift than the baseline recipes. Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models.
• 中文摘要 大型语言模型（LLM）通过微调获得新技能，但这可能会破坏它们已有的能力：灾难性遗忘仍是LLM持续学习的核心障碍。近期研究将遗忘与权重微调移动的距离及方向联系起来。策略上方法如强化学习和策略中蒸馏避免更新权重矩阵的主要奇异方向，因此遗忘更少。相比之下，SFT集中更新方向，容易出现灾难性遗忘。本研究介绍了子空间调优（SST），限制每次权重更新到底部奇异子空间，消除主要方向的变化。SST与优化器无关，能很好地组合包括SFT和自蒸馏在内的微调方案。SST结合了严格的子空间保护、密集梯度估计和简单的刷新机制，使学习比以往的正交微调方法更强。通过这些机制，保护多达99%的权重子空间显著提升了保留率，同时保持了强劲的目标任务性能。这种行为随着模型和数据规模的增加而增强，几乎消除了大模型区间的遗忘现象。SST还使模型接近其基础分布，诱导的基层级漂移比基线配方少2美元至4倍加倍。总体而言，SST推动了学习——忘记帕累托边界的进步，使其成为微调预训练语言模型的实用默认。

Learning to Steer LLM Agents and Multi-Agent Systems

学习如何引导LLM代理和多智能体系统

• Authors: Leyang Shen, Yaxin Gong, Yang Zhang, Chenxiao Fan, Jiakai Wang, Dong Wang, Yang Liu, Fuli Feng, Chun Kai Ling, Tat-Seng Chua
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.22316
• Pdf link: https://arxiv.org/pdf/2610.22316
• Abstract Large language model (LLM)-based agents and multi-agent systems increasingly tackle complex tasks through long-horizon tool use and inter-agent collaboration. However, these systems are prone to deviating from the overall goal as execution unfolds, which wastes execution resources and leads to undesired outcomes, a failure termed goal drift. In this work, we propose a steering agent that supervises execution on the fly and steers the system back on track when drift occurs. The steering agent observes the execution context, deciding when to intervene with a low-cost fast path and how to intervene with a high-cost slow path. To unlock its steering intelligence, the ability to assess goal drift and generate effective interventions, we jointly train the two paths with reinforcement learning, combining a pairwise ranking objective with group-relative policy optimization (GRPO). Experiments on information-seeking search tasks show that the steering agent provides effective supervision for both single-agent and multi-agent systems, improving their performance and efficiency. The learned steering intelligence generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency without retraining.
• 中文摘要 基于大型语言模型（LLM）的智能体和多智能体系统越来越多地通过长期工具和代理间协作来处理复杂任务。然而，这些系统在执行过程中容易偏离整体目标，浪费执行资源并导致不良结果，这种失败称为目标漂移。本研究提出一种引导代理，能够实时监督执行，并在漂移发生时引导系统回归正轨。引导代理观察执行上下文，决定何时介入低成本快速路径，如何干预高成本慢路径。为解锁其引导智能、评估目标漂移和生成有效干预的能力，我们联合训练这两条路径，结合成对排名目标与群体相对策略优化（GRPO）。信息寻寻搜索任务的实验表明，引导代理能有效监督单代理和多代理系统，提升其性能和效率。学习到的引导智能推广到域外任务，转移至未见执行者，并支持无需重新训练即可控制的干预频率。

Training Reasoning Models to Generate Diverse Scientific Ideas

训练推理模型以产生多样化的科学思想

• Authors: Manya Wadhwa, Pranav Narayanan Venkit, Hiroaki Hayashi, Yu Li, Tuhin Chakrabarty, Chien-Sheng Wu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.22324
• Pdf link: https://arxiv.org/pdf/2610.22324
• Abstract One of the first steps of a research pipeline is generating ideas. Large Language Models (LLMs) are increasingly used to automate this step, but most of this work optimizes the quality of individual ideas, while overlooking diversity. In our work, we study document-grounded ideation, focusing on increasing the diversity of LLM generated ideas and ensuring they are high quality. Given prior papers, a model must propose a set of ideas that are each high-quality and clearly distinct from one another, while also being grounded in the input documents. To optimize for this, we propose training with a set-level reinforcement learning objective. In our setup, in a single generation, the model writes K structured idea cards grounded in the source papers, and the set is scored jointly, where each idea's quality is weighted by its distance from the ideas written before it. This teaches the model to steer each new idea away from its predecessors. We show that our trained 8B model achieves higher creative utility (a measure combining quality and diversity) than Claude Opus 4.8 and GPT-6 Astra on our contamination-free benchmark. In a pilot human study with four experts, our sets are judged as promising as Claude Opus 4.8's and contain marginally more unique ideas. When a coding agent runs proof-of-concept experiments on the ideas, our model's success@k is comparable to Claude Opus 4.8's, so we see that the added diversity does not cost any executability.
• 中文摘要 研究流程的第一步之一是生成想法。大型语言模型（LLM）越来越多地用于自动化这一步骤，但大多数工作都是优化单个想法的质量，同时忽略多样性。在我们的工作中，我们研究基于文档的创意，重点是提升LLM生成的想法的多样性并确保其高质量。鉴于已有论文，模型必须提出一组高质量且彼此明显区分的观点，同时基于输入文档。为此优化，我们提出以集合级强化学习目标进行训练。在我们的设置中，模型在单一世代内写出K张基于原始论文的结构化想法卡，并对集合进行联合评分，每个想法的质量根据其与前述观点的距离加权。这教会模型引导每个新想法远离前辈。我们证明，训练好的8B模型在无污染基准测试中，比Claude Opus 4.8和GPT-6 Astra在创造效用（质量与多样性结合的指标）上更高。在一项由四位专家参与的初步人体研究中，我们的数据集被评为与Claude Opus 4.8一样有前景，且包含略多的独特想法。当编码代理对这些想法进行概念验证实验时，我们模型的success@k与Claude Opus 4.8相当，因此我们看到增加的多样性不会影响可执行性。

KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents

KnowGPO：搜索代理的知识分组策略优化

• Authors: Siqi Chen, Zhengxi Lu, Yuchen Yan, Xingyu Wu, Qipeng Chen, Xin Zhang, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Weiming Lu, Jun Xiao, Yongliang Shen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22331
• Pdf link: https://arxiv.org/pdf/2610.22331
• Abstract Deep search agents solve knowledge-intensive tasks through long-horizon, multi-turn interactions, where assigning credit to individual search decisions is fundamentally difficult. Outcome-based reinforcement learning provides only a single terminal reward shared across an entire trajectory. Recent process-supervision methods aim to assign credit at finer granularity, but they typically anchor credit to either surface-level trajectory states or model-internal signals, both of which become unreliable in deep search where observations are heterogeneous and retrieved contexts are long and noisy. We propose KnowGPO, a reinforcement learning framework that aligns trajectories by the task-relevant knowledge they have acquired rather than by their surface observations. For each query, KnowGPO decomposes the information required to derive the answer into atomic knowledge conditions, and represents the state of a trajectory at each step by the subset of conditions it has satisfied. Trajectories sharing the same knowledge state are grouped for within-group comparison, and the incremental knowledge gain between consecutive steps serves as an environment-grounded process reward. Experiments on Xbench, BrowseComp-ZH, and GAIA show KnowGPO consistently outperforms outcome-only (GRPO) and process-supervision baselines (GiGPO, IGPO) by 3.8 to 4.2 points, with our 8B model rivaling or exceeding most 32B open-source systems.
• 中文摘要 深度搜索代理通过长视野、多回合互动解决知识密集型任务，在这种机制下，将功劳归属于单个搜索决策本质上困难。基于结果的强化学习仅提供一个贯穿整个轨迹的单一终端奖励。近期的过程监督方法旨在更细粒地分配功劳，但通常将功劳锚定于表层轨迹状态或模型内部信号，而在深度搜索中，观测值异质且检索的上下文冗长且噪声大，这两者都变得不可靠。我们提出了KnowGPO，一种强化学习框架，通过任务相关知识而非表面观测来对齐轨迹。对于每个查询，KnowGPO将推导答案所需的信息分解为原子知识条件，并根据路径满足的条件子集表示每步轨迹的状态。共享相同知识状态的轨迹被分组进行组内比较，连续步骤间的增量知识增长作为基于环境的过程奖励。Xbench、BrowseComp-ZH 和 GAIA 上的实验显示，KnowGPO 在仅结果基线（GRPO）和过程监督基线（GiGPO、IGPO）上持续优于 3.8 至 4.2 分，我们的 8B 模型可与或超过大多数 32B 开源系统。

Robust Reinforcement Learning for Dynamic Treatment Regimes under Model and Measurement Uncertainty

在模型和测量不确定性下动态处理方案下的稳健强化学习

• Authors: Chengze Xie, Xuan Wang, Ying Ding, Hairong Wang, Yan Li
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22336
• Pdf link: https://arxiv.org/pdf/2610.22336
• Abstract We develop a robust reinforcement learning (RL) framework for learning adaptive treatment policies that generalize across patients and remain reliable under errors in the measured biomarker and shifts in the disease dynamics. By modeling the patient as a hidden context, we address generalization and robustness in a unified manner through robust contextual Markov decision processes. Applying our framework to adaptive therapy for metastatic prostate cancer in a simulator calibrated to clinical data, we obtain generalizable and robust policies that substantially outperform the existing adaptive therapy method. In particular, we attain robustness with almost no loss of performance when the biomarker is accurate and the disease evolves as anticipated, and find that the resulting policy Pareto-dominates the existing method across a diverse array of scenarios. Our findings suggest that robust RL holds great promise as a principled methodology for learning reliable adaptive treatment policies.
• 中文摘要 我们开发了一个稳健的强化学习（RL）框架，用于学习适应性治疗策略，这些策略在患者间推广，并在测量的生物标志物错误和疾病动态变化下保持可靠性。通过将患者建模为隐藏情境，我们通过稳健的情境马尔可夫决策过程统一处理泛化和稳健性。通过将我们的框架应用于基于临床数据的模拟器中针对转移性前列腺癌的自适应性治疗，我们获得了可广泛且稳健的策略，其表现显著优于现有的适应性治疗方法。特别是，当生物标志物准确且疾病按预期演变时，我们几乎不损失性能，并发现由此产生的策略在多种情境中比现有方法更为强大。我们的发现表明，稳健强化学习作为一种有原则的学习可靠适应性治疗策略的方法论，具有巨大潜力。

Dexterity from Supervised Play

监督游戏中的灵巧度

• Authors: Botao He, Yiming Dou, Yiannis Aloimonos, Jitendra Malik, Akash Sharma, Ruoshi Liu
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22340
• Pdf link: https://arxiv.org/pdf/2610.22340
• Abstract How can robots autonomously collect the contact-rich experience needed to learn dexterous manipulation? We present DSP: Dexterity from supervised play, a framework in which a robot learns a play policy from minutes of human play through behavior cloning, then deploys it autonomously to collect hours of interaction data to train a world model. During inference, the same play policy then guides model-predictive control (MPC) by proposing candidate action sequences for evaluation under the learned world model. To accelerate inference, we also introduce a method for distilling a closed-loop, goal-conditioned policy through self-play within the learned world model. We evaluate this framework on underactuated pushing and any-to-any in-hand pose reorientation with a multifingered hand, in simulation and real world. In real world experiment, our method outperforms hindsight behavior cloning by 69.6% in success rate and achieves comparable success to reinforcement learning with 490 times less interaction data. These results demonstrate that supervised play can autonomously collect interaction data, guide model-based planning, and enable policy learning without any task-specific demonstrations or sim-to-real transfer.
• 中文摘要 机器人如何自主收集学习灵巧操作所需的丰富接触经验？我们提出了DSP：由监督游戏中的灵巧度，该框架中机器人通过行为克隆从几分钟的人类游戏中学习游戏策略，然后自主部署以收集数小时的交互数据以训练世界模型。在推理过程中，同一游戏策略通过提出候选动作序列以供学习世界模型评估，指导模型预测控制（MPC）。为加速推理，我们还引入了一种通过学习世界模型中自我游戏提炼闭环、目标条件策略的方法。我们在多指手的欠驱动推力和任意手握姿态重新定向上，在模拟和现实世界中评估该框架。在现实实验中，我们的方法成功率比事后诸葛亮行为克隆高出69.6%，且在互动数据减少490倍的情况下，成功率与强化学习相当。这些结果表明，监督游戏能够自主收集互动数据，指导基于模型的规划，并实现策略学习，无需任何任务特定演示或模拟到现实的转移。

OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modeling

OmniVerifier-C1：用于规则控制生成世界建模的分解分级验证器

• Authors: Bowei Liu, Xinchen Zhang, Xuhuan Li, Tianlin Zhang, Kaian Jiang, Jingwei Liu, Congyang Zhao, Yifei Qian, Junhong Liu, Zhiheng Li, Yujiu Yang, Yang Cai, Ling Yang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.22355
• Pdf link: https://arxiv.org/pdf/2610.22355
• Abstract Generative visual verifiers play a crucial role in advancing multimodal intelligence. To enhance the robustness of visual verifiers in complex multimodal scenarios involving compositional requirements, we move beyond conventional binary verifiers that produce only True/False discrete judgments and introduce a verifier that automatically decomposes complex prompts or environmental rules into fine-grained components and assigns pointwise scores to provide a continuous and more informative critique signal. Our experiments further demonstrate that these structured and interpretable outputs improve the verifier's generalization to out-of-domain multimodal critic tasks. To train this verifier, we introduce a Decoupled RL training strategy that jointly optimizes sub-question-level binary judgment and global alignment scoring, using precise local feedback to develop fine-grained verification capabilities that provide a foundation for reliable continuous scoring at the global level. Under a general reinforcement learning framework, we first empirically demonstrate that the proposed pointwise graded verifier significantly outperforms binary verifiers in traditional text-to-image scenarios. We then extend our investigation to rule-governed generative world modeling where generative models are leveraged for world simulation and world reasoning under explicit rules. From a theoretical standpoint, we further show that a pairwise verification paradigm yields more accurate critiques than the pointwise formulation when ground-truth images are available, with empirical results on diverse structured world-modeling tasks such as Maze and Sudoku further validating the effectiveness of our approach. Overall, our findings suggest a key insight: the transition from generative models to structured world-modeling agents critically hinges on the availability of accurate pairwise graded visual verifiers.
• 中文摘要 生成式视觉验证器在推动多模态智能发展中发挥着关键作用。为了增强视觉验证器在涉及组合要求的复杂多模态场景中的鲁棒性，我们超越了传统的仅产生真/假离散判断的二元验证器，引入了一种自动将复杂提示或环境规则分解为细粒度组件，并分配点数以提供连续且更具信息量的批评信号的验证器。我们的实验进一步表明，这些结构化且可解释的输出提升了验证器对域外多模态批评任务的推广能力。为训练该验证器，我们引入了一种解耦强化学习训练策略，联合优化子问题级二元判断和全局对齐评分，利用精确的局部反馈开发细粒度验证能力，为全局层面可靠的连续评分奠定基础。在通用强化学习框架下，我们首先实证证明，所提出的点分级验证器在传统文本到图像场景中显著优于二元验证器。随后，我们将研究扩展到规则支配的生成世界建模，在明确规则下利用生成模型进行世界模拟和世界推理。从理论角度看，我们进一步表明，当有真实图像时，成对验证范式比点对化表述产生更准确的批评，且在迷宫和数独等多种结构化世界建模任务上的实证结果进一步验证了我们方法的有效性。总体而言，我们的发现提出了一个关键见解：从生成模型向结构化世界建模代理的转变，关键在于是否拥有准确的成对分级视觉验证器。

Value as Memory: Online Value-Guided Post-training for General Robot Policies

价值作为记忆：通用机器人政策在线价值指导后培训

• Authors: Wenye Yu, Yijin Chen, Kaiyi Xiong, Wenbo Han, Wendi Chen, Yang Jin, Han Xue, Jun Lv, Cewu Lu, Chuan Wen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.22368
• Pdf link: https://arxiv.org/pdf/2610.22368
• Abstract Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, while offline post-training methods delay feedback from deployment experience. We present Value as Memory (VaM), a framework that addresses these challenges through value caching and online real-world reinforcement learning (RL). Value caching conditions actions on recent value predictions as task-progress memory, while online updates enable timely policy improvement by combining offline demonstrations, off-policy interventions, and on-policy rollouts for sample-efficient adaptation. As a result, VaM transitions from offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks involving deformable objects and contact-rich manipulation, VaM yields a mean absolute success-rate gain of $29.4\%$ with value caching, while achieving a $32.8\%$ gain with online autonomous learning over matched offline training. The evaluated online schedule performs $8\times$ as many policy iterations under the same gradient-step budget, with more than $2\times$ lower recorded time per gradient step with concurrent data collection and training.
• 中文摘要 通用机器人策略的现实后训练面临两个挑战：部分可观察性使动作选择变得模糊，而离线训练后方法则延迟部署经验反馈。我们提出了价值即记忆（VaM）框架，通过价值缓存和在线现实强化学习（RL）来应对这些挑战。价值缓存将近期价值预测的动作视为任务进度记忆，而在线更新通过结合离线演示、非策略干预和策略内推广实现及时策略改进，实现样本高效适应。因此，VaM从离线强化学习热身过渡到在线自主学习，奖励稀疏。在涉及可变形物体和丰富接触操作的长期任务中，VaM在价值缓存下平均绝对成功率提升29.4美元，在线自主学习相比匹配离线训练提升32.8%美元。评估的在线计划在同一梯度阶梯预算下执行了8至1倍的策略迭代次数，且在并发数据收集和培训时，每个梯度步骤的记录时间比2倍倍以上。

Keyword: diffusion policy

There is no result