生成时间: 2026-11-11 00:15:16 (UTC+8); Arxiv 发布时间: 2026-11-10 20:00 EST (2026-11-11 09:00 UTC+8)

今天共有 88 篇相关文章

Keyword: reinforcement learning

Grounding Insight: An Artificial Affect Signal for Human-Centric Explainability of RL Agents

扎根洞察：一种以人为本的强化学习代理解释的人工情感信号

• Authors: Bernhard Hilpert, Kim Baraka, Joost Broekens
• Subjects: Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.09922
• Pdf link: https://arxiv.org/pdf/2611.09922
• Abstract In Human-Interactive Robot Learning (HIRL), human teachers assess learning progress by observing agent behavior. However, Reinforcement Learning (RL) agents often fail to express learning progress in human-interpretable ways, leading to suboptimal or counterproductive teaching. This work proposes "Insight": a human-interpretable affective signal grounded in the RL process. It is specifically designed to address the "Absence of learning" human teacher mental model misalignment - the misconception that a failing or exploring agent is broken, underpowered or not learning at all - by signaling an agent's "Aha!" Insight moment. The signal is derived from the absolute Temporal Difference (TD) error making it easily implementable in other RL agents. To ground this signal, five design criteria were formulated: Relevance, Interpretability, Faithfulness, Scalability, Generalizability. The structural construct validity was assessed through simulation, examining micro and macro dynamics and event-related sensitivity. Micro dynamics reveal that Insight captures meaningful learning events at the appropriate time and magnitude. Macro dynamics demonstrate that the signal follows the agent's learning phases and reflects overall progress. Event-related sensitivity shows, the signal distinguishes the saliency of environment interactions, such as positive rewards versus "trauma-like" negative events. These results demonstrate that technical RL dynamics can be effectively packaged into human-like affective substrates to close the teacher-learner loop.
• 中文摘要 在人机交互机器人学习（HIRL）中，人类教师通过观察代理的行为来评估学习进展。然而，强化学习（RL）代理常常无法以人类可理解的方式表达学习进展，导致教学效果不佳或适得其反。这项工作提出了“洞察”：一种基于强化学习过程的人类可解读的情感信号。它专门设计用于解决“缺乏学习”的人类教师心理模型错位——即误解一个失败或探索中的代理存在缺陷、能力不足或根本无法学习——通过传递代理的“啊哈！”信号。洞察时刻。该信号源自绝对时间差（TD）错误，使其易于在其他强化学习代理中实现。为使该信号有据，制定了五个设计标准：相关性、可解释性、忠实性、可扩展性、可推广性。结构构念的效度通过模拟评估，考察了微观和宏观动态以及事件相关敏感性。微观动态表明，Insight能够在适当的时间和幅度捕捉有意义的学习事件。宏观动态表明信号遵循代理的学习阶段，并反映整体进展。事件相关敏感性表明，该信号区分了环境交互的显著性，如积极奖励与“创伤类”负面事件。这些结果表明，技术强化学习动态可以有效地包装成类人情感基底，以闭合师生循环。

Preference-Augmented Distributional Reinforcement Learning for Risk-Sensitive Navigation

偏好增强分布强化学习用于风险敏感导航

• Authors: Zhaofan Zhang, Yue Chang, Rufeng Chen, Hechang Chen, Sihong Xie, Hui Xiong
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.09932
• Pdf link: https://arxiv.org/pdf/2611.09932
• Abstract Ensuring safe and reliable navigation under diverse sources of uncertainty remains a central challenge for autonomous systems. Within distributional reinforcement learning (DistRL), Implicit Quantile Networks (IQN) have attracted interest for navigation under uncertainty because they enable risk adaptation through flexible quantile sampling. Existing approaches, however, typically use a fixed or randomly sampled Conditional Value-at-Risk (CVaR) level during training and adjust the risk level during deployment using hand-tuned hyperparameters or external sensor signals. This scalar-CVaR formulation confines learning to a narrow family of tail-risk preferences, creating a mismatch between training-time preference exposure and deployment-time risk adjustment. We instead represent variations in risk preference as shifts in quantile-weight distributions and expose the agent to diverse preferences through preference-guided exploration. At deployment, we optimize against the worst-case preference distribution to derive a conservative policy that accounts for multiple risk profiles. This reduces hyperparameter tuning and reliance on limited sensor signals under partial observability. Experiments across simulated environments and vehicle platforms show that our method reduces collisions while improving task success.
• 中文摘要 确保在多样不确定性来源下安全可靠的导航仍是自主系统面临的核心挑战。在分布强化学习（DistRL）中，隐式分位数网络（IQN）因通过灵活分位采样实现风险适应而受到不确定性导航的关注。然而，现有方法通常在训练时使用固定或随机抽样的风险条件值（CVaR）水平，并在部署时使用手工调优的超参数或外部传感器信号调整风险水平。这种标量CVaR表述将学习限制在狭窄的尾部风险偏好族内，导致训练时间偏好暴露与部署时间风险调整之间存在不匹配。我们将风险偏好的变异表示为分位数权重分布的变动，并通过偏好引导探索使智能体接触到多样化偏好。部署时，我们针对最坏情况偏好分布进行优化，推导出考虑多种风险特征的保守策略。这减少了超参数调优和对部分可观测性下有限传感器信号的依赖。跨模拟环境和车辆平台的实验表明，我们的方法减少碰撞，同时提升任务成功率。

Efficient Learning of Differentiable MPC Policies on the GPU via Smoothed ADMM and First-Order Reinforcement Learning

通过平滑ADMM和一阶强化学习，高效学习GPU上的可微MPC策略

• Authors: Jianghan Zhang, Roland Schwan, Gabriel Bravo-Palacios, John Zhang, Thomas Lew, Brian Plancher
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.09937
• Pdf link: https://arxiv.org/pdf/2611.09937
• Abstract Differentiable model predictive control (MPC) enables structured, constraint-aware policies to be optimized end-to-end. However, learning MPC policies faces two bottlenecks: (1) nonsmooth sensitivities at active-set transitions and (2) expensive policy evaluation, since each requires an optimization solve. We first introduce SmoothedTurboMPC, a GPU-parallel differentiable MPC solver combining sequential quadratic programming (SQP) with a log-barrier-smoothed alternating direction method of multipliers (ADMM) inner solver. Its design enables smooth gradients, exact sensitivities at convergence, and efficient execution on GPUs. Second, to efficiently train MPC policies, we pair our solver with first-order reinforcement learning (FORL) using Short-Horizon Actor-Critic (SHAC), which reuses rollout MPC solutions instead of repeatedly re-evaluating them. Comparisons with state of the art solvers show that the approach achieves up to $4.4\times$ higher end-to-end training throughput. On a vehicle racing task, our method achieves higher returns than MPC PPO with a $46\times$ speedup, training on 3.3 million transitions in only 7 minutes. Finally, we use the framework to train a more reliable MPC racing policy via RL with domain randomization, and validate it on a full-scale race car. We integrate our advances into the open-source TurboMPC solver.
• 中文摘要 可微模型预测控制（MPC）使结构化、约束感知的策略能够端到端得到优化。然而，学习MPC策略面临两个瓶颈：（1）主动集转移时的非光滑敏感性，以及（2）昂贵的策略评估，因为每一项都需要优化求解。我们首先介绍SmoothedTurboMPC，这是一款GPU并行可微MPC求解器，结合了顺序二次规划（SQP）和对数障碍平滑的交替方向乘法（ADMM）内部求解器。其设计支持平滑梯度、收敛时的精确敏感度以及GPU上的高效执行。其次，为了高效训练MPC策略，我们将求解器与使用短视角演员批判（SHAC）的一阶强化学习（FORL）结合，该方法重用了rollout的MPC解决方案，而非反复重新评估。与最先进的求解器比较显示，该方法的端到端训练吞吐量可提升高达4.4美元/倍。在车辆赛车任务中，我们的方法比MPC PPO获得更高回报，加速46美元/倍美元，仅需7分钟训练330万次转换。最后，我们利用该框架通过带有域随机化的强化训练更可靠的MPC赛车策略，并在全尺寸赛车上验证。我们将进展集成到开源的TurboMPC求解器中。

WorldMind: Test-Time Scaling for Visual Spatial Reasoning through Latent World Simulation

WorldMind：通过潜在世界模拟进行视觉空间推理的测试时间尺度

• Authors: Haoming Wang, Yongjian Fu, Ju Ren, Wei Gao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10151
• Pdf link: https://arxiv.org/pdf/2611.10151
• Abstract Vision-Language Models (VLMs) have continuous deficiency in spatial reasoning tasks that require \emph{mental spatial transformation}, such as reasoning under hypothetical changes in viewpoint or position, known as perspective-taking tasks. Existing approaches either scale linguistic reasoning or augment VLMs with static geometric representations, but neither directly enables active transformation of the represented scene. World models offer this capability, yet existing systems typically communicate through synthesized RGB images, introducing a latent-to-pixel-to-latent bottleneck. We introduce \emph{WorldMind}, a framework for test-time scaling of spatial reasoning through latent world simulation. Given a query and observation, the VLM selects useful spatial transformations, while a world model executes them and exposes aligned intermediate latent features directly to the VLM. The resulting latent evidence is incorporated into subsequent reasoning steps, enabling iterative spatial simulation without repeatedly decoding imagined states into pixels. To optimize this interaction, WorldMind further employs process-verifiable reinforcement learning that rewards reasoning progress only when the associated action and latent transition are valid. WorldMind thus scales spatial reasoning by allocating additional inference-time computation to active world simulation rather than generating more language or pixels. Across VSI-Bench, MMSI-Bench, and MindCube-Tiny, WorldMind achieves an average improvement of 16.0% over the base VLM and 8.3% over the strongest RGB-based world simulation.
• 中文摘要 视觉语言模型（VLMs）在需要 \emph{心理空间转换}的空间推理任务中存在持续不足，例如在假设视角或位置变化下进行推理，称为视角取向任务。现有方法要么扩展语言推理，要么通过静态几何表示增强 VLM，但都无法直接实现场景的主动变换。世界模型具备此功能，但现有系统通常通过合成的 RGB 图像进行通信，导致潜在到像素再到潜在的瓶颈。我们引入 \emph{WorldMind}，这是一个通过潜在世界模拟进行空间推理测试时尺度的框架。在查询和观察下，VLM 选择有用的空间变换，而世界模型执行这些变换，并直接向 VLM 展示对齐的中间潜在特征。所得的潜在证据被纳入后续推理步骤，实现迭代空间仿真，无需反复将想象状态解码为像素。为优化这种交互，WorldMind进一步采用过程验证的强化学习，只有当相关动作和潜在转变有效时才奖励推理进展。因此，WorldMind通过为主动世界模拟分配额外的推理时间计算，而非生成更多语言或像素，从而扩展空间推理。在VSI-Bench、MMSI-Bench和MindCube-Tiny中，WorldMind平均比基础VLM提升16.0%，较最强基于RGB的世界模拟提升8.3%。

BiMuon: A Geometry-Aware Stochastic Bilevel Optimization Approach

双μ子：一种几何感知随机双层优化方法

• Authors: Zhen Qin, Zhiyao Zhang, Yingbin Liang, Jia Liu
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2611.10188
• Pdf link: https://arxiv.org/pdf/2611.10188
• Abstract Stochastic bilevel optimization provides a general framework for many machine learning applications, such as hyperparameter learning, data reweighting, language model alignment, and actor-critic reinforcement learning. However, most existing bilevel optimization methods treat matrix-valued gradients as flattened vectors and perform optimization in the resulting vectorized space, ignoring the matrix structures common in modern neural networks. Moreover, the lower-level (LL) problem curvature can induce significant spectral imbalance in the upper-level (UL) geometry. Such limitations of existing bilevel optimization methods motivate us to develop BiMuon, a geometry-aware stochastic bilevel optimization method that integrates implicit hypergradient estimation in bilevel optimization with Muon-style orthogonalized updates at the UL. To our knowledge, BiMuon is the first stochastic BLO method with Muon-style orthogonalized matrix updates and an $O(\epsilon^{-4})$ iteration-complexity guarantee, matching the best-known rates achieved by Muon-type methods based on matrix parameterizations. To improve computational and parallel efficiency for models with multiple matrix parameters, we further study BiMuon for blockwise diagonal matrix approximation and derive a relative stepsize allocation rule based on UL progress and blockwise LL sensitivity. We evaluate BiMuon on representation learning and blockwise bilevel optimization, providing empirical support for our methods.
• 中文摘要 随机双层优化为许多机器学习应用提供了通用框架，如超参数学习、数据重权重、语言模型对齐和演员-批评者强化学习。然而，大多数现有的双层优化方法将矩阵值梯度视为扁平的向量，并在所得的矢量化空间中进行优化，忽略了现代神经网络中常见的矩阵结构。此外，低层（LL）问题曲率可能导致上层（UL）几何体中显著的谱不平衡。现有双层优化方法的这些局限性促使我们开发BiMuon，这是一种几何感知型的随机双层优化方法，将双层优化中的隐式超梯度估计与UL上的Muon式正交更新相结合。据我们所知，BiMuon是首个具有Mu子式正交矩阵更新和$O（\epsilon^{-4}）$的迭代复杂度保证的随机BLO方法，匹配基于矩阵参数化的Mu子类型方法所实现的最佳速率。为提升多矩阵参数模型的计算和并行效率，我们进一步研究了BiMuon用于块对角矩阵近似，并基于UL进展和分块LL敏感度推导出相对步长分配规则。我们评估BiMuon在表示学习和分组双层优化上的应用，为我们的方法提供实证支持。

Jais 3: Introducing Arabic-centric Agentic AI

Jais 3：引入以阿拉伯语为中心的智能人工智能

• Authors: Team IFM, Haonan Li, Hadi Abdine, Nikhil Ranjan, Mohamed Anwar, Ahmad Chamma, Amr Mohamed, Ahmed Frikha, Dani Bouch, Maaz Maqsood, George Ibrahim, Mostafa Awad, Nurkhan Laiyk, Hasan Iqbal, Abhishek Maiti, Bilal Bouardi, Shivam Chauhan, Ahmer Tabassum, Owais Aijaz, Omar Arbi, Neha Sengupta, Aaryamonvikram Singh, Abdelhakim Freihat, Sarath Chandran, Rahul Pal, Mohammed Bellam, Sajid Siddiki, Sarah AlBarri, Debopriyo Banerjee, Ali El Filali, Samta Kamboj, Rifo Genadi, Ahmed Attia, Go Inoue, Shuonan Dong, Amine Khelif Khelif, Evan Dufraisse, Florian Valade, Tapas Sen, Guowei He, Sunil Sahu, Onkar Pandit, Kareem Magdy, Sheikh Khalid bin Khalifa Al Nahyan, Samujjwal Ghosh, Zainul Abedien, Ahmed Quraishi, Larry Murray, Avraham Sheinin, Natalia Vassilieva, Preslav Nakov, Guokan Shang, Xuguang Hector Ren, Zhengzhong Liu, Eric Xing
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10224
• Pdf link: https://arxiv.org/pdf/2611.10224
• Abstract We present Jais 3, a fleet of six open, Arabic-centric agentic language models ranging from 0.9B to 375B parameters. Jais 3 is built to complete tasks for Arabic speakers, not only to answer questions in plain text: it is trained to plan multi-step tasks, call tools and Model Context Protocol (MCP) servers, operate a terminal, search the web, and read, edit, and produce PDF, spreadsheet, and presentation files, while communicating with the user in Arabic. Instead of pretraining from scratch, we adapt each model of the fully open K2 Horizon fleet to Arabic through continual pretraining, followed by supervised fine-tuning (SFT), final token preference optimization (FTPO), and reinforcement learning with verifiable rewards (RLVR). We combine natural Arabic text with translated, transliterated, and synthetic Arabic, and replay the base model's own training data to keep its reasoning, agentic, and English abilities. Jais 3 covers Modern Standard Arabic and the major regional dialects, follows instructions with Arabic-specific constraints, and inherits the 512K-token context window of K2 Horizon (all models except the 0.9B). To our knowledge, its 375B model is the largest open Arabic-centric language model to date. For evaluation, we revisit how Arabic models are assessed, moving to fully generative evaluation on audited Arabic benchmarks. Across model sizes, Jais 3 matches or outperforms open-weight models of comparable size on Arabic and agentic benchmarks, with particularly strong agentic capabilities and Arabic instruction following, and far fewer hallucinations. We release the weights and the checkpoints from every training stage of all six models, together with samples of the training data, the data recipes and training configurations, the training logs, the audited benchmarks, and the evaluation code.
• 中文摘要 我们介绍Jais 3，这是一支由六个开放、以阿拉伯语为中心的代理语言模型组成的舰队，参数范围从0.9B到375B不等。Jais 3旨在完成阿拉伯语使用者的任务，而不仅仅是用明文回答问题：它经过训练，能够规划多步任务，调用工具和模型上下文协议（MCP）服务器，操作终端，搜索网页，并阅读、编辑和生成PDF、电子表格和演示文件，同时与用户用阿拉伯语交流。我们不是从零开始预训练，而是通过持续的预训练，将完全开放的K2 Horizon车队的每个模型适配为阿拉伯语，随后进行监督微调（SFT）、最终令牌偏好优化（FTPO）和带可验证奖励的强化学习（RLVR）。我们将自然阿拉伯语文本与翻译、音译和合成阿拉伯语结合，并重放基础模型自身的训练数据，以保持其推理、代理和英语能力。Jais 3涵盖现代标准阿拉伯语及主要地区方言，遵循带有阿拉伯语特有约束的指令，继承了K2 Horizon的512K标记上下文窗口（除0.9B模型外所有模型）。据我们所知，其375B模型是迄今为止最大的以阿拉伯语为中心的开放语言模型。在评估方面，我们重新审视阿拉伯语模型的评估方式，转向基于经审计阿拉伯语基准的全生成式评估。在不同模型规模中，Jais 3在阿拉伯语和智能基准测试上表现与同等规模的开放权重模型相当甚至优于，具有特别强的智能体能力和阿拉伯语指令跟踪能力，且幻觉远少于此。我们发布了六个模型每个训练阶段的权重和检查点，以及训练数据样本、数据配方和训练配置、训练日志、审计基准测试和评估代码。

Transit-Oriented Land Use Planning Using Deep Reinforcement Learning

基于交通的土地利用规划，采用深度强化学习

• Authors: Bo Yang, Yuebing Liang, Zhan Zhao, Fangyi Ding, Yunhan Zheng, Shenhao Wang, Jinhua Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10281
• Pdf link: https://arxiv.org/pdf/2611.10281
• Abstract Transit-Oriented Development (TOD) is a key strategy for advancing sustainable urban development by promoting high-density, mixed-use land development within walking distance of public transit stations. However, traditional TOD planning often relies on heuristic methods, making it difficult to efficiently identify quantitatively optimized transit-oriented land use patterns. This paper presents DRL4TOD, a deep reinforcement learning framework that automatically optimizes land use types and floor area ratios (FARs) around transit stations in alignment with TOD principles. DRL4TOD incorporates a graph neural network to capture spatial interdependencies among land parcels, road networks, and metro stations, and uses a DRL-based optimization strategy to assign land-use types and FARs to selected parcels. It jointly optimizes multiple objectives, including land use density, land use diversity, and transit capacity utilization, while minimizing reconstruction costs to account for development feasibility. Applied to two real-world station areas in Shanghai, DRL4TOD consistently outperforms rule-based approaches and genetic algorithms, achieving more effective TOD outcomes with lower reconstruction costs. By adjusting objective weights, the framework enables flexible prioritization, such as placing greater emphasis on cost minimization when financial feasibility is a concern. These results demonstrate DRL4TOD's potential as a decision-support tool for generating coordinated land-use and FAR configurations under planner-specified priorities, rather than prescribing a single fixed design. For policymakers, the framework provides parcel-level, constraint-aware alternatives that make trade-offs among competing objectives explicit and support targeted decisions about development intensity and feasibility.
• 中文摘要 以交通为导向的发展（TOD）是推动可持续城市发展的关键策略，通过在步行距离内推动高密度混合用途土地开发。然而，传统的TOD规划往往依赖启发式方法，难以高效识别定量优化的以交通为导向的土地使用模式。本文提出了DRL4TOD，一种深度强化学习框架，能够根据TOD原则自动优化交通站点周围的土地利用类型和容积比（FAR）。DRL4TOD采用图神经网络捕捉土地地块、道路网络和地铁站之间的空间相互依赖关系，并采用基于日程灯的优化策略为选定地块分配土地利用类型和FAR。它联合优化多个目标，包括土地使用密度、土地利用多样性和交通容量利用率，同时最大限度地降低重建成本以考虑开发可行性。应用于上海两个真实站点区域，DRL4TOD持续优于基于规则的方法和遗传算法，实现更有效的TOD结果且重建成本更低。通过调整目标权重，该框架实现了灵活的优先级排序，例如在财务可行性考虑时更重视成本最小化。这些结果表明DRL4TOD作为决策支持工具的潜力，能够根据规划者指定的优先级生成协调的土地利用和FAR配置，而非单一固定设计。对于政策制定者，该框架提供了地块层级、约束意识的替代方案，使竞争目标间的权衡显现，并支持针对开发强度和可行性的有针对性决策。

Plug-and-Play Verifier for Decoupling Process Reward in Stabilizing Long-Horizon Agentic Reasoning

即插即用验证器，用于稳定长视野代理推理中的过程奖励解耦

• Authors: Mingwei Xu, Jieyuan Pei, Binghao Tu, Hao Fang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10305
• Pdf link: https://arxiv.org/pdf/2611.10305
• Abstract Stabilizing long-horizon reasoning requires intelligent agents to sustain useful progress across successive reasoning steps and tool interactions. Such agents are typically trained with reinforcement learning (RL), where actor-critic RL uses a learnable critic network to estimate step advantages, while critic-free methods such as GRPO eliminate critic models and rely mostly on the group variants of sparse outcome-only rewards. Both reward types leave partially useful failed trajectories either implicit or indistinguishable from entirely unproductive ones, and as reasoning horizons grow, it becomes difficult to maintain consistent reasoning trajectories. In this work, we propose a \textit{plug-and-play verifier} framework for stabilizing long-horizon reasoning through explicit verification of intermediate reasoning artifacts. Specifically, the framework decouples process verification through a unified interface that extracts intermediate reasoning artifacts from agent running trajectories, evaluates them using domain-specific verifiers, and aggregates the verification scores as bounded process reward. Then, we combine our dense verified process rewards and the sparse outcome rewards through a two-stage post-training schedule. We showed that with plug-and-play verifiers, we further stabilize the reasoning steps as last-mile self-reflection and self-consistency. Typical evaluation on HotpotQA using Qwen3.5-4B achieves a superior accuracy (60.18\%), outperforming the standard GRPO baseline (57.73\%). Our results suggest that plug-and-play verifiers provide a modular approach to learn more stable and reliable reasoning behaviors over long horizons.
• 中文摘要 稳定长视野推理需要智能体在连续推理步骤和工具交互中持续保持有用进展。此类智能体通常通过强化学习（RL）训练，其中actor-critic RL利用可学习的批评网络估计步长优势，而无批判方法如GRPO则消除批判模型，主要依赖群体变体的稀疏结果奖励。这两种奖励类型都会留下部分有用的失败轨迹，要么隐含，要么与完全无效的轨迹无异，随着推理视野的扩大，维持推理轨迹的一致性变得困难。本研究提出一个\textit{plug-and-play验证器}框架，通过显式验证中间推理工件来稳定长视野推理。具体来说，该框架通过统一接口解耦过程验证，从运行轨迹的代理中提取中间推理工件，使用领域特定验证器进行评估，并将验证分数汇总为有界过程奖励。随后，我们通过两阶段的训练后计划，将密集的验证过程奖励和稀疏结果奖励结合起来。我们证明，通过即插即用验证器，我们进一步稳定了推理步骤，作为最后一公里的自我反思和自我一致性。使用Qwen3.5-4B在HotpotQA上进行典型评估的准确率（60.18%）优于标准GRPO基线（57.73%）。我们的结果表明，即插即用验证器提供了模块化方法，帮助在长期内学习更稳定、更可靠的推理行为。

Finite Time Analysis of Risk-Sensitive RL via Noisy Power Iteration

通过噪声幂迭代对风险敏感强化学习进行有限时间分析

• Authors: Waqar Mirza, Yashaswini Murthy, Laixi Shi, Eric Mazumdar, Adam Wierman
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10393
• Pdf link: https://arxiv.org/pdf/2611.10393
• Abstract Estimating the principal eigenpair of a positive operator from stochastic evaluations gives rise to a noisy form of normalized power iteration: each step applies a sampled operator estimate and then renormalizes. We establish finite-time stochastic-approximation guarantees for this procedure, working in Hilbert's projective metric, which is well-suited to the multiplicative geometry of Perron--Frobenius eigenproblems. Our main application is risk-sensitive average-cost reinforcement learning (RL) with exponential utility. In this setting the Bellman equations are multiplicative, and value functions are characterized by nonlinear eigenvalue problems rather than additive fixed points. We cast both policy evaluation and control in this framework, obtaining risk-sensitive TD- and Q-learning algorithms that learn from a single Markovian trajectory. Under explicit positivity, mixing, and linear function approximation assumptions, we prove finite-time bounds on the recovered eigenvector and eigenvalue with $\tilde{O}(\epsilon^{-2})$ sample complexity up to problem-dependent constants. The results cover both tabular and linear function approximation regimes, and through the duality between exponential utility and KL-robust control yield finite-time guarantees for KL-robust average-cost RL as well.
• 中文摘要 通过随机评估估计正算子的主特征对会产生一种噪声形式的归一化幂迭代：每一步应用一个采样算子估计，然后进行重整化。我们为该过程建立了有限时间的随机近似保证，采用希尔伯特射影度量，非常适合佩隆-弗罗贝尼乌斯特征问题的乘法几何。我们的主要应用是具有指数效用的风险敏感平均成本强化学习（RL）。在此环境中，贝尔曼方程是乘法的，价值函数由非线性特征值问题而非加法不动点表征。我们在该框架中构建策略评估和控制，获得从单一马尔可夫轨迹学习的风险敏感的TD和Q学习算法。在显式正值、混合和线性函数近似假设下，我们证明恢复特征向量和特征值的有限时间界限，样本复杂度为$\tilde{O}（\epsilon^{-2}）$，且为问题相关常数。结果涵盖表格和线性函数近似领域，通过指数效用与KL鲁棒控制的对偶性，也为KL鲁棒平均成本RL提供了有限时间保证。

Coding or Cheating? An Investigation of Frontier Coding LLMs

编码还是作弊？前沿编码大型语言模型的调查

• Authors: Yuntao Gui
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10404
• Pdf link: https://arxiv.org/pdf/2611.10404
• Abstract Frontier agentic LLMs are trained on verifiable rewards in reinforcement learning (RL) environments built from real software repositories. A passing test suite is treated as the reward for solving the task. We show that, on hard problems, frontier models often satisfy these rewards by obtaining the answer instead of deriving it, known as reward hacking. Prior reports document this behavior on public benchmarks. We test its persistence on unseen problems. In a hardened environment where the answer cannot leak, nearly every model we tested attempted to recover the answer, escalating through tiers of increasingly specialized exploitation: mining the repository history for the upstream fix, inspecting the git object store directly, and sweeping the filesystem for a possible newer copy of the source or hidden tests. These attempts were concentrated on the hardest problems, and the models differed in whether, how often, and how deeply they escalated. Our position is that this behavior is a learned shortcut, acquired in RL environments that leak the answer, not a sign of intelligence. We release agent traces for these models to support further study at this https URL.
• 中文摘要 前沿代理型大型语言模型在基于真实软件仓库构建的强化学习（RL）环境中，基于可验证的奖励进行训练。通过测试套件被视为完成任务的奖励。我们表明，在困难问题上，前沿模型通常通过获取答案而非推导答案来满足这些奖励，称为奖励黑客。此前报告在公开基准测试中记录了这种行为。我们在看不见的问题上测试其持久性。在一个答案无法泄露的加固环境中，几乎所有我们测试的模型都试图恢复答案，通过越来越专业化的利用层级升级：挖掘仓库历史以获取上游修复，直接检查git对象存储，以及扫描文件系统以寻找可能更新的源代码或隐藏测试副本。这些尝试集中在最难的问题上，模型在升级频率和深度上有所不同。我们的立场是，这种行为是强化学习中习得的捷径，而非智能的表现。我们发布了这些模型的代理痕迹，以支持进一步研究，链接在此 https URL。

SimDuoReal: Object-Centric Bimanual Policy for Emergent Dexterity

SimDuoReal：面向对象的双手策略，促进新兴灵巧度

• Authors: Qiyang Yan, Yulin Liu, Akash Sharma, Haozhi Qi, Guanya Shi, C. Karen Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.10410
• Pdf link: https://arxiv.org/pdf/2611.10410
• Abstract Bimanual dexterous manipulation requires coordinated arm and hand control, yet specifying such coordination manually is challenging and often task-specific. In this work, we explore whether complex bimanual dexterity can emerge from a simple object-centric learning objective. We train reinforcement learning policies under a shared object-centric reward, varying their training environments rather than tuning separate rewards for each behavior. We find that the policies discover pre-grasp manipulation, grasping and regrasping, in-hand adjustment and handover to reach object-pose goals without hand-designed motion sequences. We then refine the experts on transition states and use DAgger and reinforcement learning to distill them into a single policy that composes these behaviors for diverse bimanual tasks. On real hardware, the same policy succeeds in 61 of 80 trials (76.3\%) across four tasks with unseen objects and no task-specific fine-tuning. In simulated barcode scanning, our policy achieves 85.0\% success, compared with 60.4\% for two single-arm policies without handover. These results suggest that this objective provides a simple and scalable interface for learning general bimanual dexterity, shifting the burden from manually designing robot coordination to specifying what should happen to the object.
• 中文摘要 双手灵巧操作需要协调的手臂和手部控制，但手动指定这种协调既具挑战性又常因任务而异。本研究探讨复杂的双手灵巧是否可以从简单的以对象为中心的学习目标中产生。我们在共享的以对象为中心的奖励下训练强化学习策略，改变训练环境，而非为每种行为单独调整奖励。我们发现这些策略能够发现预抓操作、抓取与回归、手中调整和交接，以实现物体姿势目标，无需手动设计的动作序列。随后，我们完善过渡态专家，利用DAgger和强化学习将其提炼成一个策略，为不同双手任务组合这些行为。在真实硬件上，同一策略在4个未见对象且无任务特定微调的任务中，成功率为80次试验中的61个（76.3%）。在模拟条码扫描中，我们的策略成功率为85.0%，而两个单臂策略且无切换时仅为60.4%。这些结果表明，这一目标为学习一般双手灵巧度提供了一个简单且可扩展的界面，将任务从手动设计机器人协调转向指定对象应发生的处理。

Deploying Physically and Epistemically Compliant Agents by Distilling Belief-Space Task and Motion Planners

通过提炼信念空间任务和动作规划器，部署物理和认知合规的代理

• Authors: Phillip J.K. Christoffersen, Rachel Ma, Patrick Barragán, Dylan Hadfield-Menell
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10430
• Pdf link: https://arxiv.org/pdf/2611.10430
• Abstract In this work, we train agents to act under uncertainty according to specified, belief-dependent rules. Deep reinforcement learning (RL) can in theory train agents to follow rules if encoded into well-designed rewards, but in practice, RL fails to perfectly optimize these rewards, leading to ineffective or noncompliant agent behavior. Belief-space planners by construction produce behavior in line with developer specification, but require hand-programming online planners and belief-state estimators, adding significant engineering complexity at scale. We combine the two: we plan offline to generate a dataset of sample trajectories and then distill the plans into a policy that acts on raw observations. We propose two changes to planner distillation that address problems specific to history-based policies: Dual Averaging, a loss that combines per-timestep predictive loss and average loss on task changes, and $\rho$-Optimal Reinforcement, a feedback mechanism that reinforces the policy's own successful rollouts when these are near-optimal. We provide a sufficient condition under which static search problems with noiseless observation models is complete, and show that maximum-likelihood-observation planning implies this condition in static search with continuous beliefs. We create a challenging new domain, Scavenger, where an agent needs to gain information in order to succeed. We evaluate our distilled policy in two versions of Scavenger, one where information is provided visually and one where it is provided as a command in natural language. In either case, the learned policy attains a success rate above 90% in Scavenger, compared with at most 5% for RL. We also find that distilled planners transfer with greater than 60% success to following unseen and intentionally vague commands and greater than 75% success to following floorplans larger than those seen in training.
• 中文摘要 在本研究中，我们训练智能体按照指定的信念依赖规则在不确定性下行动。深度强化学习（RL）理论上可以训练智能体遵循规则，但实际上，强化学习未能完美优化这些奖励，导致智能体行为无效或不合规。信念空间规划器本质上能生成符合开发者规范的行为，但需要手工编程在线规划器和信念状态估计器，这在大规模上增加了显著的工程复杂性。我们将两者结合：我们离线规划生成样本轨迹数据集，然后将计划提炼成基于原始观测值的策略。我们提出了两个针对基于历史策略的计划器提炼的变更，针对特定问题：双重平均，即结合每时间步预测损失和任务变更平均损失的损失;以及$\rho$-最优强化，一种反馈机制，在策略自身成功部署接近最优时强化。我们给出充分条件，使得静态搜索问题在无噪声观察模型下完成，并证明最大似然观察规划在带连续信念的静态搜索中蕴含该条件。我们创建了一个具有挑战性的领域——Scavenger，主体需要获取信息才能成功。我们在两个版本的Scavenger中评估了提炼策略，一个以视觉形式提供信息，另一个以自然语言命令形式提供。无论哪种情况，学习策略在Scavenger中成功率均超过90%，而强化学习最多为5%。我们还发现，精简规划师在遵循看不见且故意模糊指令的成功率超过60%，而遵循比培训中看到的更大平面图的成功率超过75%。

Grip, Trace, Write: Force-Aware Imitation and Reinforcement Learning for High-DoF Robotic Handwriting

握、描、写：高深度机器人书写的原力模仿与强化学习

• Authors: Rongqian Chen, Yiding Chen, Jijie Zhou, Tom Z. Jiahao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.10451
• Pdf link: https://arxiv.org/pdf/2611.10451
• Abstract Most robotic handwriting rigidly mounts the pen and reduces writing to trajectory tracking. A freely held pen instead couples grasp stability, tip motion, and contact force. We present Grip, Trace, Write, a force-aware simulation environment and learning framework for high-DoF handwriting. A two-branch actor separates pen retention from arm-wrist writing while preserving their closed-loop coupling. Training learns a grasp controller, initializes writing from inverse-kinematics demonstrations by behavior cloning, and improves writing through reinforcement learning with rewards on ink, force, time, and retention. Across diverse tasks, the policy improves completion, precision, force regulation, and efficiency over behavior cloning and inverse kinematics, and generalizes without fine-tuning to unseen English and Chinese content. These results establish force-aware handwriting as a testbed for dexterous tool use.
• 中文摘要 大多数机器人手写是固定笔的，将书写简化为轨迹追踪。自由握笔则结合握力稳定性、笔尖运动和接触力。我们介绍Grip， Trace， Write，一个针对高景深手写的感应力模拟环境和学习框架。一个双分支角色将笔的保留与手臂-手腕书写分离开来，同时保持它们的闭环耦合。训练学习握把控制器，通过行为克隆从逆运动学演示中初始化书写，并通过强化学习提升书写能力，奖励墨水、力、时间和保持力。在多样化任务中，该策略提升了完成度、精度、力调控和效率，优于行为克隆和逆运动学，并在未微调的情况下推广到未见的英中内容。这些结果确立了力感书写作为灵巧工具使用的试验平台。

BunshinRL: Splitting Live Agent Workspaces for Memory-Efficient Agentic RL

BunshinRL：拆分实时代理工作区以实现内存高效的代理强化学习

• Authors: Yutong Huang, Vikranth Srivatsa, Alex Asch, Yiying Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Operating Systems (cs.OS)
• Arxiv link: https://arxiv.org/abs/2611.10453
• Pdf link: https://arxiv.org/pdf/2611.10453
• Abstract Agentic reinforcement learning (RL) rollouts interleave model reasoning with tool calls in persistent workspaces. Scaling these rollouts incurs substantial memory costs for concurrent environments, while slow tool calls delay entire rollout batches. We present BunshinRL, an agentic RL rollout system that exploits shared environment state and recurring tool calls through three techniques: memory sharing across environments, tool-prefix caching, and speculative tool execution. These techniques build on BunshinFork, a novel live whole-workspace forking mechanism for containers. BunshinFork reconstructs process trees, shares memory copy-on-write, and preserves file, communication, and GUI state, enabling cached-state reuse and rollback after incorrect speculation. We implement BunshinRL and BunshinFork through Linux kernel module and CRIU modifications. BunshinRL reduces peak memory by up to 43%, total memory consumption by up to 51%, and end-to-end rollout latency by 23%.
• 中文摘要 代理强化学习（RL）部署将模型推理与工具调用交织在持久工作区中。扩展这些部署会在并发环境中产生大量内存成本，而较慢的工具调用则会延迟整个部署批次。我们介绍BunshinRL，一种能动的强化学习（RL）滚动系统，通过三种技术利用共享环境状态和重复工具调用：跨环境内存共享、工具前缀缓存和推测工具执行。这些技术基于BunshinFork，一种针对容器的新型实时全工作空间分支机制。BunshinFork重建进程树，写时共享内存，并保存文件、通信和图形界面状态，支持缓存状态的重用和回滚，防止错误推测。我们通过Linux内核模块和CRIU修改实现BunshinRL和BunshinFork。BunshinRL 可将峰值内存降低高达 43%，总内存消耗降低 51%，端到端扩展延迟降低 23%。

Represented but Not Used: Value-Policy Dissociation in LLM Agents

表示但未使用的：LLM代理中的价值-策略解离

• Authors: Yuzhang Luo, Mohamed Aghzal, Liangming Pan, Ziyu Yao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10457
• Pdf link: https://arxiv.org/pdf/2611.10457
• Abstract Large language model (LLM) agents need to choose actions whose consequences unfold over multiple future steps, yet poor decisions do not reveal whether the model failed to represent those consequences or failed to use them. Mechanistic interpretability has studied internal representations of environment state and plans, but sequential decision making also requires evaluating how candidate actions change future outcomes. Reinforcement learning formalizes this as action value. We study this computation in Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B across a partially observed Grid environment and WebShop, a simulated web shopping environment. We find that the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. We call this gap between value representation and policy choice value-policy dissociation. To investigate how represented value relates to action selection, we compare direct interventions along the value readout direction with Natural Value Transport (NVT), a transformation learned from hidden state changes that naturally result from differences in action value. While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice.
• 中文摘要 大型语言模型（LLM）代理需要选择后果在多个未来步骤中展开的行动，但错误决策无法揭示模型是否未能代表这些后果或未能使用这些后果。机制性可解释性研究了环境状态和计划的内部表征，但顺序决策也需要评估候选行为如何改变未来结果。强化学习将此形式化为行动价值。我们在Qwen2.5-7B-Instruct、Qwen2.5-32B-Instruct和GPT-OSS-20B中，在部分观察的网格环境和WebShop（模拟网购环境）中研究了这一计算。我们发现候选行为的相对价值可从中间隐藏状态线性解码，包括政策偏好更差行为的状态。我们将这种价值表示与政策选择之间的差距称为价值-政策解离。为探讨表征价值与动作选择的关系，我们比较了沿值读出方向的直接干预与自然值传输（NVT）——一种由动作值差异自然产生的隐藏状态变化所学习的转换。虽然读出方向干预效果有限且不一致，但NVT更一致地影响行动偏好，尤其是在后期层。我们还测试了闭环控制器，该控制器将值读出作为传感器，NVT作为执行器，观察任务结果的改进。这些结果表明，LLM代理可以代表有用的潜在价值，而不必可靠地转化为策略选择。

Meta-Reinforcement Learning with Zero-Shot Reinforcement Learning

零点强化学习的元强化学习

• Authors: Jake Grigsby, Siddhant Agarwal, Yu Lei, Leonidas Varveropoulos, Yuke Zhu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10553
• Pdf link: https://arxiv.org/pdf/2611.10553
• Abstract Meta-reinforcement learning (meta-RL) trains agents to adapt to novel tasks from test-time experience, but requires diverse training environments and reward functions. While environments are becoming cheaper to generate, it remains difficult to anticipate every objective an agent may be asked to pursue at test time. Behavior Foundation Models (BFMs) learn policies from reward-free data and support zero-shot RL (ZSRL): adaptation to new objectives at test time using a large reward-labeled dataset. Recent work has shown that BFM task inference can instead be performed online without prior task-specific data. We note that this capability positions BFMs and meta-RL as alternative approaches for online adaptation to new reward functions in a fixed environment. We then ask whether we can extend BFMs to compete with meta-RL more broadly, including in novel environments, and identify four challenges: environment identification, online data collection, test-time exploration, and learning from mixed reward supervision. We study each individually, then propose MetaBFM, a hybrid method that addresses all four. During training, it combines supervised reward-following with unsupervised learning; at test time, it can interpolate between exploration, exploitation, and ZSRL-style reward inference. We demonstrate that MetaBFM can recover the in-distribution performance of supervised meta-RL while generalizing to reward functions outside the training set. Our work suggests that BFMs and meta-RL are complementary research directions whose combined strengths allow adaptive agents to learn more diverse behaviors from a fixed set of supervised objectives.
• 中文摘要 元强化学习（meta-RL）训练智能体从测试时间经验适应新任务，但需要多样化的训练环境和奖励函数。尽管环境生成成本降低，但预测智能体在测试时可能追求的每一个目标仍然困难。行为基础模型（BFM）从无奖励数据中学习策略，支持零样本RL（ZSRL）：利用大量奖励标记数据集在测试时适应新目标。最新研究表明，BFM任务推断可以在没有任务特定数据的情况下在线完成。我们注意到，这一能力使BFM和元RL成为固定环境中在线适应新奖励函数的替代方法。随后，我们探讨是否可以将BFM扩展到更广泛地与元RL竞争，包括新环境中，并确定四个挑战：环境识别、在线数据收集、测试时间探索和混合奖励监督学习。我们逐一研究每种方法，然后提出了MetaBFM，一种涵盖四种的混合方法。在训练过程中，它结合了监督式奖励跟随和无监督学习;在测试阶段，它可以在探索、利用和类似ZSRL的奖励推断之间进行插值。我们证明了MetaBFM能够恢复监督元强化学习的分布内表现，同时推广到训练集外的奖励函数。我们的研究表明，BFM和元强化学习是互补的研究方向，其综合优势使自适应智能体能够从固定的监督目标集中学习更多样的行为。

Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock Speed

Detent：批量不变LLM的服务和强化学习（RL）以接近库存速度展开

• Authors: Peixu Xin (Independent Researcher, Boston University), Runchuan Feng (Independent Researcher, Boston University), Jiazhen Guo (Google), Kan Wu (RadixArk), Ming Yin (Princeton University and Georgia Institute of Technology)
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.10611
• Pdf link: https://arxiv.org/pdf/2611.10611
• Abstract Reproducible evaluation, verifiable inference and on-policy reinforcement learning (RL) need an LLM engine to give a request the same output bits whichever requests share its batch. Batch-invariant kernels exist, but engines still select each kernel and its parameters from the batch. Two kernels that compute the same sum can round it differently. The engines' deterministic modes remove the choice by keeping one conservative kernel per operator and pay for it in latency. Many fast kernels are batch-invariant by themselves, so it is enough to fix the few batch-dependent values that kernel selection reads. Detent expresses kernel selection as a program that the engine executes. It finds the fewest values in this program that must be fixed and tunes them for the workload before deployment. On multi-GPU deployments of Llama-3.1-70B and Qwen3-30B-A3B in vLLM and SGLang, Detent is batch-invariant. Tuned per load, it serves at 1.04-1.17x the latency per output token of Stock (each engine with its standard settings), against 1.67-10.3x for the deterministic modes. In SGLang it matches the GSM8K accuracy of SGLang's deterministic mode. Detent covers multiple deployment combinations, including multiple engines, GPU generations, tensor-parallel degrees, and dense and MoE models. Seeded RL rollouts of models up to Llama-3.1-70B replay bit for bit under other batches and cache splits. A trainer on Detent's kernels and values matches every rollout logprob bit for bit at 1.03x Stock's rollout time per token. BF16 GRPO then stays stable where Stock's run collapses.
• 中文摘要 可重现的评估、可验证推理和策略强化学习（RL）需要LLM引擎为请求提供与共享批次的请求相同的输出位。存在批处理不变内核，但引擎仍从批中选择每个内核及其参数。两个计算相同和的核可以不同地四舍五入。引擎的确定性模式通过保留每个操作符一个保守内核，并以延迟代价，从而消除选择。许多快速内核本身是批不变的，因此只需固定内核选择读取的少数几项批相关值即可。Detent将内核选择表示为引擎执行的程序。它在部署前找到该程序中最少需要固定的值，并根据工作负载进行调整。在vLLM和SGLang中对Llama-3.1-70B和Qwen3-30B-A3B的多GPU部署中，Detent是批处理不变的。每次加载调优后，每个输出令牌的延迟是Stock（每个引擎及其标准设置）的1.04-1.17倍，而确定性模式为1.67-10.3倍。在SGLang中，它的精度可与SGLang确定性模式匹配GSM8K。Detent覆盖多种部署组合，包括多个引擎、GPU世代、张量-并行度以及稠密和MoE模型。在其他批次和缓存拆分下，对Llama-3.1-70B的模型进行种子重放。Detent的内核和值训练器会以每个代币1.03倍Stock的滚动时间逐位匹配每个rollout的logprob位。BF16的GRPO随后保持稳定，而斯托克的连胜则崩溃。

SLA-Aware Task Offloading in the IoT Edge-Cloud Continuum: Class-Weighted Deep Reinforcement Learning Against an Exact Optimum

物联网边缘-云连续体中的 SLA 感知任务卸载：基于精确最优的类别加权深度强化学习

• Authors: Shilpa Golla, Tharun Kumar Korine Palli, Urmila Bandaru
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.10638
• Pdf link: https://arxiv.org/pdf/2611.10638
• Abstract Deep reinforcement learning (RL) is often used to decide where Internet of Things (IoT) tasks with different latency limits run in the edge-cloud continuum, yet some methods give every missed task the same penalty and few compare their agents with an optimal policy. This benchmark study builds four offloading systems small enough to solve exactly by dynamic programming and measures reward and learner against the optimum. A bonus for a met limit makes each miss cost its penalty plus the bonus, which shifts the class weights of published per-class rewards. A class-blind penalty stays 65% to 118% above the optimal cost even with a perfect learner, while a corrected per-class penalty comes within 0.13% of the optimum. With a per-class reward, tuned proximal policy optimization (PPO) agents come within 9% to 11% of the optimum in the smallest system, and the corrected one beats a tuned reservation rule in 9 of 10 seeds. In larger systems the agents stay 41% to 76% above the optimum, and a tuned rule beats their mean. Training ten times longer does not close this gap, and a PPO-Lagrangian agent with reachable targets does no better than the per-class reward.
• 中文摘要 深度强化学习（RL）常用于决定具有不同延迟限制的物联网（IoT）任务在边缘云连续体中的运行位置，但有些方法对每个漏接任务给予相同的惩罚，且很少有方法用最优策略进行比较。这项基准研究构建了四个足够小、可通过动态规划精确求解的卸载系统，并以最优值衡量奖励和学习者。达到限制的加成会使每次未命中时，扣除其罚时加上加成，从而改变已发布的每类奖励的类别权重。即使学习者完美，类别盲惩罚仍比最优成本高出65%至118%，而修正后的每类惩罚则仅为最优值0.13%。在每类奖励下，调优近端策略优化（PPO）代理在最小系统中距离最优值9%到11%以内，修正后的代理在10个种子中有9个超过调优保留规则。在较大系统中，代理保持在最优值上41%到76%，调优规则超过平均值。训练时间延长十倍也无法弥合这一差距，且具有可达目标的PPO-拉格朗日代理表现不比每个类别奖励更好。

The Delta Gap: Cellular World Models for Perturbation Response Prediction Are Far Harder Than Global Correlation Implies

三角差距：细胞世界模型用于扰动响应预测远比全球相关性所暗示的要复杂得多

• Authors: Junhui Gao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10658
• Pdf link: https://arxiv.org/pdf/2611.10658
• Abstract Cellular world models aim to predict how cells respond to genetic perturbations, analogous to world models in reinforcement learning. Perturbation response models are evaluated almost exclusively by global Pearson correlation between predicted and observed transcriptomes. We show this metric is dominated by thousands of non-responsive genes and severely inflates apparent performance. On the Replogle K562 screen (509 unseen perturbations, 5000 genes), GEARS achieves global Pearson 0.977, but its delta Pearson is only 0.076. A trivial baseline predicting mean training delta achieves 0.267, which is 3.5x better than GEARS. Our primary contribution is methodological: we identify and quantify this delta gap as a systematic evaluation artifact, and propose delta-space evaluation as the primary benchmark. We provide two post-hoc corrections (RWR on co-expression graphs and kNN matching) and show that 2.8x of the remaining gap lies in perturbation representation geometry, not decoder capacity. We further show the gap is partially addressable end-to-end: delta-aligned training (lambda = 3.0) raises delta Pearson to 0.179 on Replogle (2.35x over GEARS) and surpasses the trivial baseline on direction accuracy (0.680 vs 0.567) and gene recovery (1.52 vs 1.19); combined with kNN it reaches 0.217. An amplitude-calibration variant (eta = 0.3) further raises gene recovery to 2.16 while reaching 0.218. Critically, these representations transfer zero-shot across cell lines: from Replogle K562 to unseen Norman A549, the delta-aligned model reaches 0.328 on 741 shared genes, 4.7x a trivial baseline and 2.1x standard GEARS. This is the first systematic quantification of the delta gap, and the first diagnostic framework localizing the residual gap to perturbation representation geometry via an oracle upper bound.
• 中文摘要 细胞世界模型旨在预测细胞对遗传扰动的反应，类似于强化学习中的世界模型。扰动响应模型几乎完全通过预测与观察到的转录组之间的全局皮尔逊相关来评估。我们发现该指标由数千个无响应基因主导，严重夸大了表观表现。在Replogle K562筛查（509个未见扰动，5000个基因）中，GEARS实现了全局皮尔逊0.977，但其δ皮尔逊仅为0.076。一个简单基线预测平均训练差值为0.267，比GEARS好3.5倍。我们的主要贡献是方法论：我们将该差值差距识别并量化为系统性评估的产物，并提出以δ-空间评估为主要基准。我们提供了两个事后修正（共表达图的RWR和kNN匹配），并显示剩余差距中有2.8%存在于扰动表示几何，而非解码器容量。我们进一步表明该差距在端到端部分可解决：Delta-aligned训练（lambda = 3.0）使Replogle的Delta Pearson提升至0.179（比GEARS的2.35倍），并在方向精度（0.680对0.567）和基因恢复率（1.52对1.19）上超过了平凡基线;结合kNN达到0.217。振幅校准变体（预计=0.3）进一步将基因恢复率提升至2.16，同时达到0.218。关键是，这些表示能够在细胞系间零射点转移：从Replogle K562到未见的Norman A549，δ比对模型在741个共享基因上达到0.328,4.7倍于平凡基线，2.1倍标准GEARS。这是首次系统量化δ间隙，也是首个通过oracle上界将残余间隙定位到扰动表示几何的诊断框架。

Two-Timescale Trajectory Control, Task Reassignment and Resource Scheduling for Multi-UAV Cooperative Mobile Edge Computing

多无人机协作移动边缘计算的两时间尺度轨迹控制、任务重新分配与资源调度

• Authors: Bin Li, Shiyu Sun, Lei Liu, Zesong Fei
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.10673
• Pdf link: https://arxiv.org/pdf/2611.10673
• Abstract Uncrewed aerial vehicle (UAV)-assisted mobile edge computing (MEC) is a promising technology, leveraging the advantages of UAVs including flexible deployment, wide coverage, and reliable wireless communication. In this paper, we study the task reassignment and offloading problem in a multi-UAV cooperative operating environment, and consider the real occurrence that the computing tasks arrive more frequently than UAVs make flight decisions. In addition, we also take into account the position offset of the UAV during flight and the unavailability of the UAV due to external interference. We propose a two-timescale optimization framework, where the trajectory of the UAV is carried out on a large timescale, while the task reassignment and computing resource allocation are jointly optimized on a small timescale. The core objective is to maximize the long-term average system gain, which takes both energy consumption and task delay into account. To address the UAV trajectory optimization problem under large timescale constraints, we propose a constrained multi-agent deep reinforcement learning algorithm. For decision-making within small timescale, to jointly optimize task offloading decisions and computing resource allocation, we integrate convex optimization with a heuristic algorithm. Finally, we conduct extensive simulations to evaluate the proposed algorithms against the baseline, achieving a 24% reduction in latency and a 28% reduction in total system cost.
• 中文摘要 无人机（UAV）辅助移动边缘计算（MEC）是一项有前景的技术，充分利用了无人机的优势，包括灵活部署、广泛覆盖和可靠的无线通信。本文研究了多无人机协作作业环境中的任务重新分配和卸载问题，并考虑计算任务到达频率超过无人机飞行决策的真实情况。此外，我们还考虑了无人机飞行中的位置偏移以及因外部干扰导致无人机无法使用的情况。我们提出了一个两时间尺度优化框架，即在较长时间尺度上执行无人机轨迹，同时在较小时间尺度上联合优化任务重新分配和计算资源分配。核心目标是最大化长期平均系统收益，同时考虑能量消耗和任务延迟。为解决大时间约束下的无人机轨迹优化问题，我们提出了一种受限多智能体深度强化学习算法。为实现短期决策，联合优化任务卸载决策和资源分配计算，我们将凸优化与启发式算法结合。最后，我们进行了大量模拟，将所提出的算法与基线进行评估，实现了延迟降低24%，总系统成本降低28%。

Safe Offline Reinforcement Learning using Behavior Regularisation and Latent Feasibility Guidance

使用行为规范化和潜在可行性指导进行安全离线强化学习

• Authors: Mahesh Keswani, Parth Bhardwaj, Ankur Kumar, Raunak P. Bhattacharyya
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10677
• Pdf link: https://arxiv.org/pdf/2611.10677
• Abstract Safe offline reinforcement learning (RL) seeks to learn policies that maximize return while satisfying safety constraints, using a static dataset. To characterize state-wise safety, reachability analysis identifies the feasible region, defined as the set of states from which constraints can be satisfied at every future step. Policies are commonly extracted from learned value functions with advantage-weighted regression (AWR). However, AWR relies on exponentiated weights that amplify value estimation errors, whereas behavior-regularized policy gradient methods directly utilize the action gradient of the value function and often make more effective use of learned critics. We propose Feasibility-Optimized Conditional Actor Learning (FOCAL), a method that incorporates behavior-regularized policy gradients into a conditional variational autoencoder (CVAE) actor. FOCAL optimizes this policy using a first-order objective partitioned by feasibility, updating the actor to increase return on transitions estimated to be feasible and to reduce constraint violations on transitions estimated to be infeasible. During inference, FOCAL dynamically adjusts the scale of the latent prior according to the estimated feasibility of the state to adjust the conservatism of action sampling. Theoretically, we show that for AWR, the policy update variance depends exponentially on the advantage error, whereas for FOCAL it scales linearly with the variance of the value estimation error. Across the DSRL benchmark, FOCAL satisfies the constraints on all evaluated tasks while achieving competitive or higher returns compared to baselines.
• 中文摘要 安全离线强化学习（RL）旨在利用静态数据集学习在满足安全约束的同时最大化回报的策略。为了描述状态层次的安全，可达性分析确定可达区域，即在每个未来步骤都能满足约束的状态集合。策略通常通过优势加权回归（AWR）从学习的价值函数中提取。然而，AWR依赖指数权重以放大价值估计误差，而行为正则化策略梯度方法则直接利用价值函数的作用梯度，且通常更有效地利用已学到的批评者。我们提出了可行性优化的条件行为者学习（FOCAL），这是一种将行为正则化策略梯度整合进条件变分自编码器（CVAE）演员的方法。FOCAL通过一阶目标按可行性划分来优化该策略，更新参与者以增加估计可行转移的收益，并减少估计不可行转移的约束违规。在推断过程中，FOCAL根据状态的可行性估计动态调整潜在先验的尺度，以调整动作采样的保守性。理论上，我们表明AWR的策略更新方差指数依赖于优势误差，而FOCAL则随价值估计误差的方差线性增长。在DSRL基准测试中，FOCAL满足所有评估任务的约束条件，同时实现与基线竞争或更高的收益。

Deep Reinforcement Learning with Massively Parallel Simulation for Low-Thrust Multi-Revolution Rendezvous

基于大规模并行模拟的深度强化学习，用于低推力多转交会

• Authors: Chenglong Wu, Yi Zhou, Bin Cheng, Nan Zhang, Hexi Baoyin
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.10712
• Pdf link: https://arxiv.org/pdf/2611.10712
• Abstract The growing demand for on-orbit servicing and constellation management in low Earth orbit requires autonomous and reliable low-thrust rendezvous. Traditional numerical methods struggle with the long-horizon nature of multi-revolution maneuvers under perturbations, while standard deep reinforcement learning (DRL) methods are limited by low simulation throughput and poor sample efficiency. To address these challenges, this paper proposes a massively parallel DRL guidance framework. A fully tensorized, GPU-resident orbital propagator is developed to integrate $J_2$-perturbed orbital dynamics and policy optimization within a zero-copy pipeline, achieving up to a 67-fold increase in simulation throughput over conventional CPU baselines. In addition, the rendezvous problem is formulated in modified equinoctial elements and trained with a progressive curriculum learning strategy to improve convergence and generalization across increasingly challenging rendezvous scenarios. Simulations show that the proposed framework can learn robust rendezvous trajectories for complex multi-revolution rendezvous over a broad range of orbital configurations. Additional evaluations under a high-fidelity model and eclipse-induced thrust interruptions further demonstrate the robustness of the learned policy under practical mission constraints. Moreover, the trajectories generated by the proposed framework can be used as effective initial guesses for optimization-based trajectory solvers, improving the convergence of high-precision trajectory optimization. The method offers a scalable approach for real-time low-thrust guidance and trajectory optimization.
• 中文摘要 低地球轨道对轨道维护和星座管理需求的增长，要求自主且可靠的低推力交会。传统数值方法难以应对多圈扰动机动的长视野特性，而标准深度强化学习（DRL）方法则受限于模拟吞吐量低和样本效率差。为应对这些挑战，本文提出了一个大规模并行的DRL导航框架。开发了一种全张量化、驻GPU轨道传播器，在零拷贝流水线内集成$J_2$扰动轨道动力学和策略优化，使模拟吞吐量比传统CPU基线提升多达67倍。此外，交会问题以修正后的春分点元素形式提出，并采用渐进式课程学习策略训练，以提升在日益复杂交会场景中的收敛和泛化能力。模拟显示，所提框架能够在广泛的轨道配置范围内学习复杂多周交会的稳健交会轨迹。在高保真度模型和日食诱导推力中断下的额外评估进一步证明了所学策略在实际任务约束下的稳健性。此外，所提出框架生成的轨迹可作为基于优化的轨迹求解器的有效初始猜测，提升高精度轨迹优化的收敛性。该方法为实时低推力制导和轨迹优化提供了可扩展的方法。

From User Behaviors to Satisfaction Judgments: Reasoning about User Satisfaction with AI Overviews

从用户行为到满意度判断：关于用户满意度的推理概述

• Authors: Xingzhu Wang, Yitong Xu, Yue Li, Daiting Shi, Jingzhou He, Jiaxin Mao
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.10715
• Pdf link: https://arxiv.org/pdf/2611.10715
• Abstract The growing presence of AI overviews in everyday search makes their effective evaluation essential. Response quality has been the primary focus of existing evaluation research, with only a small number of studies examining the user perspective. These evaluations rely on queries and response content, leaving the information conveyed by user behavior insufficiently utilized. This work investigates satisfaction prediction for AI overviews from a user-centered perspective and incorporates behavioral signals into the prediction process. Applying existing behavior-based satisfaction prediction methods from traditional information retrieval to this setting, however, presents limitations. We address these limitations with STaRS, an LLM-based approach designed specifically for predicting satisfaction with AI overviews. At its core is a structured reasoning procedure: individual behavioral signals are interpreted first, and their implications then inform the prediction of user satisfaction. Reinforcement learning further optimizes this procedure, while a reward shaping mechanism alleviates reward sparsity during training. Evaluation on two real-world datasets demonstrates that STaRS outperforms the comparison methods in satisfaction prediction, providing search providers with a practical means of accurately estimating users' satisfaction with AI overviews.
• 中文摘要 AI概览在日常搜索中的日益普及使得其有效评估变得不可或缺。响应质量一直是现有评估研究的主要关注点，只有少数研究关注用户视角。这些评估依赖于查询和回答内容，导致用户行为传递的信息未能充分利用。本研究从用户中心视角研究AI概览的满意度预测，并将行为信号纳入预测过程。然而，将传统信息检索中基于行为的满意度预测方法应用到该环境中存在局限性。我们通过STaRS解决这些局限性，这是一种基于LLM的方法，专门设计用于预测对AI概览的满意度。其核心是一个结构化推理过程：首先解读单个行为信号，然后其含义用于预测用户满意度。强化学习进一步优化了这一过程，而奖励塑造机制则缓解了训练过程中的奖励稀疏性。对两个真实世界数据集的评估表明，STaRS在满意度预测方面优于比较方法，为搜索提供商提供了切实可行的方式，准确估算用户对AI概览的满意度。

AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF

AERO：样本高效在线RLHF中用于甲骨文反馈的自适应集合分歧路由

• Authors: Shrey Singhal, Nannan Huang, Haytham M. Fayek
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10750
• Pdf link: https://arxiv.org/pdf/2611.10750
• Abstract Aligning Large Language Models (LLMs) with human preferences via Reinforcement Learning from Human Feedback (RLHF) has driven substantial improvements across a wide range of tasks, but the cost of acquiring preference labels remains the dominant bottleneck in the alignment pipeline. While many recent alignment methods improve sample efficiency through active exploration over candidate responses, they typically query the oracle uniformly across contexts, even for contexts where the learned reward estimates are already reliable enough to provide synthetic labels. We propose Adaptive Ensemble-disagreement Routing for Oracle feedback (AERO), a fully online RLHF algorithm built on a Dyna-style model-based RL architecture, in which an ensemble-based Epistemic Reward Model (ERM) serves both as a reward posterior for exploration and as a learned preference-feedback model for synthetic labeling. AERO uses ensemble vote entropy, a Query-by-Committee disagreement measure, combined with an adaptive thresholding mechanism to decide which contexts require oracle feedback and which can instead receive ERM-based synthetic preference labels, concentrating expensive labels where they are most informative. Empirical results across different model families and scales show that AERO achieves highly competitive alignment performance while reducing the required oracle-query budget compared to online DPO and strong active-exploration baselines.
• 中文摘要 通过人类反馈强化学习（RLHF）将大型语言模型（LLMs）与人类偏好对齐，已在多种任务中取得了显著改进，但获取偏好标签的成本仍然是比对流程中的主要瓶颈。虽然许多近期比对方法通过对候选回答进行主动探索提升了样本效率，但它们通常在不同上下文中对预言机查询一致，即使是在已学习的奖励估计足够可靠以提供合成标签的上下文中也是如此。我们提出了自适应集合不一致路由（AERO）oracle反馈算法，这是一种基于Dyna风格模型基础强化学习架构的全在线RLHF算法，其中基于集成的认知奖励模型（ERM）既作为探索的后验奖励，也作为合成标签的学习偏好反馈模型。AERO使用集合投票熵（一种委员会查询异议衡量指标）结合自适应阈值机制，决定哪些上下文需要oracle反馈，哪些可以接收基于ERM的合成偏好标签，并将昂贵标签集中在最具信息量的地方。跨不同模型家族和尺度的实证结果表明，AERO在实现高度竞争的对齐性能的同时，降低了在线DPO和强有力的主动探索基线所需的oracle-query预算。

BlenderFORGE: Framework for Optimizing Reactive 3D-Graphics Editing Ability of MLLMs

BlenderFORGE：优化MLLM反应式3D图形编辑能力的框架

• Authors: Zilin Guo, Tianrui Zhang, Yi Rong, Yichen Liu, Yuxin Guo, Jingcheng Ni, Lewei Lu, Zehuan Wu, Dan Xu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10768
• Pdf link: https://arxiv.org/pdf/2611.10768
• Abstract Multimodal Large Language Models (MLLMs) have shown promise as interactive agents, yet precise, programmatic 3D graphics editing remains difficult to train because existing systems largely rely on closed-source model APIs and lack deterministic, state-verifiable execution environments. We present BlenderFORGE, a trainable data-environment-optimization framework for open-weight MLLM agents that edit explicit Blender scenes through Blender Python (bpy) code generation. BlenderFORGE targets four fundamental state-verifiable editing task families: object placement, shape-key editing, lighting adjustment, and material modification. It integrates three components: (1) a scalable perturbation-based data generation pipeline built from curated assets and large-scale 3D scene repositories; (2) an isolated Blender Sandbox for deterministic code execution, structured state export, and closed-loop reward computation; and (3) a two-stage training pipeline that uses offline teacher trajectories for cold-start Supervised Fine-Tuning (SFT) and sandbox-computed rewards for multi-turn tool-agentic Reinforcement Learning (AgentLoop-RL). Experiments across the four task families show that the resulting open-weight Qwen3-VL-8B and Qwen3.5-9B BlenderFORGE agents substantially improve over their base models and achieve competitive performance with several zero-shot proprietary MLLM baselines on selected metrics. Additional cross-benchmark evaluations and camera-reframing experiments provide evidence of transfer beyond the original training setting.
• 中文摘要 多模态大型语言模型（MLLM）作为交互代理展现出潜力，但精准的程序化3D图形编辑仍难以训练，因为现有系统大多依赖闭源模型API，缺乏确定性、可状态验证的执行环境。我们介绍BlenderFORGE，一个可训练的数据环境优化框架，面向开放权重MLLM代理，通过Blender Python（bpy）代码生成编辑显式Blender场景。BlenderFORGE针对四大基本可状态验证编辑任务族：对象放置、形状键编辑、光照调整和材质修改。它集成了三个组成部分：（1）由策划资产和大规模3D场景仓库构建的可扩展基于微扰的数据生成流水线;（2）一个独立的Blender沙盒，用于确定性代码执行、结构化状态导出和闭环奖励计算;以及（3）采用两阶段培训流程，利用离线教师轨迹进行冷启动监督微调（SFT）和沙盒计算奖励，用于多回合工具代理强化学习（AgentLoop-RL）。跨四大任务家族的实验显示，所得的开权重Qwen3-VL-8B和Qwen3.5-9B BlenderFORGE代理相较其基础模型显著提升，并在选定指标上与多个零样本专有MLLM基线实现竞争性能。额外的跨基准评估和摄像头重构实验提供了超越原始训练设置的迁移证据。

A Gamified Framework for Conflict-Driven Design: Discovering Strategies for Collaborative Resilience with Multi-Agent Learning

一个游戏化的冲突驱动设计框架：通过多智能体学习发现协作韧性策略

• Authors: Jin Gao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.10841
• Pdf link: https://arxiv.org/pdf/2611.10841
• Abstract Conflicting stakeholder objectives make it difficult to coordinate investments in urban resilience. We model conflict-driven urban design as a non-zero-sum digital board game in which stakeholder agents allocate scarce resources and balance individual payoffs against shared urban development and resilience objectives. We use multi-agent reinforcement learning (MARL) to train policies and examine their behavior across three settings. On a 4 x 4 board, learned policies reach about 84% of the total return found by a simulated annealing benchmark. On an 8 x 8 board with spatial and resource constraints, trained agents favor projects that improve resilience in self-play. In a 12 x 12 simulation initialized from historical maps of Tatsumi, Tokyo, trained agents achieve a mean total score of 452.1, compared with 357.3 for a random-policy baseline across 10,000 simulation runs. Gains in adaptability, sustainability, flood resistance, and greenery come with reductions in vitality and density. A human-in-the-loop interface supports replay and lets participants take over stakeholder decisions, providing a testbed for exploring these trade-offs in participatory urban planning.
• 中文摘要 利益相关者目标冲突使得协调城市韧性投资变得困难。我们将冲突驱动的城市设计建模为非零和数字棋盘游戏，利益相关者代理分配稀缺资源，并平衡个人收益与共享的城市发展和韧性目标。我们使用多代理强化学习（MARL）训练政策并在三种环境中分析其行为。在4×4的板块上，学习的策略达到模拟退火基准总回报的约84%。在8×8板块中，空间和资源约束，受训代理偏好提升自我游戏韧性的项目。在以东京辰巳历史地图为基础的12×12模拟中，受训代理平均总得分为452.1，而随机政策基线在1万次模拟中为357.3。适应性、可持续性、抗洪性和绿化的提升伴随着活力和密度的降低。人机界面支持重播，让参与者接管利益相关者的决策，为探索参与式城市规划中的权衡提供了试验平台。

What Broke & Why: Practical Lessons from Reinforcement Learning Post-Training

坏了什么以及原因：培训后强化学习的实用经验教训

• Authors: Luv Verma
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10844
• Pdf link: https://arxiv.org/pdf/2611.10844
• Abstract Reinforcement learning with verifiable rewards (RLVR) has become an important way to improve how language models reason, solve problems, and use tools. But getting it to work in practice is not always straightforward. Training can fail because a model receives no useful reward, learns to exploit the reward system, stops exploring, or appears to improve when the evaluation itself is misleading. In this book, we give a practical introduction to reinforcement learning post-training for language models, focusing on what works, what fails, and why. We begin with the foundations of post-training, explain GRPO and the training loop step by step, and then follow five training programs covering math reasoning, search agents, training stability, mixture-of-experts models, and software engineering tasks. Each program shows the problems encountered, the changes made, and the lessons learned. The book also examines what reinforcement learning actually changes inside a model. The remaining chapters provide a practical reference for reward design, evaluation, training tool-using agents, debugging common failures, reproducibility, rollout engines, and hardware. Written for researchers and practitioners working with open models, the book aims to help readers understand how to train models, recognize when something goes wrong, and decide what to try next.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为改善语言模型推理、解决问题和使用工具的重要方式。但使其在实践中发挥作用并非总是简单。训练失败可能因模型未获得有用奖励、学会利用奖励系统、停止探索，或在评估本身误导时表现出改进。本书为语言模型后训练强化学习提供了实用介绍，重点介绍哪些有效、哪些失败及原因。我们从后训练基础开始，逐步解释GRPO和训练循环，然后跟随五个培训程序，涵盖数学推理、搜索代理、训练稳定性、专家混合模型和软件工程任务。每个程序展示了遇到的问题、所做的变化和经验教训。书中还探讨了强化学习在模型内部实际发生了哪些变化。其余章节为奖励设计、评估、工具使用智能体训练、常见故障调试、可重复性、推广引擎和硬件提供了实用参考。本书面向从事开放模型的研究人员和从业者，旨在帮助读者理解如何训练模型、识别问题并决定下一步尝试什么。

The Surprising Effectiveness of Deleting Weights in LLM Post-Training

在训练后删除LLM权重的惊人效果

• Authors: Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou, Mengye Ren
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.10866
• Pdf link: https://arxiv.org/pdf/2611.10866
• Abstract How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.
• 中文摘要 预训练大型语言模型（LLM）只需做多少变化，就能获得新能力？我们发现，在三种LLM后训练模式中，仅对其权重的0.05%归零且不做其他修改，就能在规模上匹配全微调（FFT）：可验证奖励的强化学习、监督微调和策略上蒸馏。我们的方法“删除调优”学习基于变换器块门投影的二进制掩码。以下有两个实际优势：训练后的掩码比对应的FFT检查点小数百到数千倍，且遗忘更少，这解决了持续学习中的核心挑战。在强化学习中，其前置任务精度保持在基础水平附近，而FFT、LoRA和随机稀疏微调则损失最多5.4分。为理解为何仅仅掩蔽就足够，我们分析了在强化学习下学习权重变化的几何结构，采用稳定秩、谱漂移和主权重重叠。与LoRA和随机稀疏微调相比，缺失调优在三种指标上最接近FFT。

DreamVLA: From Generated Human Motions to Humanoid Vision-Language-Action Policies

DreamVLA：从生成的人类动作到类人视觉-语言-行动政策

• Authors: Kevin Park, Izat Rasyad, Dvij Kalaria, Shankar Sastry
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.10872
• Pdf link: https://arxiv.org/pdf/2611.10872
• Abstract Training vision-language-action (VLA) policies typically depends on substantial robot demonstration data, making human teleoperation a costly and cumbersome bottleneck, particularly for whole-body humanoid interaction. Reinforcement learning in simulation offers an alternative for scaling this data, but designing task-specific rewards remains challenging. To address this, we propose DreamVLA, which uses generated motion trajectories to guide reinforcement learning (RL) in simulation, converting them into reliable task execution trajectories. These trajectories train VLA models in simulation without new task-specific teleoperation data. Our central contribution is a reference-conditioned DAgger distillation procedure: after a failed learner rollout, we restart the RL expert from a sampled learner state with the same reference suffix and aggregate successful corrective trajectories, addressing the distribution shift left by offline imitation. We evaluate picking from a platform, opening a drawer, pressing a button, and walking and picking from a platform, with ablations of DAgger iterations, residual RL, and generated dataset scale. Project website: this http URL.
• 中文摘要 视觉-语言-动作（VLA）策略的训练通常依赖大量机器人演示数据，使得人类远程操作成为成本高昂且繁琐的瓶颈，尤其是在全身人形交互中。模拟中的强化学习为数据扩展提供了替代方案，但设计任务特定奖励仍具挑战性。为此，我们提出了DreamVLA，利用生成的运动轨迹引导模拟中的强化学习（RL），将其转换为可靠的任务执行轨迹。这些轨迹在模拟中训练VLA模型，无需新的任务特定远程操作数据。我们的核心贡献是参考条件DAgger蒸馏过程：在学习者推广失败后，我们从带相同参考后缀的抽样学习者状态重启RL专家，并汇总成功的纠正轨迹，解决离线模仿留下的分布偏移。我们评估从平台选择、打开抽屉、按下按钮，以及走动和从平台选择，并结合DAgger迭代、残余强化学习和生成的数据集规模的消融。项目网站：此 http URL。

BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training

BRAT：稳定多阶段LLM后训练的有界残差对齐训练

• Authors: Xunlan Zhou, Yuzhe Zhao, Xuhong Huang, Feng Chen, Xuanlin Chen, Lang Feng, Bo An
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10873
• Pdf link: https://arxiv.org/pdf/2611.10873
• Abstract Reinforcement learning for large language model post-training often reuses each batch of sampled rollouts for several gradient passes to amortize the cost of generation. Under such multi-pass reuse, existing objectives fail in different ways: unclipped updates collapse as the policy drifts away from the sampling policy, adding a negative baseline avoids collapse but reinforces every sampled response, and PPO-style ratio clipping cuts off gradients abruptly. We first ask why reuse is fragile. Measuring how well an update estimated from a small rollout group aligns with the KL-regularized target on held-out rollouts of the same prompts, we find that this alignment drops by nearly an order of magnitude from in-sample to held-out rollouts across three base models, suggesting that repeated passes largely overfit the sampled group. Motivated by this finding, we propose Bounded Residual-Aligned Training (BRAT), motivated by a proximal update toward a bounded reweighting of the sampling policy. BRAT assigns each token a gradient weight that decreases linearly with its probability ratio and vanishes at a trust-region boundary, and normalizes group-centered rewards by their maximum magnitude, keeping advantages zero-mean and bounded without extra hyperparameters. With four reuse passes, BRAT avoids training collapse, limits policy drift, and achieves higher average accuracy with lower seed variance on most benchmarks.
• 中文摘要 大型语言模型训练后的强化学习通常会多次重复使用每批采样的展开，以摊销生成成本。在这种多重循环重复使用下，现有目标以不同方式失效：未剪辑更新因策略偏离采样策略而崩溃;添加负基线避免崩溃但强化每个采样响应;PPO式裁剪则突然切断梯度。我们首先问为什么重用是脆弱的。测量从小规模推广组估计的更新与同一提示的保留推送中KL正则化目标的匹配程度，我们发现在三个基模型中，这种对齐从样本内到保留推送几乎下降了一个数量级，表明重复遍检在很大程度上过度拟合了被抽样的组。基于这一发现，我们提出了有界残差对齐训练（BRAT），其动机是通过对采样策略进行有界重权的近端更新。BRAT 为每个代币分配一个梯度权重，权重随概率比线性递减，并在信任区域边界消失，并以最大幅度规范组中心奖励，保持优势为零均值且有界且无额外超参数。通过四次重用，BRAT 避免训练崩溃，限制策略漂移，并在大多数基准测试中实现更高平均准确率且种子方差更低。

BRACE: Belief-guided Residual Adaptive Control for Safe Navigation under Unknown Disturbances

BRACE：信念引导残留自适应控制，在未知扰动下安全导航

• Authors: Jaeyoun Choi, Niklas Schmid, Oswin So, Augustinos Saravanos, Chuchu Fan
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.10938
• Pdf link: https://arxiv.org/pdf/2611.10938
• Abstract Disturbances encountered during autonomous navigation generally vary across space, often discontinuously, as obstacles redirect wind and changing terrain alters traction. Tracking controllers typically assume sufficient control authority to mitigate disturbances and maintain safety; the regime where this assumption fails remains underexplored. We present Belief-guided Residual Adaptive Control (BRACE), a hierarchical reinforcement learning (RL) framework for reach-avoid navigation under unknown, bounded, spatially varying disturbances. BRACE infers a disturbance belief from model prediction errors and uses that to condition a high-level policy that adapts the reference trajectory. The policy combines a nominal and a belief-guided residual term, trained jointly with a likelihood-ratio advantage reweighting heuristic. We evaluate BRACE in three simulated quadcopter environments with abruptly switching wind fields, and validate it on Crazyflie hardware with fan-generated disturbances. Across these settings, BRACE improves reach-avoid performance over model predictive control and RL baselines.
• 中文摘要 自主导航过程中遇到的干扰通常在空间中变化，且常常不连续，因为障碍物会改变风向，变化的地形会改变牵引力。跟踪控制器通常假设拥有足够的控制权以减轻干扰并维持安全;但该假设失效的状态仍未被充分探索。我们提出了信念引导残差自适应控制（BRACE），这是一种用于在未知、有界、空间变化扰动下实现距离-避免导航的层级强化学习（RL）框架。BRACE从模型预测误差中推断干扰信念，并以此为条件制定一个适应参考轨迹的高级策略。该策略结合了名义值和信念引导的残差项，并与似然比优势加权启发式联合训练。我们在三种模拟四旋翼环境中评估BRACE，风场突然切换，并在Crazyflie硬件上用风扇产生的扰动验证了BRACE。在这些设置下，BRACE在距离-避免性能上优于模型预测控制和强化学习基线。

DPDP: Diffusion Policy-Steered Diffusion Planner for Offline Reinforcement Learning

DPDP：离线强化学习的扩散策略导向扩散规划器

• Authors: Ruyi An, Mingyuan Zhou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10993
• Pdf link: https://arxiv.org/pdf/2611.10993
• Abstract In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline this http URL is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.
• 中文摘要 在离线强化学习中，扩散模型承担了两个互补优势的角色。通常由批评者引导，扩散策略具有价值意识，偏好高回报、行为支持的行动，但它目光短浅：从未明确说明其打算访问的状态。扩散规划器具有远见，综合了这样一个长期视野的轨迹，但它仅在执行时以行为驱动，实现每个计划中的转移仅使用离线，而该http URL恰好在另一个较弱的地方强。我们引入DPDP将它们整合到规划者的执行步骤，即计划中的过渡成为行动。将规划者的逆动态执行者视为行动的后验：环境的过渡似然将动作与计划中的下一状态联系起来，而行为先验决定其在离线数据下的权重。用价值优化的扩散策略替代先验，可以保留计划中的转换概率，并优先考虑有价值的行动来服务于计划。我们通过两种方式针对最终的净动作分布进行目标。无训练采样器在匹配的动作噪声水平下组合策略和逆动力学分数，并应用朗之文修正;该路径使用共享动作噪声计划和近似共同终端先验。偏好目标则将转移摊销到单个扩散执行器中。在D4RL中，DPDP改进了其所构建的规划器，保持在移动中与反应扩散策略的竞争力，并在长视野导航中领先。

Differentiable Agent-Based Modeling for Multi-Echelon Supply Chains: Gradient-Based Optimization of Heterogeneous Policies at Scale

多层级供应链的可微智能体建模：大规模异构政策的梯度优化

• Authors: Bulent Soykan, Murtadha Alsayegh, Sean Mondesire, Ghaith Rabadi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.10999
• Pdf link: https://arxiv.org/pdf/2611.10999
• Abstract Global supply chains rely on the interactions of thousands of heterogeneous agents, posing significant challenges for optimization. While traditional optimization often assumes homogeneity to ensure tractability, Reinforcement Learning struggles with sample inefficiency when scaling to high-dimensional state spaces. We propose a Differentiable Agent-Based Model for multi-echelon inventory management. By implementing stochastic dynamics via differentiable tensor operations, we enable direct gradient propagation through the simulation. We validate the framework by recovering theoretical Newsvendor optima without supervision. Crucially, we demonstrate ``Mass Personalization'' by optimizing 10,000 distinct agent policies simultaneously in seconds, achieving a 53% cost reduction compared to scalar baselines. Furthermore, we utilize gradient sensitivity analysis to generate causal heatmaps, accurately recovering inherent lead-time delays from the computation graph. This approach bridges the gap between simulation and optimization, enabling the efficient training of self-correcting Digital Twins.
• 中文摘要 全球供应链依赖数千个异构代理的相互作用，给优化带来了重大挑战。传统优化通常假设同质性以确保可处理性，而强化学习在高维状态空间扩展时则面临样本效率低落的问题。我们提出了一种可微的基于代理的多阶级库存管理模型。通过通过可微张量运算实现随机动力学，我们实现了在模拟中的直接梯度传播。我们通过在无监督的情况下恢复理论上的Newsvendor最优解来验证框架。关键是，我们通过在数秒内同时优化1万个不同代理策略，展示了“大规模个性化”，相比标量基线降低了53%的成本。此外，我们利用梯度敏感性分析生成因果热图，准确还原计算图固有的前置时间延迟。这种方法弥合了仿真与优化之间的鸿沟，使得高效训练自纠正的数字孪生成为可能。

Uncertainty-Guided Reward Labeling for Reinforcement Learning under Limited Feedback

在有限反馈下进行不确定性引导的奖励标签，用于强化学习

• Authors: Renhao Zhang, Shreyas Chaudhari, Bruno Castro da Silva
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11091
• Pdf link: https://arxiv.org/pdf/2611.11091
• Abstract In reinforcement learning from limited feedback (RLLF), only a small fraction of an offline dataset can be labeled with rewards, and the central question is which samples should be labeled to learn a high-performing policy from the resulting partially labeled dataset. Prior work formalized this as a reward-selection problem by focusing on the selection stage while treating downstream policy learning as a black box, in a regime where queried rewards are not retained for reward-model training. We instead study the retained-label setting, where queried rewards can be stored and used to fit a reward model before policy learning. We bound the suboptimality of the learned policy in terms of two sources of error: one arising from offline RL on a fixed, offline dataset, and the other from uncertainty in the reward model. Since reward selection cannot change the offline dataset, the limited labeling budget must be used to strategically reduce reward uncertainty. Motivated by RLLF's observation that useful rewards tend to keep the agent on high-return trajectories, we introduce Successor-Guided Uncertainty REduction (SURE), a novel method that uses successor features to select rewards that are both uncertain and associated with states from which high-valued states are reachable. We formally derive SURE from a bound-induced design objective and characterize its exact one-step marginal gain. Empirically, SURE reaches near full-feedback performance with few reward labels across a variety of domains and offline-RL backbones, yielding an effective method for feedback-efficient reinforcement learning.
• 中文摘要 在有限反馈强化学习（RLLF）中，离线数据集中只有一小部分可以用奖励标记，核心问题是应标记哪些样本以从部分标记数据集中学习高效能策略。先前工作通过关注选择阶段，将下游策略学习视为黑箱，将该问题形式化为奖励选择问题，即查询奖励不保留用于奖励模型训练的机制。我们转而研究保留标签设置，即查询奖励可以存储并用于拟合奖励模型，然后再进行策略学习。我们将学习策略的次优性界定为两个错误来源：一是源于固定离线数据集上的离线强化学习，二是奖励模型中的不确定性。由于奖励选择无法改变离线数据集，有限的标签预算必须用于战略性地降低奖励不确定性。基于RLLF观察到有用奖励往往使智能体保持高回报轨迹，我们引入了继承引导不确定性减少（SURE）新方法，利用继承特征选择既不确定且与高价值状态可达状态相关的奖励。我们正式从绑定诱导设计目标推导SURE，并精确表征其一步边际增益。实证上，SURE在多个领域和离线强化骨干中几乎达到了几乎全反馈性能，且奖励标签较少，成为高效反馈强化学习的有效方法。

From Festival Video to the Stage: A Sim-to-Real Ladder for Long-Horizon Expressive Whole-Body Dance on a Commercial Humanoid

从节日视频到舞台：商业类人生物上长距离表现力全身舞蹈的模拟到真实阶梯

• Authors: Hesham Shehata (Research & Development Division, Arithmer Inc., Tokyo, Japan)
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.11127
• Pdf link: https://arxiv.org/pdf/2611.11127
• Abstract We present a pipeline for transferring Awa Odori, a traditional Japanese festival dance, from monocular video to a commercial Unitree G1 EDU humanoid. The pipeline combines human mesh recovery, kinematic retargeting, reinforcement learning for motion tracking in simulation, and onboard deployment via ONNX. Training ran on a single RTX 5070 GPU. An eleven-version deployment ladder evaluated on hardware examines three interventions. We detect and repair silent corruption of deployment artifacts, model state-estimator drift during training, and edit the reference motion to support stable execution and a controlled ending. The final policies achieved three consecutive rope-free completions of a 43-second routine, a 46-second performance with automatic handoff, and a continuous 9.7-minute rope-free performance composed from the captured motion. In a live handover, a single remote request let the robot start from and return to the manufacturer's walking controller on stage, with a measured torque-free window of 45-140 ms. We report the training dynamics and hardware outcomes and discuss failure boundaries involving compliant flooring and occlusion in monocular capture.
• 中文摘要 我们提出了将传统日本节日舞蹈阿波踊从单眼视频传输到商业Unitree G1 EDU人形的流水线。该流水线结合了人机网格恢复、运动学重定向、仿真中动作跟踪的强化学习以及通过ONNX进行机载部署。训练运行在一块RTX 5070显卡上。一个基于硬件评估的十一版部署梯梯分析了三种干预措施。我们检测并修复部署工件的无声损坏，训练期间建模状态估计漂移，并编辑参考动作以支持稳定执行和受控结束。最终策略实现了连续三次无绳索完成43秒的例行程序、46秒的自动切换表演，以及由捕获动作组成的连续9.7分钟无绳索表演。在实时切换中，单次远程请求允许机器人从制造商的舞台行走控制器开始并返回，测量出的无扭矩窗口为45-140毫秒。我们报告训练动力学和硬件结果，并讨论单眼捕捉中涉及合规地板和遮挡的故障边界。

From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models

从游戏生成到游戏玩法：视觉-语言-行动模型的全自动能力训练

• Authors: Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.11136
• Pdf link: https://arxiv.org/pdf/2611.11136
• Abstract Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
• 中文摘要 训练视觉-语言-动作（VLA）模型以适应不同游戏的游戏指令，受限于收集演示的成本。我们引入了一个自动生成游戏环境和专家演示的框架。一个由LLM驱动的代理利用游戏构建工具构建针对特定技能的可执行环境和任务。程序化专家随后利用特权模拟状态生成对齐指令、渲染的观察以及键盘和鼠标操作。我们通过在150.3小时的合成Minecraft和Roblox游戏过程中，通过行为克隆后训练一个4B视觉语言模型来训练我们的游戏策略Script2Play，无需人工演示或强化学习。对多个公开游戏基准测试的评估表明，Script2Play能够有效遵循游戏指令，并在多个游戏引擎间进行推广，凸显了有针对性的合成监督的有效性。

DISCOVER: Online Variance-Guided Data Discovery for Budgeted Multimodal GRPO

发现：预算多模态GRPO的在线方差引导数据发现

• Authors: Peyman Gholami, Shivam Chandhok, Saber Malekmohammadi, Shayan Shams, Robert Xiao, Leonid Sigal
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11145
• Pdf link: https://arxiv.org/pdf/2611.11145
• Abstract Recent advances in reinforcement learning have made verifiable rewards a central post-training paradigm for improving multimodal reasoning. However, methods such as GRPO remain data- and compute-inefficient, spending rollout and annotation budget on prompts that provide little learning signal. We propose DISCOVER, an online data-discovery framework for budgeted multimodal GRPO that enables vision-language models to dynamically identify prompts that are most useful for the current policy. DISCOVER first organizes the multimodal sample pool into semantic clusters and queries a small set of density-diverse representative samples to obtain initial reward observations. During training, it estimates the utility of unannotated samples by transferring sparse reward feedback from queried samples. Since sample utility changes as the policy evolves, DISCOVER dynamically updates these estimates with recency-weighted rollout observations. It then selects only high-utility prompts for annotation using a policy-conditioned score that accounts for reward variance and difficulty. Once the discovery budget is exhausted, DISCOVER continues with utility-based replay. Experiments on ViRL39K and VLAA-Thinking with Qwen3-VL-2B show that DISCOVER substantially improves multimodal mathematical reasoning under a strict 10% unique-sample budget, outperforming budget-matched random selection, hard-example sampling, full-pool GRPO under the same training-step budget, and online selection baselines. Ablations further validate the importance of semantic coverage, online reward transfer, and stable discovery cadence, establishing DISCOVER as a practical framework for data-efficient multimodal RL post-training.
• 中文摘要 强化学习的最新进展使可验证奖励成为提升多模态推理的核心训练后范式。然而，像GRPO这样的方法在数据和计算效率上仍然低效，在提供学习信号较少的提示上花费了推广和注释预算。我们提出了DISCOVER，一个用于预算化多模态GRPO的在线数据发现框架，使视觉语言模型能够动态识别对当前政策最有用的提示。DISCOVER首先将多模态样本池组织为语义簇，并查询一小部分密度多样化的代表样本以获得初始奖励观测值。在训练过程中，它通过传递来自查询样本的稀疏奖励反馈来估算未注释样本的效用。随着策略的发展，样本效用会发生变化，DISCOVER会动态更新这些估计值，并结合最近加权的推广观察值。然后仅选择高效用提示进行注释，采用考虑奖励方差和难度的策略条件评分。一旦发现预算耗尽，DISCOVER继续基于效用的重放。在ViRL39K和QWEN3-VL-2B的VLAA思维实验显示，DISCOVER在严格的10%唯一样本预算下显著提升多模态数学推理能力，优于预算匹配随机选择、硬样本抽样、同一训练步预算下的全池GRPO和在线选择基线。消融进一步验证了语义覆盖、在线奖励转移和稳定发现节奏的重要性，确立了DISCOVER作为数据高效多模态强化学习后实用框架的地位。

The Race Is Decided Early! A Kernel Law Predicts Lock-In in Majority-Vote Test-Time RL

竞选提前决出胜负！一项核心定律预测多数票测试时间的锁定

• Authors: Jaeho Hwang, Sehong Park, Daejin Choi, Uiwon Hwang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11165
• Pdf link: https://arxiv.org/pdf/2611.11165
• Abstract Test-time reinforcement learning with majority-vote rewards trains a language model on unlabeled questions by rewarding the samples that agree with its most frequent answer. During training, each question locks into one answer, on hard questions often a wrong one, but which answer it locks into is known only after training. In this paper, we show that this outcome is decided early, in a short race driven by sampling noise between the answers that are close at the start, and we derive the probability of each outcome. With a separate logit for each answer, the runner-up wins with a normal tail probability of the initial gap, scaled by an explicit noise constant. In a network, however, the answer logits depend on shared parameters, and through the kernel of these parameters the updates of the trailing answers also change the gap of the two leaders. This spillover shifts the boundary between the starts won by each leader, and a closed form computes the shifted probability from a kernel probe and the start logits. In LoRA experiments with three models from two families, predictions made before training with the kernel frozen at its start value forecast which answer each question locks into better than a logistic model refitted on the outcomes, although they use no fitted constants. The closed form matches these predictions on the runs in which a check before training finds that the two leaders stay above the other answers, and it explains why a law that ignores the spillover needs a strongly inflated step.
• 中文摘要 带有多数票奖励的测试时间强化学习通过奖励与最常见答案一致的样本，训练语言模型在无标签问题上。在训练过程中，每个问题锁定一个答案，难题常常是错误的，但锁定的答案只有训练后才能知道。本文展示了该结果在早期决定，通过起始接近答案间的采样噪声驱动的短距离竞赛，并推导每个结果的概率。每个答案单独使用logit，亚军以初始差距的正规尾概率获胜，该概率由显式噪声常数缩放。然而，在网络中，答案logit依赖于共享参数，通过这些参数的核，后方答案的更新也改变了两个领先者的差距。溢出会改变每个领先者赢得的起始边界，闭式通过核探针和起始对数计算平移概率。在LoRA实验中，来自两个模型族的三个模型，训练前用核冻结在起始值预测（每个问题的答案）的预测比基于结果调整的逻辑模型更准确，尽管它们不使用拟合常数。闭式在训练前检查发现两个领先者高于其他答案的运行中与这些预测相匹配，这也解释了为何忽略溢出的定律需要一个大幅膨胀的步骤。

Primitives and Paths: Predicting What Post-Training Can Do from What the Base Model Already Has

原件与路径：从基础模型已有内容预测后训练能做什么

• Authors: Anqi Qu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11260
• Pdf link: https://arxiv.org/pdf/2611.11260
• Abstract Does reinforcement learning teach language models new capabilities, or merely amplify ones acquired during pre-training? The literature offers evidence for both: pass@$k$ studies suggest RL reweights latent behaviours while some controlled composition studies demonstrate gains beyond simple amplification. We argue that this question has no answer independent of the capability state of the base model, and propose a two-dimensional state space for post-trainability. The first axis, \emph{effective full-solution support}, measures whether successful trajectories can be sampled under a feasible exploration budget. The second axis, \emph{atomic primitive coverage}, measures whether the constituent skills of a solution can be executed when the burden of composition is removed. Together, these dimensions motivate three regimes -- amplification, composition-limited, and acquisition-limited -- each predicting a distinct empirical signature and calling for a distinct intervention. We propose a prospective measurement protocol and investigate the framework using synthetic composition tasks across three small language models. We find that primitive mastery alone does not guarantee that RL can learn a composition. Checkpoints that execute every primitive reliably can fail to improve when complete solutions are never sampled during training, whereas modest composition supervision that establishes nonzero support enables substantial RL gains. The framework reframes mid-training as preparation for post-training: its role is to move models into states from which RL can operate effectively.
• 中文摘要 强化学习是教授语言模型新能力，还是仅仅放大了预训练中获得的能力？文献为两者提供了证据：pass@$k$研究表明强化学习会重权潜在行为，而一些受控组合研究则显示超越简单放大的益处。我们认为，这个问题没有独立于基础模型能力状态的答案，并提出了一个二维的后可训练状态空间。第一轴\emph{有效全解支持}衡量在可行探索预算下是否能采样成功轨迹。第二轴\emph{原子原始覆盖}衡量在消除组合负担后，解决方案的组成技能是否能够被执行。这两个维度共同推动三种模式——放大、组成限制和获得限制——每个模式预测不同的经验特征并呼吁不同的干预。我们提出了一种前瞻性测量协议，并利用三个小语言模型中的合成写作任务研究该框架。我们发现，仅靠原始精通并不能保证强化学习一个作文。当训练期间从未采样完整解时，可靠执行每个原语的检查点可能无法提升;而适度的作文监督建立非零支持则能实现显著的强化学习提升。该框架将训练中段重新定位为为后训练的准备：其作用是将模型移动到强化学习能够有效运行的状态。

When Credit Turns Against Tools: Transition-Aware Credit Assignment in Agentic Reinforcement Learning

当学分反对工具：代理强化学习中的过渡感知学分作业

• Authors: Xiangyu Lu, Zhanke Zhou, Jiazhe Ning, Chentao Cao, Jiangchao Yao, Bo Han
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11330
• Pdf link: https://arxiv.org/pdf/2611.11330
• Abstract Agentic reinforcement learning with verifiable rewards trains language models through multi-turn tool interactions, yet their reasoning performance and tool-calling ability can decline during training. We trace this degradation to three coupled mechanisms: (1) tool-token imbalance produces negative overall tool credit when incorrect trajectories contain more tool tokens per turn than correct ones; (2) aggregation amplification can further strengthen this negative signal as token-mean aggregation gives longer incorrect trajectories greater weight in policy updates; and (3) failure-driven retries prolong trajectories as models tend to call tools again after execution errors, extending potential negative credit to recovery attempts. To counter this suppression, we propose Transition-Aware Credit Assignment (TACA), which targets the transition from reasoning to tool use through supervision of both call initiation and generation. Specifically, TACA uses Advantage Projection to enforce a nonnegative mean advantage on tool-call tokens within each prompt group through a minimal uniform shift. Additionally, TACA guides tool-use decisions with Decision Contrast, comparing rewards for tool-assisted and direct answers to the same prompt. Through Supervised Replay, it reinforces tool-call generation using calls from previously sampled correct trajectories. Across models and tasks, TACA improves performance and sustains more reliable tool interaction during training, raising mean accuracy over MT-GRPO on six reasoning benchmarks by 4.61 percentage points for Qwen2.5-1.5B and 3.61 for Qwen2.5-3B. Our code is available at this https URL.
• 中文摘要 带有可验证奖励的代理强化学习通过多回合工具交互训练语言模型，但其推理性能和工具调用能力在训练过程中可能下降。我们将这种退化归因于三种耦合机制：（1）当错误轨迹每回合工具代币数量多于正确路径时，导致整体工具信用负;（2）聚合放大进一步强化这一负信号，因为代币均值聚合赋予更长错误轨迹在策略更新中更高的权重;（3）失败驱动重试延长轨迹，因为模型在执行错误后倾向于再次调用工具，从而将潜在的负信用扩展到恢复尝试。为应对这种抑制，我们提出了过渡感知信用分配（TACA），通过监督调用发起和生成，实现从推理向工具使用的过渡。具体来说，TACA利用优势投射（Advantage Projection）通过最小的均匀偏移，强制每个提示组内的工具调用标记保持非负的平均优势。此外，TACA通过决策对比（Decision Contrast）指导工具使用决策，比较工具辅助和直接回答对同一提示的奖励。通过监督回放，它通过从先前采样的正确轨迹中调用来强化工具调用生成。跨模型和任务，TACA提升了训练期间的表现并维持了更可靠的工具交互，Qwen2.5-1.5B在六个推理基准测试中平均准确率提升了4.61个百分点，Qwen2.5-3B提升了3.61个百分点。我们的代码可在此 https URL 获取。

OPSRL-SSP: Optimistic Posterior Sampling for Stochastic Shortest Path with Minimax-Optimal Regret

OPSRL-SSP：带有极小最大-最优遗憾的随机最短路径乐观后验抽样

• Authors: Yiming Xu, Xian Wei, Cheng Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11337
• Pdf link: https://arxiv.org/pdf/2611.11337
• Abstract We study online reinforcement learning for the stochastic shortest path (SSP) problem with unknown transition dynamics. Although existing posterior-sampling methods perform well empirically on SSPs, their regret guarantees do not achieve the minimax-optimal order. We address this limitation by proposing OPSRL-SSP, an optimistic posterior-sampling algorithm for the SSP setting. The algorithm operates in epochs and uses a logarithmic, time-dependent schedule for the number of posterior samples. To induce optimism, we augment the transition model with a pseudo-state assigned an optimistic boundary value. We establish a high-probability regret bound of $\tilde{O}(B_\star \sqrt{SAK} + B_\star S^2 A)$, where $B_\star$ is an upper bound on the expected cost of the optimal policy, $S$ and $A$ are the numbers of states and actions, respectively, and $K$ is the number of episodes. In addition, we extend the analysis to general zero-cost SSPs using a universal cost-perturbation argument. The leading term matches the minimax lower bound $\Omega(B_\star \sqrt{SAK})$, thereby addressing the open problem raised by Jafarnia-Jahromi et al. [19] for posterior sampling in the SSP setting.
• 中文摘要 我们研究了随机最短路径（SSP）问题中未知转移动力学的在线强化学习。尽管现有的后验采样方法在SSP上经验表现良好，但其遗憾保证未能达到极小极大最优阶。我们通过提出OPSRL-SSP来解决这一限制，这是一种适用于SSP设定的乐观后验采样算法。该算法以纪元为单位工作，并使用对数、时间依赖的后验样本数量排程。为诱导乐观，我们用赋予乐观边界值的伪状态来增强过渡模型。我们建立了一个高概率的后悔上限 $\tilde{O}（B_\star \sqrt{SAK} + B_\star S^2 A）$，其中 $B_\star$ 是最优策略期望成本的上界，$S$ 和 $A$ 分别是状态和动作的数量，$K$ 是发作次数。此外，我们利用普遍成本-扰动论证将分析扩展到一般零成本 SSP。首项匹配极小极大下界 $\Omega（B_\star \sqrt{SAK}））$，从而解决了 Jafarnia-Jahromi 等人[19]提出的 SSP 背景下后验抽样的未解问题。

LLM-based Multi-Criteria Deep Reinforcement Learning for Dynamic Efficient 5G UPF Selection

基于LLM的多准则深度强化学习，实现动态高效5G UPF选择

• Authors: Abdelrahman Soliman, Ahmed Refaey Hussein, Lei Lei
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.11372
• Pdf link: https://arxiv.org/pdf/2611.11372
• Abstract User plane function (UPF) selection in 5G networks is an active area of research, with various limited approaches proposed to enhance network efficiency and manage resources effectively. In this paper, we address this gap in the related literature by proposing a Deep Reinforcement Learning (DRL) UPF selection method that utilizes a dynamic, multi-objective reward function and validating it on a real-world 5G emulation platform. The proposed method is based on the large language models' (LLMs) translation of network operator intents and the network data analytics function (NWDAF) abnormality alerts. The experimental results validated in the real-world experimental setup demonstrate that our dynamic DRL solution achieves an effective balance between latency, power consumption, and traffic distribution, outperforming the heuristic baselines and fixed DRL by 16.89%. These findings highlight the real-world applicability of our approach for dynamic, intent-driven UPF selection.
• 中文摘要 用户平面功能（UPF）选择在5G网络中是一个活跃的研究领域，提出了多种有限方法以提升网络效率和有效管理资源。本文通过提出一种深度强化学习（DRL）UPF选择方法，利用动态多目标奖励函数，并在现实世界5G仿真平台上进行验证，弥补了相关文献中的这一空白。该方法基于大型语言模型（LLMs）对网络运营商意图的翻译以及网络数据分析函数（NWDAF）异常警报。在实际实验环境中验证的实验结果表明，我们的动态DRL解决方案在延迟、功耗和流量分布之间实现了有效平衡，比启发式基线和固定DRL高出16.89%。这些发现凸显了我们方法在动态、意图驱动UPF选择中的实际应用性。

How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?

策略提炼能容忍多大的训练-推断不匹配？

• Authors: Cong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai, Youhe Jiang, Eiko Yoneki, Tuo Shi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11450
• Pdf link: https://arxiv.org/pdf/2611.11450
• Abstract On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least $1.93\times$ as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality $1.62\times$ sooner than synchronous BF16 training. Code is available at this https URL
• 中文摘要 策略提纯（OPD）在固定教师指导下，基于学生语言模型的响应进行训练。生成这些响应成本高昂，因此系统采用量化推理和异步推广以降低成本。然而，这两种方法都会引入展开引擎与被训练模型之间的训练-推断不匹配。这种不匹配会使强化学习（RL）不稳定，但其对OPD的影响尚不明确。本文探讨了OPD容忍不匹配的容忍度、其影响来源，以及训练系统如何利用这种容忍度。我们在同一量表上测量所有不匹配源，考察三对师生的量化，并在一对对上额外测试权重陈旧性和注入噪声。OPD容忍显著不匹配而不显著降低训练质量，但在更高水平下会下降，显示出实证错配预算。不匹配的质量成本和量化带来的不匹配都取决于模型，某些层对精度降低特别敏感。在受控比较中，OPD在使强化学习方法不稳定的展开配置下保持稳定。在权重噪声下，不匹配还会改变生成过程中达到的前缀，引入每个令牌修正无法解决的错误。为付诸实践，我们引入了轻量级准入检查，通过校准预算筛选出展配置，而无需对每个候选对象进行完整训练。模型特定的FP8配方保持训练质量，同时提供至少1.93美元乘以BF16的KV缓存令牌容量。结合FP8与异步展开，比同步BF16训练更快达到相同的训练质量，约1.62倍。代码可在此 https URL 获取

JoMoHa: Joint Model--Harness Optimization with Reinforcement Learning

JoMoHa：联合模型——带强化学习的线束优化

• Authors: Xudong Wu, Guangchen Lan, Mingqi Yuan, Jiayu Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11499
• Pdf link: https://arxiv.org/pdf/2611.11499
• Abstract The performance of language agents depends jointly on model weights and the harness. Optimizing either alone can be constrained by the other component, limiting overall performance gains. Our analysis of naive alternating training reveals progressive incompatibility between the model and harness, which increasingly interferes with task execution and may also undermine model training, leading to performance degradation. To coordinate the evolution of models and harnesses, we introduce JoMoHa, to our knowledge the first framework to jointly train an executable harness generation policy and a task execution policy through reinforcement learning. JoMoHa outperforms the evaluated weight-only and harness-only baselines and achieves the highest success rates among the evaluated methods on ALFWorld, WebShop, and DBBench, averaging 70.3% and reaching 97.0% on ALFWorld. By learning both task execution and runtime improvement from its own generated trajectories, JoMoHa performs recursive self-improvement (RSI).
• 中文摘要 语言代理的性能与模型权重和工具束共同依赖。单独优化其中一方可能被另一部分限制，限制整体性能提升。我们对朴素交替训练的分析显示，模型与框架之间存在逐渐不兼容，这日益干扰任务执行，也可能削弱模型训练，导致性能下降。为协调模型与框架的演进，我们引入了JoMoHa，这是我们已知的首个通过强化学习联合训练可执行框架生成工具和任务执行策略的框架。JoMoHa在ALFWorld、WebShop和DBBench测试中优于仅权重和仅框架基线，在评估方法中成功率最高，平均70.3%，在ALFWorld上达到97.0%。通过从自身生成的轨迹中学习任务执行和运行时改进，JoMoHa 执行递归自我改进（RSI）。

InfCoiL: Coordinated Planner-Controller Learning for Closed-Loop Physics-Based Human-Object Interaction

InfCoiL：基于物理的闭环人机交互的协调规划者-控制器学习

• Authors: Yude Zou, Yifei Yao, Yiwei Hao, Hanqing Wang, Xing Gao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11517
• Pdf link: https://arxiv.org/pdf/2611.11517
• Abstract Text- and goal-conditioned human-object interaction (HOI) provides an intuitive interface for specifying high-level intents in physics-based character animation. Mapping such intents to physically plausible, contact-rich behaviors naturally calls for a generative planner and a physics-based controller, yet existing planner-controller systems either execute plans open-loop or close the loop while keeping the planner fixed, so execution feedback never reshapes what is planned. We present InfCoiL, a closed-loop framework for physics-based HOI from coordinated planner-controller learning. InfCoiL couples a morphology-aware flow-matching planner with a unified multi-morphology controller through a shared morphology interface in an autoregressive plan-and-execute loop, and alternately fine-tunes both components with reinforcement learning from the same physical rollouts. To optimize the flow-matching planner online, we introduce Semantic Flow Policy Optimization (SFPO), which stochasticizes selected denoising transitions with channel-wise semantic covariance over heterogeneous HOI features and assigns group-relative, sequence-level rewards from complete closed-loop executions. On held-out OMOMO sequences, InfCoiL outperforms open-loop hierarchical baselines in goal-reaching task success and physical tracking accuracy. Project page: this https URL
• 中文摘要 文本与目标条件人-对象交互（HOI）为基于物理的角色动画中指定高层意图提供了直观的界面。将这些意图映射到物理上合理、接触丰富的行为，自然需要生成规划器和基于物理的控制器，但现有的规划器-控制器系统要么开环执行计划，要么闭环但保持计划器固定，因此执行反馈不会改变计划内容。我们介绍InfCoiL，一个基于物理的HOI闭环框架，基于协调规划器-控制器学习。InfCoiL通过共享形态接口将形态感知的流动匹配规划器与统一的多形态学控制器在自回归的计划与执行循环中结合，交替通过强化学习微调这两个组件。为了优化在线流程匹配规划器，我们引入了语义流策略优化（SFPO），该方法在异构HOI特征上随机化选定的去噪转移，具有通道语义协方差，并从完整的闭环执行中分配组相对序列级奖励。在持续进行的OMOMO序列上，InfCoiL在目标任务成功率和物理跟踪准确性方面优于开环层级基线。项目页面：此 https URL

Qubit-to-Motion: Traceable Action Provenance in Hybrid QRL for Robotic Bin-Packing Sequencing

量子比特到运动：机器人箱装测序中混合QRL动作来源可追溯

• Authors: Hassen Nigatu, Howard Li
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11580
• Pdf link: https://arxiv.org/pdf/2611.11580
• Abstract Hybrid quantum reinforcement learning has been explored for robotic decision-making, but evaluations often emphasize task success, return, or convergence, leaving unresolved whether quantum circuit outputs influence constrained actions and remain traceable through robot execution. We present \texttt{Qubit-to-Motion}, an evidence-centered framework for support-constrained sequencing and simulated robotic placement. Four Pauli-$Z$ readouts provide the sole learned input to the actor and critic, while a support mask enforces admissible object selection. Each decision is recorded and linked via step indices and SHA-256 fingerprints to a sealed route, deterministic Cartesian and inverse-kinematics waypoints, prescribed joint motion, and MuJoCo contact replay. A frozen checkpoint is evaluated across five deterministic 30-object scene variants. All 150 placements satisfy support, containment, and settling criteria. A fixed-state zero-readout intervention changes $90.0\pm0.0\%$ of decisions, demonstrating dependence on the joint circuit representation. Mean macro travel is $34.537\pm0.206$~m, inference requires $8.60\pm1.27$~ms per route, and replay achieves $8.764\pm0.220~\mu$m mean final position error with 0.356~mm maximum interpenetration. These results establish reproducible decision-to-contact traceability. The implementation is available at \href{this https URL}{\textcolor{linkpink}{the anonymous repository}}.
• 中文摘要 混合量子强化学习已被探索用于机器人决策，但评估往往强调任务成功、返回或收敛，导致量子电路输出是否影响受限动作并保持可通过机器人执行追踪尚未确定。我们提出了 \texttt{Qubit-to-Motion}，一个以证据为中心的支持约束序列和模拟机器人放置框架。四个 Pauli-$Z$ 读数为行为者和评论者提供唯一的学习输入，而支持掩码则强制可接受对象选择。每个决策都被记录并通过步索引和 SHA-256 指纹连接到封闭路线、确定性笛卡尔和逆运动学路径点、规定关节运动和 MuJoCo 接触回放。在五种确定性 30 对象场景变体中评估一个冻结检查点。所有150个配置都满足支持、遏制和稳定标准。固定状态零读出干预改变了90.0\pm0.0\%$的决策，显示了对联合电路表示的依赖。平均宏行程为$34.537\pm0.206$~m，推理每条路径需8.60\pm1.27$~ms，重放平均最终位置误差为$8.764\pm0.220~\mu$m，最大穿透率为0.356~mm。这些结果建立了可重复的决策到联系可追溯性。实现可在 \href{this https URL}{\textcolor{linkpink}{the annonous repository}} 获取。

Direct GPU Energy Measurements of Batch-Level Sharing in Mixture-of-Experts and Recurrent Models

专家混合模型与循环模型中批次级共享的直接GPU能量测量

• Authors: Rafael Pina, Varuna De Silva
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11696
• Pdf link: https://arxiv.org/pdf/2611.11696
• Abstract Energy claims for efficient neural networks are often inferred from operation counts. We report direct GPU energy measurements on two workloads that batch very differently, recurrent policies in GPU-vectorised multi-agent reinforcement learning (MARL) and mixture-of-experts (MoE) language models, and ask one question of both: can the rows of a batch be made to share computation, across the batch and over time, at a measured saving? On recurrent policies the answer so far is no. Shared activity patterns exist but rarely last one step, masking a frozen policy fails, and trained-in masks succeed only because a fixed mask is equivalent to a narrow GRU, at 11x the energy of a small dense policy in our implementation. Test-time steering adds nothing beyond re-picking on either workload. On MoE models the answer is yes, and granularity decides how much: on Qwen3-30B-A3B, batch-aware routing reads 25-31% fewer expert weights within KL 0.05, with benchmark changes within noise at 500 items, and uses 12-18% less energy per token on a fused forward. We also introduce persistence-aware OEA, a one-parameter rule that keeps the batch-aware cut with about 40% fewer new experts per step than native routing at batch 32-64, though its energy has not yet been measured. These are pilot measurements on one GPU type, and we state their limits.
• 中文摘要 高效神经网络的能量要求通常通过运算次数推断。我们在两种批处理方式截然不同的工作负载上报告了直接GPU能量测量，即GPU矢量化多智能体强化学习（MARL）和专家混合（MoE）语言模型中的重复策略，并提出两者的一个问题：批次的行能否以可衡量的节省方式共享计算？关于重复性策略，目前的答案是否定的。共享活动模式存在但很少能持续一步，屏蔽冻结策略失败，训练中的掩蔽仅因固定掩码相当于狭窄的格鲁（GRU），能量是我们实现中小型密集策略的11倍。测试时间引导除了重新选择任一工作负载外，没有任何贡献。在 MoE 模型中答案是肯定的，细粒度决定了具体程度：在 Qwen3-30B-A3B 中，批次感知路由在 KL 0.05 内的专家权重读数减少 25-31%，在噪声中基准变化为 500 条，且在融合前向上每个令牌的能量消耗减少 12-18%。我们还引入了持久感知 OEA，这是一条单参数规则，使批次感知切割在每步比原生路由少约 40% 的新专家数量，尽管其能量尚未被测量。这些是针对一种 GPU 类型的试点测量，我们明确说明了其限制。

Data-Driven Precision Control of Soft Helicoid Robots

软螺旋机器人的数据驱动精密控制

• Authors: Roshan S. Kaundinya, Mark Leone, Roberto Sabella, Hugo Buurmeijer, Luis Pabon, John Irvin Alora, Marco Pavone, George Haller
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Dynamical Systems (math.DS)
• Arxiv link: https://arxiv.org/abs/2611.11709
• Pdf link: https://arxiv.org/pdf/2611.11709
• Abstract A purely data-driven method grounded in the theory of adiabatic spectral submanifolds (aSSMs) was recently used to model and control nonlinear soft robotic simulators. Here, we extend the methodology to learn aSSM-reduced models from data collected from hardware soft robotic platforms. Specifically, we learn a five dimensional (5D) aSSM-reduced nonlinear ordinary differential equation (ODE) that accurately models the dynamics of a soft-helicoid trunk robot. We then use this ODE in a model predictive control scheme to demonstrate real-time precision control across various challenging closed-loop trajectory-tracking tasks, ranging from 2D to 3D trajectories with static and dynamic constraints. We find that our methodology enhances accuracy and precision across all control tasks compared to available data-driven linear methods and model-free reinforcement learning. While we focus here on a soft-helicoid trunk robot, our methodology relies solely on hardware data and is therefore applicable to a broader class of soft robotic systems.
• 中文摘要 一种基于绝热谱子流形（aSSM）理论的纯数据驱动方法最近被用于建模和控制非线性软机器人模拟器。在这里，我们将该方法扩展到从硬件软机器人平台收集的数据中学习aSSM约化模型。具体来说，我们学习了一个五维（5D）aSSM约化的非线性常微分方程（ODE），该方程能够准确模拟软螺旋体躯干机器人的动力学。随后，我们将该常微分方程应用于模型预测控制方案，演示了在多种具有静态和动态约束的复杂闭环轨迹跟踪任务中的实时精确控制。我们发现，与现有数据驱动线性方法和无模型强化学习相比，我们的方法论在所有控制任务中都提升了准确性和精度。虽然我们这里专注于软螺旋体躯干机器人，但我们的方法论仅依赖硬件数据，因此适用于更广泛的软机器人系统类别。

Accelerated Reinforcement Learning for Soft Robotics: Hours Instead of Years

软机器人加速强化学习：用小时代替几年

• Authors: Roshan S. Kaundinya, Hehui Zheng, Robert K. Katzschmann, George Haller
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Dynamical Systems (math.DS)
• Arxiv link: https://arxiv.org/abs/2611.11718
• Pdf link: https://arxiv.org/pdf/2611.11718
• Abstract Reinforcement learning (RL) requires large amounts of interaction data or a reliable simulator, both hard to obtain for nonlinear soft robotic hardware. Recent work shows that a time-scale separation inherent to soft robot dynamics justifies their reduction to low-dimensional invariant manifolds, known as adiabatic spectral submanifolds (aSSMs). We demonstrate that an aSSM-reduced model, a simple polynomial ordinary differential equation learned purely from pre-collected data, can serve as an RL training environment. On a pneumatically actuated soft arm, we train a trajectory-tracking policy in 3 hours on a six-dimensional aSSM-reduced model and deploy it without fine-tuning, whereas reproducing this policy training extrapolates to 3 years of hardware interaction. Our policy tracks dynamic targets with 40% lower error than otherwise identical policies trained on a 4.8-million-parameter recurrent neural network (RNN) or a 1.7-million-parameter transformer, which take 13 and 100 hours to identify, against 5 minutes for our mathematically justified 6,924-parameter model.
• 中文摘要 强化学习（RL）需要大量交互数据或可靠的模拟器，而非线性软机器人硬件难以获得。最新研究表明，软机器人动力学固有的时间尺度分离性使其被简化为低维不变流形，即绝热谱子流形（aSSM）。我们证明了aSSM约化模型——一个纯粹从预收集数据中学习的简单多项式常微分方程——可以作为强化学习的训练环境。在气动驱动的软臂上，我们在六维aSSM约简模型上用3小时训练轨迹跟踪策略，并无微调地部署，而重现该策略训练则外推为3年的硬件交互。我们的策略追踪动态目标时，误差比在480万参数的循环神经网络（RNN）或170万参数变换器上训练的相同策略低40%，而这些目标的识别时间为13至100小时，而我们数学上合理的6924参数模型仅需5分钟。

Spectral Rank and Step-Direction Dynamics in Language-Model Post-Training

语言模型后训练中的谱级与阶梯方向动态

• Authors: Khurram Khalil, Khaza Anuarul Hoque
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11752
• Pdf link: https://arxiv.org/pdf/2611.11752
• Abstract Language-model post-training modifies weights through cumulative updates and successive displacements, yet existing studies report seemingly conflicting spectral properties: nearly full-rank updates and low stable-rank changes. We show these observations are compatible and characterize update directions during training. For singular values following $\sigma_i \propto i^{-\alpha}$ with $0 < \alpha \leq 1/2$, we prove that stable rank becomes negligible relative to fixed-energy rank as dimension increases. Full singular value decompositions of selected OLMo-2-7B updates confirm this separation: three proximal policy optimization (PPO) attention updates have stable ranks of 9.6-18.3 but require 532-1,222 singular components to capture 80% of their energy. We next compute consecutive checkpoint-displacement cosines from full-model inner products. Across nine publicly released reinforcement learning from verifiable rewards (RLVR) stages, all 131 adjacent pairs have negative cosines on their native checkpoint grids, including PPO and Group Relative Policy Optimization (GRPO) trajectories. Supervised fine-tuning comparisons across four model families show configuration-dependent alignment, including negative late-training cosines for Mistral and FP32 Llama. Finally, a controlled OLMo-2-1B study with three paired seeds and a 2,048-token response cap compares group-relative advantages against a learned critic with generalized advantage estimation (GAE). At matched 60-rollout windows, late-training mean cosines are +0.259 and +0.012, respectively; the paired critic-minus-group difference is -0.247 (seed-level 95% interval [-0.358, -0.137]). These findings distinguish spectral concentration, energy coverage, and trajectory direction, and demonstrate that the advantage system influences directional alignment under matched training conditions.
• 中文摘要 语言模型训练后通过累积更新和连续位移修改权重，但现有研究报告的谱性质似乎相互矛盾：几乎全秩更新，而稳定秩变化低。我们证明这些观测值是兼容的，并描述了训练期间的更新方向。对于奇异值，位于$\sigma_i \propto i^{-\alpha}$，$0 < \alpha \leq 1/2$，我们证明稳定秩相对于固定能量秩变得可忽略不计。对选定的OLMo-2-7B更新的完整奇异值分解证实了这一分离：三次近端策略优化（PPO）注意力更新的稳定秩为9.6-18.3，但需要532-1,222个奇异分量才能捕获其80%的能量。接下来我们从全模型内积计算连续检查点位移余弦。在九个公开发布的可验证奖励强化学习（RLVR）阶段中，所有131对相邻的本体检查点网格（包括PPO和组相对策略优化（GRPO）轨迹均为负余弦。在四个模型家族的监督微调比较显示出配置依赖的比对，包括Mistral和FP32 Llama的负晚训练余弦。最后，一项受控的OLMo-2-1B研究，使用三对种子和2,048个标记的响应上限，比较了组相对优势与采用广义优势估计（GAE）的学习批评者。在匹配的60次滚动窗口下，晚训练平均余弦分别为+0.259和+0.012;配对批评者-减去组差异为-0.247（种子层级95%区间 [-0.358， -0.137]）。这些发现区分了光谱浓度、能量覆盖率和轨迹方向，并表明优势系统在匹配训练条件下会影响方向对齐。

Can Hindsight Critics Guide Delayed Actors?

事后诸葛亮的批评者能否指导迟缓演员？

• Authors: Francisco M. López, Francisco Cruz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11863
• Pdf link: https://arxiv.org/pdf/2611.11863
• Abstract Sensory delays create temporal misalignments between the state of the environment and the observations available to an agent, turning Markovian control problems into partially observable ones. In this work, we inquire whether such misalignments can be mitigated during training by using hindsight to recover aligned state-action transitions. We introduce Hindsight Critics, an asymmetric extension of the Soft Actor-Critic for reinforcement learning under partial observability. The critics are trained on hindsight-aligned transitions whereas the actor learns from the resulting privileged value estimates despite only having access to delayed observations. Comparisons against delay-aware and unaware baselines on the DelayBench suite show that Hindsight Critics can partially compensate for observation delays without requiring modifications to the agent's observations during deployment.
• 中文摘要 感官延迟导致环境状态与智能体可访问的观测值之间产生时间错位，使马尔可夫控制问题变成部分可观测问题。本研究探讨，是否可以通过后见明恢复对齐状态-动作转移，在训练过程中缓解此类错位。我们引入了后见诸葛亮批评者，这是软演员-批评者的非对称扩展，用于部分可观察性下的强化学习。批评者训练于事后视角对齐的转变，而行为者即使只能访问延迟观测，仍从所得的特权值估计中学习。与DelayBench套件中延迟感知和无意识基线的比较表明，回顾批评者可以在部署过程中无需修改代理观测值的情况下，部分补偿观察延迟。

Fade to... Red? Scheduling Coloured Noise Exploration Supports Efficient Robot Learning with Reduced Movement Jerk

渐入......红色？调度彩色噪音探索支持高效机器人学习，减少动作抽动

• Authors: Francisco M. López, Francisco Cruz
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11868
• Pdf link: https://arxiv.org/pdf/2611.11868
• Abstract Exploration in deep reinforcement learning (RL) for continuous control is commonly driven by temporally uncorrelated Gaussian (white) action noise, which leads to jerky motions with locally constrained exploration. In contrast, temporally correlated coloured noise, particularly pink noise, has been shown to support more efficient exploration in RL. Motivated by these findings, we inquire whether scheduling changes to the noise colour over the course of training can outperform white noise for continuous control robotics. We perform a systematic study of 49 linear schedules between white and red noise, together with generalised state-dependent exploration (gSDE), on four sparse-reward manipulation tasks with a Fetch arm and a Shadow Dexterous Hand. We find that the noise colour at the end of training does not affect learning efficiency or end performance, whereas fading towards red noise halves the jerk of exploration in every task. Balancing learning efficiency, end performance and smoothness, red noise emerges as the preferred strategy: it learns as efficiently as pink noise with almost 2-fold less movement jerk. We therefore recommend exploration schedulers that fade towards red noise as the default for off-policy robot learning.
• 中文摘要 深度强化学习（RL）中对连续控制的探索通常由时间无相关的高斯（白色）动作噪声驱动，导致局部受限探索时动作抖动。相比之下，时间相关色彩噪声，尤其是粉色噪声，已被证明支持更高效的强化学习探索。基于这些发现，我们探讨训练过程中对噪声颜色的调度变化是否能在连续控制机器人中优于白噪声。我们系统研究了49个白噪声与红噪声之间的线性调度，结合广义状态依赖探索（gSDE），涉及四个稀疏奖励操作任务，使用取物臂和阴影灵巧手。我们发现训练结束时的噪声颜色不影响学习效率或最终表现，而向红噪声渐弱则使每个任务中的探索抖动减半。在平衡学习效率、最终性能和平滑性方面，红噪声成为首选策略：其学习效率与粉噪声相当，且动作抖动几乎减少了两倍。因此，我们推荐采用探索调度器，将红噪声作为非策略机器人学习的默认选择。

Adaptive Entropy-Sparing for Efficient Reasoning

自适应熵保存以实现高效推理

• Authors: Cong Jiang, Xiaofeng Zhang, Tom Ko, Zheng Zhang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.11901
• Pdf link: https://arxiv.org/pdf/2611.11901
• Abstract Large Reasoning Models (LRMs) produce explicit chains of thought to improve transparency, but they frequently overthink, generating long yet inefficient reasoning chains that inflate computational costs and risk introducing errors. While numerous reinforcement learning methods struggle to penalize verbosity to achieve conciseness, they typically apply the penalty uniformly across lengthy sequences, which leads to credit confounding, that is, the failure to distinguish essential reasoning steps from superfluous ones. In this paper, we study a measurable proxy for this confounding: high-entropy tokens that often coincide with surface transition points where reasoning re-enters verification. Based on this insight, we propose Adaptive Entropy-Sparing (AES), which selectively spares high-entropy tokens that occur before an efficient in-group reference, while penalizing overlength high-entropy suffix tokens associated with superfluous branching. Our AES thereby navigates the suppression-exploration trade-off, reducing overthinking while preserving average reasoning accuracy. We also present an entropy-length model that motivates why local uncertainty can serve as a proxy for future reasoning cost. Across eight reasoning benchmarks, AES reduces reasoning length by up to 52.9% while increasing mean accuracy by up to 3.8% over the base models, consistently outperforming existing efficiency-oriented methods. Our code is available at this https URL.
• 中文摘要 大型推理模型（LRM）产生显式思维链以提高透明度，但它们常常过度思考，产生冗长且低效的推理链，推高计算成本并冒引入错误的风险。虽然许多强化学习方法难以惩罚冗长以实现简洁，但它们通常在较长序列中均匀施加惩罚，导致信用混淆，即未能区分关键推理步骤与多余步骤。本文研究了该混杂的可测量代理指标：高熵符号，通常与推理重新进入验证的表面转移点重合。基于这一见解，我们提出了自适应熵保存（AES），该方法选择性地保留高效组内引用前的高熵符号，同时惩罚与多余分支相关的超长高熵后缀符号。因此，我们的AES在抑制与探索之间做出权衡，减少过度思考，同时保持平均推理准确性。我们还提出了一个熵长度模型，阐明局部不确定性为何可以作为未来推理成本的代理指标。在八个推理基准测试中，AES将推理长度缩短最多52.9%，同时平均准确率提升高达3.8%，持续优于现有的效率导向方法。我们的代码可在此 https URL 获取。

An Evaluation of Tabular Foundation Models as Training-Free World Models

作为无训练世界模型的表式基础模型评估

• Authors: Xian Yeow Lee, Ryoichi Takase, Ahmed Farahat
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.11921
• Pdf link: https://arxiv.org/pdf/2611.11921
• Abstract World models are widely considered to be a potential key component of model-based reinforcement learning and robot planning, but typically require task-specific training on substantial interaction data. We investigate whether a pretrained tabular foundation model (TabFM) can serve as a training-free dynamics model for world modeling by framing next-state prediction as in-context regression over transition pairs, and where such a model helps and where it breaks. Across synthetic dynamical systems and robot locomotion benchmarks, TabFM is more accurate than LSTM and MLP models trained from scratch on the same limited transitions in almost all settings, and among regressors fitted to the same data it is the most accurate on locomotion, while a Gaussian process is more accurate on smooth low-dimensional systems. However, models trained on the full dataset remain substantially better on most systems. When rolled out autoregressively over multiple steps, TabFM's accuracy depends strongly on how much context it is given. With a short context, its rollouts quickly degrade to the level of a trivial baseline, whereas with a long context they improve substantially and, on one locomotion task, exceed a trained LSTM. Inference is also about three orders of magnitude slower than an LSTM. Overall, in their current state, we find that training-free priors are useful as predictors when data are scarce and enough context is available, but they are not yet a substitute for trained world models in planning.
• 中文摘要 世界模型被广泛认为是基于模型的强化学习和机器人规划的潜在关键组成部分，但通常需要针对大量交互数据进行任务特定训练。我们研究了预训练的表格基础模型（TabFM）是否可以作为无训练的动力学模型，通过将下一状态预测框架为基于转移对的上下文回归，以及该模型在哪些方面有效，何时破损。在合成动力系统和机器人运动基准测试中，TabFM比几乎所有场景下从零开始训练的有限转移转换的LSTM和MLP模型更准确，在同一数据拟合的回归器中，它在运动方面最为准确，而高斯过程在光滑低维系统上更为准确。然而，在大多数系统中，基于完整数据集训练的模型仍然显著优于此。当以多步自回归方式展开时，TabFM的准确性很大程度上取决于其上下文的多少。在短上下文下，其推广速度迅速退化到平凡基线水平;而在长时间上下文中，推断显著提升，甚至在某项运动任务中超过训练过的LSTM。推断速度也比LSTM慢大约三个数量级。总体而言，在当前状态下，我们发现无训练先验在数据稀缺且上下文充足时作为预测变量很有用，但它们尚未能替代训练过的世界模型在规划中。

Constrained Variational Policy Optimization for Hybrid Action Spaces

混合作用空间的受限变分策略优化

• Authors: Shuchang Yan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11936
• Pdf link: https://arxiv.org/pdf/2611.11936
• Abstract Constrained reinforcement learning in hybrid action spaces must coordinate discrete modes and mode-conditioned continuous controls under long-horizon cost constraints. Existing constrained hybrid-action methods typically optimize factorized policies directly in parameter space, whereas Constrained Variational Policy Optimization (CVPO) separates nonparametric improvement from parametric projection in its original continuous-action formulation. We introduce Hybrid Constrained Variational Policy Optimization (H-CVPO), extending this variational framework to hybrid action spaces. Under boundedness and strict feasibility, its constrained E-step has zero duality gap and, when the dual optimum is attained at positive temperature, yields a closed-form joint target distribution over hybrid actions through exponential reward-cost reweighting. Using the exact hybrid KL decomposition, we formulate a factorized M-step that fits both policy components to this target under separate discrete and conditional-continuous trust regions. For exact E/M updates, we prove fixed-reference surrogate improvement and conditional reward-cost bounds with a sufficient feasibility-margin condition. For approximate updates, we establish the existence of a dual minimizer under continuity on a compact domain and use residual-based admissibility to determine whether to project the target. Ablations identify cost-aware reweighting as an important contributor to constraint control. Extensive experiments in simulated hybrid-control environments show that H-CVPO achieves favorable reward-cost trade-offs compared with hybrid-action baselines.
• 中文摘要 混合动作空间中的受限强化学习必须在长视野成本约束下协调离散模式和模式条件连续控制。现有受限混合动作方法通常直接在参数空间中优化分解策略，而受限变分策略优化（CVPO）在其原始连续动作表述中将非参数改进与参数投影分离。我们引入混合受限变分策略优化（H-CVPO），将该变分框架扩展到混合动作空间。在有界性和严格可行性下，其受限E步长为零对偶缺口，当在正温度下达到对偶最优时，通过指数奖励-成本重权重产生混合动作上的闭合联合目标分布。利用精确的混合KL分解，我们构建了一个分解化的M步，使该目标在独立的离散和条件连续信任区域下拟合两个策略组件。对于精确的E/M更新，我们证明固定参考代理改进和条件性奖励-成本界限，条件条件充足且具备可行性-裕度条件。对于近似更新，我们确定在紧凑域连续性下存在对偶最小化器，并利用基于残差的可容许性判断是否投射目标。消融技术指出成本感知性加权是约束控制的重要贡献者。在模拟混合控制环境中的大量实验表明，H-CVPO相比混合行动基线实现了有利的奖品-成本权衡。

OASIS: Optimization-Aware Subgoal Inference with Safety Constraints for Autonomous Mobile Robot Navigation

OASIS：自主移动机器人导航的优化感知子目标推断，并具安全约束

• Authors: Arian Hajizadeha, Javad Pourrafia, Negar Shenavara, Sana Bashiria, Seyed Majid Esmailzadeh
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.11996
• Pdf link: https://arxiv.org/pdf/2611.11996
• Abstract Navigation in dynamic environments requires mobile robots to make progress toward their goals while respecting motion limits and avoiding surrounding obstacles. Learned subgoal policies can guide model predictive control beyond its local prediction horizon, but a geometrically suitable subgoal does not necessarily lead to a successfully solved control problem. This paper proposes OASIS, a hierarchical navigation framework that incorporates recent optimization outcomes into reinforcement learning-based subgoal generation. A high-level policy trained using Soft Actor--Critic generates local subgoals, while a nonlinear model predictive controller tracks them subject to motion limits, geometric collision-avoidance constraints, and embedded high-order control barrier function constraints. The outcomes of controller optimizations over each subgoal interval are summarized and provided to the policy at its next decision, without requiring additional optimization calls. The framework is evaluated in five navigation scenarios with static and moving obstacles, including dense interactions and sensing uncertainty. Comparisons with hierarchical and direct-goal control variants show that OASIS achieves the highest aggregate navigation success. Matched comparisons across three training seeds further show a consistent success advantage over the otherwise identical hierarchy without optimization-status input. The selected-policy comparison also shows fewer primary optimization failures with similar mean controller time.
• 中文摘要 动态环境中的导航要求移动机器人在尊重运动限制和避开周围障碍物的前提下，朝目标推进。学习到的子目标策略可以引导模型的预测控制超出其局部预测视野，但几何上合适的子目标不一定能成功解决控制问题。本文提出了OASIS的分层导航框架，将近期优化结果纳入基于强化学习的子目标生成中。使用软演员--Critic训练的高级策略生成局部子目标，而非线性模型预测控制器则在运动限制、几何碰撞避免约束和嵌入的高阶控制障碍函数约束下跟踪这些子目标。每个子目标区间内的控制器优化结果会被总结并提供给策略的下一步决策，无需额外的优化调用。该框架在五种有静态和移动障碍的导航场景中进行评估，包括密集交互和感知不确定性。与分层和直接目标控制变体的比较显示，OASIS在总量导航成功率上最高。三个训练种子的匹配比较进一步显示，在没有优化状态输入的情况下，相对于其他相同层级结构，取得了一致的成功优势。选择策略比较还显示，在相似的平均控制器时间下，主要优化失败率较低。

HyInv Technical Report

HyInv 技术报告

• Authors: Haowei Lin, Cai Zhou, Xiangxin Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Biomolecules (q-bio.BM)
• Arxiv link: https://arxiv.org/abs/2611.11997
• Pdf link: https://arxiv.org/pdf/2611.11997
• Abstract Inverse folding designs amino-acid sequences for a given protein backbone, and screening needs many candidates that are both diverse and well folded. We present HyInv, a 1.96-million-parameter backbone-conditioned graph network with an encoder-decoder architecture, trained by Hyra 2.0, the research agent developed by our team: pretrained with random-order autoregressive sequence modeling, followed by structure-reward reinforcement learning. At inference, a grouped draft-and-rewrite sampler that uses separate draft and rewrite temperatures improves refolding TM-score at matched diversity. An optimized implementation with BF16 numerics designs 128 sequences for a 402-residue backbone in 50.6 ms on a single NVIDIA H20 GPU, about 52 times faster than the reference sampler, and an exact FP32 path that returns the reference sequences is 8.7 times faster. Across five benchmark datasets and seven dataset-predictor combinations, HyInv reaches a higher average scTM than all ten external baselines at comparable or higher sequence diversity, with large margins over ProteinMPNN and ProteinZero V2. The same default setting supports de novo backbones, pocket-constrained libraries, enzyme redesign with the catalytic site kept, and loop redesign. We have released our code and model at this https URL.
• 中文摘要 逆折叠设计给定蛋白主链的氨基酸序列，筛选需要多个既多样又折叠良好的候选序列。我们介绍HyInv，一个拥有196万参数的主链条件图网络，采用编码-解码器架构，由我们团队开发的研究代理Hyra 2.0训练：先用随机序自回归序列建模预训练，随后进行结构-奖励强化学习。在推断时，采用分组草图和重写采样器，使用不同的草稿和重写温度，在匹配多样性下提升重叠TM评分。采用BF16数值的优化实现，在单个NVIDIA H20 GPU上设计128条序列，402残基骨干链需50.6毫秒，约快于参考采样器的52倍，精确FP32路径返回参考序列时快8.7倍。在五个基准数据集和七个数据集-预测变量组合中，HyInv在相当或更高序列多样性下达到的10个外部基线平均scTM均高，且与ProteinMPNN和ProteinZero V2有较大优势。相同的默认设置支持全新骨架、口袋约束文库、保留催化位点的酶重设计以及环路重设计。我们已在此https URL发布代码和模型。

Reward-Estimated Hypergradient for Bilevel Reinforcement Learning with Black-Box Follower

黑箱跟随者双级强化学习的奖励估计高梯度

• Authors: Shigeki Kusaka, Mikoto Kudo, Takumi Tanabe, Akifumi Wachi, Youhei Akimoto
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.11998
• Pdf link: https://arxiv.org/pdf/2611.11998
• Abstract Hypergradient (HG) methods for bilevel reinforcement learning (RL) typically assume a white-box follower whose reward is known. In practice, however, the leader rarely knows the follower's reward function and can only observe its trajectories. We address this black-box setting and propose Reward-Estimated Hypergradients (RE-HG), which recovers the follower's reward from these trajectories via inverse RL (IRL). Yet IRL identifies the reward only up to potential-based shaping, a transformation that leaves the follower's behavior unchanged but biases the HG. We prove that, for linear rewards, this bias cancels when IRL estimates are aggregated across sufficiently diverse dynamics, which the leader's own updates naturally supply. To handle finite samples, RE-HG truncates small eigenvalues of the aggregated feature covariance, restoring the identifiable subspace and stabilizing the estimation. Experiments show that RE-HG estimates gradients aligned with a white-box Oracle and achieves comparable performance; empirically, the truncation also suppresses HG explosions, letting RE-HG avoid local optima that trap the Oracle in sharp reward landscapes.
• 中文摘要 双层强化学习（RL）中的超梯度（HG）方法通常假设一个白箱跟随者，其奖励已知。然而，实际上领导者很少知道跟随者的奖励函数，只能观察其轨迹。我们针对这一黑箱设置提出奖励估计超梯度（RE-HG），通过逆强化学习（IRL）从这些轨迹中恢复跟随者的奖励。然而，IRL仅识别基于势能的整形奖励，这种转换使跟随者的行为保持不变，但对HG产生偏向。我们证明，对于线性奖励，当现实估计在足够多样的动态中汇总时，这种偏差会抵消，而领导者自身的更新自然提供了这些动态。为处理有限样本，RE-HG截断聚合特征协方差的小型特征值，恢复可识别子空间并稳定估计。实验表明RE-HG估计与白盒预言机对齐的梯度，并实现相当性能;经验上，截断还抑制了HG爆炸，使RE-HG避免局部最优解，避免将神谕困在锐利的奖励景观中。

SkillDuet: Co-Evolving Models and Skills through Response Feedback

SkillDuet：通过响应反馈共同进化模型与技能

• Authors: Xingguang Ji, Ziqiang Dong, Yubo Zhang, Xi Leng, Yang Yang, Guanjun Jiang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12109
• Pdf link: https://arxiv.org/pdf/2611.12109
• Abstract During reinforcement learning, experience can be both internalized in LLM parameters and retained externally as reusable textual skills. Parameter updates alter the model's behavior and the interaction experience from which new skills are extracted. These skills, in turn, guide the model's subsequent interactions and parameter updates. As the model's capabilities evolve, previously beneficial skills may become redundant or detrimental. We present SkillDuet, a framework that coordinates parameter updates and skill evolution through model response feedback. Skills extracted from on-policy trajectories guide parameter updates through OPD at every step. A response-based score, Strength, ranks candidate skills by the magnitude of the shift they induce in the current model's responses, and the highest-scoring skills are periodically consolidated into a persistent global skill set (Global). The updated model and Global jointly guide subsequent interactions, generating experience for further parameter and skill updates. With Qwen3-4B, SkillDuet achieves success rates of 98.57% on ALFWorld and 92.97% on WebShop, exceeding the matched SEED-API baseline by 2.86 and 5.47 percentage points, respectively. Ablations show that response-based skill selection and Global consolidation improve task performance. Further experiments show that the benefits of retained skills depend on both their content and their compatibility with the model. SkillDuet provides a concrete mechanism for model-skill co-evolution and skill-level recursive self-improvement.
• 中文摘要 在强化学习过程中，经验既可以内化在LLM参数中，也可以作为可重复使用的文本技能外部保留。参数更新改变模型的行为和从中提取新技能的交互体验。这些技能反过来指导模型后续的交互和参数更新。随着模型能力的发展，原本有益的技能可能变得多余甚至有害。我们介绍SkillDuet，一个通过模型响应反馈协调参数更新和技能演进的框架。从政策轨迹中提取的技能在每个环节通过OPD的参数更新指导。基于响应的评分“强度”根据候选技能对当前模型反应造成的变化幅度进行排名，得分最高的技能会定期整合为持续的全局技能集（Global）。更新后的模型与Global共同指导后续互动，为后续参数和技能更新积累经验。通过Qwen3-4B，SkillDuet在ALFWorld上达到98.57%和WebShop上的92.97%成功率，分别比匹配的SEED-API基线高出2.86个百分点和5.47个百分点。消融显示，基于响应的技能选择和全局巩固能提升任务表现。进一步实验表明，保留技能的益处取决于其内容及其与模型的兼容性。SkillDuet为模型-技能共进化和技能层面递归自我提升提供了具体机制。

Let the Fixer Decide: Training a Code Localizer from the Fixer's Likelihood

让修复者自己决定：从解决者的可能性训练代码本地化器

• Authors: Khalil Hennara, Muhammad Hreden, Zeina Aldallal, Sara Chrouf, Safwan AlModhayan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2611.12164
• Pdf link: https://arxiv.org/pdf/2611.12164
• Abstract Code localizers are usually trained to predict the files a reference patch edits, but the code a fixer needs to read is not the same as the code it changes. We propose Fixer-Guided Localization (FGL): the localizer proposes several localizations, a frozen fixer scores each by how likely it makes a verified patch, and the localizer is trained on these scores with weighted supervised fine-tuning, without running tests and without reinforcement learning. We build the full pipeline on Gemma 4 E4B for the Kaggle Gemma-4 Developer Agent competition, from 1,629 test-verified Kimi K3 trajectories. The FGL target helps the fixer find the bug as often as the teacher's localization while using 27.5% less code. On the development set, one EM round moves the agent in the expected direction compared with a matched control: more tasks reach a patch (53 vs. 46), fewer tasks get stuck reading files that do not exist (43 vs. 54), and more tasks are solved (8 vs. 6). Training also changes how the agent works: trained agents explore about three times more than the base model and write far fewer wrong patches, and the single agent solves up to 11 of 78 tasks, against 8 for the base model. An analysis of 13 evaluation runs identifies what limits a 4B agent under an 8-minute budget (time, repetition loops and edit quality) and gives practical lessons for training small coding agents. All data, code and adapters are publicly available.
• 中文摘要 代码本地化器通常训练为预测参考补丁编辑的文件，但修复者需要读取的代码与其修改的代码不同。我们提出修复者引导本地化（FGL）：本地化器提出多个本地化方案，冻结的修复器根据其完成验证补丁的可能性评分，本地化器则通过加权监督微调训练这些分数，无需运行测试，也未进行强化学习。我们在Gemma 4 E4B上构建了完整的流程，用于Kaggle Gemma-4开发代理竞赛，基于1629条经过测试验证的Kimi K3轨迹。FGL目标帮助修复者发现漏洞的频率与教师本地化相当，同时使用27.5%的代码。在开发组中，一次EM轮次使代理朝预期方向前进，相较匹配对照组：更多任务达到补丁（53次对46次），更少的任务卡在读取不存在的文件上（43次对54次），解决的任务也更多（8次对6次）。训练还改变了代理的工作方式：受过训练的代理探索量约是基础模型的三倍，写错补丁更少，单个代理解决了78个任务中的11个，而基础模型仅解决8个。对13次评估运行的分析揭示了4B代理在8分钟预算内的限制因素（时间、重复循环和编辑质量），并为训练小型编码代理提供了实用经验。所有数据、代码和适配器均公开。

Graph-Based RL for Sequential Rendezvous Planning with Multiple Patrolling USVs

基于图的强化学习，用于多巡逻USV的顺序会合规划

• Authors: Seung Woo Han, Hyeon Bin Yeo, Seung Heon Oh
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.12205
• Pdf link: https://arxiv.org/pdf/2611.12205
• Abstract In patrol missions involving multiple unmanned surface vehicles (USVs), a designated command USV can support fleet management by sequentially rendezvousing with each patrolling USV. These operations require motion-feasible route planning and priority-aware rendezvous sequencing, with the objective of minimizing mission completion time and total tardiness relative to priority-dependent due dates. We propose a framework combining grid-based path planning, surrogate travel-time prediction, and graph-based reinforcement learning. The path planner accounts for ocean currents and approximates turn-rate constraints through multi-stage maneuvers, with path feasibility empirically validated through trajectory-tracking simulations. For rendezvous sequencing, a Feasible Transition Graph (FTG) represents candidate rendezvous events and temporally feasible transitions, enabling a graph neural network to exploit spatiotemporal rendezvous relationships for policy learning. A surrogate predicts travel times from compact current and motion features to avoid repeated path searches during FTG construction and policy training. Evaluation using grid-based path planning shows statistically significant average objective reductions of 10.5%, 3.5%, and 1.5% compared with greedy, MLP-RL, and 2-step lookahead greedy, respectively. The policy achieves these improvements while requiring approximately 1/35 of the latter's computation time. It maintains its advantage over MLP-RL under unseen priority-tier settings, with larger gains as fleet size increases.
• 中文摘要 在涉及多艘无人水面飞行器（USV）的巡逻任务中，指定的指挥USV可通过与每艘巡逻USV顺序会合来支持舰队管理。这些操作需要可行的路线规划和优先级感知的会合顺序，目标是相较于优先级相关的截止日期，最大限度地减少任务完成时间和总迟到率。我们提出了一个结合基于网格路径规划、替代行进时间预测和基于图的强化学习的框架。路径规划器考虑洋流，并通过多阶段机动近似转弯速率约束，并通过轨迹跟踪模拟实证验证路径可行性。对于交会序列，可行过渡图（FTG）表示候选会合事件和时间可行转移，使图神经网络能够利用时空交会关系进行策略学习。代理人通过紧凑电流和运动特征预测旅行时间，以避免在FTG构建和策略训练过程中重复路径搜索。基于网格的路径规划评估显示，与贪婪、MLP-RL和两步前瞻贪婪相比，目标平均减少率分别为10.5%、3.5%和1.5%，具有统计学显著性。该策略实现了这些改进，同时耗费了后者约1/35的计算时间。在看不见的优先级设置下，它依然保持相较于MLP-RL的优势，随着舰队规模增加，收益更大。

Prospective Action-Resource Graph Multi-Agent Reinforcement Learning for Electric HGV Fleets under Shared Charging Infrastructure

电动重型货车车队共享充电基础设施下的前瞻性行动资源图多智能体强化学习

• Authors: Yiming Xu, Fulong Yao, Maurizio Albano, Rigel Gjomemo, Liana Cipcigan
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.12244
• Pdf link: https://arxiv.org/pdf/2611.12244
• Abstract Electric heavy goods vehicle (eHGV) fleet coordination requires each vehicle to choose driving, charging and resting actions whose consequences depend on shared infrastructure. Existing multi-agent reinforcement learning (MARL) methods mainly represent current states, leaving action-specific futures and resource interactions to be learned indirectly. We develop Horizon-Preview and Graph-Coordinated Proximal Policy Optimisation (HGC-PPO), which integrates horizon preview and candidate-resource graph coordination into the PPO decision architecture. Horizon preview represents each legal action through its short-horizon operational consequences. Graph message passing then updates these candidates through station, charger-type and time-slot resources shared across the fleet. On the 794km Folkestone-Glasgow task, HGC-PPO achieved the lowest mean charging cost among five RL methods at £504.32 per vehicle, with a fleet completion time of 24.920h. It also completed the task 1.763h faster and cost £36.51 less per vehicle than a replay-verified finite-budget MILP benchmark. Across six unseen routes and fleet sizes from 5 to 20 vehicles, HGC-PPO delivered the lowest macro-average completion time, charging cost and charging frequency among the RL methods. A 2 by 2 ablation showed that the full preview-graph architecture performed best across all four operational metrics. These results show that prospective action information and fleet-wide resource coordination improve eHGV decisions under charging and driver-hours constraints.
• 中文摘要 电动重型货车（eHGV）车队协调要求每辆车选择驾驶、充电和休息动作，其后果取决于共享基础设施。现有的多智能体强化学习（MARL）方法主要表示当前状态，行动特定的未来和资源相互作用则间接学习。我们开发了地平线预览和图协调近端策略优化（HGC-PPO），将地平线预览和候选资源图协调整合进PPO决策架构。地平线预览通过短视野运营后果来表示每项法律行动。图消息传递随后通过车队共享的站点、充电器类型和时隙资源更新候选信息。在794公里的福克斯通-格拉斯哥任务中，HGC-PPO在五种RL方法中实现了最低的平均充电成本，每辆车为504.32英镑，车队完成时间为24.920小时。它还比经过回放验证的有限预算MILP基准快完成任务1.763小时，每辆车成本低36.51英镑。在六条未见路线和5至20辆车队规模中，HGC-PPO在RL方法中实现了最低的宏观平均完成时间、充电成本和充电频率。2×2消融显示，完整的预览图架构在所有四个运营指标中表现最佳。这些结果表明，前瞻性行动信息和全车队资源协调在充电和驾驶小时数限制下能改善eHGV决策。

RAIL: Representation-Aligned Imitation Learning for Student-Compatible Teacher Policies

RAIL：代表性对齐的模仿学习，促进学生兼容教师政策

• Authors: Meraj Mammadov, Pedro Zuidberg Dos Martires, Johannes Andreas Stork
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.12251
• Pdf link: https://arxiv.org/pdf/2611.12251
• Abstract Reinforcement learning (RL) from raw sensory inputs such as images or onboard sensors can be challenging due to high-dimensional observations and sparse rewards. A common strategy is to train a teacher policy with access to privileged state information and then distill it into a student policy that acts from raw inputs alone. However, when the teacher relies on information unavailable to the student, exact imitation may be impossible, creating an irreducible imitation gap. Existing approaches typically mitigate this mismatch through careful reward shaping or additional reinforcement learning on the student policy after the teacher has been trained. Instead, we introduce Representation-Aligned Imitation Learning (RAIL), which addresses the mismatch during teacher training itself. RAIL learns a latent representation shared across teacher and student observations using contrastive learning, and trains the teacher policy directly in this space. Because the teacher policy is constrained to operate on this shared space, it learns behaviors that are reproducible from student observations. This substantially reduces the imitation gap while preserving task performance. Across multiple environments, RAIL outperforms strong baselines without reward modification or post-hoc student fine-tuning, and surpasses direct reinforcement learning from raw observations. The learned representations further enable zero-shot transfer to new tasks through teacher-only training. Project site: this https URL.
• 中文摘要 由于高维观察和稀疏奖励，从原始感官输入（如图像或内置传感器）进行强化学习（RL）可能具有挑战性。一种常见策略是先通过访问特权状态信息来训练教师政策，然后将其提炼成仅凭原始输入作用的学生政策。然而，当教师依赖学生无法获得的信息时，精确模仿可能不可能，从而造成不可减少的模仿差距。现有方法通常通过精心的奖励塑造或在教师培训后对学生政策进行额外的强化学习来缓解这种不匹配。相反，我们引入了表征对齐模仿学习（RAIL），在教师培训过程中解决了这一不匹配问题。RAIL通过对比学习学习教师和学生观察共享的潜在表征，并直接在此领域培训教师政策。由于教师政策受限于该共享空间，它从学生观察中学习可重复的行为。这大大减少了模仿差距，同时保持任务表现。在多种环境中，RAIL在无奖励修改或事后学生微调的情况下表现优于强基线，且超越了直接强化学习。学习到的表征进一步支持通过教师专属培训实现零样本转移至新任务。项目网站：此 https URL。

Agent4Crypto: A Dual-Reflective and Market-Feedback-Aligned LLM-Based Multi-Agent System for Cryptocurrency Trading

Agent4Crypto：一个双重反射且市场反馈对齐的基于大型语言模型的多智能体加密货币交易系统

• Authors: Guangyan Gan, Jiahao Li, Jiashun Wang, Lin William Cong
• Subjects: Subjects: Multiagent Systems (cs.MA); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.12269
• Pdf link: https://arxiv.org/pdf/2611.12269
• Abstract Cryptocurrency markets provide a natural testbed for autonomous AI agents because they are digital-native, high-frequency, and machine-speed environments, yet existing methods often struggle with non-stationary market regimes, opaque decisions, and fixed aggregation rules. In this paper, we propose Agent4Crypto, a dual-reflective and market-feedback-aligned multi-agent system. The framework coordinates specialized agents for data collection, analysis, risk assessment, reflection, and trading, synthesizing heterogeneous signals from prices, indicators, news, and charts in an interpretable decision pipeline. Our central design is a trader-centric decision mechanism: three style-specific trader agents, Aggressive, Neutral, and Conservative, generate candidate actions and natural-language rationales from the same market context, while their influence is learned rather than manually fixed. A PPO-based gating policy dynamically allocates trader weights through a dual-reflective feedback process. One reflection stream evaluates post-trade decision quality from realized portfolio outcomes and distills reusable lessons into a Financial ReasoningBank for retrieval in subsequent trader prompts, while the other estimates trader-level counterfactual contribution, indicating which style has recently aligned better with market movements. The PPO state combines Counterfactual Contribution Memory with market features, previous trader weights, and current trader actions, so trader influence adapts across bull, sideways, and bear regimes. Experiments on BTC, ETH, and XRP show that Agent4Crypto improves total return and Sharpe ratio over traditional strategies, machine learning and reinforcement learning baselines, and existing LLM-based trading systems, demonstrating the value of interpretable trader competition and market-feedback-driven adaptation in volatile cryptocurrency markets.
• 中文摘要 加密货币市场为自主AI代理提供了自然的试验场，因为它们是数字原生、高频且机器速度极快的环境，但现有方法常常在非平稳市场机制、不透明决策和固定聚合规则方面遇到困难。本文提出Agent4Crypto，一个双重反思与市场反馈对齐的多智能体系统。该框架协调专门的代理进行数据收集、分析、风险评估、反思和交易，综合价格、指标、新闻和图表中的异构信号，形成可解释的决策管道。我们的核心设计是一个以交易者为中心的决策机制：三个风格特定的交易者代理——激进、中立和保守——从同一市场语境生成候选行动和自然语言理由，其影响力是学习而非手动固定的。基于PPO的门槛政策通过双重反思反馈过程动态分配交易者权重。一种反思流评估交易后决策质量，基于实现的投资组合结果，并将可重用的经验教训提炼成财务推理库，供后续交易者提示检索;另一种则估计交易者层面的反事实贡献，指出哪种风格最近更符合市场走势。PPO状态将反事实贡献记忆与市场特征、以往交易者权重及当前交易者行为结合起来，因此交易者影响力在牛市、横盘和熊市阶段都能适应。对BTC、ETH和XRP的实验显示，Agent4Crypto相较传统策略、机器学习和强化学习基线以及现有基于LLM的交易系统提升了总回报和夏普比率，展示了在波动性加密货币市场中，可解释的交易者竞争和市场反馈驱动的适应价值。

Learning Evidence-Grounded Reasoning for Context-Aware Time Series Forecasting

学习基于证据的推理以实现情境感知时间序列预测

• Authors: Xingpeng Gao, Mingyue Cheng, Daoyu Wang, Huajian Zhang, Xiaoyu Tao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12285
• Pdf link: https://arxiv.org/pdf/2611.12285
• Abstract LLM-driven time series forecasting has become increasingly promising by leveraging contextual features to generate future values directly or refine forecast priors. However, incorporating contextual features alone does not ensure that models can identify which features matter for forecasting and understand what they suggest about future dynamics. We argue that the fundamental challenge is how to extract predictive evidence from contextual features and when and how to leverage it to guide forecasting decisions. In this paper, we propose CastReasoner, an evidence-grounded reasoning model for context-aware time series forecasting. Specifically, CastReasoner uses a structured chain of thought (CoT) to extract predictive evidence from contextual features, infer the future dynamics suggested by this evidence, and assess these dynamics against the forecast prior to guide adaptive forecasting decisions. To enable this capability, we design a reasoning-centric two-stage post-training framework including supervised fine-tuning (SFT) and reinforcement learning (RL). First, we synthesize reasoning trajectories through controllable multi-teacher guidance and select high-quality ones using a novel student-aware difficulty metric. Second, we design fine-grained multi-objective rewards to further improve reasoning and forecasting. To address heterogeneous learning needs across domains in both stages, we introduce a convergence-guided data balancing strategy. Through this two-stage training, CastReasoner improves both reasoning quality and predictive accuracy. Extensive experiments across multi-domain forecasting benchmarks demonstrate that CastReasoner achieves superior forecasting performance.
• 中文摘要 利用上下文特征直接生成未来值或细化预测先验，利用大型语言模型驱动的时间序列预测变得越来越有前景。然而，仅仅纳入上下文特征并不能保证模型能够识别哪些特征对预测至关重要，并理解它们对未来动态的暗示。我们认为根本挑战在于如何从上下文特征中提取预测证据，以及何时以及如何利用这些证据来指导预测决策。本文提出了CastReasoner，一种基于证据的情境感知时间序列预测推理模型。具体来说，CastReasoner利用结构化思维链（CoT）从上下文特征中提取预测证据，推断这些证据所暗示的未来动态，并在指导自适应预测决策前评估这些动态与预测的关系。为实现这一能力，我们设计了一个以推理为中心的两阶段后培训框架，包括监督式微调（SFT）和强化学习（RL）。首先，我们通过可控的多教师指导综合推理轨迹，并利用一种新的学生感知难度指标选择高质量的轨迹。其次，我们设计细粒度的多目标奖励，进一步提升推理和预测能力。为满足跨领域异质学习需求，我们引入了收敛引导的数据平衡策略。通过这两阶段训练，CastReasoner 提高了推理质量和预测准确性。跨多领域预测基准的大量实验表明 CastReasoner 实现了更优越的预测性能。

VLA-MOPD: On-Policy Distillation for Multi-Task VLA Policy Consolidation

VLA-MOPD：多任务VLA策略整合的策略蒸馏

• Authors: Yongxuan Lv, Zhen Fang, Lichen Man, Hao Li, Pu Li, Hengyi Wang, Langfeng Cao, Lin Chen, Jie Huang, Xiu Su, Shan You, Fei Wang, Feng Zhao
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.12286
• Pdf link: https://arxiv.org/pdf/2611.12286
• Abstract Physical intelligence requires embodied agents to ground language in observations and execute actions while retaining useful skills. Vision-language-action (VLA) policies offer a shared interface, but task-specific post-training often produces separate specialists. Consolidation is difficult: mixed-task reinforcement learning provides sparse trajectory-level credit, while offline distillation may miss student-visited observations and endpoint distillation does not supervise intermediate action-generation states. We introduce VLA Multi-teacher On-Policy Distillation (VLA-MOPD), a protocol for consolidating flow-matching VLA specialists. After acquiring task or suite experts, a shared student collects robot rollouts and queries the instruction-routed frozen teacher at each visited observation and intermediate ordinary differential equation (ODE) state. Detached per-flow-step transition targets supervise valid action chunks through a masked local matching loss; an optional reference-policy term supports retention on other tasks. Starting from one trajectory per LIBERO task, VLA-MOPD raises mean success-once from 75.4% to 98.9%, compared with 98.15% for endpoint action distillation. It improves mean RoboTwin 2.0 success from 73.0% to 89.3%; on a Piper robot, two consolidation rounds improve average success from 70.0% to 90.0%. In a near-matched Spatial-adaptation study, reference matching limits the Object suite decrease to 3.0 points, versus 13.2 without the reference term.
• 中文摘要 物理智能要求具身代理在观察中扎根语言并执行行动，同时保留有用技能。视觉-语言-行动（VLA）策略提供共享接口，但任务特定后培训通常会产生独立的专家。整合困难：混合任务强化学习提供稀疏的轨迹级学分，离线蒸馏可能遗漏学生访问的观察，终点蒸馏不监督中间动作生成状态。我们引入了VLA多教师策略上提炼（VLA-MOPD），这是一种整合流匹配VLA专家的协议。获得任务或套件专家后，共享学生收集机器人展开数据，并在每次访问的观察和中间常微分方程（ODE）状态查询指令路由的冻结教师。分离的每流步骤转移目标通过掩蔽局部匹配损耗监督有效动作块;可选的参考策略项支持其他任务的保留。从每个LIBERO任务的一条轨迹开始，VLA-MOPD将平均成功率从75.4%提升到98.9%，而终点动作提炼则为98.15%。它将RoboTwin 2.0的平均成功率从73.0%提升到89.3%;在Piper机器人上，进行两轮合并后，平均成功率从70.0%提升到90.0%。在一项近乎匹配的空间适应研究中，参考匹配限制了对象套件的3.0分，而没有参考项时为13.2分。

A Modular Perspective for Modeling MARL Systems

MARL系统建模的模块化视角

• Authors: Tiziano Maisonhaute, Jean-Christophe Soulié, Fabien Michel
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.12314
• Pdf link: https://arxiv.org/pdf/2611.12314
• Abstract The growing complexity of Multi-Agent Reinforcement Learning (MARL) has fragmented the literature, obscuring underlying architectural choices behind diverse implementations. This fragmentation makes it difficult to evaluate critical modeling dimensions in isolation, identify how components affect the resulting system dynamics, and compare, reuse, or combine contributions across the literature. This paper proposes a modular perspective for modeling, analyzing, and implementing MARL systems. We decompose them into eight functional modules: agents, environments, learning, reward modeling, training and execution modes, communication, modeling of other agents, and evaluation criteria. For each module, we clarify its role and review representative approaches. We instantiate this decomposition in MARLKIT, a framework whose modules are explicit and interchangeable. MARLKIT demonstrates how the proposed perspective supports transparent experimental assumptions and the controlled variation and combination of MARL mechanisms. Experiments illustrate its ability to prototype, compare, reuse and combine modules while making differences between configurations explicit. Overall, the paper provides a structured perspective on MARL modeling and implementation, offering a common language for making modeling choices explicit, supporting clearer descriptions of MARL approaches, and supporting more controlled experimentation through the systematic isolation and combination of MARL mechanisms.
• 中文摘要 多智能体强化学习（MARL）日益复杂，使文献分散，掩盖了多元实现背后的架构选择。这种碎片化使得孤立评估关键建模维度、识别组件如何影响系统动态，以及比较、重用或合并文献中的贡献变得困难。本文提出了一种模块化视角，用于建模、分析和实现MARL系统。我们将它们分解为八个功能模块：代理、环境、学习、奖励建模、训练与执行模式、通信、其他代理建模以及评估标准。对于每个模块，我们明确其角色并回顾代表性方法。我们将这种分解实例化为MARLKIT框架，该框架的模块显式且可互换。MARLKIT展示了所提出视角如何支持透明的实验假设以及MARL机制的受控变异与组合。实验展示了其原型、比较、重用和组合模块的能力，同时明确区分配置之间的差异。总体而言，本文为MARL建模与实现提供了结构化视角，提供了一种通用语言，使建模选择显明化，支持对MARL方法的更清晰描述，并通过系统隔离和组合MARL机制支持更受控的实验。

PhysOPSD: Code as Privilege for Physically Plausible Video Generation

PhysOPSD：作为物理上合理视频生成特权的代码

• Authors: Harold Haodong Chen, Rui Peng, Lingteng Qiu, Kejie Qiu, Chao Xu, Zilong Dong
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.12411
• Pdf link: https://arxiv.org/pdf/2611.12411
• Abstract While modern video generative models achieve remarkable visual fidelity, they remain noticeably brittle when generating physically plausible dynamics and complex interactions. Targeted post-training is crucial, yet supervised fine-tuning suffers from offline state mismatch, and reinforcement learning is hindered by sparse, ill-defined physical reward functions. On-Policy (Self-)Distillation (OPSD) reconciles on-policy rollouts with dense corrective feedback, shifting the core challenge to designing effective privileged context. However, conventional modalities fall short: text prompts are inherently declarative, whereas real reference videos heavily entangle physical laws with visual appearance without parametric control. To this end, we propose PhysOPSD, introducing executable code as privileged physical context for video generation. Unlike static visual observations, executable code instantiates an editable physical world with intrinsic abstraction and controllability. Powered by PhysOPSD-Engine, we translate real-world physical scenarios into verifiable simulation programs. PhysOPSD then incorporates (i) dual-level physical privilege to bridge the information gap via an intermediate bottleneck, alongside (ii) controlled physical variations under shared rollout noise to expose causal dynamic shifts. Extensive experiments demonstrate that PhysOPSD substantially improves physical plausibility across diverse backbones, outperforming SFT and RL baselines with only $3$K samples, while preserving general video generation quality.
• 中文摘要 虽然现代视频生成模型实现了卓越的视觉真实度，但在生成物理可信动态和复杂交互时仍明显脆弱。有针对性的后期训练至关重要，但监督式微调存在离线状态不匹配的问题，强化学习受限于稀疏且定义模糊的物理奖励函数。策略（自）蒸馏（OPSD）将策略上的推广与密集的纠正反馈相结合，将核心挑战转向设计有效的特权上下文。然而，传统模式不足：文本提示本质上是声明性的，而真实参考视频则将物理定律与视觉外观纠缠在一起，缺乏参数控制。为此，我们提出了PhysOPSD，引入可执行代码作为视频生成的特权物理上下文。与静态视觉观察不同，可执行代码实例化一个具有内在抽象和可控性的可编辑物理世界。借助PhysOPSD引擎，我们将现实世界的物理场景转化为可验证的模拟程序。PhysOPSD随后结合了（i）双层物理特权，通过中间瓶颈弥合信息差距，同时（ii）在共享的滚动噪声下控制物理变化，揭示因果动态变化。大量实验表明，PhysOPSD在多样骨干链上显著提升物理可信度，仅用3万美元样本就优于SFT和RL基线，同时保持了整体视频生成质量。

K-prop: Deep Online Learning by Backpropagating Temporal Kernels

K-prop：通过反向传播的时序核进行深度在线学习

• Authors: Bingkun Liu, Paul Haider, Federico Benitez, Mihai A. Petrovici, Walter Senn
• Subjects: Subjects: Machine Learning (cs.LG); Neurons and Cognition (q-bio.NC)
• Arxiv link: https://arxiv.org/abs/2611.12530
• Pdf link: https://arxiv.org/pdf/2611.12530
• Abstract Self-recurrence represents an important component for temporal processing in neural networks, both artificial and biological, where it appears naturally in the form of leaky integration. However, due to the complex temporal dependencies of the gradient, an efficient and effective method of online training is still lacking. Here we propose K-prop, a novel online learning method for leaky neural networks. It dramatically reduces the computational and memory cost of real time recurrent learning (RTRL) by propagating kernel gains. K-prop also outperforms existing RTRL approximations and achieves performance comparable to backpropagation through time in sequence learning, classification, and reinforcement learning tasks. We further analyze how its sole approximation affects gradient quality and identify conditions under which K-prop remains accurate and effective.
• 中文摘要 自递性是神经网络（无论是人工还是生物）中时间处理的重要组成部分，在这些网络中自然以泄漏积分的形式出现。然而，由于梯度的时间依赖性复杂，仍然缺乏高效有效的在线训练方法。这里我们提出了K-prop，一种用于泄漏神经网络的新型在线学习方法。它通过传播核增益，显著降低了实时循环学习（RTRL）的计算和内存成本。K-prop 还优于现有的 RTRL 近似，在序列学习、分类和强化学习任务中实现了与反向传播时间相当的性能。我们还进一步分析了其唯一近似如何影响梯度质量，并识别 K-prop 在哪些条件下保持准确和有效。

AgentOPSD: Turn-Level Credit Assignment via Recursive Belief Revision

AgentOPSD：通过递归信念修正实现回合级信用分配

• Authors: Yizhuo Li
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.12587
• Pdf link: https://arxiv.org/pdf/2611.12587
• Abstract Reinforcement learning with verifiable rewards ordinarily spreads a single advantage over a whole trajectory, a granularity too coarse to isolate the few decisions on which success in a long, multi-turn interaction actually turns. Privileged self-distillation supplies denser supervision, but it leaves unanswered how a set of local teacher-student discrepancies should be converted into credit along a sequence. We introduce AgentOPSD, a critic-free scheme that assigns credit recursively, one interaction turn at a time. Inside a turn, the token-level log-probability gaps between a skill-conditioned self-teacher and the student are summed, and that sum acts as evidence in a recursive Bayesian update carried out in log-odds space; the resulting revisions reallocate the outcome-derived advantage across turns through a multiplier that is bounded and preserves its sign. With Qwen2.5-3B and Qwen2.5-7B as policy backbones on ALFWorld, the method improves consistently on GRPO and on self-distillation baselines, and attains 89.1% success at the 7B scale. Ablations attribute the gain to aggregating evidence at turn boundaries and to making the belief revision depend on history, rather than to the local gap by itself.
• 中文摘要 带有可验证奖励的强化学习通常将单一优势分布在整个轨迹上，其粒度过于粗糙，无法区分长时间多回合互动中实际决定成功的少数决策。特权自蒸馏提供了更密集的监督，但未解答如何将一组局部师生差异转化为沿序列的积分。我们引入AgentOPSD，一种无批评的方案，递归地逐回合分配积分。在一回合内，技能条件自教师与学生之间的代币级对数概率差距被加和，该和作为递归贝叶斯更新的证据，在对数赔率空间中进行;由此产生的修订通过有界乘数重新分配结果衍生优势，该乘数保持其符号。以Qwen2.5-3B和Qwen2.5-7B作为ALFWorld上的政策骨干，该方法在GRPO和自提纯基线上持续提升，并在7B尺度上达到89.1%的成功率。消融法将收益归因于在转弯边界汇总证据，使信念修正依赖于历史，而非单纯的局部差距。

Simulation-aided Reinforcement Learning with Control Variates

带控制变量的仿真辅助强化学习

• Authors: Omer Hemo, Ido Sefi, Mirco Mutti, Aviv Tamar, Kfir Y. Levy
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.12643
• Pdf link: https://arxiv.org/pdf/2611.12643
• Abstract Reinforcement Learning (RL) algorithms often rely on data collected from a target environment, which can be expensive and noisy. Although simulators can generate large amounts of additional data, standard simulation mixing approaches, which directly combine simulated and real experience, can degrade performance due to simulator inaccuracies, especially when they bias the solution away from the optimal policy. We propose a conservative simulation-aided learning method based on control variates, where simulated data is used solely to reduce variance in the learning process rather than directly shaping the learning target. We develop our approach for Policy-Evaluation, which is a fundamental building block of RL algorithms. Then we combine it within the actor-critic framework towards policy improvement, as well as extend it towards offline RL scenarios. Experiments on multiple MuJoCo benchmark tasks demonstrate improvements over standard simulation mixing approaches, highlighting control variates as an effective tool for simulation-aided reinforcement learning.
• 中文摘要 强化学习（RL）算法通常依赖于从目标环境中收集的数据，这可能成本高昂且噪声较大。尽管模拟器可以生成大量额外数据，但直接结合模拟和真实经验的标准模拟混合方法，由于模拟器不准确，尤其是当解偏离最优策略时，可能会降低性能。我们提出了一种基于控制变量的保守模拟辅助学习方法，其中模拟数据仅用于减少学习过程的方差，而非直接塑造学习目标。我们开发了策略评估方法，这是强化学习算法的基本构建模块。然后我们将其结合在actor-critic框架中用于策略改进，并推广至离线强化场景。在多个MuJoCo基准测试任务上的实验展示了相较标准模拟混合方法的改进，凸显控制变量作为模拟辅助强化学习的有效工具。

VLA Adaptation with Real and Simulated Data: A Case Study on Robotic Chess

基于真实与模拟数据的VLA适应：机器人国际象棋案例研究

• Authors: Matěj Kripner, Milan Straka
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12676
• Pdf link: https://arxiv.org/pdf/2611.12676
• Abstract Vision-language-action (VLA) models are trained as general-purpose foundation models for robot learning, and require fine-tuning for each downstream task and robot. This raises practical questions such as how much data and training updates are needed, how checkpoints should be selected, and whether simulation can be leveraged to alleviate the cost of real data collection. We study these questions on a task that requires both precise manipulation and spatial understanding of the scene. In this task, an SO-101 arm moves a chess king between two squares that are given only by their names, such as e4 and a5. We fine-tune $\pi_{0.5}$ with various dataset sizes and update budgets, using teleoperated demonstrations as well as simulated demonstrations from a randomized scripted expert that imitates the variability of human demonstrations. In simulation, the success rate scales with both data and updates. Importantly, we find that a short learning-rate cool-down adds up to 31 points and is necessary for comparing checkpoints, and that validation loss and action error are unreliable for model selection. On the real arm, we fine-tune on up to 256 teleoperated demonstrations and evaluate ten policies in 25 trials each. Partial results indicate that pretraining on the simulated demonstrations helps substantially, most clearly when real data is scarce. We also present a negative result, in which training on the policy's own rollouts with filtered behavior cloning, RECAP-style advantage conditioning, corrections from an automatic coach, or residual reinforcement learning did not yield a measurable improvement over plain behavior cloning in our setting. We release the simulator, the expert and the real demonstrations.
• 中文摘要 视觉-语言-动作（VLA）模型被训练为机器人学习的通用基础模型，需要针对每个下游任务和机器人进行微调。这引发了实际问题，如需要多少数据和训练更新、如何选择检查点，以及是否可以利用仿真来减轻真实数据收集的成本。我们在一项既需要精确操作又需要对场景空间理解的任务中研究这些问题。在该任务中，一只SO-101机械臂将一只象棋国王移动到两个仅以名称命名的格子之间，如e4和a5。我们通过远程操作演示以及由随机脚本专家模拟人类演示的变异性，微调不同数据集大小和更新预算的$\pi_{0.5}$。在模拟中，成功率会随着数据和更新而增长。重要的是，我们发现学习率冷却时间较短可累积31分，且对比较检查点是必要的，且验证丢失和动作误差对模型选择不可靠。在实臂上，我们对多达256个远程操作演示进行了微调，并在25次试验中评估了10个策略。部分结果表明，在模拟演示上的预训练有显著帮助，尤其是在真实数据稀缺时。我们还给出了一个负面结果，即在策略自身的推广中，使用过滤行为克隆、RECAP式优势条件、自动教练修正或残余强化学习的训练，在我们的环境中并未相比纯行为克隆产生可测量的改善。我们发布了模拟器、专家和真实演示。

Safe Reinforcement Learning for Multi-Turn Clinical Agents: When Dense Rewards Beat Outcome Rewards

多回合临床代理的安全强化学习：当密集奖励胜过结果奖励时

• Authors: Brendan P. Murphy, Agustina Saenz
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.12728
• Pdf link: https://arxiv.org/pdf/2611.12728
• Abstract Reinforcement learning with verifiable rewards is increasingly used to improve language model reasoning, yet its effects on sequential decision procedures remain poorly understood. We study this problem in clinical multi-turn agents that must decide what information to request, when to stop gathering evidence, and which action to take. We introduce three controlled environments: medication refill triage, acute diagnostic work-up, and dosing titration, each with hidden clinical states, query costs, exact adaptive oracles, and deterministic outcome verifiers. Across these tasks, outcome-only group relative policy optimization often improves aggregate reward while producing undesirable procedures, including premature commitment, one-question collapse, and failures on rare high-risk cases. In diagnostics, assigning dense credit to questions that reduce uncertainty improves held-out value and moderate-case support across three training seeds relative to outcome-only reinforcement learning, without the high-risk seed collapse observed under outcome-only training. Separating exploration from exploitation further increases high-risk support at its best checkpoint, but at the cost of excessive information gathering on low-acuity cases. Additional experiments show that the optimal strength of process credit is task-dependent; that after a supervised warmup that already gathers carefully, reinforcement learning raises aggregate reward by trimming questions while degrading high-risk performance; and that reinforcement learning does not reliably repair rare behavior gaps inherited from biased supervised data. Verifiable final outcomes are therefore insufficient for training reliable multi-turn agents: reward design must account for the procedure used to reach an answer, and evaluation must distinguish adaptive information gathering from reward-exploiting shortcuts.
• 中文摘要 带有可验证奖励的强化学习越来越多地用于提升语言模型推理，但其对顺序决策过程的影响仍不充分。我们在临床多回合代理中研究该问题，这些代理必须决定请求哪些信息、何时停止收集证据以及采取何种行动。我们引入了三种受控环境：药物续方分诊、急性诊断检查和剂量调整，每个环境都隐藏了临床状态、查询成本、精确自适应预言机和确定性结果验证器。在这些任务中，仅结果组相对策略优化常常提升总体奖励，同时产生不良操作，包括过早承诺、单问崩溃以及罕见高风险病例的失败。在诊断中，对降低不确定性的问题赋予密集信用，相较于仅结果强化学习，能提升三个训练种子的保留价值和中等案例支持，同时避免仅结果训练中观察到的高风险种子崩溃。将探索与利用分离进一步提升了高风险支持在其最佳检查点，但代价是对低严重案例进行过多信息收集。更多实验表明，过程认可的最佳强度依赖任务;在已经仔细收集的监督热身后，强化学习通过修剪问题提升了总体奖励，同时降低了高风险表现;且强化学习无法可靠地修复由偏见监督数据继承的罕见行为缺口。因此，可验证的最终结果不足以训练可靠的多回合代理：奖励设计必须考虑得出答案所用的过程，评估还必须区分自适应信息收集与利用奖励的捷径。

Large Language Models as a Universal Searcher for Open-World Automated Machine Learning

大型语言模型作为开放世界自动化机器学习的通用搜索器

• Authors: Yuxuan Hu, Jihao Liu, Weikang Shi, Xijie Huang, Linda Wei, Mingjie Zhan, Aojun Zhou, Rui Liu, Hongsheng Li
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.12791
• Pdf link: https://arxiv.org/pdf/2611.12791
• Abstract Developing effective machine learning models requires complex manual tuning, while current Automated Machine Learning (AutoML) frameworks remain domain-specific and isolate model architecture, training hyperparameter, and data ratio optimization. Here, we bridge this gap by proposing a universal, large language model (LLM)-based AutoML framework that jointly optimizes these configurations for the first time without the need for task-specific fine-tuning. We propose NCode, a unified numerical encoding scheme that translates diverse AutoML design choices into universal discrete sequences and effectively reframes the optimization process as a unified ranking task. Trained solely on optimization trajectories derived from a limited, single-domain vision task, the proposed LLM-based searcher demonstrates impressive cross-task, cross-modality and joint search-space optimization generalizability, achieving state-of-the-art performance across image classification, image segmentation, speaker verification and text-to-image benchmarks. Crucially, it exhibits superior scalability in the large-scale training and joint optimization of a billion-parameter Diffusion Transformer, requiring 896 GPU-days and attaining a state-of-the-art GenEval score of 0.91. By leveraging reinforcement learning to instill Chain-of-Thought reasoning, we provide a systematic way to interpret the LLM-based searcher's internal decision-making mechanisms, revealing its optimization strategy shifted from the early-stage diversity-driven exploration to the late-stage anchor exploitation. Our proposed framework provides a scalable, generalizable, and interpretable pathway toward general-purpose open-world machine learning optimization.
• 中文摘要 开发有效的机器学习模型需要复杂的手动调优，而现有的自动化机器学习（AutoML）框架仍保持领域特定，并隔离模型架构、训练超参数和数据比例优化。在此，我们提出一个通用的大型语言模型（LLM）框架，首次联合优化这些配置，无需任务特定微调。我们提出了NCode，一种统一的数值编码方案，将多样化的AutoML设计选择转换为通用离散序列，有效将优化过程重新框架为统一排名任务。仅基于有限单域视觉任务的优化轨迹训练，基于LLM的搜索器展现了令人印象深刻的跨任务、跨模态和联合搜索空间优化能力，在图像分类、图像分割、说话者验证和文本到图像基准测试中实现了最先进的性能。关键是，它在十亿参数扩散变换器的大规模训练和联合优化中展现出卓越的可扩展性，需要896个GPU日，并实现了最先进的GenEval评分0.91。通过利用强化学习灌输思维链推理，我们提供了一种系统解读基于LLM的搜索器内部决策机制的方法，揭示了其优化策略从早期多样性驱动探索转向后期锚点利用的转变。我们提出的框架提供了一条可扩展、可推广且可解释的通用开放世界机器学习优化路径。

Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Agentic Tasks

代理OPD：通过策略上蒸馏恢复被遗忘的代理能力，用于代理任务

• Authors: Tingcheng Bian, Kangye Gong, Jinchang Luo, MingQuan Cheng, Feiran Liu, Ni Li, Xiaoling Xia, Haiwei Wang, Hui Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.12799
• Pdf link: https://arxiv.org/pdf/2611.12799
• Abstract Reinforcement learning is now the mainstream paradigm for training LLMs on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest. We show that this is lossy: in GRPO training of LLM agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved, and the union of problems solved across checkpoints far exceeds any single one. The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation (OPD) can mine to recover the lost coverage. Mining this pool raises two challenges: i) which checkpoint to distill from, as no checkpoint dominates the others and complementarity to the student is problem-specific; ii) where to trust it, as even a well-chosen teacher is only locally superior, so indiscriminate distillation overwrites behavior the student has already mastered. We propose Agentic-OPD, which treats the best-validation checkpoint as the student and all saved checkpoints as frozen teacher candidates, using the student's evolving capability gaps to guide both teacher selection and supervision filtering: the HCN policy (Highest Coverage Next) dynamically selects teachers by weighted coverage of the student's unsolved problems, while the LAD module (Locality-Aware Distillation) confines transfer to failed student rollouts on problems where the teacher performs better. Across five agentic task domains and 24 benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to 7.4 points, with a 4B student surpassing the latest frontier models GPT-5.6 and DeepSeek-V4.1 on deep search. Agentic-OPD recasts a finished RL run as a reusable pool of policies and establishes post-RL capability recovery as a stage orthogonal to improvements in the RL algorithm itself.
• 中文摘要 强化学习现已成为训练大型语言模型执行代理任务的主流范式，但标准做法只保留最佳验证检查点，丢弃其余。我们证明这是有损的：在LLM代理的GRPO训练中，最佳验证检查点在之前同一运行中已解决的问题上失败，且跨检查点解决的问题合并远超任何单个检查点。因此，丢弃的检查点形成了一个教师池，无需额外培训费用，策略蒸馏（OPD）可以挖掘以恢复失去的覆盖。挖掘该检查点带来了两个挑战：i）从哪个检查点提炼，因为没有检查点主导其他检查点，且对学生的互补性是特定问题的;ii）信任哪里，因为即使是精心挑选的教师也只是局部优越，因此无差别的蒸馏覆盖学生已掌握的行为。我们提出了Agentic-OPD，将最佳验证检查点视为学生，所有保存检查点视为冻结教师候选人，利用学生不断演变的能力差距来指导教师选择和监督过滤：HCN政策（Highest Coverage Next）通过对学生未解决问题的加权覆盖动态选择教师，而LAD模块（Locality-Aware Distillation）则将转移限制在教师表现更好的问题上失败的学生推广。在五个代理任务领域和24个基准测试和任务子集中，代理OPD较初始化检查点提升了最多7.4分，4B学生在深度搜索上超过最新的前沿模型GPT-5.6和DeepSeek-V4.1。代理OPD将完成的强化学习运行重新定义为可重用的策略池，并将强化后能力恢复建立为与强化学习算法本身改进正交的阶段。

Simulating Vein Compressibility for Ultrasound-Guided Femoral Access: Speed, Accuracy, and Phantom Validation

超声引导股骨通路的静脉压缩模拟：速度、准确性与幻影验证

• Authors: Amit Manicka, Laurent Coulon, Darwin Ang, Michael Loushin, Timothy M. Kowalewski
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.12818
• Pdf link: https://arxiv.org/pdf/2611.12818
• Abstract In ultrasound-guided femoral access, probe pressure deforms the tissue and changes the vascular anatomy a robot observes. A simulator must reproduce these mechanics fast enough for robotic interaction, yet the computational effort needed for a clinically meaningful signal like vein collapse before artery is unclear. We test the feasibility of a physics-based simulation in NVIDIA's Isaac Sim for Healthcare, with Stable Neo-Hookean tissue solved by Vertex Block Descent on one GPU. A simulation of a low-fidelity cylindrical benchtop phantom with known geometry and mechanical properties, pressed by a robot, predicted its artery within 5 percentage points at every depth. It matched the vein within 3 points through 11 mm of a 25.4 mm press and followed the collapse behavior of the low-fidelity physical phantom. Across element size, time step, and solver iterations, wall clock time per frame scaled linearly with solver work, but accuracy did not improve with more work. The chosen setting reduced solver time per frame by a factor of 976, from 29,700 ms to 30.4 ms, while remaining within 1.03 points of the finest reference setting. The physics solver alone took 30.4 ms per frame, while the full stack, including ultrasound and graphics rendering, took 56.4 ms per frame. Finally, when seeded with realistic literature-based mechanical properties, the digital leg model compressed the vein far more than the artery at every setting, reproducing the expected clinical signal of vein collapse before artery, which is used to distinguish them. In total, these results suggest that the NVIDIA Isaac Sim platform may be suitable for augmenting reinforcement learning training or online simulation for autonomous femoral access with ultrasound.
• 中文摘要 在超声引导股骨通路中，探针压力会变形组织并改变机器人观察到的血管解剖结构。模拟器必须足够快地重现这些机制以实现机器人交互，但对于动脉前静脉塌陷等临床有意义信号所需的计算工作尚不明确。我们在NVIDIA的Isaac Sim for Healthcare中测试基于物理的可行性，该模拟通过一个GPU上的顶点块下降解决稳定新胡克组织。一个由机器人按压的低保真圆柱形台面幻影（几何和机械性能已知）的模拟，预测其动脉在每个深度的5个百分点内。它匹配了25.4毫米压迫后11毫米内3点内的静脉，并跟踪低保真物理幻影的塌陷行为。在元素大小、时间步和求解器迭代中，墙钟每帧时间随求解器工作线性缩放，但精度并未因工作量增加而提升。所选设置将求解器每帧时间减少了976倍，从29,700毫秒降至30.4毫秒，同时保持在与最细参考设置相差1.03个点以内。物理求解器本身每帧耗时30.4毫秒，而全栈（包括超声和图形渲染）每帧耗时56.4毫秒。最后，当植入基于文献的真实机械性质时，数字腿模型在每个设置下压缩静脉远超动脉，重现静脉塌陷前的预期临床信号，用于区分静脉。总体而言，这些结果表明NVIDIA Isaac模拟平台可能适合用于增强学习训练或超声波自主股骨通路在线模拟。

DreamLander: A 6-DOF World Model based Reinforcement Learning Planetary Lander

梦境着陆器：一艘基于6自由度的世界模型强化学习行星着陆器

• Authors: George Nehma, Patrick Quinn, Madhur Tiwari, Camilo Riano-Rios
• Subjects: Subjects: Robotics (cs.RO); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2611.12824
• Pdf link: https://arxiv.org/pdf/2611.12824
• Abstract Soft powered descent landings are becoming evermore important in the field of space exploration and scientific advancement, because of their ability to deliver mission critical payloads, both highly accurately and safely. The high efficiency restriction on planetary landings due to the severely limited fuel supply results in the need for optimal guidance trajectories and control. Model based reinforcement learning is proving to be an important tool in simplifying the highly complex process of generating optimal guidance trajectories and control. In this work, we present DreamLander, a state of the art model based reinforcement learning algorithm based on DreamerV3, to generate optimal guidance and control for a realistic 6 degree of freedom entry descent landing problem. The model based approach creates a sample efficient and highly generalizable model that can be applied to a variety of scenarios whilst maintaining a high degree of accuracy, landing within 2 meters of the designated landing zone, and with velocity norm less than 0.3 meters per second. DreamLander makes use of state sensor observations, making progress toward End-to-End solutions of the Entry Descent Landing problem.
• 中文摘要 软动力下降着陆在空间探索和科学进步领域变得越来越重要，因为它们能够高效且安全地投放关键任务有效载荷。由于燃料供应极度有限，行星着陆的高效限制导致了对最佳制导轨迹和控制的需求。基于模型的强化学习被证明是简化生成最优制导轨迹和控制过程的重要工具。本研究中，我们提出了DreamLander，一种基于DreamerV3的先进基于模型的强化学习算法，用于为现实的六自由度入降着陆问题生成最优的制导和控制。基于模型的方法创建了一个样本高效且高度可推广的模型，可在多种场景中应用，同时保持高度准确性，着陆距离指定着陆区2米以内，速度规范低于0.3米每秒。梦境着陆器利用状态传感器观测，朝着进入下降着陆问题的端到端解决方案取得进展。

MaxIM: Maximally Informative Incremental Summarization via Reinforcement Learning

MaxIM：通过强化学习实现的最大信息性增量总结

• Authors: Jihwan Jeong, Guy Tennenholtz, Yinlam Chow, Chih-Wei Hsu, Craig Boutilier
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12825
• Pdf link: https://arxiv.org/pdf/2611.12825
• Abstract AI systems must often maintain bounded memory of unbounded interaction histories. However, optimizing the memory representation, and its updates, for downstream task utility is difficult for standard reinforcement learning (RL) due to severe credit assignment bottlenecks induced by sparse, delayed rewards. We introduce MaxIM (Maximally Informative Incremental Memory), a framework that formulates incremental summarization as a capacity-constrained, information-theoretic reward-shaping problem. Grounded in value suboptimality bounds, we show that bounding sequential information loss motivates maximizing the predictive log-likelihood of future, task-relevant targets, a variational lower bound on predictive sufficiency. We introduce a recursive consistency condition to ensure summaries are grounded by maximizing mutual information (MI) with the observed history. We mitigate credit assignment issues with a parameterized asymmetric critic that computes step-wise variational MI bounds for dense reward shaping. Empirical evaluation on long-horizon scientific reading and multi-turn dialogue benchmarks shows that MaxIM outperforms RL, supervised fine-tuning, and prompt-focused baselines on factual grounding, comprehensive autorater metrics, and downstream task utility across strict token budgets and extended horizons, without the need for expensive human preference labels.
• 中文摘要 AI系统通常必须保持无界交互历史的有界记忆。然而，由于奖励稀疏、延迟导致严重的信用分配瓶颈，标准强化学习（RL）难以优化内存表示及其更新以实现后续任务效用。我们引入了MaxIM（最大化信息增量记忆），这一框架将增量总结表述为容量受限的信息理论奖励塑造问题。基于价值次最优界限，我们证明了序列信息丢失的边界激励最大化未来任务相关目标的预测对数似然，这是预测充分性的变分下界。我们引入递归一致性条件，确保摘要通过最大化与观察历史的互信息（MI）来建立基础。我们通过参数化非对称批评器来缓解信用分配问题，该批评器计算密集奖励塑形的逐步变分MI界限。对长期科学阅读和多回合对话基准的实证评估表明，MaxIM在事实基础、综合权威指标和下游任务效用方面，在严格代币预算和扩展视野下，表现优于强化学习、监督微调和提示导向基线，无需昂贵的人为偏好标签。

Shortcuts and Boundaries: Verifiable RL Misalignment and Contextual Safety Learning

捷径与界限：可验证的强化学习错位与情境安全学习

• Authors: Xinjie Shen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.12865
• Pdf link: https://arxiv.org/pdf/2611.12865
• Abstract Reinforcement learning (RL) can improve an agent's task capability while reinforcing behavior that violates contextual execution requirements. We study this reward-driven misalignment, including reward hacking, through verifiable reward--context conflicts, where compliant task completion remains feasible while some executable shortcuts earn more reward than every compliant strategy. Encoding these conflicts in task dynamics lets us generate diverse interactive environments at scale and derive compliance feedback from execution histories. In resource and reporting workflows, task optimization increases real productivity through unauthorized resource use or raises reported scores while actual work declines. Controlled changes in the training distribution show that minority conflict exposure can still lead to widespread shortcut acquisition, reaching 97.9\% certified profitable violations on an API task under a 25\% conflict mixture after 128 updates. The same contextual checks supply a simple binary penalty that supports joint capability and compliance gains and partially mitigates acquired violations. Learned safety adapts to expanded authorization and transfers across mechanisms into workflows with evolving evidence and approval requirements. On 32 unseen changing-evidence tasks, resource-trained safety increases safe completion from 53.1\% to 89.1\% relative to task-only training, while both policies achieve full real work. These results connect a verifiable account of reward-driven misalignment with scalable task construction and safety learning that remains useful under changing execution requirements.
• 中文摘要 强化学习（RL）可以提升智能体的任务能力，同时强化违反上下文执行要求的行为。我们通过可验证的奖励——情境冲突研究了这种奖励驱动的错位，包括奖励黑客，即合规任务完成仍可行，而某些可执行捷径获得的奖励超过所有合规策略。将这些冲突编码进任务动态中，使我们能够大规模生成多样化的交互环境，并从执行历史中获得合规反馈。在资源和报告工作流程中，任务优化通过未经授权的资源使用提高实际生产力，或在实际工作下降时提高报告分数。训练分布的受控变化显示，少数群体冲突暴露仍可能导致广泛的捷径获取，在128次更新后，API任务在冲突比例为25%的情况下达到97.9%的认证盈利违规率。同样的上下文检查提供了简单的二元惩罚，支持联合能力和合规的提升，并部分减轻了已获得的违规行为。学习安全会适应扩展授权，并跨机制转移到具有不断演变证据和批准要求的工作流程中。在32个未见的证据变化任务中，资源培训的安全完成率从53.1%提升至89.1%，同时两项策略均实现了完整的实际工作。这些结果将可验证的奖励驱动错位描述与可扩展的任务构建和安全学习联系起来，这些在不断变化的执行需求下依然有用。

Adaptive Quantization Preserves Fast Rates for TD Learning over Rate-Limited Channels

自适应量化保持了在速率受限信道上TD学习的快速速率

• Authors: Sreejeet Maity, Aritra Mitra
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12878
• Pdf link: https://arxiv.org/pdf/2611.12878
• Abstract Motivated by reinforcement learning subject to communication constraints, we study a policy evaluation problem with linear function approximation over a rate-limited channel. In our setup, an agent interacting with an unknown environment (modeled as a Markov Decision Process) transmits quantized estimates of a temporal-difference (TD) learning update direction to a server over a finite-capacity channel. Our goal is to investigate to what extent one can preserve the statistical convergence rates achievable when the channel has infinite capacity. To that end, we propose a new algorithm called Batched Adaptively Quantized TD (BAQ-TD), which uses batching to reduce the variance of the transmitted information, and adaptive quantization to encode innovations. We prove a high-probability finite-time error bound for BAQ-TD, and show that, above a certain finite channel capacity, BAQ-TD preserves the convergence rate of unquantized TD up to logarithmic factors. Specifically, for a $d$-dimensional value function parameter, we establish that transmitting a total of $\mathcal{O}(d \log(T))$ bits over a horizon of length $T$ suffices to achieve the order-optimal rate of $\tilde{\mathcal{O}}(1/T)$, marking an exponential improvement in communication-complexity over prior work.
• 中文摘要 基于在通信约束下的强化学习，我们研究了一个在速率限制信道上线性函数近似的策略评估问题。在我们的设置中，一个与未知环境（建模为马尔可夫决策过程）交互的代理，通过有限容量信道向服务器发送时间差分（TD）学习更新方向的量化估计值。我们的目标是探究当信道具有无限容量时，在多大程度上可以保持统计收敛率。为此，我们提出了一种名为批量自适应量化TD（BAQ-TD）的新算法，利用批处理减少传输信息的方差，并利用自适应量化编码创新。我们证明了BAQ-TD的高概率有限时间误差界限，并证明在一定有限信道容量以上，BAQ-TD能保持未量化TD的收敛率，保持对数倍。具体来说，对于$d$维价值函数参数，我们确定在长度为$T$的视界上传输总共$\mathcal{O}（d \log（T）$的比特，足以达到阶级最优速率$\tilde{\mathcal{O}}（1/T）$，这标志着通信复杂度相较之前工作呈指数级提升。

Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning

重新框架：观察对齐的多教师策略提炼用于代理视觉推理

• Authors: Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.12886
• Pdf link: https://arxiv.org/pdf/2611.12886
• Abstract Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.
• 中文摘要 代理强化学习（RL）和多教师策略提炼（MOPD）为后期训练提供了互补的途径。然而，对于视觉智能体来说，工具使用带来了两个相关挑战。模型可能忽视有用的工具返回观察，而缺乏基础的行为则限制后续提炼的证据。我们引入了Reframe，一种将观察状态而非仅输出分布视为对齐对象的后训练框架。首先，Ground-Drop-GRPO（GD-GRPO）通过证据感知奖励和随机移除原始视觉输入来训练空间和时间专家，鼓励他们获取有根据的证据并利用返回的观察。其次，观察-门控-MOPD（OG-MOPD）在部署过程中用规范工具动作替换不足的工具动作，并恢复从修复后的观察中提取。这两个阶段相互补充，学习依赖工具返回的观测数据有助于政策从观测修复中获益。这两个阶段都不需要针对特定任务的SFT冷启动。对14个图像和视频基准测试的广泛实验表明，Reframe的表现优于Mix-RL和原版MOPD。消融结果显示，GD-GRPO相比基于结果的GRPO提高了观测依赖性和工具收益。

Keyword: diffusion policy

Tiresias: Towards Language-Conditioned World Models

泰瑞西亚斯：迈向语言条件世界模型

• Authors: Fengyi Wu, Yifei Dong, Yetong Sha, Lingdong Kong, Qiyu Hu, Bohan Xiong, Qi Dai, Alexander G. Hauptmann, Zhi-Qi Cheng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.10392
• Pdf link: https://arxiv.org/pdf/2611.10392
• Abstract World models enable planning through internal simulation, while natural language provides intuitive task specification. Yet, language-conditioned world modeling remains fragmented across domains with disparate action spaces, datasets, and evaluation protocols. We introduce LCWM, a shared task that maps an initial agent-view observation and a language instruction to an action sequence that fulfills the instruction. Its accompanying benchmark, LCWM-Bench, integrates 15 datasets across navigation, robotics, 3D gaming, and driving under a shared action interface and evaluation protocol, pairing 142,987 episodes with 428,961 instructions in three styles. We further propose Tiresias, an agent that plans via selective imagination: a language-conditioned diffusion policy generates candidate trajectories, a trajectory ranker shortlists them, and a video-pretrained world model imagines their visual consequences. A visual ranker evaluates these rollouts, complementing trajectory scores to select the final plan. Tiresias achieves competitive planning and imagination performance on LCWM-Bench. Ablations demonstrate the complementary benefits of trajectory ranking and visual verification, while experiments with alternative proposal methods show that the trained rankers remain effective beyond our diffusion policy. Together, LCWM-Bench and Tiresias provide a unified evaluation setting and an effective starting point for studying language-guided planning through world models.
• 中文摘要 世界模型通过内部仿真实现规划，自然语言则提供直观的任务指定。然而，语言条件世界建模仍分散在不同领域，动作空间、数据集和评估协议各异。我们引入了LCWM，这是一个共享任务，将初始代理视图观察和语言指令映射到执行指令的动作序列。其配套基准测试LCWM-Bench整合了15个涵盖导航、机器人、3D游戏和驾驶的数据集，采用共享动作界面和评估协议，配对142,987集剧集和428,961条指令，采用三种风格。我们还提出了Tiresias，一种通过选择性想象来规划的代理：语言条件扩散策略生成候选轨迹，轨迹排序器筛选，视频预训练世界模型想象其视觉后果。视觉排名器评估这些推广，辅以轨迹分数以选择最终方案。Tiresias 在 LCWM-Bench 上实现了竞争性规划和想象力表现。消融展示了轨迹排名与视觉验证的互补优势，而替代提案方法的实验表明，训练有素的排名器在扩散政策之外依然有效。LCWM-Bench 和 Tiresias 共同提供了一个统一的评估环境，并成为通过世界模型研究语言引导规划的有效起点。

Explicitly capturing demonstrator multi-modality for distillation into diffusion policies

明确捕捉示范器多模态以提炼为扩散策略

• Authors: Shao Yuan Chew Chia, Rebecca H. Jiang, Abhinav Agarwal, Andreas Hofmann, Leslie Pack Kaelbling, Russ Tedrake
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.10949
• Pdf link: https://arxiv.org/pdf/2611.10949
• Abstract The diversity of a distilled diffusion policy's outputs matters beyond how it impacts the policy's success rate. In particular, downstream applications like test-time search and policy improvement require exploration. Thus, we want to distill multi-modal demonstrators into diffusion policies which reproduce their conditional action distributions. Standard data collection for behavior cloning applies a sliding window to demonstration episodes, which pairs unique observation histories with actions. These datasets can be fit by a policy that is either multi-modal, or unimodal with high Lipschitz constant to capture a diversity of actions seen at nearby observation samples. Prior work finds that diffusion policies trained on sparse data learn the latter, memorizing one action per observation. We adapt vine sampling to behavior cloning data collection to resolve this ambiguity: at some states the demonstrator visits, query it $k$ times from exactly the same observation. On a synthetic task, we study how the number of actions per observation, $k$, affects the learned policy, and find that larger $k$ yields a policy that better matches the demonstrator's conditional distribution, including its multi-modality and diversity. On a 2D truck unloading task, vine sampling likewise leads to a policy closer to the demonstrator and substantially improves both best-of-$N$ and expert iteration performance.
• 中文摘要 精简扩散策略输出的多样性不仅仅影响策略成功率。特别是，像测试时搜索和策略改进等下游应用需要探索。因此，我们希望将多模态演示器提炼为复制其条件动作分布的扩散策略。行为克隆的标准数据收集对演示片段应用滑动窗口，将独特的观察历史与动作配对。这些数据集可以通过多模态或单模态且利普希茨常数高的策略拟合，以捕捉附近观察样本中观察到的多样动作。先前研究发现，训练在稀疏数据上的扩散策略学习后者，每个观察只记住一个动作。我们将藤采样应用于行为克隆数据收集，以解决这一模糊性：在某些州，示范者访问时，从完全相同的观察中查询$k$次。在综合任务中，我们研究每个观察的动作数$k$如何影响所学策略，发现更大的$k$会得到更符合演示者的条件分布的策略，包括其多模态性和多样性。在二维卡车卸货任务中，藤采样同样使策略更接近演示者，显著提升了最佳的 $N 美元和专家迭代性能。

DPDP: Diffusion Policy-Steered Diffusion Planner for Offline Reinforcement Learning

DPDP：离线强化学习的扩散策略导向扩散规划器

• Authors: Ruyi An, Mingyuan Zhou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.10993
• Pdf link: https://arxiv.org/pdf/2611.10993
• Abstract In offline reinforcement learning, diffusion models fill two roles with complementary strengths. Typically guided by a critic, a diffusion policy is value-aware and favors high-return, behavior-supported actions, but it is short-sighted: it never lays out the states it intends to visit. A diffusion planner is far-sighted, synthesizing such a long-horizon trajectory, but it is only behavior-driven at execution, realizing each planned transition using only offline this http URL is strong exactly where the other is weak. We introduce DPDP to bring them together at the planner's execution step, where a planned transition becomes an action. Reading the planner's inverse-dynamics executor as a posterior over actions: the environment's transition likelihood relates an action to the planned next state, while the behavioral prior determines its weight under the offline data. Replacing this prior with a value-optimized diffusion policy preserves the planned transition likelihood and favors valuable actions that can serve the plan. We target the resulting clean-action distribution in two ways. A training-free sampler composes policy and inverse-dynamics scores at matched action noise levels and applies Langevin correction; this route uses a shared action noise schedule and an approximately common terminal prior. A preference objective instead amortizes the transfer into a single diffusion executor. Across D4RL, DPDP improves the planner it builds on, staying competitive with reactive diffusion policies on locomotion and leading on long-horizon navigation.
• 中文摘要 在离线强化学习中，扩散模型承担了两个互补优势的角色。通常由批评者引导，扩散策略具有价值意识，偏好高回报、行为支持的行动，但它目光短浅：从未明确说明其打算访问的状态。扩散规划器具有远见，综合了这样一个长期视野的轨迹，但它仅在执行时以行为驱动，实现每个计划中的转移仅使用离线，而该http URL恰好在另一个较弱的地方强。我们引入DPDP将它们整合到规划者的执行步骤，即计划中的过渡成为行动。将规划者的逆动态执行者视为行动的后验：环境的过渡似然将动作与计划中的下一状态联系起来，而行为先验决定其在离线数据下的权重。用价值优化的扩散策略替代先验，可以保留计划中的转换概率，并优先考虑有价值的行动来服务于计划。我们通过两种方式针对最终的净动作分布进行目标。无训练采样器在匹配的动作噪声水平下组合策略和逆动力学分数，并应用朗之文修正;该路径使用共享动作噪声计划和近似共同终端先验。偏好目标则将转移摊销到单个扩散执行器中。在D4RL中，DPDP改进了其所构建的规划器，保持在移动中与反应扩散策略的竞争力，并在长视野导航中领先。

PAIR: Proprioception-Aligned Image-Ray Representations for Viewpoint-Robust Manipulation

PAIR：本体感觉对齐图像-射线表示，用于视点稳健操作

• Authors: Sung-Wook Lee, Xuhui Kang, Dylan Losey, Yen-Ling Kuo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.11798
• Pdf link: https://arxiv.org/pdf/2611.11798
• Abstract While imitation learning has made substantial progress toward generalizable robot manipulation through scaling, viewpoint robustness remains a major bottleneck. Visuomotor policies rely heavily on visual inputs, yet images provide an inherently ambiguous signal under viewpoint change: similar scenes may appear markedly different across viewpoints, while distinct states may share certain visual overlaps. A policy must therefore infer not only the task-relevant state, but also how it is visually projected under a particular camera configuration. This entanglement of viewpoint-dependent appearance and task-relevant state can make learning inefficient and limit generalization to unseen viewpoints. Recent work has begun to address this challenge through camera-ray conditioning and representation learning using geometric cues to induce invariance. However, providing camera information as additional context does not by itself ground visual representations in how the task scene is visually projected, while geometric alignment without camera context can introduce spurious correspondences by ignoring viewpoint-dependent projection. We propose PAIR, a contrastive learning framework that aligns joint image-ray representations using robot proprioception. PAIR encourages observations with similar robot states to remain close in representation space across viewpoints, providing a structured grounding signal for learning viewpoint-robust features. PAIR is a highly adaptable framework: it can be used to train Diffusion Policy (DP) or post-train vision-language-action (VLA) models. In simulated and real-world manipulation benchmarks with camera variation, PAIR consistently outperforms the behavior cloning baseline without ray conditioning by 12.7% and 26.7% points, respectively, demonstrating stronger generalization to unseen viewpoints. Project website: this http URL.
• 中文摘要 尽管模仿学习已在通过缩放实现机器人通用操作方面取得了显著进展，但视点鲁棒性仍是一个主要瓶颈。视觉运动策略高度依赖视觉输入，但图像在视点切换下本质上提供了模糊信号：相似场景在不同视角之间可能明显不同，而不同状态可能存在某些视觉重叠。因此，策略不仅要推断任务相关状态，还要推断其在特定摄像机配置下的视觉投射方式。视点依赖的外观与任务相关状态的纠缠可能使学习效率降低，限制对未可见视角的泛化。近期工作开始通过相机-射线条件反射和使用几何线索诱导不变性进行表征学习来应对这一挑战。然而，仅仅提供相机信息作为额外上下文，并不能在任务场景的视觉投影中实现地面视觉表示，而没有相机上下文的几何对齐可能会通过忽略视点依赖投影引入虚假对应关系。我们提出了PAIR的对比学习框架，利用机器人本体感觉对联合图像-射线表示进行对比。PAIR鼓励机器人状态相似的观察在各视点的表示空间中保持接近，为学习视点稳健特征提供结构化的基础信号。PAIR是一个高度适应性的框架：可用于训练扩散策略（DP）或训练后视觉-语言-动作（VLA）模型。在模拟和真实操作基准测试中，具有摄像头变化，PAIR在无光线调控时的行为克隆基线表现持续优于12.7%和26.7%，展示了对未见视角更强的推广性。项目网站：此 http URL。

Grasp-Taxonomy Conditioning as a Mode Switch for Dexterous Grasping of Climbing Holds

抓取分类制约作为灵巧攀爬抓握的模式开关

• Authors: Rumi Loghmani, Abraham George, Amir Barati Farimani
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.12017
• Pdf link: https://arxiv.org/pdf/2611.12017
• Abstract Dexterous grasping in robotics relies on distinct hand configurations depending on object geometry. We introduce BetaGrasp, a 3D diffusion policy that conditions a DP3-style point-cloud policy on a four-class grasp label (crimp, sloper, pinch, and jug), running on a Franka arm with a 16-DoF LEAP hand. We also introduce a climbing-hold benchmark: a spring testbed whose ratchet gives a slip-force reading in newtons. In our analysis the policy's point-cloud encoder did not separate the four holds, so the scan alone gave it little to choose a grasp from. Without the label (a DP3-style baseline), simulated rollouts from the robot's reset pose all end in a jug grasp, whatever the hold (25% correct grasp type, versus 94% for BetaGrasp). The label works as a switch for which grasp the policy performs. On 80 paired hardware trials, BetaGrasp raises average slip force from 9.6 N to 14.3 N, and grasps slipping fully off the hold drop from 44% of trials to 10%.
• 中文摘要 机器人中的灵巧抓取依赖于根据物体几何形状的不同手部配置。我们引入了BetaGrasp，一种三维扩散策略，将DP3风格的点云策略建立在四类抓取标签（压接、斜接、捏和壶）上，运行在Franka臂上，手为16度LEAP。我们还引入了攀爬保持基准：一个弹簧测试床，其棘轮给出以牛顿为单位的滑动力读数。在我们的分析中，策略的点云编码器未能分离这四个抓点，因此仅扫描时几乎没有选择抓握的依据。没有标签（类似DP3的基线），机器人复位姿态的模拟滚动全部以壶抓结束，无论握法类型为25%正确，而BetaGrasp为94%）。该标签作为一个开关，控制策略对此起作用。在80次配对硬件试验中，BetaGrasp将平均滑动力从9.6牛进提升到14.3牛顿，握把完全脱离抓握的次数从44%降至10%。

PCTac: Mechanochromic Photonic-Crystal Tactile Sensor for Dexterous Manipulation

PCTac：机械变色光子晶体触觉传感器，用于灵巧操作

• Authors: Ruilin Zhang, Xuanshuo Liu, Junzhe Dai, Kefei Wu, Xuhao Qin, Ziyuan Jiao, Wanlin Li, Yurui Qu, Chenxi Xiao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.12383
• Pdf link: https://arxiv.org/pdf/2611.12383
• Abstract Modern visuotactile sensors primarily sense contact by imaging the deformation of a compliant interface. Using techniques such as photometric stereo, they can recover detailed contact geometry, often represented as surface normals or depth. However, force-related information typically still needs to be inferred indirectly through calibration, geometric reconstruction, or learned mappings. Mechanochromic photonic-crystal (PC) materials offer an alternative physical encoding mechanism: contact loading changes their periodic microstructure and spectral response, directly transforming local stress into a spatially resolved structural-color field. Despite this potential, PC materials have rarely been explored as visuotactile sensors for closed-loop robot learning. We present PCTac, an open-source mechanochromic PC fingertip that uses a dense hue field as its tactile observation. Loading characterization demonstrates a repeatable relationship between the applied normal load and the observed hue response. We further integrate this representation directly into a Diffusion Policy and evaluate it on three real-robot manipulation tasks. Across 90 physical rollouts, PCTac Hue achieves 68 successes (75.6\%), compared with 60 (66.7\%) for PCTac RGB, 57 (63.3\%) for Wedge RGB, and 29 (32.2\%) for vision-only. PCTac Hue also achieves the highest observed success rate on each evaluated task, including Perturbed Test-Tube Insertion, where it achieves 16/30 successes versus 13/30 for tactile RGB baselines and 1/30 for vision-only. These results demonstrate the potential of mechanochromic structural color as a compact, load-sensitive tactile representation for robot learning and closed-loop manipulation. Project page: this https URL.
• 中文摘要 现代视触传感器主要通过成像顺应性界面的变形来感知接触。利用光度立体技术，它们可以恢复详细的接触几何，通常以表面法线或深度表示。然而，与力相关的信息通常仍需通过校准、几何重建或学习映射间接推断。机械色相光子晶体（PC）材料提供了另一种物理编码机制：接触加载改变其周期性的微观结构和光谱响应，直接将局部应力转化为空间分辨的结构色场。尽管有此潜力，PC材料很少被作为闭环机器人学习的视觉触觉传感器。我们介绍PCTac，一种开源的机械色素PC指尖传感器，利用密集的色场作为触觉观察。载荷表征展示了施加的法向载荷与观测到的色相响应之间的可重复关系。我们进一步将该表示直接整合进扩散策略，并在三项真实机器人操作任务中进行评估。在90次物理部署中，PCTac Hue取得了68次成功（75.6%），而PCTac RGB为60次（66.7%），楔形RGB为57次（63.3%），仅视觉为29次（32.2%）。PCTac Hue在包括扰动试管插入在内的每项评估任务中均取得了最高的观察成功率，在该测试中，其成功率为16/30，而触觉RGB基线为13/30，仅视觉为1/30。这些结果展示了机械色色结构色彩作为紧凑且负载敏感的触觉表示，用于机器人学习和闭环操作的潜力。项目页面：此链接 URL。

Learning Heterogeneous Tactile Representation for General Dexterous Manipulation

学习异质触觉表征以实现通用灵巧操作

• Authors: Bing Han, Yichuan Peng, Haoyu Dong, Yuanmeng Hu, Yongyi Jia, Fangjing Wang, Kecheng Zheng, Yiming Zhang, Yanhong Zeng, Xiaoqing Zhang, Chen Song, Rundong Zou, Anqi Dong, Yuan Guan, Ka Leong Cheng, Xing Zhu, Yujun Shen, Nan Xue, Wei Ning
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.12806
• Pdf link: https://arxiv.org/pdf/2611.12806
• Abstract Tactile sensing is an indispensable modality for contact-rich dexterous manipulation, a capability that hints at the ultimate frontiers of embodied intelligence. However, commercially available tactile sensors are still at a relatively early stage of development, with each type of sensor offering irreplaceable strengths alongside its own drawbacks. A key challenge is thus to free tactile understanding from sensor-specific physics, so that embodied models can effectively leverage heterogeneous tactile data. To this end, we develop LingBot-Touch, a tactile foundation model characterized by the following properties. (i) Our model yields a unified representation across different tactile sensors (i.e., piezoresistive, magnetic, and visuotactile sensors), and is easily extensible to new sensor types. (ii) The resulting tactile tokens are plug-and-play: they can be seamlessly integrated into policy models across paradigms, from diffusion policy to vision-language-action (VLA) models such as $\pi_{0.5}$ and LingBot-VLA, consistently enhancing manipulation dexterity. (iii) Extensive experiments, spanning simulation and real-robot deployments across multiple hardware platforms, demonstrate that our tactile representation substantially benefits complex manipulation tasks while maintaining strong computational efficiency. We believe the findings in this work demonstrate the potential of leveraging cross-sensor data to scale tactile learning, and offer encouraging insights for the design of future hardware and embodied models.
• 中文摘要 触觉感知是实现接触丰富且灵巧操作的不可或缺的模式，这一能力预示着具身智能的终极前沿。然而，市售触觉传感器仍处于相对早期的发展阶段，每种传感器都具有不可替代的优势和自身的缺点。因此，一个关键挑战是将触觉理解从传感器特定物理中解放出来，使具身模型能够有效利用异构的触觉数据。为此，我们开发了LingBot-Touch，这是一种具有以下特性的触觉基础模型。（i）我们的模型在不同触觉传感器（如压阻、磁性和视觉触觉传感器）之间实现了统一的表示，并且易于扩展到新型传感器类型。（ii）所得的触觉令牌即插即用：可无缝集成到跨范式的策略模型中，从扩散策略到视觉-语言-行动（VLA）模型，如$\pi_{0.5}$和LingBot-VLA，持续提升操作灵活性。（iii）跨多个硬件平台的模拟和实机器人部署的广泛实验表明，我们的触觉表征在复杂操作任务中显著提升，同时保持了强大的计算效率。我们相信，这项研究的发现展示了利用跨传感器数据扩展触觉学习的潜力，并为未来硬件和具身模型的设计提供鼓舞人心的见解。