生成时间: 2026-11-06 23:45:25 (UTC+8); Arxiv 发布时间: 2026-11-06 20:00 EST (2026-11-07 09:00 UTC+8)

今天共有 41 篇相关文章

Keyword: reinforcement learning

Octopus: One Brain, Many Branches for Latent-KV Multi-Agent Reinforcement Learning

章鱼：一个大脑，多分支用于潜在KV多智能体强化学习

• Authors: Tianming Sha
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.06956
• Pdf link: https://arxiv.org/pdf/2611.06956
• Abstract Multi-agent LLM systems explore in parallel by spawning sub-agents with isolated context windows, whereas single-agent systems append every search and verification step into one growing trajectory. However, existing multi-agent frameworks duplicate prefix KV caches across branches, exchange findings by re-prefilling text summaries, and flatten or drop cross-branch dependencies during reinforcement learning (RL). We introduce Octopus, a single-model multi-agent RL framework in which all threads share one backbone and coordinate over a GPU-resident directed acyclic graph (DAG) of KV states through five primitives (fork, read_thread, wait_threads, rewind, commit). Octopus combines three co-designed mechanisms: (1) Zero-Copy Radix Forking, which spawns branches as $\mathcal{O}(1)$ cursor views over shared physical KV pages; (2) Attention-Only KV Mounting, which mounts a sibling thread's terminal KV blocks directly into full-attention layers while bypassing linear-recurrent states (Gated DeltaNet / Mamba-2), cutting off-task hijacking by $21\times$ ($57.9\% \to 2.7\%$) while retaining $92.2\%$ of full-text mutual information at zero extra KV allocation; and (3) a Packed DAG-GRPO Trainer, which packs all episode branches into one block-sparse attention pass and assigns counterfactual branch credit via leave-one-branch-out marginal rewards. Trained end-to-end with GRPO on Qwen3.5-4B over a sparse BM25 corpus, Octopus reaches $49.6\%$ accuracy on BrowseComp-Plus ($+17.5$ points over single-agent GRPO at 143K context and $+12.5$ points over text-summary multi-agent RL) and $51.2\%$ on OpenResearcher, outperforming dense-retrieval agents up to $250\times$ larger while increasing training throughput $3.3\times$ and cutting peak KV memory $2.8\times$.
• 中文摘要 多智能体LLM系统通过生成具有隔离上下文窗口的子智能体并行探索，而单智能体系统则将每个搜索和验证步骤附加到一个增长轨迹中。然而，现有多智能体框架在分支间重复前缀KV缓存，通过重新填充文本摘要交换发现，并在强化学习（RL）期间平整或删除跨分支依赖关系。我们介绍了Octopus，一个单模型多智能体RL框架，所有线程共享一个骨干，并通过GPU驻留的有向无环图（DAG）协调KV状态，通过五个原语（fork、read_thread、wait_threads、rewind、commit）进行协调。Octopus结合了三种共同设计的机制：（1）零副本基分叉，在共享的物理KV页面上生成分支，作为$\mathcal{O}（1）$光标视图;（2） 仅注意 KV 挂载，将兄弟线程的终端 KV 块直接挂载到全注意力层，绕过线性循环状态（Gated DeltaNet / Mamba-2），将任务劫持减少 $21\times$（$57.9% 到 2.7%$），同时保留 $92.2\%$ 的全文互信息，且无额外 KV 分配;（3） Packed DAG-GRPO Trainer，将所有集数分支打包为一个区块稀疏的注意力传递，并通过保留一分支的边际奖励分配反事实分支积分。通过在Qwen3.5-4B上端到端训练，基于稀疏的BM25语料库，章鱼在BrowseComp-Plus上达到49.6美元准确率（在143K上下文下比单代理GRPO高+17.5美元，比文本摘要多智能体强化语言+12.5美元）和OpenResearcher上的51.2美元，性能优于高达250美元/倍数的密集检索智能体，同时提升训练吞吐量3.3倍/倍，峰值KV内存减少2.8倍/倍。

Resource-Efficient Reinforcement Learning Post-Training through Elastic Expert Parallelism

通过弹性专家并行处理的资源高效强化训练后学习

• Authors: Jia Wei, Ziyi Qiu, Yi Xu, Xuehai Qian
• Subjects: Subjects: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.06984
• Pdf link: https://arxiv.org/pdf/2611.06984
• Abstract Reinforcement learning post-training improves the capabilities of large language models, but generating training responses is a major computational bottleneck. In mixture-of-experts models, workers that finish their assigned responses may remain occupied because their experts are still needed by workers processing longer responses. Consequently, approximately one-third of occupied worker time in the baseline occurs after local response generation finishes. Here we present FlexFloor, a system that enables earlier resource release during synchronous response generation while preserving the expert computation required by ongoing responses. FlexFloor jointly adapts response scheduling and the active group of workers. It uses historical response lengths and memory constraints to guide batching and prompt placement, then progressively releases workers while maintaining expert availability and correct communication. Compared with the verl baseline, FlexFloor generates approximately 65.40% more response tokens per unit of occupied computing time and increases response generation throughput by 52.74%, with little observed variation in rollout reward. By making reinforcement learning post-training more efficient, FlexFloor supports faster development and iterative improvement of large language models and AI agents.
• 中文摘要 训练后强化学习提升了大型语言模型的能力，但生成训练响应是一个主要的计算瓶颈。在专家混合模型中，完成分配响应的工人可能仍然忙碌，因为他们的专家仍被处理较长响应的工人所需要。因此，大约三分之一的基线工作者被占用时间发生在局部响应生成结束之后。这里介绍FlexFloor系统，该系统在同步响应生成时实现资源提前释放，同时保持持续响应所需的专家计算。FlexFloor联合调整响应调度和活跃工作组。它利用历史响应长度和内存约束来指导批处理和提示放置，然后在保持专家可用性和正确沟通的同时逐步释放工作者。与verl基线相比，FlexFloor每单位占用计算时间生成的响应代币数量约多出65.40%，响应生成吞吐量提升52.74%，且部署奖励变化较小。通过提高训练后强化学习的效率，FlexFloor支持大型语言模型和AI代理的快速开发和迭代改进。

Cooperative Multi-Agent Reinforcement Learning via Epigraph-Form Guided Exploration

通过Epigraph形式引导探索的合作多智能体强化学习

• Authors: Sungil Son, Hoseong Jung, Dahyun Oh, H. Jin Kim
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.07066
• Pdf link: https://arxiv.org/pdf/2611.07066
• Abstract Discovering cooperative behavior in multi-agent reinforcement learning (MARL) is challenging due to the combinatorial complexity of joint state-action spaces, which hinders the emergence of coordinated behaviors from trial-and-error alone. Intrinsic rewards are often used to aid discovery, but naively combining them with team objectives can distort the learning signal, compromising task performance. In this paper, we propose EFXPLORER, a constrained exploration framework that maximizes exploration objectives subject to a constraint that preserves established task performance. We solve the resulting problem using an epigraph reformulation that introduces adaptive exploration budgets. This approach separates intrinsic rewards from task objectives and regulates exploration through task feasibility. To further encourage diverse and temporally extended exploration, we incorporate a successor distance-based intrinsic reward that captures long-horizon dependencies. Empirically, our method outperforms strong baselines and induces novel cooperative strategies across SMAX, VMAS, and MPE benchmark suites.
• 中文摘要 在多智能体强化学习（MARL）中发现合作行为具有挑战性，因为联合状态-行动空间的组合复杂性阻碍了仅靠试错形成协调行为。内在奖励常被用来辅助发现，但若天真地将其与团队目标结合，可能会扭曲学习信号，损害任务表现。本文提出EFXPLORER，一种受限探索框架，在保留既有任务表现的约束条件下最大化探索目标。我们通过引入自适应探索预算的引言重述解决了问题。该方法将内在奖励与任务目标分离，并通过任务可行性调节探索。为进一步鼓励多样化且时间延伸的探索，我们引入了基于距离的内在奖励，捕捉长期依赖关系。从实证角度看，我们的方法优于强基线，并在SMAX、VMAS和MPE基准测试套件中引入了新的合作策略。

Finite-Time Convergence and Regret Guarantees for Neural Reinforcement Learning in POMDPs

POMDP中神经强化学习的有限时间收敛与遗憾保证

• Authors: Jingjuan Huang, Zhiyao Zhang, Alvaro Velasquez, Nathaniel D. Bastian, Jia Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.07069
• Pdf link: https://arxiv.org/pdf/2611.07069
• Abstract Reinforcement learning (RL) in Partially Observable Markov Decision Processes (POMDPs) is computationally challenging due to the curse of long-range dependence on history. Fortunately, recent studies have shown that a range of POMDPs that satisfy the filter stability assumption can be well approximated by a superstate Markov Decision Process (MDP) based on a truncated history in a finite-memory window, with a value gap that decays with the window length. However, existing finite-time convergence analyses on such superstate MDP surrogates are either limited to over-simplified linear function approximation or rely on an i.i.d. sampling oracle that is violated by POMDP trajectories, both of which severely limit the applicability of these theoretical results in practice. This motivates us to bridge the gap by investigating RL in POMDPs with nonlinear neural value function approximation. Our main contributions are threefold: i) We first establish a new finite-state MDP point-wise approximation error bound, which lays the foundation for our RL algorithm design and analysis in POMDPs; ii) By applying Temporal-Difference (TD) learning to POMDP-generated superstate transitions and targeting the Bellman fixed point of the superstate MDP, we prove a finite-time critic error bound with nonlinear neural value function approximation; and iii) By combining the neural critic function approximation error bound with the policy mirror descent framework, we obtain a finite-time sublinear regret against the optimal POMDP value. To the best of our knowledge, such a finite-time sublinear regret based on neural function approximation is the first in the literature for finite-memory POMDPs. Our numerical experiments also verify our theoretical results.
• 中文摘要 部分可观测马尔可夫决策过程（POMDPs）中的强化学习（RL）由于对历史的长期依赖性，计算上具有挑战性。幸运的是，近期研究表明，满足滤波器稳定性假设的一系列POMDP可以通过基于有限内存窗口中截断历史的超态马尔可夫决策过程（MDP）很好地近似，MDP的值缺口随窗口长度衰减。然而，现有对此类超态MDP代理的有限时间收敛分析要么局限于过度简化的线性函数近似，要么依赖于被POMDP轨迹破坏的独立自变量采样预言机，这两者都严重限制了这些理论结果在实际中的适用性。这促使我们通过非线性神经价值函数近似研究POMDP中的强化学习来弥合这一差距。我们的主要贡献有三方面：i）首先建立了新的有限状态MDP点对点近似误差界限，为我们在POMDP中的强化学习算法设计和分析奠定基础;ii）通过对POMDP生成的超状态转变应用时间差分（TD）学习，并针对超状态MDP的Bellman不动点，我们证明了一个有限时间批判误差界限，且非线性神经价值函数近似是有效的;iii）通过结合神经批判函数近似误差界限与政策镜像下降框架，我们获得了对最优POMDP值的有限时间亚线性遗憾。据我们所知，基于神经函数近似的有限时间亚线性遗憾是文献中首个针对有限记忆POMDP的应用。我们的数值实验也验证了理论结果。

Regulating Autonomous Mobility-on-Demand Operators with Fixed and Adaptive Penalties: A Multi-Agent Reinforcement Learning Approach

以固定和自适应惩罚监管自主出行按需运营者：多智能体强化学习方法

• Authors: Korbinian Kügler, Carolin Schmidt, Florian Dandl, Filipe Rodrigues
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.07122
• Pdf link: https://arxiv.org/pdf/2611.07122
• Abstract Profit-maximizing Autonomous Mobility-on-Demand (AMoD) operators may exploit consumer surplus through excessive prices and underserve less profitable regions by relocating vehicles away from them. We study how penalty-based regulation of an AMoD operator in a multimodal mobility system can limit surge pricing and improve regional service availability. We model regulation as an outer control loop that acts on a learning-based operator. Static regulation with fixed penalties is formulated as a Markov Decision Process for an operator that jointly controls pricing and rebalancing, and dynamic regulation as a Partially Observable Stochastic Game in which a regulator periodically adjusts region-specific penalties in closed loop. Operator and regulator policies are learned with graph-based multi-agent reinforcement learning. In experiments with real-world Manhattan data, fixed surge pricing penalties reduce the share of travelers facing excessive prices from 22% to 1%. For regional service availability, dynamic regulation improves the regulator's reward by up to 15.6% over fixed penalties of equal mean intensity. The gains in service availability are strongest when spare fleet capacity exists, whereas service deficits persist in peak periods with highly utilized fleets. The learned regulator policy responds by reducing penalties where limited fleet capacity restricts compliance. The results identify trade-offs between regulatory targets, operator profitability, and traveler utility.
• 中文摘要 追求利润最大化的自主出行按需（AMoD）运营商可能通过高价利用消费者剩余，并通过将车辆移离较低盈利地区来服务不足。我们研究了多模式出行系统中基于罚款的AMoD运营商如何限制突发定价并改善区域服务可用性。我们将监管建模为一个外部控制循环，作用于基于学习的运营商。固定罚款的静态监管被表述为马尔可夫决策过程，用于运营商共同控制定价和再平衡;动态监管则作为部分可观测随机博弈，监管者在闭环中周期性调整区域特定罚款。运营商和监管者策略通过基于图的多智能体强化学习来学习。在使用真实世界曼哈顿数据的实验中，固定高涨涨价罚款将面临高价费用的旅客比例从22%降至1%。对于区域服务可用性，动态监管使监管者的奖励比同等平均强度的固定罚款提升最多15.6%。当存在备用车队容量时，服务可用性的提升最为显著，而服务赤字在高使用率车队的高峰期持续存在。监管者政策通过减少因车队容量有限而限制合规的处罚来应对。结果识别了监管目标、运营商盈利能力和旅客效用之间的权衡。

Teaching Agents the "Principle", Not the "Reward"

教导代理人“原则”，而非“奖励”

• Authors: Shan Yang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.07130
• Pdf link: https://arxiv.org/pdf/2611.07130
• Abstract Reinforcement learning has achieved strong performance in many tasks by letting agents learn from nothing more than a reward signal. However, in more complex tasks, agents still struggle to learn good, generalizable policies, even with carefully designed rewards and abundant samples. We argue that this difficulty stems from the reward decoding burden: designers encode their knowledge of the desired behavior into rewards and penalties, and agents must infer this knowledge from long sequences of noisy scalars. To address this burden, we propose prescriptive policy learning (PPL), which directly teaches agents higher-level knowledge about the tasks, termed principles of behavior, such as how to maintain balance, how to accumulate energy, and how to land smoothly. Through appropriately selected key quantities and references, PPL can directly indicate at every step whether the agent follows the principles and to what extent. Our experiments show that agents can learn good, generalizable policies without any reward. Taught only simple heuristic principles, PPL agents outperform reward-based methods with far fewer samples. Their policies also transfer directly to harder settings and keep performing well, demonstrating their strong generalization ability. In summary, PPL provides a new approach in which rough knowledge of the desired behavior is enough for agents to learn good, generalizable policies, even when the system dynamics cannot be accurately modeled.
• 中文摘要 强化学习通过让智能体仅凭奖励信号学习，在许多任务中取得了优异表现。然而，在更复杂的任务中，智能体即使在精心设计的奖励和丰富样本下，仍难以学习良好且可推广的策略。我们认为，这种困难源于奖励解码负担：设计者将期望行为的知识编码为奖励和惩罚，智能体则必须从长序列的噪声标量推断这些知识。为解决这一负担，我们提出了规范性政策学习（PPL），它直接教授智能体关于任务的高层次知识，称为行为原则，如如何保持平衡、如何积累能量以及如何顺利着陆。通过适当选择的关键量和引用，PPL可以在每个环节直接指示智能体是否遵循这些原则以及遵循的程度。我们的实验表明，智能体可以在没有任何奖励的情况下学习良好且可推广的策略。PPL代理仅教授简单的启发式原则，且样本量远少于基于奖励的方法。他们的策略也能直接转移到更高难度的环境中并持续表现出色，展现了其强大的泛化能力。总之，PPL提供了一种新方法，即便对期望行为的粗略了解，就足以让代理学到良好且可推广的策略，即使系统动态无法被准确建模。

IRIS: Intelligent Random Access with Integrated Shared Learning for Decentralized LEO Direct-to-Satellite Uplinks

IRIS：智能随机接入与集成共享学习，用于去中心化的LEO直连卫星上行链路

• Authors: Abdullah Al Noman, Gaurav Singh, Rameez Ahmed, Debashri Roy
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.07201
• Pdf link: https://arxiv.org/pdf/2611.07201
• Abstract Grant-free uplink access in low-Earth-orbit (LEO) non-terrestrial networks (NTNs) poses a coordination challenge when multiple UEs share a satellite channel without real-time centralized scheduling under long propagation delays and time-varying channels. Conventional random access does not exploit peer context, leading to collisions and inefficient channel use as contention increases. To address this problem, we propose IRIS, a multi-agent reinforcement learning framework for decentralized grant-free direct-to-satellite (D2S) uplink access, with A2A-IRIS as its sidelink-assisted Agent-to-Agent (A2A) realization. Each UE executes a shared binary transmission policy using local traffic/channel information augmented by compact peer states exchanged over UE-to-UE sidelink, while Vanilla-IRIS serves as a local-only ablation. Evaluations in rural and suburban LEO D2S deployments use TLE-based trajectories propagated using the Simplified General Perturbations 4 (SGP4), with a Walker-Delta constellation as fallback. A2A-IRIS achieves 2.1579 and 5.4478 Mbps, corresponding to 99.9% and 96.1% of the respective scenario-specific upper bounds. Relative to Vanilla-IRIS, throughput improves by 40.8% and 325.0%, with collision rates of 0.0000 and 0.0433, while one- and two-epoch delayed centralized schedulers are outperformed by approximately 51--130%. A2A-IRIS also improves fairness, service coverage, and transmission efficiency with moderate Deep Q-Network (DQN) inference overhead. Active-UE load scalability results further show sustained performance over the evaluated load range. These results demonstrate the effectiveness of sidelink-assisted A2A coordination for decentralized LEO D2S uplink access under propagation delay and delayed channel feedback.
• 中文摘要 在低地球轨道（LEO）非地面网络（NTNs）中，当多个UE共享一个卫星信道且缺乏实时集中调度且存在长传播延迟和时变信道时，无资助上行接入会带来协调挑战。传统的随机接入无法利用对等上下文，导致冲突和信道使用效率低下，争用增加。为解决这一问题，我们提出了IRIS，这是一个多代理强化学习框架，用于去中心化无资助直达卫星（D2S）上行接入，A2A-IRIS作为其侧链辅助的代理对代理（A2A）实现。每个UE执行共享的二进制传输策略，利用本地流量/信道信息，辅以通过UE到UE侧链交换的紧凑对等状态，而Vanilla-IRIS则作为仅本地的消融。农村和郊区LEO D2S部署的评估采用基于TLE的轨迹，使用简化一般扰动4（SGP4）传播，沃克-德尔塔星座作为备选。A2A-IRIS实现了2.1579和5.4478 Mbps，分别占了各自场景特定上限的99.9%和96.1%。与原版IRIS相比，吞吐量提升了40.8%和325.0%，碰撞率分别为0.0000和0.0433，而一和两晚延迟的集中调度器则被优于约51%--130%。A2A-IRIS还提升了公平性、服务覆盖率和传输效率，同时实现适度的深度Q网络（DQN）推断开销。主动-联合终端负载可扩展性结果进一步显示，在评估的负载范围内持续表现。这些结果展示了侧链辅助A2A协调在分散式LEO D2S上行访问中，在传播延迟和通道反馈延迟下的有效性。

MAC-v2: Long-horizon model-based value expansion from pixels

MAC-v2：基于长视野的模型从像素的价值扩展

• Authors: Kwanyoung Park, Sergey Levine
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.07245
• Pdf link: https://arxiv.org/pdf/2611.07245
• Abstract We study whether \emph{long-horizon model-based value expansion} can serve as an effective framework in pixel-based offline reinforcement learning (RL). Long-horizon model-based value expansion from pixels is substantially harder than state-based environments, as the agent must learn both the representation and its dynamics, and errors from either can accumulate recursively over imagined trajectories. We introduce MAC-v2, which enables long-horizon model-based expansion from pixels by training a reconstruction-free latent world model with action chunking. By learning representations from prediction and control objectives, MAC-v2 does not need to encode control-irrelevant visual details that can hinder the dynamics prediction. Action chunking further mitigates the prediction error by reducing the number of recursive model predictions required for value expansion. Across diverse visual benchmarks including visual OGBench, MMBench2, Robomimic, and LIBERO-10, MAC-v2 achieves strong performance against model-free and model-based offline RL baselines. Our analyses show that long-horizon model-based value expansion is critical for the performance of MAC-v2, and that both action chunking and latent learning objectives are essential for reliable long-horizon latent imagination.
• 中文摘要 我们研究了 \emph{长视界模型基于价值扩展}是否能作为基于像素的离线强化学习（RL）中的有效框架。从像素进行长视野模型基于价值的扩展远比基于状态的环境更为困难，因为智能体必须学习该表示及其动态，且其中的错误可能递归地在想象轨迹上累积。我们引入了 MAC-v2，通过训练无重建的潜世界模型，使用动作分块实现基于像素的长视界模型扩展。通过从预测和控制目标学习表示，MAC-v2 无需编码可能阻碍动态预测的控制无关视觉细节。动作分块通过减少递归模型预测数量，进一步降低了预测误差。在包括视觉 OGBench、MMBench2、Robomicic 和 LIBERO-10 在内的多种视觉基准测试中，MAC-v2 在无模型和基于模型的离线强化学习基线中表现出色。我们的分析表明，基于长期模型的价值扩展对 MAC-v2 的性能至关重要，动作分块和潜在学习目标对于可靠的长期潜在想象力至关重要。

An Empirical Study of Reinforcement Learning for Keypoint Positioning of Deformable Linear Objects

关于可变形线性物体关键点定位的强化学习的实证研究

• Authors: Vinay Rodrigues, Sara Zeynalpour, Gokhan Solak, Sebastian Hjorth, Arash Ajoudani
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.07261
• Pdf link: https://arxiv.org/pdf/2611.07261
• Abstract Deformable linear object (DLO) manipulation is challenging because high-dimensional configurations, nonlinear contact dynamics, and diverse material and geometric properties produce a broad range of object responses that are both difficult to model and data-intensive to learn. This work studies reinforcement learning for planar manipulation of a one-end-grasped DLO, focusing on training efficiency, observation design, robustness, and zero-shot sim-to-real transfer. We compare three Proximal Policy Optimization training stacks and find that the GPU-parallel MuJoCo Playground stack enables faster convergence and more consistent performance than CPU-based alternatives under the same wall-clock budget. Relative keypoint observations and an asymmetric actor-critic formulation further improve learning efficiency. Policies trained with observation noise, delay, domain randomization, and disturbances achieve simulation success rates of 96.4% for single-point tracking, 98.0% for disturbance recovery, and 88.8% for two-point tracking. Using markerless RGB-D keypoint estimation, the policies transfer without real-world fine-tuning, succeeding in 9/10, 10/10, and 8/10 physical trials, respectively.
• 中文摘要 可变形线性对象（DLO）操作具有挑战性，因为高维构型、非线性接触动力学以及多样的材料和几何属性会产生广泛的对象响应，这些响应既难以建模，也需要大量数据来学习。本研究研究单端抓取 DLO 平面操作的强化学习，重点关注训练效率、观测设计、鲁棒性以及零样本模拟到实的传输。我们比较了三种近端策略优化训练栈，发现 GPU 并行的 MuJoCo Playground 协议栈在相同时钟预算下比基于 CPU 的替代方案实现更快的收敛和更稳定的性能。相对关键点观察和非对称演员-批评者表述进一步提升了学习效率。使用观测噪声、延迟、域随机化和扰动训练的策略，单点跟踪的模拟成功率为96.4%，扰动恢复98.0%，两点跟踪88.8%。利用无标记RGB-D关键点估计，策略转移时无需真实世界微调，分别成功完成了9/10、10/10和8/10的物理试验。

Efficient Diffusion Policy Fine Tuning with Latent Noise Representation Bridging

高效扩散策略与潜在噪声表示桥接的微调

• Authors: Yuanchu Liang, Yue Yang, An-Chi He, Hanna Kurniawati
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.07275
• Pdf link: https://arxiv.org/pdf/2611.07275
• Abstract Pre-trained diffusion and flow policies have emerged as powerful backbones for robotics, yet adapting them to novel data via reinforcement learning (RL) remains computationally expensive, sample inefficient and lacks theoretical guarantees. Prior approaches that fine-tune the initial noise bypass costly backpropagation through time but struggle with the "noise aliasing" problem: there does not exista natural link between the noise space value and the action space value. This paper resolves these challenges by introducing the Latent Noise Linear Markov Decision Process (LNL-MDP). We derive the Bridge Equation for LNL-MDPs which eliminates the noise aliasing problem by aligning values with a shared representation between the noise and the action space. Furthermore, our LNL-MDP fine tuning framework falls under the FineTuneRL theory with near-optimal guarantee. Leveraging these insights, we propose Latent Representation Bridging for Diffusion (LARBRID), an algorithm that efficiently learns the representation to optimize the initial distribution that steers the pre-trained model to act optimally in the environment. It yields up to 2-3$\times$ performance increases on challenging control problems across a diverse suite of robotic locomotion and manipulation tasks. Code can be found at: this http URL.
• 中文摘要 预训练的扩散和流策略已成为机器人的强大骨干，但通过强化学习（RL）将其调整到新数据仍计算成本高、样本效率低下且缺乏理论保证。先前方法通过微调初始噪声旁略，代价高昂的反向传播，但在“噪声混叠”问题上遇到困难：噪声空间值与动作空间值之间不存在自然联系。本文通过引入潜在噪声线性马尔可夫决策过程（LNL-MDP）解决了这些挑战。我们推导了LNL-MDP的桥梁方程，通过将值与噪声与作用空间之间的共享表示对齐，消除噪声混叠问题。此外，我们的LNL-MDP微调框架属于FineTuneRL理论，且保证近乎最优。基于这些见解，我们提出了潜在表征桥接扩散（LARBRID）算法，该算法高效学习表征，优化初始分布，引导预训练模型在环境中表现最佳。在多样化的机器人运动和操作任务中，该算法在复杂控制问题上可带来多达2-3美元\倍数的性能提升。代码链接如下：http URL。

ArtiVidRM: Verifiable Artifact-as-Rubric Reward Modeling for Explicit Quality Articulation in Video Generation

ArtiVidRM：可验证的伪造作为评分标准的奖励建模，用于视频生成中明确的质量表达

• Authors: Caorui Li, Yangguang Ji, Shuxia Lin, Yu Chen, YiCheng Xiao, Yongliang Wu, Zhenxiang Jiang, Licheng Tang, Deng Li, Zhuxuanzi Wang, Jiahao Sun, Chonghan Liu, Lirian Su, Jay Wu, Xu Yang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.07301
• Pdf link: https://arxiv.org/pdf/2611.07301
• Abstract Reinforcement learning has become an effective approach for improving video generation, but its success depends heavily on reliable reward models. Existing methods either map videos directly to opaque scalar scores or generate free-form rationales that are not explicitly optimized as the basis of the final judgment. We propose ArtiVidRM, a verifiable, artifact-as-rubrics video reward model that follows an observe-then-score paradigm. It first identifies visible generation artifacts and organizes them into a structured artifact list, then uses these artifacts as explicit rubrics for quality assessment. This intermediate representation grounds each score in concrete video-level evidence, making the reward more interpretable and informative for optimization. We further introduce a multi-reward reinforcement learning strategy to jointly optimize artifact prediction and quality assessment. The reward decomposes into complementary objectives that evaluate artifact prediction quality, including semantic coverage and prediction efficiency, together with a pairwise preference objective that supervises quality ranking through score differences. By jointly optimizing the intermediate artifact representation and the downstream preference prediction, the model learns to produce informative artifact lists while improving the accuracy of video quality assessment. Experiments show that ArtiVidRM achieves state-of-the-art performance on video preference benchmarks, provides effective signals for video diffusion optimization, and enables artifact-guided refinement of closed-source video generators.
• 中文摘要 强化学习已成为提升视频生成的有效方法，但其成功在很大程度上依赖于可靠的奖励模型。现有方法要么直接将视频映射到不透明的标量分数，要么生成未明确优化的自由形式理由作为最终判断的基础。我们提出了ArtiVidRM，一种可验证的、以人工物为评分标准的视频奖励模型，遵循观察后评分的范式。它首先识别可见生成的人工物，并将其组织成结构化的人工物列表，然后将这些人工物作为明确的评分标准用于质量评估。这种中间表示为每个评分建立在具体的视频级证据基础上，使奖励更具可理解性和信息量，便于优化。我们还进一步引入了一种多奖励强化学习策略，以联合优化人工物预测和质量评估。奖励分解为互补目标，评估伪影预测质量，包括语义覆盖率和预测效率，以及通过分数差异监督质量排名的两两偏好目标。通过联合优化中间伪造物表示和下游偏好预测，模型学会生成有信息的伪影列表，同时提高视频质量评估的准确性。实验表明，ArtiVidRM在视频偏好基准测试上实现了最先进的性能，为视频扩散优化提供了有效信号，并支持伪影导向的闭源视频生成器的细化。

Permanent and Transient Representations for Continual Reinforcement Learning

持续强化学习的永久与瞬态表示

• Authors: Nishanth Anand, Doina Precup
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.07327
• Pdf link: https://arxiv.org/pdf/2611.07327
• Abstract Continual Reinforcement Learning (CRL) agents struggle to adapt to new situations while retaining prior knowledge, leading to a stability-plasticity trade-off. One approach to solving this problem is by using complementary learning systems: one for slow, long-term learning and another for quick, transient adaptation. Anand & Precup (2023) instantiated this idea using permanent and transient value functions, but used the same features in both systems. Intuitively, long-term learning should rely more on parametric representations for broad generalization, while transient learning would be best served by non-parametric representations for situation-specific adaptation. In this paper, we explore this idea both theoretically and empirically. By introducing a novel non-parametric approximation for estimating the transient value function in complex tasks, we demonstrate that our method enables online learning and outperforms competitive baselines on image-based tasks and Craftax-Classic, underscoring the effectiveness of our system-level decomposition.
• 中文摘要 持续强化学习（CRL）智能体在适应新情境时难以保留先验知识，导致稳定性与可塑性之间的权衡。解决该问题的一种方法是使用互补学习系统：一种用于缓慢、长期学习，另一种用于快速、瞬态适应。Anand 和 Precup（2023）通过永久和瞬态价值函数实现了这一思想，但两者都使用相同的特性。直观上，长期学习应更多依赖参数表示进行广泛推广，而瞬态学习则最好依赖非参数表示以适应特定情境。本文从理论和实证角度探讨这一观点。通过引入一种新颖的非参数近似方法估算复杂任务中的瞬态价值函数，我们证明了该方法支持在线学习，并在基于图像的任务和 Craftax-Classic 中优于竞争基线，强调了系统级分解的有效性。

Deep reinforcement learning-based optimization of multi-RAT handovers

基于深度强化学习的多重RAT切换优化

• Authors: Pavan Kumar Mangipudi, Satya Krishna Pothapragada, Alina Zare, Janise McNair
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.07390
• Pdf link: https://arxiv.org/pdf/2611.07390
• Abstract A major evolutionary goal and challenge for next-generation wireless systems is seamless multiple radio access technology (multi-RAT) mobility. In a multi-RAT environment, handover decisions must account for both user quality of service and network priorities, including signal quality, resource availability, energy consumption, latency, and security. Although WiFi/cellular mobility is well explored in research and standardization, many conventional handover mechanisms continue to rely primarily on received signal strength. Such decisions may overlook link delay, congestion, and the distinct operating characteristics of heterogeneous RATs. In this work, we present the Reinforced Flexible Multi-RAT Selection framework (ReFlexMRS), a configurable Deep Reinforcement Learning (DRL)-based approach for multi-criteria handover target selection. ReFlexMRS is evaluated using an SDN-enabled multi-RAT testbed. The decision problem is formulated as a Markov Decision Process with a finite action space and is addressed using a Deep Q-Network (DQN) with convolutional layers that extract relationships among network measurement features. The agent is trained offline from collected network observations using reward guided Q-value updates, then used to select handover targets during inference. A 5G-WiFi case study demonstrates that ReFlexMRS can trade a small amount of throughput for substantial improvements in link delay, handover stability, and WiFi offloading. Compared with RSSI-based selection, the proposed formulations retain approximately 90-95% of the baseline throughput while reducing total handovers by up to 18% and WiFi-to-5G handovers by up to 13%, while maintaining the required RSSI levels.
• 中文摘要 下一代无线系统的一个主要进化目标和挑战是无缝多无线接入技术（multi-RAT）移动性。在多RAT环境中，切换决策必须同时考虑用户服务质量和网络优先级，包括信号质量、资源可用性、能耗、延迟和安全性。尽管WiFi/蜂窝移动性在研究和标准化中得到了充分探讨，许多传统切换机制仍主要依赖接收信号强度。此类决策可能忽视链路延迟、拥塞以及异构RAT的独特操作特性。本研究介绍了强化灵活多RAT选择框架（ReFlexMRS），这是一种基于深度强化学习（DRL）的可配置多准则切换目标选择方法。ReFlexMRS通过支持SDN的多RAT测试平台进行评估。该决策问题被表述为具有有限动作空间的马尔可夫决策过程，并通过具有卷积层的深度Q网络（DQN）来解决，这些方法提取网络测量特征之间的关系。智能体通过奖励引导的Q值更新离线训练，然后在推断过程中选择切换目标。一项5G-WiFi案例研究表明，ReFlexMRS可以以少量吞吐量换取链路延迟、切换稳定性和WiFi卸载的显著提升。与基于RSSI的选择相比，所提出的方案保留了约90-95%的基线吞吐量，同时将总切换次数减少最多18%，WiFi到5G切换减少最多13%，同时保持所需的RSSI水平。

Dynamic Watermarking and Action-Conditioned Latent Dynamics for Secure Vision-Based Controllers

动态水印与动作条件潜在动力学，用于安全视觉控制器

• Authors: Aditya Parameshwaran, Navid Aftabi, Abhishek Hanchate, Satish T. S. Bukkapatnam, Dan Li, Yue Wang
• Subjects: Subjects: Robotics (cs.RO); Cryptography and Security (cs.CR); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.07403
• Pdf link: https://arxiv.org/pdf/2611.07403
• Abstract Replay attacks are difficult to detect in vision-based control systems because previously recorded observations can remain plausible while the physical system follows a different trajectory. This paper develops an adaptive dynamic-watermarking framework for detecting replay attacks against vision-based controllers. A private Gaussian watermark is added to the nominal control input, and a control input-conditioned latent dynamics network (LDN) predicts the corresponding evolution in a compact latent space learned from image observations. Attack evidence is obtained from the discrepancy between the predicted latent response and the latent representation of the received image observation. A reinforcement learning policy adjusts the watermark covariance using the latent state and detector confidence, thereby balancing attack detectability, watermark energy, and control distortion. A Lagrangian penalty on task-specific safety costs is included during training to limit the adverse effect of the excitation. The framework is evaluated on a vision-guided UR5 manipulation task in which the robot reaches a target while avoiding a moving human arm under nominal and replay-attacked observations.
• 中文摘要 在基于视觉的控制系统中，重放攻击难以被检测，因为先前记录的观测可能保持合理，而物理系统则遵循不同的轨迹。本文开发了一个自适应的动态水印框架，用于检测针对基于视觉控制器的回放攻击。在名义控制输入中添加私有高斯水印，控制输入条件潜动力学网络（LDN）预测从图像观测中学习到的紧凑潜在空间中的相应演变。攻击证据来自预测的潜在响应与接收图像观测的潜在表征之间的差异。强化学习策略利用潜态和检测器置信度调整水印协方差，从而平衡攻击可检测性、水印能量和控制失真。在训练过程中包含任务特定安全成本的拉格朗日惩罚，以限制激发的不利影响。该框架在视觉引导的UR5操作任务中进行评估，该任务中机器人在正常和回放攻击观察下，在避开移动的人类手臂的情况下到达目标。

Enhancing Repository-Level Code Completion with Reinforcement Learning in Real-World Scenarios

在现实场景中通过强化学习提升仓库级代码完成

• Authors: Zhenlong Dai, Zhuoluo Zhao, Xiu Tang, Sai Wu, Chang Yao, Zhipeng Gao, Jingyuan Chen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.07510
• Pdf link: https://arxiv.org/pdf/2611.07510
• Abstract The rapid development of large language models (LLMs) for code, which closely aligns with the practical needs of real-world software development, has sparked growing attention in repository-level code completion. However, existing Code LLMs present challenges in focusing on suitable contexts from the repository and in deeply reasoning about cross-file dependencies. To address these challenges, we propose a novel reinforcement learning framework for repository-level code completion. To better understand both in-file and cross-file contexts, we employ identifier-driven intent recognition to capture completion intent, thereby improving the model's performance in real-world scenarios. To enhance cross-file reasoning, we propose a reward-driven completion learning to effectively reward in complex repository completion scenarios. To guide LLMs in completing code that aligns with intended functionality and repository context, we introduce a selective-exploration strategy that directs the model to focus on low-confidence, high-reward tokens, promoting the exploration of valuable, underexplored completion patterns. Experimental results show that our approach significantly improves the performance of Code LLMs. The code and the dataset are available at this https URL.
• 中文摘要 大型语言模型（LLMs）的快速开发与现实软件开发的实际需求高度契合，激发了对代码库级代码完成的日益关注。然而，现有代码LLM在聚焦于合适的代码库上下文以及深入推理跨文件依赖方面存在挑战。为应对这些挑战，我们提出了一种新的代码级代码完成强化学习框架。为更好地理解文件内和跨文件上下文，我们采用标识符驱动的意图识别来捕捉完成意图，从而提升模型在现实场景中的表现。为增强跨文件推理，我们提出一种奖励驱动的完成学习，以有效奖励复杂的仓库完成场景。为引导LLM完成符合预期功能和仓库上下文的代码，我们引入选择性探索策略，引导模型聚焦于低信心、高回报的代币，促进对有价值且未被充分探索的完成模式的探索。实验结果表明，我们的方法显著提升了代码大型语言模型的性能。代码和数据集可在该 https URL 访问。

RLGPA: A Reinforcement Learning-Guided Granularity Planning Attack for Decision-Based Sparse Adversarial Examples

RLGPA：基于决策的稀疏对抗实例的强化学习引导粒度规划攻击

• Authors: Chao Zhou, Shenghua He, Jiaxin Mo, Qiangguo Yu, Fengchuang Xing
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.07512
• Pdf link: https://arxiv.org/pdf/2611.07512
• Abstract Decision-based sparse attacks must reduce perturbation support using only the victim's predicted label. The best restoration strategy varies as support contracts: coarse removal works early, while tail states need finer, history-aware tests. We propose RLGPA, an online reinforcement learning method that schedules support-reduction executors and their granularity under a hard query budget. A masked Dueling Double DQN selects among seven stable policy--scale actions; each action produces support-relative restoration groups that are rebuilt after every macro decision. A feasibility-preserving executor commits only adversarial restorations and feeds support gain and query cost back to the planner. Under the matched top-1-label oracle, RLGPA reaches 61\% targeted ASR at sparsity 0.01 on CIFAR-10 (vs. 25\% SparseEvo, 0\% Pointwise), and 75.5\%/73.0\% on ResNet50/ViT at sparsity 0.005 (vs. 29.0\%/15.5\%). A fixed-probe test reduces median (L_0) from 165 to 23 after 40 images without any update. State-blind and state-shuffled controls degrade significantly, confirming that the learned policy uses attack state; the main gains come from state-conditioned executor scheduling and dynamic repartition. The source code is publicly available at this https URL.
• 中文摘要 基于决策的稀疏攻击必须仅使用受害者预测的标签来减少扰动支持。最佳恢复策略会随着支持合同而变化：粗移除早期有效，而尾部状态则需要更细致、历史感知的测试。我们提出了RLGPA，一种在线强化学习方法，可在硬查询预算下调度支持减量执行者及其细度。一个掩蔽的双重双重DQN从七个稳定的策略尺度动作中选择;每个动作产生支持相对的恢复组，每次宏决策后重建。可行性保持执行者只提交对抗性恢复，并将支持收益和查询成本反馈给规划器。在匹配顶级标签oracle下，RLGPA在CIFAR-10稀疏度0.01时目标ASR达到61%（对比SparseEvo为25%，点数为0%），在ResNet50/ViT上稀疏度0.005时达到75.5%/73.0%（对比29.0\%/15.5%）。固定探针测试在40张图像无更新时，中L_0位数ASR从165降至23。状态盲和状态洗牌控制显著下降，证实学习策略使用攻击状态;主要优势来自状态条件执行器调度和动态重分。源代码公开于此https网址。

Semantics-to-Contact: A Stagewise Framework for Robust Contact-Rich Manipulation

语义到联系：一个分阶段的稳健联系人丰富操作框架

• Authors: Guanren Qiao, Ruixiang Ouyang, Sheng Xu, Yueci Deng, Yunxin Tai, Kui Jia, Guiliang Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.07532
• Pdf link: https://arxiv.org/pdf/2611.07532
• Abstract Contact-rich manipulation requires precise control under complex contact dynamics while remaining robust to diverse visual conditions. However, real-world reinforcement learning often overfits to the visual appearance of training scenes. We propose Semantics-to-Contact (S2C), a stagewise framework for visually robust contact-rich manipulation. The insight is to decompose the problem into two stages: semantic focusing and contact refinement. In the first stage, a vision-language model localizes task-relevant regions across visually diverse scenes, providing coarse spatial guidance and reducing the burden of exploration. In the second stage, a real-world human-in-the-loop residual reinforcement learning policy learns fine-grained contact behaviors within the localized region using force feedback. To further improve visual robustness, we introduce an object-centric visual augmentation strategy that randomizes background appearance while preserving the robot and task-relevant objects. Experiments on six contact-rich real-world manipulation tasks exhibit that S2C consistently improves success rates over strong baselines and maintains reliable performance under significant visual and positional variations. These results demonstrate that stagewise semantic focusing and contact refinement provide a practical path toward visually robust real-world contact-rich manipulation. Our demos can be seen in this https URL.
• 中文摘要 丰富的接触操作需要在复杂接触动态下进行精确控制，同时保持对多样视觉条件的鲁棒性。然而，现实世界强化学习常常会过度拟合训练场景的视觉外观。我们提出了语义到接触（S2C）框架，这是一种分阶段的视觉鲁棒接触丰富操作框架。该见解是将问题分解为两个阶段：语义聚焦和接触细化。第一阶段，视觉语言模型将任务相关区域定位于视觉多样场景中，提供粗略的空间引导并减轻探索负担。第二阶段，现实世界的人在环中残余强化学习策略通过力反馈学习局部区域内的细粒度接触行为。为进一步提升视觉鲁棒性，我们引入了以对象为中心的视觉增强策略，该策略在保持机器人和任务相关对象的同时随机化背景外观。在六个接触丰富、真实世界操作任务上的实验显示，S2C在强基线下持续提升成功率，并在显著的视觉和位置变化下保持可靠表现。这些结果表明，分阶段语义聚焦和接触细化为实现视觉上强大、真实世界接触丰富的操作提供了切实可行的路径。我们的演示可在此 https 网址中观看。

Provable Zeroth-Order Safe Exploration for Reinforcement Learning from Human Feedback

可证实的零阶安全探索，用于人类反馈强化学习

• Authors: Weiqin Chen, Xinmeng Huang, Dongsheng Ding, Santiago Paternain
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.07561
• Pdf link: https://arxiv.org/pdf/2611.07561
• Abstract Safe reinforcement learning from human feedback (Safe-RLHF) seeks to improve helpfulness while satisfying harmlessness constraints. Existing primal-dual approaches can converge to feasible policies but do not prevent constraint violations during training. We study a stronger requirement, safe exploration for RLHF, under which every policy executed during training, including auxiliary exploration policies, remains feasible. We propose Zeroth-Order Log-Barrier Policy Gradient (ZO-LBPG), a reward-model-free method that optimizes a log-barrier objective using zeroth-order estimates derived directly from human preferences. To ensure safety under noisy feedback, we combine conservative constraint margin estimates with coupled rollouts and an adaptive step size that prevents updates from crossing constraint boundaries. We prove that, with high probability, all iterates and exploration policies remain strictly feasible, and that ZO-LBPG globally converges to an approximately optimal constrained policy. We establish a sample complexity of $\widetilde{\mathcal O}(\varepsilon^{-6}+d^3\varepsilon^{-5})$ trajectories and $\widetilde{\mathcal O}(\varepsilon^{-8}+d^5\varepsilon^{-7})$ human queries. Experiments on a synthetic CMDP and a language-model alignment task corroborate the training-time safety guarantees.
• 中文摘要 从人类反馈中安全强化学习（Safe-RLHF）旨在提高帮助性，同时满足无害约束。现有的原始-对偶方法可以收敛到可行策略，但无法防止训练期间的约束违规。我们研究了更强的要求——RLHF的安全探索，在该要求下，训练期间执行的所有策略，包括辅助探索策略，都依然可行。我们提出了零阶对数障碍策略梯度（ZO-LBPG），这是一种无奖励模型的方法，利用直接从人类偏好得出的零阶估计来优化对数障碍目标。为确保在噪声反馈下的安全性，我们结合了保守的约束裕度估计、耦合展开以及防止更新跨越约束边界的自适应步长。我们证明，所有迭代和探索策略在高概率下依然严格可行，ZO-LBPG全局收敛为近似最优的约束策略。我们建立了样本复杂度为$\widetilde{\mathcal O}（\varepsilon^{-6}+d^3\varepsilon^{-5}）$轨迹和$\widetilde{\mathcal O}（\varepsilon^{-8}+d^5\varepsilon^{-7}）$人类查询。合成CMDP和语言模型比对任务的实验支持了训练时间的安全保障。

What Gets Hacked in a Collaborative Reward for LLM Recommenders: The Part That Does Not Depend on the User

协作奖励中对LLM推荐者被入侵的部分：不依赖用户的部分

• Authors: Zhuangsheng Li, Liang Dong, Yu Wu, Dailin Li, Yu Xia, Yunhan C. Zhang, Rui Cai, Yi Wu, Julian McAuley
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.07588
• Pdf link: https://arxiv.org/pdf/2611.07588
• Abstract Collaborative-filtering (CF) scores provide graded, personalized rewards for training large language model (LLM) recommenders through reinforcement learning. However, these rewards can cause exposure collapse: even though the CF model itself produces diverse top recommendations across users, an LLM optimized with its rewards concentrates on fewer items. We decompose each score into an item-mean component, the item's average CF score across user histories, and a residual component, the current user's score deviation from the average. Through controlled interventions, we identify the item-mean component as a contributor to exposure collapse. Accordingly, we propose two reward corrections: per-item centering removes each item's estimated mean score, while per-item standardization further rescales the residual using its estimated standard deviation. Experiments on two datasets with two identifier schemes show that both corrections substantially improve catalog coverage and exposure evenness. Compared with training on the uncorrected CF reward, our corrections increase the number of distinct items ranked first across users by up to $9.8\times$, while maintaining comparable ranking accuracy. Code is available at this https URL.
• 中文摘要 协同过滤（CF）评分为通过强化学习训练大型语言模型（LLM）推荐者提供分级、个性化的奖励。然而，这些奖励可能导致暴露崩溃：尽管CF模型本身在用户间产生多样化的顶级推荐，但用奖励优化的LLM则关注更少的项目。我们将每个评分分解为一个项目均值成分，即该项目在用户历史中的平均CF分数，以及一个残差成分，即当前用户得分偏离平均值的部分。通过受控干预，我们识别出项目均值成分是暴露崩溃的促成因素。因此，我们提出了两种奖励修正方案：逐条目中心化去除每个项目的估计平均得分，而每条目标准化则利用其估计的标准差进一步重新标度残差。在两个数据集上采用两种标识符方案的实验显示，这两种修正都显著提升了目录覆盖率和暴露均匀性。与未纠正的CF奖励训练相比，我们的修正使用户中排名第一的项目数量增加最多9.8美元，同时保持了相当的排名准确性。代码可在此 https 网址获取。

Monotone Viability Learning for Offline Safe Reinforcement Learning

单调可行性学习用于离线安全强化学习

• Authors: Ziqian Wang, Zhen Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.07616
• Pdf link: https://arxiv.org/pdf/2611.07616
• Abstract Offline safe reinforcement learning requires reliable long-horizon safety estimates from fixed data. Learning Hamilton-Jacobi (HJ) viability in this setting calls for semantic preservation, convergence, and numerical stability. To address these challenges, we propose Monotone Viability Learning (MoViL), which learns viable sets from offline transitions through monotone finite-horizon set recursion. The recursion preserves finite-horizon safety semantics and converges to the infinite-horizon viability kernel. Binary supervision maintains a fixed target scale across horizons, avoiding discount-induced numerical attenuation. We further establish sufficient conditions for recovering behavior-supported viable subsets. The learned viability estimates are used to certify actions at deployment. Dubins car experiments validate the numerical accuracy of the learned viable sets. Evaluations on safe RL benchmarks demonstrate state-of-the-art reward-cost performance. MoViL satisfies the average cost constraint on every task and achieves the lowest average cost among the compared methods. These results highlight MoViL's potential to extend long-horizon HJ viability analysis to safety-critical systems where reliable dynamics models are unavailable and online exploration is unsafe.
• 中文摘要 离线安全强化学习需要从固定数据中获得可靠的长视野安全估计。在该环境中学习Hamilton-Jacobi（HJ）可行性需要语义保持、收敛和数值稳定性。为应对这些挑战，我们提出了单调可行性学习（MoViL），通过单调有限视界集递归从离线转移学习可行集合。递归保持有限视野安全语义，收敛至无限视界可行性核。二元监督在视野间保持固定目标尺度，避免折扣诱导的数值衰减。我们进一步建立了恢复行为支持可行子集的充分条件。学习到的可行性估计用于在部署时认证动作。Dubins汽车实验验证了已学可行集合的数值准确性。安全强化学习基准测试的评估展示了最先进的奖赏成本性能。MoViL满足每个任务的平均成本约束，并在比较方法中实现最低的平均成本。这些结果凸显了MoViL将长期HJ可行性分析扩展到缺乏可靠动力学模型且在线探索不安全的关键系统中的潜力。

AiXapply: Fast and Reliable Full-File Code Integration with Specialized Small Models for IDE Workflows

AiXapply：快速可靠的全文件代码集成，与专用小型模型集成，适用于IDE工作流程

• Authors: Siyuan Jiang, Xiang Cai, Peixu Wang, Yu Han, Yihong Dong, Wei Ning, Xuyuan Guo, Jincheng Wen, Dexin Liu, Wei Zhao, Ge Li
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2611.07634
• Pdf link: https://arxiv.org/pdf/2611.07634
• Abstract Large language models increasingly assist developers by proposing small code edits, yet reliably integrating a partial update snippet into an existing file while preserving unrelated content remains poorly supported by current code-generation workflows. We formalize the Code Integration (Apply) task as follows: given an original file and an update snippet, the system must produce the complete merged file while preserving all content outside the requested edit. We construct a high-quality multi-language dataset from real commits via a structured pipeline that converts code changes into update snippets paired with full-file ground truth and enforces consistency through verification and filtering. We then train a specialized 4B model, aiXapply, using supervised fine-tuning and reinforcement learning (GRPO). The supervised variant achieves 94.4% accuracy under our language-aware comparison, surpassing a strong large-model (Deepseek-V3.2) baseline (91.6%). Speculative decoding enables aiXapply-RL to reach $\sim$2700 tokens/s on a single A100 in a decoding subset; on the full benchmark, it achieves roughly 75$\times$ lower latency than the API-served Deepseek-V3.2 baseline. We further conduct an industrial deployment of aiXapply-RL in the aiXcoder IDE plugin as a dedicated Apply service, reducing mean Apply latency from 50 seconds to 1.89 seconds by offloading full-file generation from the upstream large model. Under time-based capacity planning, replacing one of five upstream serving instances would save 20% of upstream machine cost before the additional shared A100 service cost. Overall, our results demonstrate that treating code integration as a standalone task enables small models to achieve high accuracy, practical efficiency, and concrete industrial benefits in real-world IDE workflows where latency, cost and privacy are critical.
• 中文摘要 大型语言模型越来越多地通过提出小范围的代码编辑来辅助开发者，但当前代码生成流程对部分更新片段的可靠整合，同时保留无关内容，仍支持不足。我们将代码集成（应用）任务形式化为：给定原始文件和更新摘要，系统必须生成完整的合并文件，同时保留请求编辑以外的所有内容。我们通过结构化流水线，从真实提交构建高质量多语言数据集，将代码变更转换为更新片段，配合完整文件的真实数据，并通过验证和过滤强制一致性。随后，我们通过监督微调和强化学习（GRPO）训练一个专门的4B模型aiXapply。在语言感知比较下，监督变体的准确率达到94.4%，超过强大型模型（Deepseek-V3.2）基线（91.6%）。推测解码使 aiXapply-RL 在解码子集内单个 A100 上达到 $\sim$2700 代币/秒;在完整基准测试中，其延迟比 API 服务的 Deepseek-V3.2 基线低约 75 美元/倍数。我们还在 aiXcoder IDE 插件中作为专用 Apply 服务进行工业部署，将平均应用延迟从 50 秒降至 1.89 秒，通过将全文件生成从上游大型模型卸载。在基于时间的容量规划下，替换五个上游服务实例中的一个，可节省 20% 的上游机器成本，扣除额外的共享 A100 服务成本。总体而言，我们的结果表明，将代码集成视为独立任务，使小模型能够在延迟、成本和隐私至关重要的现实 IDE 工作流中实现高准确性、实用效率和具体的工业效益。

Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning

衡量基于验证器的强化学习在监督微调后的边际价值

• Authors: Amit Sagtani, Saloni Garg
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.07639
• Pdf link: https://arxiv.org/pdf/2611.07639
• Abstract Reinforcement learning from verifiable rewards (RLVR) is usually evaluated as an alternative to supervised fine-tuning (SFT), or as a stage that follows it, with the implicit assumption that the RL stage contributes gains of its own. We measure that contribution directly, in a controlled setting where both stages can be run to completion and compared exactly. We built gpu-train-debug, a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks across 30 archetypes, graded by a programmatic verifier with partial credit, and calibrated so that a mid-tier model scores in a 0.60 to 0.75 band. A pilot produced a surprise: 1,200 gold demonstrations nearly solved the benchmark (SFT 0.785 on the hard split), while 15 steps of GRPO from the base model did not move it (0.314 vs 0.311 base). We therefore amended the study, in writing and before the full run, to ask the sharper question: what does verifier-based RL add beyond demonstrations? At the pre-registered scale (60 GRPO steps, learning rate 1e-6), the answer is: no detectable improvement in-distribution (+0.22 points, 95% CI [-0.06, +0.51]), and a modest but measurable improvement under distribution shift (+0.33 points on a held-out OOD split, CI [+0.02, +0.68]). SFT accounted for most of the available improvement, while verifier-based RL produced only modest additional gains under distribution shift. The residual errors SFT leaves are mostly systematic, identical across samples, which likely reduces the within-group learning signal available to GRPO. A full 60-step cold-start GRPO arm confirms the pilot at the full horizon. The whole study cost $8.69 of compute and is fully reproducible from archived generations, frozen configs, and tagged code.
• 中文摘要 可验证奖励强化学习（RLVR）通常作为监督微调（SFT）的替代方案，或作为其之后的一个阶段进行评估，隐含假设强化学习阶段自身也贡献了收益。我们直接测量这种贡献，在一个可控的环境中，两个阶段均可运行至完成并精确比较。我们构建了GPU-train-debug，这是一个确定性基准测试，涵盖30个原型的1877个合成GPU和LLM训练失败诊断任务，由程序化验证器部分评分，并校准使中端模型得分在0.60至0.75区间。一项试点项目带来了惊喜：1200次金示范几乎解决了基准问题（硬拆分时SFT 0.785），而基础模型中GRPO的15步并未使基准提升（0.314对0.311基准）。因此，我们在完整运行前书面修订了研究，提出更尖锐的问题：基于验证器的强化学习除了演示之外还能带来什么？在预注册尺度（60个GRPO步，学习率1e-6）下，答案是：分布内无明显改善（+0.22分，95% CI [-0.06， +0.51]），而在分配转移下则有适度但可测量的改善（OOD分配延迟+0.33分，置信区间[+0.02， +0.68]）。SFT贡献了大部分可用改进，而基于验证器的RL在分布偏移下仅带来适度的额外增益。SFT留下的残余误差大多系统性且样本间相同，这可能降低了GRPO可获得的组内学习信号。完整的60步冷启动GRPO臂确认了试点的全面预期。整个研究计算成本为8.69美元，且可完全从归档的代数、冻结配置和标记代码中复现。

BiGrounder: Bidirectional Training Unlocks the Potential of GUI Grounding

双向训练：双向培训释放图形界面接地潜力

• Authors: Dengyao Shi, Tianyu Fu, Fei Hu, Anyang Su, Hanning Wang, Shuo Wang, Minghui Wu, Chenxu Zhao, Zhen Lei
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.07765
• Pdf link: https://arxiv.org/pdf/2611.07765
• Abstract Graphical User Interface (GUI) grounding aims to map natural language instructions to precise operational coordinates, serving as a critical task for autonomous agent interaction. Current approaches formalize GUI grounding as predicting action-coordinate pairs from instructions, with end-to-end training supervised solely by final coordinates. Training datasets typically provide only a small number of descriptions per target coordinate, whereas real-world scenarios involve a much larger set of varied descriptions for the same target, which limits robustness to linguistic variation. While recent GUI grounding paradigms have begun employing reinforcement learning to enhance localization accuracy, models with limited exploratory capability before reinforcement yield constrained improvements even with identical training data during the reinforcement phase. To overcome these limitations, we propose a bidirectional reward paradigm that enables model self-reinforcement. During SFT, bidirectional training equips the model to generate action descriptions from target coordinates. During RL, the model samples semantically consistent instruction variations on-policy and grounds each variation back to its target. A cycle-consistency gate couples semantic quality to grounding success, turning the model's own instruction policy into a source of continually refreshed reinforcement trajectories. Our 4B-parameter model achieves competitive accuracy across five grounding benchmarks in our evaluation, indicating the effectiveness of the proposed training paradigm. this https URL
• 中文摘要 图形用户界面（GUI）基础化旨在将自然语言指令映射到精确的操作坐标，是自主智能体交互的关键任务。当前方法将图形界面基础化形式化为根据指令预测动作-坐标对，端到端训练仅由最终坐标监督。训练数据集通常每个目标坐标只提供少量描述，而现实场景涉及大量多样化描述，限制了语言变异的鲁棒性。虽然近期GUI基础范式开始采用强化学习提升定位准确性，但强化前探索能力有限的模型，即使在强化阶段使用相同训练数据，改进效果也受限。为克服这些限制，我们提出一种双向奖励范式，支持模型自我强化。在SFT阶段，双向训练使模型能够从目标坐标生成动作描述。在强化学习中，模型采样语义一致的指令变体，并将每个变体归根结为目标。循环一致性门将语义质量与基础成功相结合，将模型自身的指令策略转化为不断更新的强化轨迹来源。我们的4B参数模型在评估中五个基准测试中达到了竞争精度，表明所提训练范式的有效性。此链接

Towards Foundation Decision Models: Drex

迈向基金会决策模型：Drex

• Authors: Nischay Dhankhar, Alisher Amantay, Zhanibek Datbayev, Darkhan Akhmetov, Yash Shah, Nurmukhamed Ubaidullayev, Raman Saparkhan, Dos Baha, Abulhair Saparov
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.07854
• Pdf link: https://arxiv.org/pdf/2611.07854
• Abstract Autoregressive approaches with large language models, such as long chain-of-thought reasoning, have become the default for new AI applications because they reach higher accuracy on a wide range of tasks. In production, however, they give inconsistent answers, require parsing of outputs, and add cost and latency. Decision models address this: given a question and a set of options supplied by the user, they return a probability distribution over the options without autoregressive decoding. We present Drex, a family of four open decision models with 8B (DLM), 9B (v1.5), 35B with 3B active (v2.0), and 27B (Max) parameters. Each request consists of a state (a document, record, or agent context, optionally with images) and questions about it, each asking for a choice among options, a yes/no judgment, or a score on an ordered scale. Training combines supervised fine-tuning with reinforcement learning from adjusted feedback (RLAF), plus vision tuning for Max. Ablations show that RLAF improves accuracy and that consistency losses improve robustness to option order. On Decision Index 0.4.1, Max sets a new state of the art among the decision models we compare (67.52 on the public suite vs. 60.70 for Jev 1.14.0). On Decision Bench v4, it is statistically tied with d3-turbo and GPT-6 Sol and trails Claude Opus 5.5 by 1.6 points, and it has the highest accuracy among decision models on DecideBench. Max answers a typical request in 183 ms, against about 2.0 s for GPT-6 Sol and 3.5 s for Claude Opus 5.5, and its outputs are stable under option reordering. All four models are open source: this https URL
• 中文摘要 利用大型语言模型的自回归方法，如长链思考推理，已成为新型人工智能应用的默认方法，因为它们在广泛任务中实现了更高的准确率。然而，在生产环境中，它们给出的答案不一致，需要解析输出，并增加成本和延迟。决策模型解决了这个问题：给定一个问题和用户提供的一组选项，它们返回了对这些选项的概率分布，而无需自回归解码。我们介绍Drex，这是一个由四个开放决策模型组成的家族，参数分别为8B（DLM）、9B（v1.5）、35B（3B激活，v2.0）和27B（Max）。每个请求由一个状态（文档、记录或代理上下文，可选配图片）及其相关问题组成，每个问题询问选项选择、是非判断或有序评分。训练结合了监督微调、调整反馈强化学习（RLAF）以及 Max 的视觉调优。消融显示 RLAF 提高了准确性，一致性损失增强了选项顺序的鲁棒性。在 Decision Index 0.4.1 中，Max 在我们比较的决策模型中树立了新的最前沿（公开套件为 67.52，Jev 1.14.0 为 60.70）。在 Decision Bench v4 中，它在统计上与 d3-turbo 和 GPT-6 Sol 持平，落后于 Claude Opus 5.5 1.6 个百分点，并且在 DecideBench 决策模型中准确率最高。Max 在典型请求响应时间为 183 毫秒，而 GPT-6 Sol 约 2.0 秒，Claude Opus 5.5 约 3.5 秒，其输出在选项重排序下保持稳定。这四个型号都是开源的：这个 https URL

TactileLab: Sim-to-Real Dexterous Robot Manipulation with Efficient Shear-Sensitive Tactile Simulation

TactileLab：模拟到真实的灵巧机器人操作，具备高效的剪切敏感触觉模拟

• Authors: Yijiong Lin, Wenhao He, Toby Jack, Max Yang, Haoran Li, Nathan F. Lepora
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.07881
• Pdf link: https://arxiv.org/pdf/2611.07881
• Abstract Robotic simulators that represent shear within tactile sensing are becoming increasingly important for contact-rich manipulation. However, simulating contact shear with high-fidelity deformable or soft-body simulation is computationally demanding, limiting scalability to large-scale reinforcement learning. Here we introduce \textit{TactileLab}, a scalable multimodal tactile simulation framework built on IsaacLab, with a particular focus on a new optical-flow-based tactile representation termed \textit{simulated tactile flow}. Instead of relying on soft-body physics to model shear deformation, TactileLab uses rigid-body simulation to efficiently estimate dense tangential motion at the contact interface, producing tactile flow fields that encode shear-relevant contact dynamics. This flow field complements geometric features from a contact depth field. We further introduce a novel policy architecture for jointly processing long-horizon proprioceptive signals and rich tactile observations. Across multiple contact-rich tasks, policies trained using simulated tactile flow learn efficiently at scale and exhibit emergent contact-aware behaviours. To facilitate sim-to-real policy deployment, we then propose a novel real-to-sim tactile translation method that maps real tactile image streams to simulated tactile flow and depth fields, together with a new tactile data collection method, and validate it across diverse tactile sensor designs. Using this pipeline, we demonstrate real-world deployment on tasks encompassing dual-arm and dexterous in-hand manipulation. Overall, TactileLab provides an end-to-end sim-to-real framework for scalable tactile policy learning, enabling diverse shear-sensitive manipulation across heterogeneous tactile robotic systems.
• 中文摘要 在触觉感知中表示剪切的机器人模拟器在接触丰富操作中变得越来越重要。然而，用高精度可变形或软体模拟接触剪切计算需求较大，限制了大规模强化学习的可扩展性。这里我们介绍 \textit{TactileLab}，这是一个基于 IsaacLab 构建的可扩展多模态触觉仿真框架，特别关注一种基于光流的新型触觉表示，称为 \textit{模拟触觉流动}。TactileLab 不依赖软体物理来模拟剪切变形，而是利用刚体仿真高效估计接触界面上的密集切向运动，生成编码剪切相关接触动力学的触觉流场。该流场补充了接触深度场的几何特征。我们进一步引入了一种新颖的策略架构，用于联合处理长视野本体感觉信号和丰富的触觉观察。在多个接触丰富的任务中，使用模拟触觉流训练的策略能够大规模高效学习，并表现出新出现的接触感知行为。为促进模拟到现实的策略部署，我们提出了一种新型实到模拟触觉转换方法，将真实触觉图像流映射到模拟的触觉流和深度场，结合一种新的触觉数据收集方法，并在不同触觉传感器设计中进行验证。利用该流程，我们展示了在涵盖双臂和灵巧手持操作任务上的实际部署。总体而言，TactileLab提供了一个端到端的模拟到现实框架，支持可扩展的触觉策略学习，支持跨异构触觉机器人系统实现多样化的剪切敏感操作。

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

注意SR：长视野能动强化学习的注意力阶梯奖励

• Authors: Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.07932
• Pdf link: https://arxiv.org/pdf/2611.07932
• Abstract Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at this https URL .
• 中文摘要 长期视野的LLM代理会做出多步决策，但结果奖励仅强化学习对单个步骤几乎没有直接监督。我们介绍了AttnSR，这是一种步骤级奖励增强方法，通过训练重放时提取的策略注意力来分配奖励，无需单独训练的奖励模型或额外的推广。AttnSR回溯策略模型注意力图，估计早期步骤与最终反应的相关性。相对于运行基线的终端步骤，决定这些奖励是强化还是惩罚所选步骤。在一项受控的Qwen2.5-7B-Instruct研究中，AttnSR在ALFWorld所有评估检查点上持续优于PPO基线，最终保留成功率从91.2%提升至97.1%。消融实验表明，收益取决于中间奖励的分配方式，而非简单地随时间累积步骤奖励。AttnSR还在每个检查点提升了ScienceWorld，同时减少了29.9%的环境交互。这些结果支持将政策关注作为长期代理强化学习中步骤级奖励分配的实用先验。代码可在此 https 访问 。

The Blindfolded Controller: Decision Model-Guided Training for Reinforcement Learning

蒙眼控制者：基于决策模型的强化学习培训

• Authors: Muhammad Ibrahim Khan, Aniket Dixit, Faizan Ahmed, James Brusey
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.07934
• Pdf link: https://arxiv.org/pdf/2611.07934
• Abstract Reinforcement learning (RL) provides a framework for continuous improvement through interaction with an environment. However, RL agents struggle against tasks that offer exploration challenges and sparse rewards, because learning efficient policies from scratch is slow and sample-inefficient. In contrast, decision models (DMs), given a description of the environment and task, can perform well in sparse-reward tasks, but, as they do not learn from reward, they are not able to improve towards the optimal solution. In this paper, we present Decision Model-Guided Training (DMGT), a technique that uses a DM with a description of the environment and task to initialise an RL policy, combining the DM's ability to act without training with RL's ability to improve from reward. Instead of exploring from scratch, the agent reaches the goal and receives reward from its first episodes, and RL then improves its policy beyond what the DM achieves. We demonstrate through experiments on seven tasks that our approach outperforms RL in all seven and DM in five. In addition, we provide comprehensive guidelines, derived from ablation studies, for describing an environment and task to a DM.
• 中文摘要 强化学习（RL）为通过与环境互动实现持续改进提供了框架。然而，强化学习者在面对具有探索挑战和奖励稀疏的任务时会遇到困难，因为从零学习高效策略速度慢且样本效率低。相比之下，决策模型（DM）在环境和任务的描述下，在稀疏奖励任务中表现良好，但由于不从奖励中学习，无法向最优解方向改进。本文介绍了决策模型引导训练（DMGT），这是一种利用DM对环境和任务描述来初始化强化学习策略的技术，结合了DM无需训练即可行动的能力和RL从奖励中改进的能力。主体不再从零探索，而是在最初的阶段达到目标并获得奖励，随后强化学习会改进其策略，超越DM的实际表现。我们通过七个任务的实验证明，我们的方法在七个任务中均优于强化学习，五个任务中优于DM。此外，我们还提供了基于消融研究的全面指南，用于向DM描述环境和任务。

SEER: A Scalable Event-Driven Runtime for Parallel Execution of Multi-Agent Reinforcement Learning Environments

SEER：一种可扩展的事件驱动运行时，用于多智能体强化学习环境的并行执行

• Authors: Jiming Su, Feng Zhu, Yuhao Xiao, Miao Zhang
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.07996
• Pdf link: https://arxiv.org/pdf/2611.07996
• Abstract Multi-agent reinforcement learning (MARL) is increasingly used in systems where many entities interact in a shared environment. In these workloads, policy inference can be batched on GPUs, but the CPU-side environment path still updates entity states, resolves local interactions, and builds observations inside each $\mathrm{this http URL}$. This path can dominate latency in interaction-intensive workloads after policy batching. Existing vectorized environment interfaces and general event-driven runtimes do not directly address the fine-grained cross-entity work inside one bounded MARL step while preserving the step-level semantics expected by learning systems. We present SEER (Scalable Event-Driven Environment Runtime), a runtime for MARL environment execution under an explicit event-locality contract. SEER models each environment step as a bounded stream of timestamped events. If direct state updates are entity-local and cross-entity effects are represented as ordered future events, SEER can run independent events in parallel while preserving serial-equivalent boundary outputs. To make this contract practical, SEER uses a CAS-based dual-layer event queue: local event queues preserve per-entity causal order, while a parallel executive list exposes ready work to global worker threads. SEER further uses STPA, a preflight-guided thread-pool selection mechanism that combines short runtime traces with static workload and hardware features. We evaluate SEER on two parameterized MARL workloads: compute-intensive UAV swarm control (Aircraft) and communication-intensive wireless management (PCS). On Aircraft, SEER reduces inference latency by 16.2\% over the baseline. On PCS, SEER reduces environment-execution latency by 46.9\%, 19.5\%, and 64.9\% over ROSS-T, NBSE, and ROOT-Sim, respectively.
• 中文摘要 多智能体强化学习（MARL）越来越多地应用于多个实体在共享环境中交互的系统。在这些工作负载中，策略推理可以在GPU上批量处理，但CPU端环境路径仍会更新实体状态，解决本地交互，并在每个$\mathrm{this http URL}$内构建观测值。该路径在策略批处理后，在交互密集型工作负载中可能主导延迟。现有的矢量化环境接口和通用事件驱动运行时，无法直接处理一个有界MARL步中的细粒度跨实体工作，同时保持学习系统预期的步级语义。我们介绍SEER（可扩展事件驱动环境运行时），这是一个基于显式事件-局部性契约的MARL环境执行时。SEER将每个环境步骤建模为有界的时间戳事件流。如果直接状态更新为实体-本地，跨实体效应以有序未来事件表示，SEER可以在保持串行等效边界输出的同时并行运行独立事件。为使该合同更实用，SEER采用基于CAS的双层事件队列：本地事件队列保持每个实体的因果顺序，而并行执行列表则向全局工作线程开放现成工作。SEER进一步使用STPA，一种预检引导线程池选择机制，结合了短的运行时追踪与静态工作负载和硬件特性。我们在两种参数化MARL工作负载上评估SEER：计算密集型无人机群控制（飞机型）和通信密集型无线管理（PCS）。在飞机上，SEER比基线降低了16.2%的推理延迟。在PCS上，SEER相比ROSS-T、NBSE和ROOT-Sim分别将环境执行延迟降低了46.9%、19.5%和64.9%。

Detecting Silent Faults at the Environment-Trainer Boundary in Reinforcement Learning

强化学习中环境-训练器边界的静默缺陷检测

• Authors: Zuo Gou, Chenhao Lu
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2611.08144
• Pdf link: https://arxiv.org/pdf/2611.08144
• Abstract Robot reinforcement learning increasingly runs environments outside the trainer: in another process, on another machine, in another language. Many faults at this boundary raise no error. We make the boundary a wire protocol and ask what notices a fault in it. On Pendulum, HalfCheetah, Hopper and a 7-joint arm (Pusher), we injected up to 23 faults into what crosses the wire and 2 into the trainer's log, from one value to every value. Unless every value was wrong, a learning-curve threshold caught at most one run in ten, and a comparison with ten fault-free seeds at most 21%. A clean evaluation of all 1,060 final policies agreed with the curves: of the 728 boundary-fault runs both curve checks passed, 723 scored no more than three standard deviations below fault-free seeds, though nearly all ended on other weights and, for reward faults, logged other returns. Checks of the run itself, not of its outcome, saw every fault. A step ledger kept on both sides and compared step by step caught and located every fault that crossed, on Pendulum, Hopper and Pusher on one machine and on Pendulum across two, with no fault-free run and, on Pendulum, a median 3% more training time. An episode record kept at the environment caught the faults in the trainer's log. We list such faults found in practice, in Gymnasium 1.3 and our own code and, reported by others, in five multi-agent libraries, and recommend a step ledger and episode counts kept at the environment.
• 中文摘要 机器人强化学习越来越多地运行在训练器外部的环境：在另一个进程、另一台机器上、另一种语言中运行。该边界处的许多错误不会引发错误。我们将边界设为线协议，并询问什么能察觉到其缺陷。在Pendulum、HalfCheetah、Hopper和一个7关节臂（Pusher）上，我们向穿线的部分注入最多23个错误，向训练器日志注入2个错误，从一个值到每个值。除非每个值都错误，否则学习曲线阈值最多捕捉到十次运行中一次，与十个无故障种子比较最多21%。对所有1060条最终策略的全面评估与曲线一致：在728次边界故障运行中，两次曲线检查均通过，其中723次得分仅比无故障种子低三个标准差，尽管几乎所有测试均以其他权重结束，奖励错误则记录了其他回报。检查运行本身，而非结果，发现了所有故障。双侧的步进账本逐步比较，捕捉并定位了所有交叉的故障，在Pendulum、Hopper和Pusher上用一台机器，Pendulum在两台机器上，没有无故障运行，Pendulum的训练时间中位数增加了3%。环境记录中记录了训练师日志中的故障。我们列出了在Gymnasium 1.3和我们自身代码中发现的此类缺陷，并据他人报告的五个多智能体库中发现的，并建议在环境中保存步数账本和集数。

Agent-R$^2$: Scaling Agents for Realistic Productivity Work via Real-Source Data Synthesis

Agent-R$^2$：通过真实源数据综合实现真实生产力工作的智能体规模化

• Authors: Teng Pan, Zixuan Wang, Yuchen Yan, Qifei Wu, Yao Chen, Yingqi Qu, Ruiqing Zhang, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.08219
• Pdf link: https://arxiv.org/pdf/2611.08219
• Abstract Large language models have made rapid progress in agentic reasoning and tool use, raising expectations for agents that can take over realistic productivity work from humans. However, current agents still struggle to finish such tasks, where success requires following long-horizon professional workflows, understanding heterogeneous file contexts, and producing usable deliverables under fine-grained requirements. A key bottleneck is the lack of scalable post-training data that reflects real work intent, authentic file environments, and deliverable-level evaluation. To close this gap, we introduce Agent-R$^2$, a $\textbf{r}$eal-source data synthesis framework that scales training data for $\textbf{r}$ealistic productivity work. Instead of relying on manually specified workflows or LLM-generated files, Agent-R$^2$ grounds synthesis throughout the pipeline in real occupational seeds, corpus-derived workflow evidence, and authentic web-sourced files. From 10k web-corpus candidates, Agent-R$^2$ distills 4k task instances that pair real multi-format attachments with fine-grained, deliverable-level rubrics. On this data, we train Agent-R$^2$-4B via supervised fine-tuning and rubric-based reinforcement learning, improving both end-to-end productivity performance and file-grounded reasoning. Trained on only 4k instances, Agent-R$^2$-4B surpasses Qwen3-235B-A22B-Thinking-2507 on both GDPval (51.0$\%$ vs. 48.0$\%$) and OfficeQA (14.8$\%$ vs. 13.8$\%$). Further analysis shows that Agent-R$^2$ mitigates shortcut behavior, where agents produce superficially plausible deliverables before sufficiently grounding, constructing, or verifying them, while promoting sustained exploration.
• 中文摘要 大型语言模型在智能体推理和工具使用方面取得了快速进展，提高了对能够接替人类真实生产力工作的智能体的期望。然而，当前智能体仍在难以完成此类任务，成功需要遵循长期的专业工作流程，理解异构文件上下文，并在细粒度要求下生成可用的交付物。一个关键瓶颈是缺乏反映真实工作意图、真实文件环境和交付物级评估的可扩展训练后数据。为弥合这一差距，我们引入了Agent-R$^2$，这是一个$\textbf{r}$eal源数据综合框架，可扩展训练数据以实现$\textbf{r}$ealistic生产力工作。Agent-R$^2$ 不依赖手动指定的工作流或大型语言模型生成的文件，而是基于真实的职业种子、语料库衍生的工作流程证据和真实的网络文件来构建整个流程的综合。从 10k 个网络语料库候选中，Agent-R$^2$ 提炼出 4k 任务实例，将真实的多格式附件与细粒度的交付级评分标准配对。基于这些数据，我们通过监督微调和基于评分标准的强化学习训练 Agent-R$^2$-4B，提升端到端的生产力表现和文件基础推理。仅在4000个实例上训练，Agent-R$^2$-4B在GDPval（51.0$\%$对48.0$\%$）和OfficeQA（14.8$\%$对13.8$\%$）上均超过Qwen3-235B-A22B-Thinking-2507。进一步分析显示，Agent-R$^2$能缓解捷径行为，即代理在充分扎实、构建或验证前就先做出表面合理的交付物，同时促进持续探索。

FewStepRL: Fine-tuning Few-step Diffusion Models with Reinforcement Learning

FewStepRL：通过强化学习微调少步扩散模型

• Authors: Haitao Lin, Minsi Ren, Yang Li, Xingtong Ge, Zheyuan Hu, Peiyan Hu, Tailin Wu
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.08240
• Pdf link: https://arxiv.org/pdf/2611.08240
• Abstract Reinforcement learning (RL) has become a standard post-training method for diffusion models, but most RL objectives roll out the current policy at every iteration, and with a many-step sampler these rollouts typically dominate the cost of training. Few-step models cut this cost, yet they model an average velocity over an interval, whereas existing RL objectives are built for an instantaneous velocity with a many-step sampler. We adapt RL objectives to few-step models systematically in three phases, namely a transformation, a decomposition and an adaptation. The transformation to $x$-space expresses each objective we consider through a common policy prediction, at the clean end by default and at an explicitly marked level for transition or flow-map objectives, so that objectives stated over densities, velocities or rewards become comparable. The decomposition writes the gradient of each of these objectives as the sum of a self-regression toward a reference and a reward augmentation, so that a method reduces, at the gradient level, to a choice of anchors and weights. The adaptation computes each anchor from a few-step model at the end of its transport or along its tangent, with the instantaneous velocity recovered from the network itself or from an auxiliary network. This yields few-step versions of FlowGRPO, AWM, DiffusionNFT, DiffusionOPSD and ReFL in one modular codebase. In our experiments, reward-gradient-based methods are the most effective when the reward is differentiable and its gradient is affordable; otherwise DiffusionNFT-style methods generally reach high rewards in relatively few iterations despite a sometimes higher per-iteration cost. Extra inference steps show no systematic aggregate benefit in our experiments, although selected outputs visibly improve with two to four times as many sampling steps as in training.
• 中文摘要 强化学习（RL）已成为扩散模型的标准训练后方法，但大多数强化学习目标在每次迭代中都会推出当前策略，而多步采样器通常主导这些推广成本。少步模型降低了这一成本，但它们能模拟区间内的平均速度，而现有的强化学习目标则是基于多步采样器构建瞬时速度。我们将强化学习目标系统地适应为少步模型，分为三个阶段，即变换、分解和适应。向$x美元空间的转换通过共同策略预测表达每个目标，默认在干净端，并在过渡或流图目标明确标记的层级，使密度、速度或奖励的目标变得可比。分解将每个目标的梯度写为对参考的自回归和奖励增强的和，使得方法在梯度层面简化为选择锚点和权重。适应从传输末端或切线的几步模型计算每个锚点，瞬时速度从网络本身或辅助网络恢复。这在一个模块化代码库中生成了FlowGRPO、AWM、DiffusionNFT、DiffusionOPSD和ReFL的少步版本。在我们的实验中，基于奖励梯度的方法在奖励可微且梯度可负担时最有效;否则，DiffusionNFT风格方法通常在相对较少的迭代中获得高回报，尽管每次迭代成本有时更高。额外推断步骤在我们的实验中未显示出系统性的总体收益，尽管部分输出在抽样步骤数是训练的两到四倍时明显提升。

Value-oriented policy distillation for interpretable wind farm power control

面向价值的政策提炼，用于可解释的风电场电力控制

• Authors: Max Bruninx, Timothy Verstraeten, Jan Decuyper, Ann Nowé, Jan Helsen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.08245
• Pdf link: https://arxiv.org/pdf/2611.08245
• Abstract Wind farm power control can help the system maintain balance and avoid blackouts. In Europe, this can be achieved through implicit balancing, where a wind farm adjusts its power setpoint in anticipation of the imbalance price using information published by the Transmission System Operator. Thereby, the wind farm helps restore system balance while also reducing its own imbalance costs resulting from forecast errors. However, implicit balancing is a challenging problem since the settlement price is unknown in advance and can be impacted by the pursued control strategy. Reinforcement learning methods could help to solve this problem, but lack the interpretability needed to be deployed on real wind farms. Therefore, we propose a policy distillation framework using explainable boosting machines (EBMs), a variant of generalized additive models, to obtain a transparent control policy. To increase the value captured by the EBM model, we introduce a value-oriented perspective to policy distillation inspired by research in the field of energy forecasting. Furthermore, we suggest to include expert knowledge during model design, such that the control strategy is also justifiable to stakeholders. Using real data of the Belgian grid and an offshore wind farm, we find that our value-oriented and expert-based controller can reduce imbalance costs by 82%, while outperforming a price threshold controller as well as a policy distillation baseline using decision trees.
• 中文摘要 风电场的电力控制可以帮助系统维持平衡，避免停电。在欧洲，这可以通过隐式平衡实现，即风电场根据输电系统运营商发布的信息，预见不平衡价格调整功率设定值。这样，风电场有助于恢复系统平衡，同时降低因预测误差导致的自身失衡成本。然而，隐式平衡是一个具有挑战性的问题，因为结算价格事先未知，且可能受到所采用控制策略的影响。强化学习方法可以帮助解决这一问题，但缺乏在真实风电场上部署所需的可解释性。因此，我们提出了一个利用可解释助推机（EBM）——一种广义加法模型变体——的策略提炼框架，以获得透明的控制策略。为了提升EBM模型捕捉的价值，我们引入了受能源预测领域研究启发的价值导向政策提炼视角。此外，我们建议在模型设计时加入专家知识，使控制策略对利益相关者来说也是合理的。利用比利时电网和海上风电场的真实数据，我们发现以价值为导向且基于专家的控制器能够将失衡成本降低82%，同时优于价格阈值控制器以及使用决策树的政策提炼基线。

Accelerating Safe Reinforcement Learning with Massive Parallelism

加速安全强化学习，采用大规模并行

• Authors: Joonyoung Lim, Younghwan Yoo
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.08273
• Pdf link: https://arxiv.org/pdf/2611.08273
• Abstract We investigate the problem of scaling safe reinforcement learning (safe RL) to massively parallel training regimes, where a large number of environments are exe- cuted with short rollout horizons. Although this setting improves data throughput and wall-clock efficiency, it creates a mismatch with constrained Markov decision processes (CMDPs), whose safety constraints are defined over full-episode cost returns. We address this mismatch by analyzing staggered environment resets as a phase-mixture distribution over short trajectory segments. This analysis charac- terizes the bias induced by phase coverage and motivates a phase-aggregated cost estimator that reconstructs episode-level cost estimates from short rollout batches. We further incorporate finite-sample uncertainty and distribution mismatch into a constraint-tightening framework, yielding a scalable Safe RL method compatible with on-policy optimization. Using an MJX-based implementation of Safety Gym, our experiments show that the proposed method achieves performance comparable to non-massively parallel implementations while substantially reducing wall-clock training time.
• 中文摘要 我们研究如何将安全强化学习（safe RL）扩展到大规模并行训练模式的问题，在这些环境中执行大量环境并行且部署时间较短。尽管该设置提升了数据吞吐量和墙时钟效率，但与受限马尔可夫决策过程（CMDP）存在不匹配，后者的安全约束是基于全集成本回报定义的。我们通过分析错峰环境重置，作为短轨迹段上的相位混合分布来解决这一不匹配。该分析揭示了相位覆盖引起的偏向，并激励了一种阶段汇总成本估计器，该算法从短部署批次重建了每集的成本估算。我们还将有限样本不确定性和分布不匹配纳入约束紧密框架，形成了一种可扩展的安全强化学习方法，兼容策略优化。利用基于MJX的Safety Gym实现，我们的实验表明，所提方法在实现与非大规模并行实现相当的性能下，同时显著减少了墙钟训练时间。

ManimCode: Learning Visual Self-Correction for Educational Animation

ManimCode：学习视觉自我修正以实现教育动画

• Authors: Zhengyang Liang, Lizi Liao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.08294
• Pdf link: https://arxiv.org/pdf/2611.08294
• Abstract Manim supports precise, programmatic video generation, but a successfully rendered educational animation can still have visual defects, missing explanations, or incorrect content. Rendering does not directly identify the code changes needed to correct these visual failures, so feedback-driven revision is needed. Existing agentic frameworks use iterative rendering and revision but typically rely on large models, often proprietary. Existing Manim datasets primarily contain final programs and often lack quality verification and the trajectories of feedback and revisions needed to learn this refinement process. We introduce ManimData and ManimCode, a data and training framework for feedback-driven educational animation generation. We construct ManimData through a render-guided pipeline that iteratively generates, executes, reviews, and revises Manim programs with retrieved API documentation. Its 24,034 render-verified, quality-filtered records preserve intermediate programs, execution errors, visual critiques, and corrective revisions. We use role-conditioned supervised fine-tuning on these trajectories to train ManimCode, a single 4B model, to write, review, and revise animation programs. To improve correction of errors that differ from those in the collected trajectories, we then train the Reviser with reinforcement learning on self-generated states, using rendering outcomes and reference-free quality feedback. ManimCode achieves 96.6% and 97.0% render success on ManimData-test and ManimBench, compared with 52.6% and 82.0% for the untrained 4B model under the same inference setup, while also improving video quality scores from 2.48 to 2.94 and from 3.00 to 3.44. Integrating ManimCode into two long-form educational video generation systems increases completed videos from 19/117 to 107/117 and from 53/239 to 239/239, alongside improvements in educational effectiveness and visual quality.
• 中文摘要 Manim 支持精确的程序化视频生成，但成功渲染的教育动画仍可能存在视觉缺陷、缺失解释或内容错误。渲染无法直接识别纠正这些视觉缺陷所需的代码变更，因此需要反馈驱动的修订。现有的代理框架使用迭代渲染和修订，但通常依赖大型模型，通常是专有的。现有的 Manim 数据集主要包含最终程序，且常缺乏质量验证以及学习此完善过程所需的反馈和修订轨迹。我们介绍 ManimData 和 ManimCode，这是一个用于反馈驱动教育动画生成的数据和培训框架。我们通过渲染引导流水线构建 ManimData，迭代生成、执行、审查和修订 Manim 程序，并结合检索到的 API 文档。其24,034条经过渲染验证、质量过滤的记录保存中间程序、执行错误、视觉批评和修正性修订。我们使用角色条件监督微调，训练ManimCode这一单一4B模型编写、审查和修订动画程序。为改善与收集轨迹不同错误的纠正，我们通过强化学习训练修订器，基于自生成状态，使用渲染结果和无引用的质量反馈。ManimCode在ManimData测试和ManimBench上实现了96.6%和97.0%的渲染成功率，而同一推理设置下未训练的4B模型分别为52.6%和82.0%，同时将视频质量分数从2.48提升至2.94，从3.00提升至3.44。将ManimCode整合进两个长视频生成系统，使完成视频数量从19/117提升至107/117，从53/239提升至239/239，同时提升了教育效果和视觉质量。

RM-VLM: From Natural-Language Task Descriptions to Vision-Grounded Reward Machines

RM-VLM：从自然语言任务描述到基于视觉的奖励机器

• Authors: Roko Parac, Christos Ziakas, Leo Ardon, Alessandra Russo
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.08308
• Pdf link: https://arxiv.org/pdf/2611.08308
• Abstract Reward machines (RMs) have been proposed as a reward formulation in reinforcement learning (RL) to capture non-Markovian tasks where credit assignment depends on history. By representing reward functions as automata, RMs expose the task's temporal structure and allow such rewards to be handled within a Markovian framework. However, they assume a labelling function, a proposition-grounding oracle for observations, which must be manually engineered for each environment, limiting their wider applicability. In this work, we address this limitation by proposing RM-VLM, a novel approach in which a large language model (LLM) generates an RM from a natural-language task description, and a vision-language model (VLM) acts as the labelling function, grounding the RM's propositions over image-based observations during RM-guided policy learning. A VLM-based value function estimator is used to assess the agent's progress towards each RM transition and derive a dense reward signal. We evaluate RM-VLM on three domains covering non-Markovian grid-world navigation, robotic manipulation, and 3D object collection. Our experimental results show that RM-VLM consistently achieves higher task success rates than existing VLM-based reward methods across all tested domains.
• 中文摘要 奖励机（RM）被提出作为强化学习（RL）中的一种奖励表述，用于捕捉依赖于历史的非马尔可夫任务。通过将奖励函数表示为自动机，RM揭示了任务的时间结构，并允许在马尔可夫框架内处理此类奖励。然而，它们假设了一个标记函数，即观察的命题基础预言机，必须针对每个环境手动设计，限制了其更广泛的适用性。在本研究中，我们提出了RM-VLM这一新颖方法，其中大型语言模型（LLM）从自然语言任务描述生成RM，视觉语言模型（VLM）作为标记函数，将RM的命题基于基于图像的观察，在RM引导的策略学习中建立基础。使用基于VLM的价值函数估计器评估代理在每个RM转变中的进展，并推导出密集的奖励信号。我们在三个领域评估RM-VLM，涵盖非马尔可夫网格导航、机器人操作和三维对象收集。我们的实验结果表明，RM-VLM在所有测试领域中始终比现有基于VLM的奖励方法实现更高的任务成功率。

Discovering Learning Rules that Shape Others

发现塑造他人的学习规则

• Authors: Hong Jun Lee, Ziyan Wang, Jun Wang, Stefanos Leonardos, Yali Du
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.08335
• Pdf link: https://arxiv.org/pdf/2611.08335
• Abstract In general-sum games, self-interested learners can reach outcomes that are poor for everyone, which has motivated methods that shape the opponent's learning. Methods like Learning with Opponent-Learning Awareness (LOLA), which differentiates through the opponent's learning process, and Model-Free Opponent Shaping (M-FOS), which uses model-free optimisation methods for long-horizon opponent shaping, have enabled mutual cooperation and exploitation of learners. However, these methods require access to the opponent's parameters, policy, rewards or learning rule during deployment. In single-agent reinforcement learning, update rules discovered by meta-learning have achieved state-of-the-art results (e.g. DiscoRL). These rules are separate from the policy they train and read only the agent's own experience. We hereby extend this framework to general-sum settings by introducing Discovered Update Rule for Opponent Shaping (DUOS). To the best of our knowledge, this is the first method to discover an update rule for an agent in general-sum games. DUOS meta-trains the rule to maximise the agent's expected return over its lifetime, and updates an inner network using the agent's own experience. We show that our discovered rule exhibits behaviour consistent with opponent shaping without reading opponent policies or parameters, and ranks 1st in normalised mean score and 1st in self-play among tournament entrants.
• 中文摘要 在一般求和博弈中，利己学习者可能得出对所有人都不利的结果，这促使对手的学习方法被塑造。诸如通过对手学习过程区分的对手学习意识学习（LOLA）和使用无模型优化方法进行长期对抗塑造的模型无对手塑造（M-FOS）等方法，促进了学习者的相互合作与利用。然而，这些方法在部署时需要访问对手的参数、策略、奖励或学习规则。在单智能体强化学习中，元学习发现的更新规则取得了最先进的成果（如DiscoRL）。这些规则与它们训练的策略是分开的，只读取智能体自身的经验。我们在此通过引入对手塑造的发现更新规则（DUOS）将该框架扩展到一般求和设置。据我们所知，这是在一般和博弈中发现代理更新规则的首个方法。DUOS 通过元训练该规则以最大化代理在其生命周期内的预期回报，并利用代理自身的经验更新内部网络。我们证明，发现的规则表现出与对手塑造行为一致的行为，无需读取对手策略或参数，并在参赛者中归一化平均分排名第一，在自我对弈中排名第一。

SHORE: Stability-Hybrid On-Policy Reinforcement Learning for Online Bin Packing

SHORE：稳定性-混合策略强化学习，用于在线垃圾箱包装

• Authors: Chang Shi, Ann Majewicz Fey
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.08364
• Pdf link: https://arxiv.org/pdf/2611.08364
• Abstract Online 3D bin packing with a processing buffer requires choosing which box to pack, its orientation, and its placement while keeping the stack stable under unknown future arrivals. A collapse ends the episode and forfeits the remaining volume. Existing methods use geometric rules or hard stability masks, limiting how risk can influence the joint choice of box, orientation, and placement. We propose Stability-Hybrid On-Policy Reinforcement Learning (SHORE), which combines a learned stability probability of a placement with an on-policy estimate of its return conditional on stability. A 3D U-Net learns stability from executed outcomes and simulated alternatives, revealing outcomes of actions underrepresented in early rollouts. A discrete on-policy actor--critic uses stability predictions in both its action values and policy prior, with gates that limit the effect of an untrained or changing model. In deployment, SHORE does not require additional search or simulation overhead. In Isaac Gym, SHORE reaches 73\% space utilization by 0.4M environment steps, while the same learner without stability modeling reaches that level at about 1M. On the expanded 60-box set, SHORE achieves 0.734 space utilization versus 0.588 for the GOPT state-of-the-art baseline, with a higher placement-failure frequency (1.03\% vs. 0.63\%), trading more risk for greater packed volume. The results show faster learning and higher utilization on the more demanding realistic box arrival set. Ablations suggest that simulated alternatives provide useful early supervision, while executed outcomes alone sustain later performance. We also demonstrate the method on a real robot to verify the simplicity of integrating the method to real bin packing pipelines.
• 中文摘要 带有处理缓冲区的在线3D箱打包需要选择要打包的箱子、其朝向和位置，同时保持堆栈在未知未来到达时保持稳定。崩溃则结束该事件并失去剩余体积。现有方法使用几何规则或硬稳定性掩码，限制风险对箱子、方向和放置的联合选择的影响。我们提出了稳定性混合策略强化学习（SHORE），它结合了学习到的配置稳定性概率和基于稳定性的策略内返回估计。3D U-Net从执行结果和模拟的替代方案中学习稳定性，揭示早期部署中被忽视的动作结果。离散的策略上行为者-批判者在其动作值和策略先验中都使用稳定性预测，并通过门限制未训练或变化模型的影响。在部署中，SHORE无需额外的搜索或仿真开销。在Isaac Gym中，SHORE通过0.4百万环境步达到73\%的空间利用率，而未进行稳定性建模的同一学习者在约100万步时达到该水平。在扩展后的60盒套中，SHORE实现了0.734的空间利用率，而GOT最先进基线为0.588，且放置失败频率更高（1.03%对0.63%），以更高的风险换取更大的装箱量。结果显示，在更严苛的真实箱子到达集上，学习更快，利用率更高。消融表明，模拟替代方案提供了有用的早期监督，而仅执行的结果则能维持后续表现。我们还在真实机器人上演示该方法，验证将该方法集成到真实垃圾包装管道中的简单性。

Information Elicitation by Ordinal Agreement

按序协议获取信息

• Authors: Yiling Chen, Shi Feng, Grant Schoenebeck, Fang-Yi Yu
• Subjects: Subjects: Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2611.08366
• Pdf link: https://arxiv.org/pdf/2611.08366
• Abstract Ordinal information is central to tasks ranging from conference peer review to Reinforcement Learning from Human Feedback for large language models. In many such settings, there is no objective ground truth against which reports can be directly verified, making incentive-compatible elicitation challenging. Peer-prediction mechanisms have been designed to achieve truthful elicitation despite the lack of ground-truth verification. However, existing peer-prediction methods do not provide incentives for agents to truthfully compress rich private information (e.g. reviews) into coarser ordinal reports (e.g. rankings) requested by the mechanisms. We propose a simple ordinal peer-prediction mechanism that rewards agents based on pairwise agreement, measured by Kendall tau. Under a broad class of Bayesian Stochastically Order Consistent models---which includes classical models such as Mallows and Thurstone---we prove that truthful reporting is stochastically dominant, and strongly truthful under index-equivariant strategies. This provides robust incentives even for agents with non-linear utility functions, such as risk aversion. Our mechanism further accommodates heterogeneous agents, top-K reports, and subset assignments, where each agent is scored against one randomly drawn peer. Experiments on synthetic Mallows and Thurstone data, together with a real-data stress test on the SUSHI preference dataset, show right-shifted truthful-payment distributions and the highest normalized sensitivity among tested baselines.
• 中文摘要 序数信息在从会议同行评审到大型语言模型的人类反馈强化学习等任务中至关重要。在许多此类环境中，没有客观的基础真理可以直接验证报告，这使得激励兼容的引出具有挑战性。同伴预测机制设计用于实现真实的引出，尽管缺乏基础真理验证。然而，现有的同伴预测方法并未激励代理将丰富的私人信息（如综述）真实压缩为机制请求的更粗略的序数报告（如排名）。我们提出了一种简单的序数同伴预测机制，基于两两一致（以Kendall tau衡量）奖励代理。在一类广泛的贝叶斯随机顺序一致模型---包括经典模型如Mallows和Thurstone---我们证明了真实报告在随机上占优势，且在指数等变策略下具有强真实性。这为具有非线性效用函数（如风险厌恶）的代理提供了强有力的激励。我们的机制还兼容异质代理、top-K报告和子集分配，每个代理与一个随机抽取的对等代理进行评分。对合成Mallows和Thurstone数据的实验，以及对SUSHI偏好数据集的真实数据压力测试，显示真实支付分布右移，且在测试基线中具有最高的归一化敏感度。

Spectral and Subspace Dynamics of LoRA in Supervised Fine-Tuning and Reinforcement Learning

监督式微调与强化学习中LoRA的谱动态和亚空间动力学

• Authors: Chenruo Liu, Zihan Wang, Yijun Dong, Qi Lei
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2611.08414
• Pdf link: https://arxiv.org/pdf/2611.08414
• Abstract Low-rank adaptation (LoRA) is commonly used in both supervised fine-tuning (SFT) and reinforcement learning (RL), often under similar optimizer settings. Yet, how the different dynamics of the two stages manifest in the low-rank increment that LoRA optimizes remains unclear. Tracking the singular spectrum and subspaces of LoRA updates on mathematical reasoning tasks, we find that RL concentrates the spectrum at every learning rate, while SFT expands it at moderate learning rates; moreover, RL rotates the singular subspaces much faster than SFT at small learning rates. Interpolating between SFT and RL one ingredient at a time, we pinpoint group-mean baseline subtraction as the source of both contrasts. A quadratic multi-index model updated via LoRA unveils the underlying mechanism: group-mean baseline subtraction tilts the balance between two competing forces, coherent fitting and gradient noise, toward spectral concentration and faster subspace rotation, while forcing the risk-favorable learning rate to be low. Guided by these dynamics, we tailor LoRA updates to each stage: in SFT, we flatten the spectrum of the full update to spread adaptation across singular directions, whereas in RL, we flatten the part of the update outside the current LoRA increment span to promote subspace rotation. Experiments on mathematical reasoning tasks across multiple models show that the respective configurations for SFT and RL yield the best performance among the compared methods in most evaluated settings.
• 中文摘要 低秩适应（LoRA）常用于监督微调（SFT）和强化学习（RL），通常在类似的优化器设置下。然而，LoRA优化的低秩增量中这两个阶段的不同动态如何表现仍不清楚。通过追踪LoRA在数学推理任务中的奇异谱和子空间更新，我们发现RL在每个学习率下集中谱，而SFT在中等学习率时扩展谱;此外，RL在低学习率下旋转奇异子空间的速度远快于SFT。通过一次插值SFT和RL的一种成分，我们确定了群均基线减法作为两种对比的来源。通过LoRA更新的二次多指标模型揭示了其潜在机制：群均基线减法使相干拟合和梯度噪声这两个竞争力之间的平衡倾向于谱集中和更快的子空间旋转，同时迫使风险有利的学习率降低。基于这些动态，我们针对每个阶段调整LoRA更新：在SFT中，我们将完整更新的频谱平坦化，以在单一方向上扩散适应;而在强化学习中，我们平整当前LoRA增量跨外的部分，以促进子空间旋转。跨多个模型的数学推理任务实验表明，SFT和RL的各自配置在大多数评估环境中表现最佳。

Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO

解析强到弱OPD对GRPO的计算效率分析

• Authors: Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen, Anurag Koul, Srijan Bansal, Erik Nijkamp, Shafiq Joty, Semih Yavuz
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.08453
• Pdf link: https://arxiv.org/pdf/2611.08453
• Abstract On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with $50\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD (DeepSeek-AI, 2026) consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.
• 中文摘要 策略提纯（OPD）在固定教师的每个代币监督下，对学生进行自身推广训练。这种密集信号是否降低了达到目标准确率所需的训练FLOP，与强化学习相比？我们系统地比较了基础反向基准OPD与组相对策略优化（GRPO），涵盖任务难度、学生和教师规模、领域和优化选择，衡量其与训练FLOP的延迟准确率。使用外部后期培训教师时，OPD在匹配设置下每训练FLOP的准确率通常会落后于GRPO，即使教师接受了GRPO进一步培训，且常常出现响应长度膨胀。但不同的教师会改变局面：当教师成为学生自己的GRPO检查点时，OPD用比GRPO所需的50倍倍的提炼FLOP数量恢复该检查点的数学准确性。在相同的教师设置下，我们持续观察到较高的学生、编程和第二模型家族中的高效率。然而，在单一领域，GRPO检查点已是理想模型，因此提炼后可增加总计算量。跨域，不同的GRPO检查点具有不同的能力，且没有单一检查点服务所有领域。多教师OPD（DeepSeek-AI，2026）将每个领域的GRPO专家整合为一个学生，使每位专家的准确度匹配，蒸馏阶段在两个领域仅占专业培训FLOP的5.2%和8.0%（四个领域）。将专业培训和蒸馏合并计算，该流程达到GRPO在两域混合中的联合表现，且总FLOP减少了42.0%。

Latent RL Priors for Data-Efficient Dexterous Manipulation

数据高效灵活操作的潜在强化学习先验

• Authors: Yixiao Wang, Peiyi Chen, Mustafa Mukadam, Shreekant Gayaka, Masayoshi Tomizuka, Son Tran
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.08458
• Pdf link: https://arxiv.org/pdf/2611.08458
• Abstract Learning dexterous manipulation from limited visual demonstrations is challenging because policies must acquire both perceptual grounding and precise control, while reinforcement learning (RL) in simulation can acquire rich control behaviors from large-scale interaction with privileged information. We introduce \textbf{Latent RL Prior}, a two-stage framework that reuses this control knowledge for data-efficient visual policy learning. During privileged pretraining, we distill large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. During visual adaptation, we freeze the decoder and learn a visual interface that maps images to its latent context using action supervision alone, without privileged labels or latent targets. The same decoder can further be shared across arms by learning coordinated visual contexts for bimanual manipulation. Across three simulated tasks, our method outperforms full-parameter fine-tuning of $\pi_{0.5}$ and GR00T N1.7 in the low-data regime; on door lifting and bimanual assembly, it surpasses full-data baselines using $10\times$ and $5\times$ fewer demonstrations, respectively. Further analysis shows that direct visual-to-latent conditioning improves adaptation performance and training stability over routing visual predictions through the pretrained state encoder. Robustness to unseen mass variation and greater linear accessibility of contact-force information in decoder representations further provide evidence of retained RL-acquired control knowledge. Project website: this https URL.
• 中文摘要 从有限的视觉演示中学习灵巧操作具有挑战性，因为策略必须获得感知基础和精确控制，而模拟中的强化学习（RL）则能通过与特权信息的大规模互动获得丰富的控制行为。我们引入 \textbf{Latent RL Prior}，这是一个两阶段框架，重用该控制知识以实现数据高效的视觉策略学习。在特权预训练中，我们将大规模强化学习（RL）推导出基于本体感觉和潜在任务上下文的流程匹配动作解码器。在视觉适应过程中，我们冻结解码器，学习一个仅通过动作监督将图像映射到潜在上下文的视觉界面，无需特权标签或潜在目标。同一解码器还可以通过学习协调的视觉上下文进行双手操作，实现跨臂共享。在三项模拟任务中，我们的方法在低数据区段优于全参数微调（$\pi_{0.5}$）和GR00T N1.7;在门抬起和双手组装方面，它分别以$10\乘数和$5\倍数的演示次数超越全数据基线。进一步分析显示，直接的视觉到潜在条件反射比通过预训练状态编码器路由视觉预测更能提升适应性能和训练稳定性。对未见质量变化的鲁棒性以及解码器中接触力信息的更线性可及性进一步证明了强化学习获得的控制知识依然保留。项目网站：此 https URL。

Keyword: diffusion policy

There is no result