生成时间: 2026-11-14 00:00:11 (UTC+8); Arxiv 发布时间: 2026-11-13 20:00 EST (2026-11-14 09:00 UTC+8)

今天共有 52 篇相关文章

Keyword: reinforcement learning

Enhancing Resolution and Reliability with Attention Mechanisms in Nuclear Accident Surrogate Modeling

通过注意力机制提升核事故替代建模中的分辨率和可靠性

• Authors: Wonung Jeong, Sangam Khanal, Jeongik Lee, Sangseung Lee, Joongoo Jeon
• Subjects: Subjects: Computational Engineering, Finance, and Science (cs.CE); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.14532
• Pdf link: https://arxiv.org/pdf/2611.14532
• Abstract Severe accident (SA) analysis of nuclear power plants relies on system codes such as MAAP, whose computational cost limits large-scale applications including dynamic probabilistic safety assessment and reinforcement learning. This study proposes the Accident Binary-Continuous Transformer (ABC-Transformer), a decoder-only surrogate combining causal self-attention with temporal attention pooling. It jointly embeds continuous thermal-hydraulic variables with binary safety-system and severe accident management guideline (SAMG) states, enabling 5-minute-resolution forecasting over 72-hour transients. Trained on 11,000 MAAP simulations of a total loss of component cooling water scenario for the OPR1000, it achieved lower one-step teacher-forcing error than CNN and recurrent baselines at fine resolutions. Autoregressive reliability was characterized as a full per-scenario distribution over 1100 held-out scenarios. For scheduled-sampling (SS) training, the median $R^2$ is $0.947 \pm 0.006$ and $3.8 \pm 1.0$% of scenarios fail ($R^2<0$; mean $\pm$ SD over four training seeds), while errors accumulate with rollout length. A Mahalanobis self-distance computed from each surrogate rollout, before comparison with the MAAP reference, ranks these failures (ROC-AUC 0.895, reference seed). A threshold at the 40th percentile of calibration scores accepts $36.8 \pm 0.5$% of 550 evaluation scenarios, with no accepted rollout below $R^2 = 0$ for any seed, and routes the rest to MAAP.
• 中文摘要 核电站的严重事故（SA）分析依赖于如MAAP等系统代码，其计算成本限制了包括动态概率安全评估和强化学习在内的大规模应用。本研究提出了事故二元连续变压器（ABC-Transformer），这是一种仅解码器的替代品，结合了因果自关注和时间注意力集中。它结合了连续热-液压变量，并结合了二元安全系统和严重事故管理指南（SAMG）状态，实现了72小时瞬变的5分钟分辨率预测。通过训练11,000次MAAP模拟，模拟该OPR1000的组件冷却水完全丢失情景，在细分辨率下实现了比CNN和循环基线更低的一步教师强迫误差。自回归可靠性被描述为对1100个保留场景的全场景分布。在计划抽样（SS）训练中，中位数$R^2$为$0.947 \pm 0.006$，$3.8 \pm 1.0$的失败率（$R^2<0$;四个训练种子的平均$\pm SD），而错误随推广长度累积。在与MAAP参考比较前，从每个替代部署计算的Mahalanobis自距离对这些失败进行排名（ROC-AUC 0.895，参考种子）。校准得分第40百分位的阈值接受550个评估场景中36.8 \pm 0.5$的比例，且任何种子中没有低于$R^2 = 0$的公认推广，其余将路由至MAAP。

StructRL: Model-Free Reinforcement Learning with Replay Graphs

StructRL：无模型强化学习与重放图

• Authors: Ivo Nowak
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.14599
• Pdf link: https://arxiv.org/pdf/2611.14599
• Abstract StructRL turns the experience collected during reinforcement learning into an explicit replay graph. Once the graph contains a path to the task goal, dynamic programming selects actions on the observed part of the state space; these actions form a graph policy that is defined only where the required transitions have been observed. StructRL transfers these actions to model-free learners: as action preferences for a DQN Q-model and as target directions along shortest observed paths for a SAC actor. For DQN on three sparse-reward MiniGrid tasks, final success rises from 0.10 to 0.90 (DoorKey), 0.10 to 0.80 (KeyCorridor) and 0 to 0.50 (ObstructedMaze); plain DQN does not catch up with three times the budget. For SAC on continuous PointMaze, graph-derived direction labels raise final success from 0.17 to 1.0 on seven matched runs that reached the goal. Distilling the graph policy into a fresh Q-model gives a DoorKey 6x6 controller that reaches the goal from 91% of reachable states, against 38% for plain DQN. Scaling experiments identify three bottlenecks: incomplete graph support, structural aliasing, and loss of graph information in high-dimensional inputs.
• 中文摘要 StructRL 将强化学习过程中收集的经验转化为显式重放图。一旦图中包含通往任务目标的路径，动态规划会选择观察到状态空间部分的动作;这些动作形成图策略，仅在观察到所需转移时定义。StructRL 将这些动作传递给无模型的学习者：作为 DQN Q 模型的动作偏好，以及作为 SAC 演员沿最短观测路径的目标方向。对于三个稀疏奖励 MiniGrid 任务的 DQN，最终成功率从0.10升至0.90（DoorKeyKey）、0.10 至 0.80（KeyCorridor）和0至0.50（ObstructedMaze）;纯DQN无法达到三倍预算。对于连续PointMaze上的SAC，图源方向标签将七次匹配运行达到目标的最终成功率从0.17提升至1.0。将图策略提炼成新的Q模型，得到一个DoorKey 6x6控制器，能从91%的可达状态中达到目标，而普通DQN为38%。缩放实验识别出三个瓶颈：图支持不完整、结构混叠和高维输入中图信息丢失。

Trait Hitchhiking in LLM Post-Training

LLM培训后特质搭便车

• Authors: Adil Soubki, Elizabeth S.Z. Tan, Miles Cranmer
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.14683
• Pdf link: https://arxiv.org/pdf/2611.14683
• Abstract Reinforcement learning (RL) is now a standard stage of language model post-training, and the compute devoted to it is growing rapidly. While it has been shown to improve capabilities, particularly for tasks with verifiable rewards, it can also change behaviours and internal properties that the reward does not explicitly target. We present a formalism to describe the evolution of "traits" during RL post-training and, in doing so, recover the Price equation from evolutionary biology. We use this to develop a mathematical model for what we call "trait hitchhiking": the amplification or suppression of non-targeted traits as training redistributes probability across completions. We evaluate this Price update expression on language models trained with GRPO and show that the Price estimates closely track the sampled trait dynamics. These results identify trait hitchhiking as a mechanism by which RL post-training can silently amplify undesirable traits.
• 中文摘要 强化学习（RL）现已成为语言模型训练后标准阶段，相关计算量正在迅速增长。虽然已被证明能提升能力，尤其是对可验证奖励任务，但它也可能改变奖励未明确针对的行为和内部属性。我们提出了一种形式主义，描述强化学习后训练中“特质”的演变，并由此从进化生物学中恢复了Price方程。我们利用此构建了所谓“特质搭便车”的数学模型：即在训练过程中，非定向特质的放大或抑制，使概率重新分配到完成过程。我们在用GRPO训练的语言模型上评估了Price更新表达式，表明Price估计与抽样的特质动态紧密相符。这些结果表明，特质搭便车是强化训练后默默放大不良特质的机制。

Open Challenges and Opportunities in Human-Agent Systems

人机系统中的未解决挑战与机遇

• Authors: Reuth Mirsky, Sarath Sreedharan, Nitay Alon, Stefan Sarkadi, Munindar P. Singh, Liz Sonenberg, Peter Stone, Silvia Tulli, Stylianos Loukas Vasileiou, Shlomo Zilberstein
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.14846
• Pdf link: https://arxiv.org/pdf/2611.14846
• Abstract Agentic AI systems increasingly undertake extended tasks and act on people's behalf, making human-agent interaction central to their effectiveness. Successful interaction requires understanding human perspectives, coordinating complementary capabilities, and maintaining productive relationships within socio-technical environments. This paper examines how foundations developed by the Autonomous Agents and Multi-Agent Systems (AAMAS) community inform these demands. We consider interactions ranging from individual human-agent partnerships to teams and larger groups, where participants differ in their roles, decision authority, capabilities, and objectives. We organize computational approaches around the connected challenges of understanding others and deciding how to interact with them. Theory of Mind, belief-desire-intention architectures, goal and plan recognition, inverse planning, and recursive reasoning provide mechanisms for representing and inferring partially observable mental states. Observer-aware planning, model reconciliation, and decision-theoretic communication connect these inferences to action, explanation, and information exchange. Together with shared autonomy, argumentation, and cooperative inverse reinforcement learning, these approaches offer foundations for agentic systems that account for evolving human knowledge, intentions, and expectations. We argue that progress requires treating human interaction as an ongoing decision problem shaped by uncertainty, interdependence, and potentially conflicting objectives.
• 中文摘要 智能人工智能系统越来越多地承担扩展任务并代表人们行动，使人与智能体交互成为其有效性的核心。成功的交互需要理解人类视角、协调互补能力，并在社会技术环境中维持富有成效的关系。本文探讨了自主智能体与多智能体系统（AAMAS）社区开发的基础如何指导这些需求。我们考虑了从个人人主伙伴关系到团队和更大群体的交互，参与者在角色、决策权、能力和目标上存在差异。我们围绕理解他人及决定如何与他人互动的相关挑战组织计算方法。心智理论、信念-欲望-意图架构、目标与计划识别、逆向规划以及递归推理提供了表示和推断部分可观察心理状态的机制。观察者意识规划、模型调和和决策理论交流将这些推论与行动、解释和信息交换联系起来。结合共享自主性、论证和合作逆强化学习，这些方法为能够体化系统奠定基础，涵盖人类知识、意图和期望的演变。我们认为，进步需要将人际互动视为一个由不确定性、相互依存和潜在冲突目标塑造的持续决策问题。

FOE-Dreamer: Deployment-Efficient Learning of Cyber Defense Policies in Operational Networks

FOE-Dreamer：在运营网络中高效部署学习网络防御政策

• Authors: Yinuo Du, Christopher Kiekintveld
• Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.14880
• Pdf link: https://arxiv.org/pdf/2611.14880
• Abstract Autonomous cyber defense is widely studied with deep reinforcement learning, but most work trains policies inside simulators because learning requires many trial-and-error interactions. Simulator-based training carries a fidelity gap: trained policies face authentication delays, partial action failures, service dependencies, and user traffic patterns that the simulator did not represent, and degrade on transfer. Training directly in the network being defended removes the gap but makes every interaction slow and costly, so sample efficiency becomes the binding constraint. Model-based reinforcement learning has reduced sample requirements on games and physical robots by orders of magnitude by planning inside a learned world model, but it has not been applied to defenders trained in operational networks. We present FOE-Dreamer (Factored Opponent-Enhanced Dreamer), a model-based reinforcement learning framework that trains a defender online and directly in a network on cloud infrastructure, without a simulator stage. FOE-Dreamer separates three sources of variation in the latent state: a controllable component driven by defender actions, an exogenous component driven by benign user activity, and an opponent embedding inferred from defender-observable signals alone. Separating the components reduces the variance in policy gradients during planning in imagination. Against two scripted attacker profiles with user activity emulated by the GHOSTS framework on an 8-host OpenStack testbed, FOE-Dreamer trains within a 3-day budget on a single GPU and achieves about 2$\times$ better episode loss than Rainbow and IQN baselines under matched compute. Both the factoring and the opponent model contribute to the gain, and the learned latents are consistent with the intended separation.
• 中文摘要 自主网络防御在深度强化学习中被广泛研究，但大多数工作是在模拟器内训练策略，因为学习需要大量试错交互。基于模拟器的训练存在保真度差距：训练后的策略面临认证延迟、部分动作失败、服务依赖和用户流量模式，模拟器未曾表示这些问题，且在传输过程中会退化。直接在被防御网络中训练消除了差距，但使每次交互都缓慢且成本高昂，因此样本效率成为约束条件。基于模型的强化学习通过在学习世界模型内规划，将游戏和物理机器人的样本需求降低了几个数量级，但尚未应用于在运营网络中训练的防御者。我们介绍FOE-Dreamer（分解对手增强Dreamer），这是一个基于模型的强化学习框架，可在云基础设施上在线和直接网络中训练防御者，无需模拟器阶段。FOE-Dreamer将潜在状态的三个变异源分离出来：由防御者行动驱动的可控组件、由良性用户活动驱动的外生成分，以及仅凭防御者可观察信号推断的对手嵌入。分离这些组件减少了在想象中规划时策略梯度的变异。面对两个由GHOSTS框架在8主机OpenStack测试平台上模拟用户活动的脚本攻击者配置文件，FOE-Dreamer在单GPU上以3天预算训练，在匹配计算下比Rainbow和IQN基线高约2$/时间点的集数损失。分解和对手模型都对增益有贡献，且学习的潜能与预期分离一致。

Epigraph Form Reinforcement Learning-based Safe Trajectory Tracking Control for Connected and Autonomous Vehicles

Epigraph形式基于强化学习的互联和自动驾驶车辆安全轨迹跟踪控制

• Authors: Xiaoke Yang, Jin Chen, Yafei Wang, Qi Zhang, Ming Li, Haiwen Wu, Chengliang Yin, Yong Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.14903
• Pdf link: https://arxiv.org/pdf/2611.14903
• Abstract Safe trajectory tracking (STT) aims to enable autonomous vehicles to follow a reference trajectory accurately while avoiding collisions. This problem is particularly challenging for Connected and Autonomous Vehicles (CAVs) in highly dynamic environments. Existing non-learning-based STT methods often use simplified vehicle and interaction models to meet real-time requirements, which may result in suboptimal tracking accuracy. Although Deep Reinforcement Learning (DRL) can help address this limitation, widely used Lagrangian DRL methods often suffer from unstable training because increasing Lagrange multipliers amplify policy-gradient magnitudes. To address these issues, we propose a novel reinforcement learning framework for STT, called STT Control Epigraph Form RL (STTC-EFRL). In this framework, the STT problem is formulated as an epigraph-form min-max optimization problem. Based on this formulation, we develop a learning mechanism that uses a decreasing scalar related to the epigraph to adjust the policy gradient descent direction without increasing the gradient magnitude, thereby improving training stability. We also develop a computationally efficient ray-casting method to compute a polyhedron scaling factor for safety quantification. Experimental results, including Vehicle-in-the-Loop tests, show that STTC-EFRL achieves higher tracking accuracy than the considered non-learning-based baseline and more stable training than widely used Lagrangian DRL methods, while satisfying safety requirements in connected environments.
• 中文摘要 安全轨迹跟踪（STT）旨在使自动驾驶车辆能够在避免碰撞的同时，准确地跟踪参考轨迹。对于在高度动态环境中的联网和自动驾驶车辆（CAV）来说，这一问题尤为具有挑战性。现有的非基于学习的STT方法常常使用简化的车辆和交互模型以满足实时需求，这可能导致跟踪精度不理想。尽管深度强化学习（DRL）可以帮助解决这一限制，但广泛使用的拉格朗日DRL方法常常存在训练不稳定的问题，因为拉格朗日乘数的增加会放大策略梯度的幅度。为解决这些问题，我们提出了一种新的STT强化学习框架，称为STT控制引文形式RL（STTC-EFRL）。在该框架下，STT问题被表述为引言表式最小极大优化问题。基于该表述，我们开发了一种学习机制，利用与引字相关的递减标量调整策略梯度下降方向而不增加梯度大小，从而提升训练稳定性。我们还开发了一种计算高效的射线投射方法，用于计算多面体尺度因子以实现安全量化。实验结果，包括环路载体测试，表明STTC-EFRL在跟踪精度上优于考虑的非基于学习的基线，训练稳定性优于广泛使用的拉格朗日日程日行灯方法，同时满足连接环境中的安全要求。

Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data

重新思考“强化学习泛化，SFT记忆”：SFT数据的作用

• Authors: Yunlong Hou, Fengzhuo Zhang, Yuan Cheng, Jiachun Pan, Xingyao Li, Zhuoran Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.14912
• Pdf link: https://arxiv.org/pdf/2611.14912
• Abstract Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.
• 中文摘要 使用强化学习（RL）训练且奖励可验证的大型语言模型展现出较强的推理能力和广泛的泛化能力，而用监督微调（SFT）训练的模型通常被认为更容易记忆和有限的迁移。本文通过广泛使用的Qwen2.5和Qwen3系列模型上的SFT训练数据重新思考这一区别。首先，我们研究了数据源的作用，并证明其至关重要：精心混合的SFT数据集远远优于仅由更大模型生成的数据。其次，我们研究数据尺度的作用，表明匹配SFT和RL之间正确展开次数极大提升SFT泛化，同时匹配总推广预算使SFT能够实现和强化学习。结合这两者进一步使SFT的泛化效果甚至优于强化学习。第三，通过使用LLM注释来描述训练部署中的解法，我们表明更大的数据集覆盖更多尾法，这些尾法提供了可推广的推理信号。最后，我们通过分析浅层变换器在强化和SFT下的训练动态，理论上支持这些实证发现。

Inverse-Optimal Fixed-Time Integral Reinforcement Learning for Unknown Nonlinear Leader-Follower Multi-Agent Systems

未知非线性先导-跟随多智能体系统的逆最优固定时间积分强化学习

• Authors: Tien Dat Vu, Minh Doan
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.14927
• Pdf link: https://arxiv.org/pdf/2611.14927
• Abstract This paper develops a distributed inverse-optimal fixed-time integral reinforcement learning framework for unknown nonlinear leader-follower multi-agent systems. The proposed method first reconstructs the unknown agent dynamics from finite integral data. Based on the identified dynamics, a distributed inverse-optimal formulation is constructed by prescribing a desired fixed-time value function and recovering a compatible running cost. A critic-only integral reinforcement learning architecture is then developed using current integral data and finitely informative experience-replay data, thereby avoiding the persistent-excitation requirement. Practical fixed-time convergence of the critic-weight errors is established in the presence of identification and function-approximation residuals. Finally, a network-level composite Lyapunov analysis proves practical fixed-time leader-following consensus of the true unknown nonlinear multi-agent system. Numerical simulations are provided to validate the developed theory.
• 中文摘要 本文开发了一种分布式逆最优固定时间积分强化学习框架，应用于未知非线性领先-跟随多智能体系统。所提出的方法首先从有限积分数据重建未知智能体动态。基于识别的动力学，通过规定期望的固定时间值函数并恢复兼容运行成本，构建了分布式逆最优表述。随后利用当前积分数据和有限信息经验回放数据开发纯批评积分强化学习架构，从而避免持久激励要求。在识别和函数近似残差存在的情况下，已建立批判权重误差的实际固定时间收敛性。最后，网络层级的复合李雅普诺夫分析证明了真实未知非线性多智能体系统的固定时间领先跟随共识。提供数值模拟以验证已建立的理论。

Binding Mode Matters: Hotspot-Aware Drug Discovery via Explorative Preferences

结合模式重要：通过探索性偏好实现热点感知药物发现

• Authors: Dingshuo Chen, Hao Yang, Kuangqi Zhou, Zhixun Li, Qiang Liu, Xiangxiang Zeng, Shu Wu, Yaosen Min
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.14956
• Pdf link: https://arxiv.org/pdf/2611.14956
• Abstract Discovering hit molecules requires not just high binding affinity, but also the identification of diverse binding modes, which are critical for experimental assays. Existing generative approaches have predominantly relied on optimizing a scalar docking score, obscuring the distinct contributions of key binding determinants. To this end, we introduce a paradigm shift by formulating target-based drug design as a multi-objective exploration task, where each objective explicitly corresponds to enhancing interactions with a specific hotspot. Here, we introduce BindMol, a novel generative framework driven by a customized multi-objective reinforcement learning algorithm. By incorporating explorative preferences during training, our approach efficiently uncovers molecules with diverse and desirable binding profiles. Empirical results demonstrate that BindMol facilitates the discovery of high-affinity compounds characterized by both structural novelty and diverse binding modes. Validated across target-based drug discovery and multi-property optimization tasks, our approach provides a versatile paradigm for goal-oriented drug discovery.
• 中文摘要 发现命中分子不仅需要高结合亲和力，还需识别多样结合模式，这对实验检测至关重要。现有生成方法主要依赖于优化标量对接分数，掩盖了关键结合决定因素的独特贡献。为此，我们通过将基于靶点的药物设计制定为多目标探索任务，明确对应增强与特定热点相互作用，从而引入范式转变。这里，我们介绍了BindMol，一种由定制多目标强化学习算法驱动的新型生成框架。通过在训练过程中融入探索偏好，我们的方法高效发现具有多样且理想的结合谱的分子。实证结果表明，BindMol促进了高亲和力化合物的发现，这些化合物具有结构新颖性和多样结合模式。我们的方法在基于靶点的药物发现和多特性优化任务中得到了验证，为目标导向的药物发现提供了多样化的范式。

LLM-Enhanced Deep Reinforcement Learning with Human-in-the-Loop Calibration for Real-Time Optimal Power Flow

LLM增强深度强化学习，采用人机在环校准实现实时最优功率流

• Authors: Jiayang Tu, Bozhen Jiang, Bojun Du, Yunxiao Ling, Ershun Du, Qin Wang
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.15003
• Pdf link: https://arxiv.org/pdf/2611.15003
• Abstract Real-time optimal power flow (RT-OPF) requires fast and reliable decision-making under time-varying operating conditions. However, many practical operating preferences and safety related requirements are difficult to encode using conventional numerical rewards alone, which limits the effectiveness and controllability of deep reinforcement learning (DRL) for RT-OPF. To address this issue, this paper proposes an LLM-enhanced DRL framework with human-in-the-loop (HITL) for RT-OPF. The proposed method augments the physics-based reward with sparse language evaluative feedback during offline training, so that semantic operational requirements can be incorporated into policy learning. To improve reliability in RT-OPF, a selective HITL calibration mechanism is further introduced to inspect and correct questionable LLM judgments before they continuously shape the learning process. Case studies on the IEEE 30-bus and IEEE 118-bus systems show that LLM feedback improves policy safety over vanilla TD3, while further HITL calibration achieves the best overall performance and more stable training behavior. These results demonstrate the effectiveness of combining language guidance and expert calibration for RT-OPF.
• 中文摘要 实时最优功率流（RT-OPF）需要在时间变化的操作条件下快速且可靠的决策。然而，许多实际操作偏好和安全相关要求难以仅用传统数值奖励编码，这限制了深度强化学习（DRL）在RT-OPF中的有效性和可控性。为解决这一问题，本文提出了一个基于LLM增强的DRL框架，采用人机环路（HITL）支持RT-OPF。该方法在离线训练期间通过稀疏的语言评估反馈增强基于物理的奖励，从而将语义操作需求纳入策略学习中。为提高RT-OPF的可靠性，进一步引入了选择性HITL校准机制，用于在LLM判断持续影响学习过程前检查和纠正其问题。IEEE 30-总线和IEEE 118-总线系统的案例研究显示，LLM反馈相较于普通TD3提升了策略安全性，而进一步的HITL校准则能实现最佳的整体性能和更稳定的训练行为。这些结果证明了结合语言指导与专家校准对RT-OPF的有效性。

Elastra: A Simulation Framework for Elastic Bodies

Elastra：弹性体模拟框架

• Authors: Youran Sun, Chugang Yi, Xingyu Ren, Haizhao Yang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.15112
• Pdf link: https://arxiv.org/pdf/2611.15112
• Abstract When robots interact with elastic surfaces such as mattresses and trampolines, changes in loading cause the surfaces to deform and rebound, altering subsequent contact locations and forces. We present Elastra, a simulation framework for modeling this interaction and supporting the training and evaluation of robot control policies. Elastra represents each surface using elements that move only vertically, with elastic connections and damping to model deformation and recovery. Contact forces drive surface motion, and the updated surface geometry is used to compute subsequent contact forces. We systematically validate our simulations against an independent implementation and a real mattress load curve. We further use Elastra to train the Unitree G1 humanoid to stand up on mattresses and trampolines. Using the pretrained HoST policy and ProtoMotions motion tracker as original controllers, we train residual policies through reinforcement learning to add corrections to their joint targets. Adding these residual policies increases the success rate with HoST from 44.1\% to 72.5\%, and from 55\% to 89\% on the trampoline. With ProtoMotions, the success rate increases from 19.0\% to 45.0\% for initial poses not encountered during training. The code and video are available on this https URL.
• 中文摘要 当机器人与床垫和蹦床等弹性表面相互作用时，载荷变化导致表面变形和反弹，从而改变后续接触位置和力。我们提出了Elastra，一个用于模拟这种相互作用并支持机器人控制策略训练和评估的模拟框架。Elastra用仅垂直运动的元素表示每个表面，结合弹性连接和阻尼来模拟变形和恢复。接触力驱动表面运动，更新后的表面几何形状用于计算后续接触力。我们系统地验证模拟结果，基于独立实现和真实床垫载荷曲线。我们还进一步使用Elastra训练Unitree G1类人生物站立在床垫和蹦床上。使用预训练的HoST策略和ProtoMotions运动追踪器作为原始控制器，我们通过强化学习训练残余策略，对其联合目标添加修正。添加这些残余策略后，HoST的成功率从44.1%提升到72.5%，蹦床成功率从55%提升到89%。使用ProtoMotions时，初始姿势未在训练中遇到的成功率从19.0%提升至45.0%。代码和视频可在该 https 网址访问。

Sim-to-Real Powertrain Vibration Control via Model-Based Assisted Domain-Randomized Reinforcement Learning

通过基于模型的辅助领域随机强化学习实现仿真到真实动力总成振动控制

• Authors: Heisei Yonezawa, Ansei Yonezawa, Itsuro Kajiwara
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.15133
• Pdf link: https://arxiv.org/pdf/2611.15133
• Abstract Controllers designed from nominal models often degrade on mechanical systems with nonlinearities and parametric variations, such as automotive powertrains with backlash. Deep reinforcement learning (DRL) with domain randomization can acquire robustness to such model discrepancies, but wide randomization destabilizes training and yields conservative policies, and experimental transfer of such policies has rarely been demonstrated in vibration control. This paper presents model-based controller assisted domain randomization (MBCADR), a framework that integrates model-based control into domain-randomized DRL. An $H_{2}$ controller designed from the nominal model supplies a baseline action, while a residual policy trained with twin delayed deep deterministic policy gradient under randomized dynamics compensates for nonlinearities and uncertainties. This model-based assistance relieves the agent of synthesizing the entire control input, thereby stabilizing training under extensive randomization. For each realization of the uncertain parameters, the augmented plant--controller state is shown to be a Markov state of the closed loop, so that domain-randomized training is rigorously formulated as policy optimization over a parameterized family of Markov decision processes. A Lyapunov-based stability analysis further derives a forward-invariant set and an explicit ultimate bound, in which the learned action enters only through its saturation level. Monte Carlo simulations with multiple training seeds show that MBCADR reduces performance variability compared with either component alone. Experiments on a powertrain test bench with backlash, using the simulation-trained policy on an embedded processor without further tuning, confirm successful sim-to-real transfer.
• 中文摘要 基于名义模型设计的控制器在具有非线性和参数变化的机械系统上常常会退化，例如汽车动力系统中伴随反冲。带域随机化的深度强化学习（DRL）可以获得对此类模型差异的鲁棒性，但宽随机化会使训练不稳定并产生保守策略，且此类策略的实验转移在振动控制中很少被证明。本文提出了基于模型的控制器辅助域随机化（MBCADR）框架，该框架将基于模型的控制整合进域随机化日间移动。基于名义模型设计的$H_{2}$控制器提供基线动作，而在随机动力学下用双延迟深确定性策略梯度训练的残差策略则补偿非线性和不确定性。这种基于模型的辅助减轻了代理合成全部控制输入的麻烦，从而稳定了在广泛随机化下的训练。对于每个不确定参数的实现，增强的植物-控制状态被证明为闭环的马尔可夫状态，因此域随机训练被严格表述为参数化马尔可夫决策过程族的策略优化。基于李雅普诺夫的稳定性分析进一步推导出一个前向不变集合和显式的终极界限，其中所学动作仅通过其饱和水平进入。蒙特卡洛模拟与多训练种子显示，MBCADR相比单独任一组件能降低性能变异性。在带有回荡的动力传动测试台上实验，使用仿真训练策略在嵌入式处理器上进行且未进一步调优，确认模拟到实物的成功转移。

HyperSkill: Training-Free Omnimodal GRPO via Hypergraph-Indexed Skill-Library Evolution

HyperSkill：通过Hypergraph索引技能库演进实现无训练的全模态GRPO

• Authors: Haoran Luo, Shangkai Lin, Jinyang Wu, Xinliang Zhou, Luu Anh Tuan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.15178
• Pdf link: https://arxiv.org/pdf/2611.15178
• Abstract Reinforcement learning sharpens LLM reasoning, but parameter updates are expensive, forgetful, and opaque; training-free libraries swap gradients for a textual store but still use trajectory-level, correlation-based admission and flat structure. To address both, we propose HyperSkill, a Training-Free Omnimodal GRPO framework that keeps the backbone frozen and replaces gradient updates with auditable edits to a Hypergraph-Indexed Skill-Library, where skills are advantage-bearing nodes and compositions are hyperedges. It has three components: training-time distillation admits counterfactually checked skills and learns a protocol skill from failures; gated hypergraph retrieval injects advantage-ranked skills where a per-dataset gate admits them; and an execution protocol makes the frozen backbone commit to an answer. On 11 text, vision-language, and omnimodal benchmarks, HyperSkill is best or tied on every dataset and beats the best baseline by 13.0 average F1 points while keeping the backbone frozen. Our software and data are publicly available at this https URL.
• 中文摘要 强化学习提升了大型语言模型推理能力，但参数更新成本高、容易遗忘且不透明;无训练库用文本存储取代梯度，但仍使用轨迹级、基于相关性且采用平面结构。为解决这两点，我们提出了HyperSkill，一个无训练的全模态GRPO框架，保持骨干链冻结，用可审计的超图索引技能库编辑替代梯度更新，其中技能为优势节点，组合为超边。它包含三个部分：训练时间蒸馏允许反事实检查技能，并从失败中学习协议技能;门控超图检索注入优势等级技能，按数据集门准入;执行协议使冻结骨干节点承诺答案。在11项文本、视觉语言和全模态基准测试中，HyperSkill在所有数据集上均领先或并列，平均F1积分比最佳基线高出13.0分，同时保持核心结构冻结。我们的软件和数据在此 https 网址公开。

Towards Robust Vision-Language Reasoning: A Multi-Stage Optimization Pipeline with Masked Diffusion Model

迈向稳健的视觉语言推理：采用掩蔽扩散模型的多阶段优化流水线

• Authors: Shunming Liu, Pei Huang, Yajing Sun, Huaxiao Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.15238
• Pdf link: https://arxiv.org/pdf/2611.15238
• Abstract Discrete diffusion generation is emerging as a competitive alternative to autoregressive generation by denoising the entire token sequence through iterative refinement. Its global planning and progressive correction capabilities are particularly well suited to autonomous driving. However, transferring this generation paradigm from general-purpose settings to driving introduces substantial domain gaps in both training and inference. To unlock its potential, we propose a multi-stage optimization pipeline for standard vision-language model architectures, without auxiliary action experts or specialized mixture-of-experts Transformer, spanning post-training and inference. During supervised fine-tuning, we optimize the representation of planning inputs to suppress hallucinations in task responses. During reinforcement learning, we introduce metric-guided diversity filtering to identify stronger learning starting points and improve policy exploration. At inference time, we propose Dynamic Autonomous Driving Decoding (DAD), which adaptively adjusts decoding modes according to scene complexity. Based on this pipeline, we develop Masked Diffusion Drive (MD-Drive), improving its performance by 29.7\% over the unoptimized model. Experiments on the NAVSIM closed loop benchmark and the nuScenes and DriveCoT open loop datasets show that MD-Drive outperforms autoregressive VLM baselines with the same architecture, while achieving a more favorable balance between performance and computational cost than systems with different architectures through discrete diffusion training and inference.
• 中文摘要 离散扩散生成正作为自回归生成的竞争替代方案崭露头角，通过迭代细化去噪整个令牌序列。其全局规划和渐进修正能力特别适合自动驾驶。然而，将这一代范式从通用环境转移到驾驶，会在训练和推理上带来显著的领域差距。为释放其潜力，我们提出了一个多阶段的标准视觉语言模型架构优化流水线，无需辅助动作专家或专业专家混合的Transformer，涵盖训练后和推理阶段。在监督微调过程中，我们优化规划输入的表示，以抑制任务响应中的幻觉。在强化学习过程中，我们引入度量导向多样性过滤，以识别更强的学习起点并提升策略探索。在推断阶段，我们提出了动态自主驱动解码（DAD），该方法根据场景复杂度自适应调整解码模式。基于该流水线，我们开发了掩蔽扩散驱动（MD-Drive），其性能比未优化模型提升29.7%。在NAVSIM闭环基准测试以及nuScenes和DriveCoT开环数据集上的实验显示，MD-Drive在相同架构下优于自回归VLM基线，并通过离散扩散训练和推断实现了性能与计算成本之间的更优平衡，优于不同架构的系统。

Dual Representation Enhanced Diffusion Policy for Flexible Job Shop Scheduling

双重代表增强扩散政策，实现灵活的工坊排班

• Authors: Yunyi Zhao, Wei Zhang, Cheng Xiang, Vadakkepat Prahlad, Zuming Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.15246
• Pdf link: https://arxiv.org/pdf/2611.15246
• Abstract The Flexible Job-Shop Scheduling Problem (FJSP) is a fundamental and challenging optimization problem in smart manufacturing. Although recent deep reinforcement learning (DRL) approaches can learn effective scheduling policies through interaction with the environment, their performance often degrades as problem scale and processing-time uncertainty increase. In this paper, we propose DredRL, a dual representation enhanced diffusion-policy reinforcement learning scheduler. DredRL first constructs enhanced dual representations of operations and machines through self- and cross-attention mechanisms to capture both intra-entity and inter-entity dependencies. A diffusion model is further employed as the policy network, generating scheduling decisions through iterative denoising conditioned on the encoded state to enhance policy expressiveness and robustness. Extensive experiments on synthetic, public, large-scale, and stochastic FJSP benchmarks demonstrate that DredRL outperforms state-of-the-art learning-based schedulers across most configurations. Compared with SOTA learning-based methods, DredRL reduces makespan by up to 5.44% on synthetic instances. It also achieves the best learning-based performance on three of four public benchmarks and outperforms the stochasticity-aware algorithms on stochastic benchmarks. These results demonstrate that combining enhanced dual representations with diffusion-based policies provides an effective and robust approach for large scale scalable FJSP scheduling in complicated environment.
• 中文摘要 灵活工坊调度问题（FJSP）是智能制造中一个基础性且具有挑战性的优化问题。尽管最新的深度强化学习（DRL）方法可以通过与环境交互学习有效的调度策略，但随着问题规模和处理时间不确定性的增加，其性能常常下降。本文提出了 DredRL，一种双重表示增强的扩散策略强化学习调度器。DredRL 首先通过自关注和交叉注意机制构建了操作和机器的增强双重表示，以捕捉实体内和实体间的依赖关系。进一步采用扩散模型作为策略网络，通过基于编码状态的迭代去噪生成调度决策，以增强策略表达性和鲁棒性。在合成、公共、大规模和随机FJSP基准测试上的广泛实验表明，DredRL在大多数配置中优于最先进的基于学习的调度器。与SOTA学习方法相比，DredRL在合成实例上可将剩余时长缩短多达5.44%。它还在四个公开基准测试中实现了三个最佳基于学习的表现，并在随机基准测试中优于随机感知算法。这些结果表明，将增强的对偶表示与基于扩散的策略结合，为复杂环境中大规模可扩展的FJSP调度提供了有效且稳健的方法。

Knowledge in the Harness, Reliability in the Weights: Teacher-Free Co-Evolution of an LLM Agent

知识在约束带，权重可靠性：无教师的LLM代理共进化

• Authors: Yujia Zhang, Xinqi Ren, Shixiong Zhou, Jia Meng
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.15268
• Pdf link: https://arxiv.org/pdf/2611.15268
• Abstract Small open models suit private customer-service deployments but trail frontier models in two ways: they miss what the written policy requires in a given situation, and they act inconsistently on what they know. We study whether such a model can close both gaps from its own deployment logs and their reference outcomes, without a stronger teacher model. Our starting point is a Gemma 4 12B agent that reads its domain policy from on-demand skills rather than from its system prompt. In each round, the deployed system serves a fresh batch of tasks; the base model diagnoses failed conversations and writes rules into the skills, and a rule survives only if it passes code checks, counterexample self-review, a paired retest, and a noise-calibrated validation gate. Reinforcement learning (GRPO) then trains a LoRA adapter on complete multi-turn, tool-using conversations. On the retail and airline domains of $\tau^2$-bench, without training or selecting on official tasks, the agent after two rounds improves on this starting point by 12.5 pass$^1$ points in retail and 19.5 in airline, and by 18.4 and 22.0 pass$^4$ points. It also exceeds the benchmark's reference agent, which runs the same model with the whole policy in its system prompt, by 11.8 and 8.0 pass$^1$ points and by 22.8 and 16.0 pass$^4$ points, closing most of the gap to three frontier models under the same protocol. A $2\times2$ ablation shows that the rules rescue tasks the agent never solved and carry policy knowledge that RL with the rules in context does not internalize, whereas the trained weights make solved tasks reliable.
• 中文摘要 小型开放模型适合私有客户服务部署，但在前沿模型方面有两个方面：它们错过了特定情境下书面政策的要求，并且基于已知内容的行为不一致。我们研究此类模型是否能在没有更强教师模型的情况下，通过自身部署日志和参考结果弥补两者之间的差距。我们的起点是一个Gemma 4 12B代理，它通过按需技能而非系统提示读取其领域策略。每轮部署的系统执行一批新的任务;基础模型诊断失败的对话并将规则写入技能，规则只有通过代码检查、反例自我审查、配对重测和噪声校准验证门后才能存活。强化学习（GRPO）随后训练LoRA适配器完成多回合工具使用对话。在零售和航空公司领域$\tau^2$-bench上，未经训练或选择官方任务，代理在两轮后在零售领域提升了12.5个通过点^1$，在航空领域提升了19.5个通过点，在22.0通过点方面提升了18.4和22.0个通过点^4$。它还比基准的参考代理多出11.8和8.0的pass$^1$点，比22.8和16.0通过$^4$的分数高出，缩小了与三个前沿模型在同一协议下大部分差距。$2\times2$的消融显示，规则能够挽救代理从未解决过的任务，并携带了带有上下文规则的强化学习不会内化的策略知识，而训练出的权重使已解决任务更可靠。

APG-Planner: Learning Vision-Based High-Speed Trajectory Planning via Analytic Policy Gradient

APG-规划器：通过分析政策梯度学习基于愿景的高速轨迹规划

• Authors: Jinwoo Lee, Geunsik Bae, Changseung Kim, Jiwoo Kim, Woojae Shin, Hyondong Oh
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.15296
• Pdf link: https://arxiv.org/pdf/2611.15296
• Abstract Vision-based high-speed navigation requires a trajectory planner that can reactively generate smooth, collision-free trajectories from onboard visual observations. Deep reinforcement learning offers a promising approach to vision-based trajectory planning, as it enables the optimization of sequential decision-making policies in complex environments. However, model-free policy gradient methods rely on sample-based policy gradient estimation, which yields noisy gradient estimates and inefficient training. To this end, we propose a framework for learning trajectory planning policies using the analytic policy gradient (APG) method and use it to train APG-Planner, a vision-based high-speed trajectory planner. The proposed framework leverages differential flatness to analytically reconstruct vehicle states from planned trajectories, thereby enabling differentiable trajectory-level state transitions without requiring an explicit low-level dynamics model. By iteratively performing trajectory planning and differentiable state transitions, we construct a differentiable episode rollout and analytically compute policy gradients from episode-level losses, improving both training efficiency and policy performance. After approximately 30 minutes of training on a single GPU, APG-Planner demonstrates comparable or superior performance to existing baseline methods in simulation and achieves speeds of up to 11\,m/s in real-world flight experiments. We further validate the proposed framework on a skid-steer ground robot, demonstrating its applicability to differentially flat robotic systems beyond quadrotors.
• 中文摘要 基于视觉的高速导航需要一种能够从机载视觉观测中被动生成平滑、无碰撞轨迹的轨迹规划器。深度强化学习为基于视觉的轨迹规划提供了有前景的方法，因为它能够在复杂环境中优化顺序决策策略。然而，无模型策略梯度方法依赖基于样本的策略梯度估计，导致波噪声的梯度估计和低效的训练。为此，我们提出了一个利用分析策略梯度（APG）方法学习轨迹规划策略的框架，并用它来训练基于视觉的高速轨迹规划器APG-Planner。该框架利用差分平坦性从计划轨迹中解析重建车辆状态，从而实现可微的轨迹级状态转变，而无需显式的低层动力学模型。通过迭代执行轨迹规划和可微态转换，我们构建了可微分的事件展开，并从事件层级损失分析计算策略梯度，提升训练效率和策略性能。经过约30分钟的单GPU训练后，APG-Planner在模拟中展现出与现有基线方法相当甚至更优的性能，并在真实飞行实验中实现高达11\，m/s的速度。我们进一步验证了该框架在滑移转向地面机器人上的应用，证明其适用于四旋翼以外的差分平面机器人系统。

Towards Better Legal Reasoning LLMs: Signal Balancing and Reward Scheduling

迈向更优法律推理LLM：信号平衡与奖励调度

• Authors: Yinghao Hu, Qian Xiao, Yuting Huang, Kaisong Song, Yiquan Wu, Ang Li, Junjie Cao, Jun Lin, Kun Kuang, Fei Wu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.15346
• Pdf link: https://arxiv.org/pdf/2611.15346
• Abstract In the legal domain, LLMs can improve reliability and interpretability by generating explicit reasoning traces. However, training LLMs to produce high-quality reasoning traces remains challenging. Existing supervised fine-tuning methods struggle when reasoning traces are significantly longer than final answers, as the learning signal for the answer becomes diluted. Meanwhile, reinforcement learning methods such as Group Relative Policy Optimization also have several limitations, including costly reward design, performance plateaus, and reward hacking. We therefore propose a two-stage training framework. In Stage I, JurisCoT-SFT employs a length-normalized dual objective to balance learning signals between reasoning traces and final answers. In Stage II, Lifecycle-Aware Backtrackable Policy Optimization dynamically evaluates and retires auxiliary reward signals based on their impact on final evaluation objectives, enabling efficient reward utilization. Our framework substantially improves legal reasoning performance on relevant tasks selected from multiple legal benchmarks, including ChJGBench, LawBench, and LegalBench. Experimental results demonstrate consistent improvements in both reasoning trace quality and final answer accuracy, while our models remain competitive with larger models.
• 中文摘要 在法律领域，LLMs通过生成显式推理痕迹可以提升可靠性和可解释性。然而，训练LLM产生高质量推理轨迹仍具挑战。现有监督式微调方法在推理轨迹明显长于最终答案时会遇到困难，因为答案的学习信号会被稀释。与此同时，强化学习方法如群体相对策略优化也存在若干局限性，包括高昂的奖励设计、性能平台期和奖励黑客。因此，我们提出了一个两阶段的训练框架。在第一阶段，JurisCoT-SFT采用长度归一化的对偶目标，以平衡推理痕迹与最终答案之间的学习信号。在第二阶段，生命周期感知可回溯策略优化根据辅助奖励信号对最终评估目标的影响动态评估并退役它们，从而实现高效的奖励利用。我们的框架显著提升了从多个法律基准中选出的相关任务的法律推理性能，包括 ChJGBench、LawBench 和 LegalBench。实验结果显示，推理追踪质量和最终答案准确性均有持续提升，同时我们的模型仍能与更大型模型竞争。

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

对于LLMs中的稳定强化学习来说，重要性比是必要的吗？

• Authors: Shuibai Zhang, Junhyuck Kim, Gyeongman Kim, Jaewoong Cho
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.15404
• Pdf link: https://arxiv.org/pdf/2611.15404
• Abstract Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
• 中文摘要 强化学习（RL）已成为大型语言模型（LLMs）训练后的核心。然而，像GRPO这样的流行RL方法通过计算旧策略和当前策略似然来形成重要性抽样比，会产生不可忽视的开销。在本研究中，我们提出了似然门控策略优化（LGPO），通过似然门控强制软信任区域约束，免去计算旧策略似然的必要。通过实证，我们表明移除重要性抽样修正项不会损害训练稳定性，而移除信任区域机制可能导致崩溃。此外，基于比率的裁切在完全策略训练中可能失败：重要性比保持在1，因此基于比率的信任区域约束永远不会激活。在GRPO稳定的标准训练环境中，LGPO在训练稳定性和峰值性能上可达相当，且相较于重新计算旧策略似然的标准实现，平均减少约18%的训练时间。在完全策略训练中，GRPO失败，LGPO保持稳定，使跨训练体系的LLM强化学习更高效、更稳健。

SimRec: Learning to Search, Ask, and Recommend from Simulated Interaction

SimRec：学习从模拟交互中搜索、询问和推荐

• Authors: Yaxin Zhang, Xingyu Wu, Yuchen Yan, Yuqin Dai, Hong Zhao, Lixing Shen, Zhan Chen, Jian Shao, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.15426
• Pdf link: https://arxiv.org/pdf/2611.15426
• Abstract LLM-based recommendation agents can search catalogs, inspect products, ask users, and revise rejected recommendations. However, these behaviors are often governed by hand-designed workflows, while existing training methods optimize isolated components such as query rewriting or list ranking rather than the interaction policy as a whole. We present SimRec, a reinforcement learning framework that trains the full interaction policy of a tool-integrated recommendation agent. SimRec introduces two components. A task-anchored user simulator provides feedback on every recommendation: conditioned on the current task need rather than on user profile, and it can be an external LLM or the recommender itself under self-play. A task-aligned reward optimizes the whole trajectory with GRPO, crediting natural retrieval quality and valid tool use alongside the final hit, with training-only target injection to handle sparse hits over large catalogs. On ESCI and Amazon C4, SimRec improves the NDCG@100 of a Qwen3-4B agent from 32.23% to 35.42% and from 22.94% to 24.72%}, outperforming training-free agents on far larger backbones and task-aligned baselines on NDCG. Analysis shows that the trained agent verifies candidates, uses preferences, and recovers from rejected recommendations far more often than its untrained counterpart. Our code is available at this https URL .
• 中文摘要 基于LLM的推荐代理可以搜索目录、检查产品、询问用户并修改被拒绝的推荐。然而，这些行为通常由手工设计的工作流程控制，而现有训练方法则优化了查询重写或列表排名等孤立组件，而非整体交互策略。我们介绍SimRec，一个强化学习框架，训练工具集成推荐代理的完整交互策略。SimRec引入了两个组件。任务锚定的用户模拟器对每个推荐提供反馈：基于当前任务需求而非用户配置文件，它可以是外部LLM或推荐者自身，处于自我游玩状态。任务对齐奖励通过GRPO优化整个轨迹，将自然检索质量和有效工具使用与最终命中并列，仅训练目标注入以处理大型目录中的稀疏命中。在ESCI和Amazon C4上，SimRec将Qwen3-4B代理的NDCG@100提升率从32.23%提升到35.42%，从22.94%提升到24.72%，在NDCG上表现优于无训练代理在更大主干网和任务对齐基线上。分析显示，训练后的代理验证候选人、使用偏好，并从被拒绝推荐中恢复的频率远高于未受训练的代理。我们的代码可在此 https 网址获取。

DeALR: Decoupled Audio-Latent Optimization with Acoustic Backtracking for Test-Time Reasoning

DeALR：带声学回溯的解耦音频潜在优化，用于测试时间推理

• Authors: Haoyu He, Yan Rong, Li Liu
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2611.15439
• Pdf link: https://arxiv.org/pdf/2611.15439
• Abstract Large audio-language models (LALMs) typically encode audio once and conduct subsequent multi-step reasoning in a language-dominated space, weakening access to fine-grained acoustic evidence. Although recent multimodal approaches have explored re-engaging perceptual inputs during reasoning, whether a frozen LALM can actively revisit crucial acoustic evidence in the latent space at test time remains underexplored. To bridge this gap, we introduce Decoupled Audio Latent Reasoning (DeALR), a novel test-time latent reasoning framework that shifts the paradigm from text-dominated deliberation to active audio-latent interaction. Inspired by the human cognitive process of "backtracking" to specific acoustic cues during complex listening tasks, DeALR establishes a dynamic interaction between learnable thought tokens and acoustic segments. Specifically, we introduce Span-Level Acoustic Backtracking, which traces initial attention distributions to dynamically isolate contiguous, key acoustic spans. Furthermore, we propose Decoupled Audio-Latent Optimization (DALO), a test-time reinforcement learning algorithm that jointly refines thought vectors and acoustic patches via confidence-guided policy gradients. Crucially, DALO explicitly decouples trajectory search from response generation by reinstating uncorrupted, high-fidelity audio embeddings prior to text synthesis, entirely eliminating optimization-induced distribution drift while retaining the navigation of the refined thoughts. Comprehensive experiments on benchmarks demonstrate that DeALR consistently outperforms prior test-time optimization approaches, delivering substantial improvements in both confidence calibration and reasoning accuracy.
• 中文摘要 大型音频语言模型（LALMs）通常只编码一次音频，随后在语言主导的空间中进行多步推理，削弱了对细粒度声学证据的获取。尽管近期多模态方法探索了推理时重新激活感知输入，但冻结的LALM是否能在测试时主动重访潜在空间中的关键声学证据，仍缺乏探讨。为弥合这一空白，我们引入了解耦音频潜在推理（DeALR），一种新型测试时间潜在推理框架，将范式从文本主导的审议转向主动音频潜在互动。DeALR受人类在复杂听觉任务中“回溯”特定声学线索的认知过程启发，建立了可学习思维符号与声学片段之间的动态互动。具体来说，我们引入了跨度级声学回溯，通过追踪初始注意力分布动态隔离连续的关键声学跨度。此外，我们提出了解耦音频潜在优化（DALO），这是一种测试时强化学习算法，通过置信导向策略梯度共同细化思维矢量和声学补丁。关键是，DALO通过在文本合成前恢复未损坏的高保真音频嵌入，明确将轨迹搜索与响应生成解耦，完全消除优化引起的分布漂移，同时保持精细思维的导航。基准测试的综合实验表明，DeALR持续优于以往测试时优化方法，在置信度校准和推理准确性上均有显著提升。

What Makes a VLA Reactive? A Study of Execution Strategies for Dynamic Manipulation

什么使VLA具有反应性？动态操作执行策略研究

• Authors: Zhuoyue Li, Zhendong Han, Pankhuri Vanjani, Wenting Li, Rudolf Lioutikov
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.15491
• Pdf link: https://arxiv.org/pdf/2611.15491
• Abstract Vision-language-action (VLA) models predict chunks of future actions, and when objects move, those chunks go stale before they are executed. We study what makes a pretrained VLA reactive, using \pihalf{} on two dynamic simulation benchmarks and three real-robot tasks. We vary the prediction and execution horizons, compare synchronous and asynchronous inference, test methods that deliver actions sooner or keep successive chunks continuous, and adapt the policy with motion cues and latent-space reinforcement learning. The best prediction horizon is about one second, and the best execution horizon depends on the checkpoint. Short horizons help more with asynchronous execution, and methods that keep the trajectory continuous outperform those that only reduce latency. Adapting the policy repairs targeted failures, but on the conveyor task changing the execution horizon gained as much. Reactivity depends on the execution pipeline as much as on the policy, so both should be evaluated together.
• 中文摘要 视觉-语言-动作（VLA）模型预测未来行动的片段，当物体移动时，这些片段在执行前就变得迟钝。我们研究了预训练VLA为何具有反应性，使用\pihalf{}在两个动态模拟基准和三个真实机器人任务中进行分析。我们调整预测和执行视野，比较同步与异步推理，测试更快交付动作或保持连续分块的方法，并通过运动提示和潜空间强化学习调整策略。最佳预测视野约为一秒，最佳执行视野取决于检查点。短视野对异步执行更有帮助，保持轨迹连续的方法优于仅降低延迟的方法。调整策略修复针对性失败，但在传送带任务中改变执行视野的效果同样显著。反应性既依赖于执行流程，也取决于策略，因此两者应同时评估。

The Illusion of Diversity: Aligning LLM Exploration via Effective Entropy

多样性的幻象：通过有效熵对齐LLM探索

• Authors: Xiaoliang Fu, Jiaye Lin, Yangyi Fang, Cong Qin, Binbin Zheng, Chaowen Hu, Zekai Shao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.15493
• Pdf link: https://arxiv.org/pdf/2611.15493
• Abstract Reinforcement learning with verifiable rewards has shifted the alignment paradigm of large language models from subjective preference to objective correctness, yet a fundamental disconnect persists between exploration diversity and reasoning validity. Existing strategies predominantly rely on "blind" entropy metrics, e.g., token-level or global sequence-level entropy, which structurally fail to distinguish between effective reasoning and potential hallucinations. This misalignment creates an illusion of diversity, where high entropy tends to signal chaotic degeneration rather than valuable exploration. To bridge this gap, we introduce Effective Entropy, a validity-aware entropy concept that quantifies diversity exclusively within the valid subspace and systematically filters out the noise of incorrect trajectories. Built on this concept, we instantiate VEPO, a simple Validity-aware Entropy Policy Optimization method that directly incorporates effective entropy into RLVR training. VEPO induces an adaptive attraction dynamic that drives up the probability of under-explored valid trajectories, a mechanism that matches the practical regime of large-scale autoregressive models where sequence probabilities remain extremely small. Experiments on six mathematical reasoning benchmarks with 1.5B and 7B DeepSeek-R1-Distill-Qwen models show consistent aggregate accuracy gains over GRPO and competitive repeated-sampling success. A direct evaluation with matched correct counts finds more distinct strategies on commonly solved MATH500 problems, supported by paired statistical tests and three-seed training comparisons.
• 中文摘要 带有可验证奖励的强化学习已将大型语言模型的对齐范式从主观偏好转向客观正确，但探索多样性与推理效度之间仍存在根本脱节。现有策略主要依赖“盲”熵指标，例如代币级或全局序列级熵，这些熵结构上无法区分有效推理与潜在幻觉。这种错位造成多样性错觉，高熵往往预示混乱退化，而非有价值的探索。为弥合这一差距，我们引入有效熵，一种有效性意识熵概念，仅量化有效子空间内的多样性，并系统性过滤错误轨迹的噪声。基于这一概念，我们实现了VEPO，一种简单的有效度感知熵策略优化方法，直接将有效熵纳入RLVR训练。VEPO诱导自适应吸引力动态，提高未被充分探索有效轨迹的概率，这一机制符合大型自回归模型中序列概率极低的实际机制。在六个数学推理基准测试中，使用1.5B和7B的DeepSeek-R1-Distill-Qwen模型，显示总量准确率相较GRPO和竞争性重复抽样成功率持续提升。直接评估匹配正确计数，发现在常见MATH500问题上出现更明显的策略，并辅以配对统计检验和三种子训练比较。

ReShap: Interpretable Reinforcement Learning for High-Dimensional Multi-Action Settings

ReShap：适用于高维多行动场景的可解释强化学习

• Authors: Yiyu Qian, Pengfei Guo, Liyuan Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.15507
• Pdf link: https://arxiv.org/pdf/2611.15507
• Abstract Deep Reinforcement Learning (RL) plays a crucial role in tackling complex tasks, yet its reliance on deep neural networks makes the learned policies opaque. To enhance the trustworthiness of RL policies, existing work (e.g., the SILVER framework [Li et al., 2025]) has focused on Shapley-based interpretability, but is typically restricted to low-dimensional state spaces, binary-action settings, and a single class of linear surrogate. We propose ReShap, a policy-distillation framework that extends Shapley-based interpretability to high-dimensional, multi-action environments. ReShap extracts compact feature representations from raw image observations, performs SHAP-based feature attribution to characterize how each feature contributes to the policy's decisions, and distills the deep RL policy into an interpretable surrogate trained on the full set of (state, policy-action) pairs along the policy's visited distribution. We instantiate the surrogate with three interpretable model families (decision trees, linear regression, and logistic regression) and compare their fidelity and task return. We evaluate ReShap on Atari and Box2D environments using three deep RL algorithms and conduct a human-subject study to assess the clarity and trustworthiness of the resulting policies. Results show that ReShap preserves competitive task performance, that decision trees generally outperform linear and logistic regression, and that linear regression is weakest in most settings.
• 中文摘要 深度强化学习（RL）在处理复杂任务中起着关键作用，但其对深度神经网络的依赖使得所学策略变得不透明。为增强强化策略的可信度，现有工作（如SILVER框架[Li等，2025]）专注于基于Shapley的可解释性，但通常局限于低维状态空间、二元动作设置和单一类线性代理。我们提出了ReShap，一种策略提炼框架，将基于Shapley的可解释性扩展到高维多动作环境。ReShap从原始图像观测中提取紧凑的特征表示，进行基于SHAP的特征归因以描述每个特征如何对策略决策，并将深度强化学习策略，训练于策略访问分布上的全部（状态、策略-动作）对。我们用三个可解释的模型族（决策树、线性回归和逻辑回归）实例化该代理，并比较它们的保真度和任务回报。我们使用三种深度强化学习算法在Atari和Box2D环境中评估ReShap，并进行人受试者研究以评估策略的清晰度和可信度。结果显示，ReShap保持了竞争性任务表现，决策树通常优于线性和逻辑回归，且线性回归在大多数环境中表现最弱。

DecArena: Evaluating Large-Scale Decentralized Multi-Agent Collaboration

DecArena：评估大规模去中心化多智能体协作

• Authors: Jiali Cheng, Oleg Rokhlenko, Joo-Kyung Kim, Hadi Amiri, Xingzhi Guo
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.15521
• Pdf link: https://arxiv.org/pdf/2611.15521
• Abstract Emerging personal AI agents raise fundamental scalability and privacy concerns for centralized multi-agent systems (MASs), which rely on a global orchestrator with broad access to agents, resources, and communication. In decentralized environments, global visibility is unavailable. Agents need to proactively discover non-local resources (e.g. tools, skills), and collaborate with other agents to solve tasks. We study decentralized multi-agent collaboration under graph-constrained communication, a natural reflection of human structures. Each agent observes only its local state and can interact only with neighboring agents. We introduce a benchmark with synthetic scalable distributed tool calling with 100K+ tools. Our experiments reveal that recent LLM agents struggle to discover non-local relevant resources and collaborate in decentralized setting. Moreover, standard reinforcement learning fails to reliably overcome this limitation, suggesting that decentralized MAS needs more explicit harness mechanisms for effective collaboration, even using SOTA LLMs. We then present a strong baseline \textbf{GraphHarness}, combining topology-aware warmup with self-improving LLM-based routing. Results show that GraphHarness substantially improves task success (+48.5) and produces higher quality collaboration (+30.6), while using fewer costs (+34.0) than baselines.
• 中文摘要 新兴的个人人工智能代理为中心化多代理系统（MAS）带来了根本性的可扩展性和隐私问题，这些系统依赖于全球编排器，广泛访问代理、资源和通信。在去中心化环境中，全球可见性不可用。代理需要主动发现非本地资源（如工具、技能），并与其他代理协作解决任务。我们研究了图约束通信下的去中心化多代理协作，这是一种人类结构的自然反映。每个代理仅观察其本地状态，只能与邻近代理交互。我们引入了基于100K+工具的合成可扩展分布式工具调用基准测试。我们的实验显示，近年来的大型语言模型代理在去中心化环境中难以发现非本地相关资源和协作。此外，标准强化学习未能可靠克服这一局限，表明去中心化MAS需要更明确的赋能机制以实现有效协作，即使使用SOTA LLMs也是如此。随后，我们提出了一个强有力的基线 \textbf{GraphHarness}，结合了拓扑感知的预热与自我改进的基于LLM的路由。结果显示，GraphHarness显著提升任务成功率（+48.5），并产生更高质量的协作（+30.6），同时成本（+34.0）低于基线。

Reward-Driven Collider Physics Challenges for Self-Improving Agentic AI via Reinforcement Learning

奖励驱动的碰撞器物理挑战，通过强化学习实现自我改进的代理人工智能

• Authors: Alexander Yue, Qibin Liu, Julia Gonski
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.15885
• Pdf link: https://arxiv.org/pdf/2611.15885
• Abstract Recent advances in agentic AI have demonstrated the potential to automate complex high-energy physics (HEP) workflows. We focus on the regime of HEP tasks where frontier models still struggle to provide correct and complete solutions, and introduce an open-ended optimization task format with continuous, automatically evaluated rewards. Unlike conventional benchmarks focused on task completion, these tasks target reinforcement learning (RL) environments to generate agentic improvement on state- of-the-art HEP challenges. We benchmark frontier reasoning models on several HEP tasks and demonstrate an initial proof of concept in which a small language model improves through RL using automated feedback without human intervention. By connecting challenging HEP problems with verifiable scientific rewards and RL, we aim to advance agentic AI from reproducing established workflows to autonomously discovering better scientific solutions, paving the way toward self-improving scientific agents.
• 中文摘要 代理人工智能的最新进展展示了自动化复杂高能物理（HEP）工作流程的潜力。我们聚焦于前沿模型仍难以提供正确完整解决方案的HEP任务体系，并引入了开放式优化任务格式，并配备持续自动评估的奖励。与专注于任务完成的传统基准不同，这些任务针对强化学习（RL）环境，以在最先进的HEP挑战中实现代理改进。我们在多个HEP任务上基准测试前沿推理模型，并展示了一个初步概念验证，展示一个小型语言模型通过强化学习通过自动反馈改进，无需人工干预。通过将具有挑战性的HEP问题与可验证的科学奖励和强化学习连接起来，我们旨在推动代理人工智能从复制既有工作流程向自主发现更优科学解决方案，铺平自我改进科学代理的道路。

Your Hypergradient is Skewed: Antithetic Neumann Estimation for Bilevel Optimization

你的超梯度偏斜：双层优化中的反诺依曼估计

• Authors: Jason Bohne, Pawel Polak, Gary Kazantsev, David Rosenberg
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2611.15895
• Pdf link: https://arxiv.org/pdf/2611.15895
• Abstract Bilevel optimization is a core tool in practical machine learning workflows, including hyperparameter tuning, data reweighting, robust learning, reinforcement learning, and imbalanced classification. Hypergradient computation requires the inverse of the inner Hessian, typically estimated via a truncated Neumann series. While existing analyses focus on bias and variance, we show that the product structure of the Neumann series introduces a third source of error: skewness. We identify that, unlike ordinary gradients, whose asymmetry is suppressed by mini-batch averaging, the Neumann estimator amplifies skewness across factors, with the third-moment error growing, under weak contraction, with truncation depth, inner condition number, and per-factor noise asymmetry. This arises in many practical settings such as corrupted labels, outliers, heterogeneous or class-imbalanced datasets. To address this, we introduce the Antithetic Neumann Estimator (ANE), which pairs each Hessian sample with a mirrored counterpart around a reference Hessian and averages the resulting inverse-Hessian estimates. With an exact reference, the antithetic product retains only even-order noise, eliminating all odd-order perturbation terms; with an approximate reference, the residual odd-order contribution is bounded linearly in the reference error. ANE requires only a modification of the Hessian vector step and combines with existing stochastic or variance-reduced bilevel methods. We instantiate ANE-SGD and ANE-VR, the latter adding recursive variance reduction. Across robust regression, data cleaning, offline reinforcement learning, and imbalanced classification, ANE-based methods improve on stochastic and variance-reduced baselines on every task (20 seeds each), lowering loss by 4 to 25% relative to the best baseline; the largest gains, on robust regression, are a 25% lower test loss and a 54% higher test r-squared.
• 中文摘要 双层优化是实际机器学习工作流中的核心工具，包括超参数调优、数据重加权、稳健学习、强化学习和不平衡分类。超梯度计算需要内黑森矩阵的逆，通常通过截断诺依曼级数估计。现有分析主要关注偏置和方差，我们表明诺依曼级数的乘积结构引入了第三个误差源：偏态。我们发现，与普通梯度通过微批量平均抑制不对称不同，诺依曼估计器放大了因子间的偏态，在弱收缩下，三度误差随着截断深度、内条件数和每因子噪声不对称而增加。这在许多实际环境中都会出现，如标签损坏、离群值、异质或类不平衡数据集。为此，我们引入了反诺依曼估计量（Antithetic Neumann Estimation，ANE），它将每个黑森样本与参考黑森样本周围的镜像对应样本配对，并对所得的逆黑森估计值进行平均。对于精确参考，反积仅保留偶数阶噪声，消除所有奇数阶扰动项;有了近似参考，剩余的奇数阶贡献在参考误差中线性有界。ANE只需修改黑森向量步骤，并与现有的随机或方差约简双水平方法结合。我们实例化ANE-SGD和ANE-VR，后者增加了递归方差缩减。在稳健回归、数据清理、离线强化学习和不平衡分类中，基于ANE的方法在每个任务（各20个种子）上均优于随机基线和方差减少基线，相较最佳基线降低4%至25%;在稳健回归中，最大收益为测试损失降低25%，检验r平方提高54%。

Reinforcement Learning for Inventory Control under Regime-Switching Demand: A Controlled Comparison of LSTM, MLP, and (s,S) Policies

在体制切换需求下库存控制的强化学习：LSTM、MLP及（s，S）政策的受控比较

• Authors: Alex Haines, Nathanael L. Baisa
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.15945
• Pdf link: https://arxiv.org/pdf/2611.15945
• Abstract Classical inventory policies such as (s,S) are provably optimal under stationary demand but cannot adapt when demand switches between hidden regimes, and memoryless deep reinforcement learning policies, confined to a fixed observation window, cannot maintain a belief over latent demand structure. We ask whether a recurrent policy offers a measurable advantage over both a memoryless policy and a classical baseline under regime-switching demand, and how that advantage scales with supply-chain lead time. We build a custom Gymnasium environment in which daily demand is generated by a two-state hidden Markov process with Poisson emissions, so that the regime is known at evaluation time but hidden from the agent. An optimised (s,S) heuristic, an MLP PPO agent, and an LSTM Recurrent PPO agent are trained under matched compute and evaluated at lead times of 0, 3, 6, and 10 days over 30 deterministic episodes each. The recurrent policy outperforms both alternatives at every lead time, and its margin over the MLP widens monotonically with lead time, from parity at lead time 0 to 2.17$\times$ at lead time 10. Principal component analysis of the LSTM hidden state explains the widening gap: the representation shifts from encoding a binary regime indicator at short lead times to encoding multi-step pipeline state at long lead times. Recurrent memory becomes the decisive architectural factor as the credit-assignment horizon lengthens.
• 中文摘要 经典库存策略如（s，S）在平稳需求下可证明最优，但无法适应需求在隐藏状态间切换时，且无记忆深度强化学习策略被限制在固定观察窗口内，无法维持对潜在需求结构的信念。我们探讨循环策略在体制切换需求下是否相较无记忆策略和经典基线具有可测量优势，以及这种优势如何随供应链交期扩展。我们构建了一个定制的体育馆环境，其中每日需求由带有泊松排放的双态隐马尔可夫过程生成，使得该状态在评估时已知，但对智能体隐藏。在匹配计算下训练了一个优化（s，S）启发式、一个MLP PPO智能体和一个LSTM重复PPO智能体，并在30次确定性事件中分别在0、3、6和10天的前置时间进行评估。循环策略在每次交期内都优于两种方案，且其对MLP的利润率随着交期单调扩大，从交期0的平等到交期10的2.17美元/时间$。对LSTM隐藏状态的主成分分析解释了差距扩大：表示方式从在短交期编码二进制状态转向在长交期编码多步流水线状态。随着信用分配时间的延长，循环记忆成为决定性的架构因素。

Wayfinder: Adaptive Resource Routing from Agent Citations

Wayfinder：从代理引用中实现自适应资源路由

• Authors: Miguel Romero, George Karypis
• Subjects: Subjects: Machine Learning (cs.LG); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.16053
• Pdf link: https://arxiv.org/pdf/2611.16053
• Abstract To act effectively, agents must gather contextual information across resources such as chats, issue trackers, email and knowledge bases, to answer questions like "what was decided in the last project meeting?" or "where is the FAQ for our internal tooling?" Where this information lives in a user's environment is unknown to the agent, so it re-explores the same resources across consecutive interactions. Common routing approaches represent each resource as a tool and rely on high-level descriptions to generalize to unseen environments. Such descriptions often fail to separate similar or overlapping resources and cannot improve from the agent's subsequent interactions. We introduce Wayfinder, an adaptive router that uses the documents cited by the downstream agent as feedback about where relevant information resides. These citations update Wayfinder's internal resource representations, which subsequent interactions use to prioritize the resources likely to contain relevant information. We also introduce resource augmentation during offline reinforcement learning, which anonymizes and permutes resource identities to discourage training-time overfitting of which resource holds which topic; our ablations suggest it helps most when few training environments are available. Because no openly available professional environments pair realistic queries with resource-access patterns, we construct and release three controlled testbeds derived from existing datasets: NFCorpus, FEVER and Tech Startup. Compared with supervised, tool-retrieval and contextual bandit baselines under a shared closed-loop protocol, Wayfinder gives the best routing when resources contain semantically overlapping content that static descriptions cannot distinguish, while description-based tool retrieval suffices when resources are drawn from distinct corpora.
• 中文摘要 为了有效行动，代理必须在聊天、问题跟踪、电子邮件和知识库等资源中收集上下文信息，以回答诸如“上次项目会议决定了什么？”或“我们内部工具的常见问题解答在哪里？”等问题。代理不知道这些信息在用户环境中的位置，因此会在连续的交互中重新探索相同的资源。常见的路由方法将每个资源视为工具，并依赖高层描述来泛化到未见环境。此类描述往往无法分离相似或重叠的资源，且无法从代理后续的交互中得到改进。我们引入了Wayfinder，一种自适应路由器，利用下游代理引用的文档作为关于相关信息所在位置的反馈。这些引用更新了Wayfinder的内部资源表示，后续交互用以优先排序可能包含相关信息的资源。我们还在离线强化学习中引入资源增强，匿名化和置换资源身份，以防止训练时间对哪个资源包含哪个主题进行过度拟合;我们的消融表明，当可用的训练环境较少时，这种方法最为有效。由于没有公开的专业环境将现实查询与资源访问模式配对，我们构建并发布了三个基于现有数据集的受控测试平台：NFCorpus、FEVER 和 Tech Startup。与共享闭环协议下的监督式、工具检索和上下文盗用基线相比，Wayfinder 在资源包含语义重叠且静态描述无法区分的内容时提供最佳路由，而基于描述的工具检索则适用于资源从不同语料库提取。

HyperPage: Learned Page Layouts for Disk-Resident Graph Vector Indexes

HyperPage：磁盘驻留图向量索引的学习页面布局

• Authors: Xinyu Zhang, Jingze Li, Baihua Zheng, Weiwei Sun
• Subjects: Subjects: Databases (cs.DB); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.16149
• Pdf link: https://arxiv.org/pdf/2611.16149
• Abstract Disk-resident graph indexes such as DiskANN store each vector with its graph edges as one record on SSD pages. A search needs the page of every record it expands, so its page reads depend on the page layout, that is, which records share a page. Search agents trained with reinforcement learning query one such index run after run, and every graph traversal can be logged with the records it expands. We therefore use the logs of a finished run to lay out the index for a declared memory budget, so later runs read fewer pages. In contrast, existing layouts such as Starling and MARGO are computed from the graph and vectors and use no logs. Learning a layout from logs is challenging, because the logs must stay valid under every layout, the saving depends on what the budget caches, and the best layout is NP-hard to find. Our key observation is that DiskANN-style search routes on in-memory codes and the records it expands, never on their page-mates. As a result, the logged traversals price every layout by replay, and cached records add no page reads. Accordingly, we propose HyperPage, which first builds layouts for the declared budget on the logged traversals of uncached records. It then selects one layout, together with the budget split between hot records and a page cache, by replaying validation steps. In addition, a Lagrangian lower bound shows how far a layout may be from the optimum. We learn from one Search-R1 agent run and test on a later run with a different reward, over 1M and 21M passages. Across seven cache budgets, HyperPage reads 5.2% to 21.2% fewer pages than MARGO, and no compared layout reads fewer. The advantage is largest when the cache is small. For example, at 1% of the 1M index, MARGO needs nearly 3 times the budget to read as few pages. On the 21M index, HyperPage also cuts search time per step by up to 10.4%. The saving holds for a second agent model and a second corpus.
• 中文摘要 磁盘驻留图索引如DiskANN将每个向量及其图边作为SSD页面上的一条记录存储。搜索需要扩展的每条记录的页面，因此其页面的读取取决于页面布局，即哪些记录共享同一页面。通过强化学习训练的搜索代理一次又一次地查询此类索引，每次图遍历都可以记录其扩展的记录。因此，我们使用完成运行的日志来规划声明内存预算的索引，因此后续运行读取的页面更少。相比之下，现有布局如Starling和MARGO是基于图和向量计算的，不使用日志。从日志学习布局具有挑战性，因为日志必须在每个布局下保持有效，节省的程度取决于预算缓存，最佳布局难以找到。我们的关键观察是，DiskANN风格的搜索路径是内存代码及其扩展记录，而非其同页记录。因此，记录的遍历会为每个布局定价回放，缓存记录则不增加页面读取次数。因此，我们提出了HyperPage，它首先根据未缓存记录的记录遍历构建声明预算的布局。然后通过重放验证步骤，选择一个布局，以及热记录和页面缓存之间的预算分配。此外，拉格朗日下限显示布局距离最优值的距离。我们从一次Search-R1代理运行中学习，并在后续一次奖励不同的情况下测试，分别为100万和2100万段。在七个缓存预算中，HyperPage的读数比MARGO少5.2%至21.2%，且无对比布局读数少。当缓存较小时，优势最大。例如，在100万索引的1%下，MARGO需要近3倍预算才能读取同样的页面。在21M索引中，HyperPage还将每步搜索时间缩短了最多10.4%。节省的效果适用于第二个代理模型和第二个语料库。

Fault Injection Attacks on Q-Learning: Analysis, Vulnerabilities, and Countermeasures

Q学习中的故障注入攻击：分析、漏洞与对策

• Authors: Tom Slooff, Anthony Etim, Jiaqi Yu, Francesco Regazzoni, Jakub Szefer
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.16151
• Pdf link: https://arxiv.org/pdf/2611.16151
• Abstract Hardware fault injection attacks threaten machine learning systems, yet their impact on reinforcement learning (RL) has received little attention despite the use of RL in safety-critical applications. This paper presents a security analysis of Q-Learning implementations under voltage and clock glitching attacks on a ChipWhisperer platform, evaluating five variants: Standard, Robust, Resilient, Linear, and Deep Q-Learning. In standard Q-Table implementations, faults corrupt reward computation, action selection, and Q-value updates during learning, and change the selected action during inference. Robust Q-Learning does not prevent these attacks: NaN and infinite values injected during learning persist in the Q-table, and its greedy policy uses the same code as standard Q-Learning. This paper also evaluates a Resilient Q-Learning variant that applies triple modular redundancy with majority voting. It corrects corruption of a single Q-table copy, but inputs shared by the copies, such as the reward, and the voting code itself remain outside its protection. The analysis extends to Linear and Deep Q-Learning, where faults corrupt stored weights during learning and change actions during inference. Moving beyond benchmark environments, this work evaluates an online traffic light controller in a microscopic traffic simulator, where a single injected infinite value degrades accumulated vehicle waiting time by 148x while corrupting under 4% of the Q-table, and where triple modular redundancy corrected the faults injected in our tests, although glitching could still halt the device. Altogether, this work provides a basis for understanding and securing RL deployments against fault injection attacks.
• 中文摘要 硬件故障注入攻击威胁着机器学习系统，尽管强化学习在安全关键应用中被广泛应用，但其对强化学习（RL）的影响却鲜有关注。本文对ChipWhisperer平台上Q-Learning在电压和时钟故障攻击下的安全性分析，评估了五种变体：标准、稳健、韧性、线性和深度Q学习。在标准Q表实现中，错误会破坏奖励计算、动作选择和Q值更新，并在推理过程中改变所选动作。稳健Q-Learning无法防止这些攻击：学习过程中注入的NaN和无限值会在Q表中保留，其贪婪策略使用与标准Q-Learning相同的代码。本文还评估了一种带有多数投票的三重模块冗余的韧性Q-Learning变体。它修复单个Q表副本的损坏，但副本共享的输入，如奖励和投票码本身，仍不受其保护。分析扩展到线性和深度Q学习，其中故障在学习过程中破坏存储权重，在推理中破坏变更动作。超越基准环境，本研究评估了一个微观交通模拟器中的在线交通信号控制器，其中单个注入的无限值可使累计车辆等待时间降低148倍，且损坏Q表不到4%，三重模块冗余修复了测试中注入的故障，尽管故障仍可能使设备停机。总体而言，这项工作为理解和保障强化学习部署免受故障注入攻击提供了基础。

Turning Multi-Agent Communication into Token-Level Credit with On Policy Self Distillation

通过政策自蒸馏将多代理通信转化为令牌级信用

• Authors: Zixu Wang, Yu Zhao, Xing Zheng, Bingbing Xu, Tian Xueyun, JiaTang Luo, Weiqiang Wang, Bowen Song, Huawei Shen, Xueqi Cheng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.16206
• Pdf link: https://arxiv.org/pdf/2611.16206
• Abstract Reinforcement learning improves problem solving in large language model multi-agent systems. Existing approaches overlook communication knowledge or compress it into scalar rewards, leaving coarse credit with little guidance on which decisions each agent should reinforce or revise. OPSD4MAS converts future communication into fine-grained token-level credit through On Policy Self Distillation (OPSD). An external model extracts facts and assesses their trajectory relations for coarse-grained filtering, followed by fine-grained leave-one-out filtering of target-token influence against agent-wise outcome advantages. The fact-conditioned policy supplies token-level credit, combined with GRPO through a linearly annealed joint objective. Our method improves mathematical pass@4 over GRPO by up to 20.00 percentage points. Ablations support complete filtering, while frozen-history injection diagnostics support fact utility. Together, these results support combining agent-level outcome guidance with communication-derived token-level credit. Code is available at this https URL.
• 中文摘要 强化学习提升了大型语言模型多智能体系统中的问题解决能力。现有方法忽视了沟通知识或将其压缩为标量奖励，导致粗略的信用对每个代理应强化或修订哪些决策缺乏指导。OPSD4MAS通过策略自蒸馏（OPSD）将未来的沟通转化为细粒度的代币级信用。外部模型提取事实并评估其轨迹关系，进行粗粒度过滤，随后进行细粒度的留一过滤，针对目标代币影响与代理间结果优势进行筛选。事实条件策略提供代币级信用，结合线性退火联合目标的GRPO。我们的方法使数学pass@4相较于GRPO提升了高达20.00个百分点。消融支持完全过滤，冻结历史注入诊断支持事实效用。这些结果共同支持将代理级结果指导与通信衍生的令牌级信用结合起来。代码可在该 https URL 访问。

Indic-Task Gains and Regressions after Reinforcement Learning

强化学习后的印度任务增益与回归

• Authors: Sutanshu Saksena Raj, Daksh Maan, Avinash Benki, Parijat Chatterjee, Pasunuti Prasanjith, Purusharth Malik
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.16228
• Pdf link: https://arxiv.org/pdf/2611.16228
• Abstract We run reinforcement learning (RL) with GRPO across four environments, starting from a supervised fine-tuned 30B-A3B mixture-of-experts model covering English and eight Indic languages. We compare the model before and after RL across 34 benchmarks in one evaluation campaign. MATH improves by 36.07 points, and most Indic benchmarks also improve. Yet XQuAD-Indic F1 falls 19.32 points as English answers rise from 8.7% to 91.7%; IndicQA falls 8.64, while English XQuAD gains 11.71 F1. A second RL run from the same supervised initialization adds an Indic environment together with language and repetition rewards. Compared with the released model, it improves both passage-answering and instruction-following benchmarks, along with preserving MATH performance. But it loses ground on reasoning and coding benchmarks. Several training choices changed together, so we cannot isolate the effect of any one of the reward terms. Script compliance remains unresolved. Adding a language reward did not ensure use of the requested script. Passage answering and script compliance therefore need direct evaluation, even when other benchmark scores improve.
• 中文摘要 我们在四个环境中运行GRPO强化学习（RL），起点是一个监督微调的30B-A3B专家混合模型，涵盖英语和八种印度语言。我们在一次评估活动中比较了RL前后34个基准测试中的模型。数学提升了36.07分，大多数印度基准也有所提升。然而，XQuAD-Indic F1下降了19.32个百分点，英文答案从8.7%升至91.7%;IndicQA下降8.64个百分点，而英语XQuAD获得了11.71个F1。同一监督初始化的第二次强化学习（RL）增加了印度环境，同时增加了语言和重复奖励。与发布模型相比，它提升了文章答题和指令跟踪基准，同时保持了数学（MATH）的表现。但在推理和编码基准上，这会失去优势。多个训练选项同时发生变化，因此我们无法单独判断任何一个奖励词的效果。脚本合规性仍未解决。添加语言奖励并不能保证使用所请求的脚本。因此，即使其他基准分数有所提升，文章答题和脚本合规仍需直接评估。

DER Aggregators in Wholesale Electricity Markets: A Hybrid Mean-Field Framework with Endogenous Price Feedback

批发电力市场中的DER聚合器：带有内生价格反馈的混合平均场框架

• Authors: Jun He, Andrew L. Liu
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.16230
• Pdf link: https://arxiv.org/pdf/2611.16230
• Abstract The rapid growth of distributed energy resources (DERs), including rooftop solar and energy storage, is transforming the grid edge, where customer-side assets increasingly interact with the broader power grid. DER aggregators, entities that coordinate the actions of many small-scale DERs, play a key role in this transformation. This paper presents a hybrid Mean-Field Control (MFC) and Mean-Field Game (MFG) framework for integrating DER aggregators into wholesale electricity markets. Unlike traditional approaches that treat market prices as exogenous, our model captures the feedback between aggregators' collective actions and electricity prices. The MFC component optimizes DER operations within each aggregator, while the MFG models strategic interactions among multiple aggregators. To account for various uncertainties, we incorporate reinforcement learning, which allows aggregators to learn optimal bidding strategies in dynamic market conditions. We prove the existence and uniqueness of a mean-field equilibrium and validate the framework through a case study of the Oahu Island power system. Results show that our approach reduces price volatility and improves market efficiency, offering a scalable and decentralized solution for DER integration in wholesale markets.
• 中文摘要 分布式能源资源（DER），包括屋顶太阳能和储能，正在改变电网边缘，客户方资产日益与更广泛的电网互动。DER聚合器——协调众多小型DER行动的实体——在这一转型中发挥关键作用。本文提出了一个混合的均值场控制（MFC）和平均场游戏（MFG）框架，用于将DER聚合器整合进批发电力市场。与传统将市场价格视为外生方法不同，我们的模型捕捉了聚合者集体行为与电价之间的反馈。MFC组件优化每个聚合商内的DER运营，而MFG则模拟多个聚合商之间的战略互动。为考虑各种不确定性，我们引入强化学习，使聚合者能够在动态市场条件下学习最优竞价策略。我们证明了平均场均衡的存在性和独特性，并通过对欧胡岛电力系统的案例研究验证了该框架。结果表明，我们的方法降低了价格波动性，提高了市场效率，为批发市场中去中心化方程式集成提供了可扩展且去中心化的解决方案。

LoCoJAX: Host-Local Coordination for Small-Model RL on TPUs

LoCoJAX：TPU上小模型强化学习的主机-本地协调

• Authors: Loki Chen, Gregory B. Rehm, Yang Song, Hongye Xie, Ming Lei, Chongyang Bai, Bob Kamma, Mingfu Liang, Kavosh Asadi, Yufei Li, Xi Liu, Jay Xu, Rui Liu, Maziar Sanjabi, Luke Simon, Hamed Firooz
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.16301
• Pdf link: https://arxiv.org/pdf/2611.16301
• Abstract Reinforcement learning (RL) increasingly targets small language models, and executing these workloads efficiently across multiple accelerators requires effective coordination. On TPUs, Pathways-based RL uses a single controller to coordinate computation across hosts. This execution model is effective for large models, but its coordination computational costs are harder to amortize when small models repeatedly invoke short computations. To characterize these computational costs, we sweep dense models from 0.6B to 405B parameters on TPU v7x under a decode-heavy workload and measure how host and network computational overhead vary with model size. At 0.6B, host orchestration occupies 36% of step time and network communication 2%, whereas for large models, estimated network computational costs dominate the non-compute time. Motivated by these findings, we present Local Coordination JAX (LoCoJAX), which implements local rollout and reward processing on each TPU host with synchronous global training. In controlled comparisons against a tuned and optimized Pathways baseline on Qwen3-0.6B across three workloads and 8-64 TPU v7x chips, its colocated mode (LoCoJAX-CO) is 1.3-3.6x faster per step; on GSM8K, it completes a step faster on 8 chips than Pathways does on 64. Finally, a Ray-managed disaggregated mode (LoCoJAX-DA) overlaps rollout and training on separate hosts, and placement experiments establish when this overlap compensates for dividing devices and transferring weights: only when rollout dominates the step and receives most of the hosts. On NuminaMath-CoT at 256 chips, giving rollout 75% of hosts cuts step time by about 15% compared with LoCoJAX-CO.
• 中文摘要 强化学习（RL）越来越多地针对小型语言模型，在多个加速器间高效执行这些工作负载需要有效的协调。在TPU上，基于Pathways的RL使用单一控制器协调跨主机的计算。该执行模型对大型模型有效，但当小模型反复调用短计算时，其协调计算成本更难摊销。为表征这些计算成本，我们在TPU v7x上对0.6B到405B参数的密集模型进行解码重负载的扫描，测量主机和网络计算开销随模型规模的变化。在0.6B时，主机编排占步长的36%，网络通信占2%，而大型模型中估计的网络计算成本主导非计算时间。基于这些发现，我们提出了本地协调JAX（LoCoJAX），该系统在每个TPU主机上实现了本地部署和奖励处理，并实现同步全局训练。在Qwen3-0.6B上调优和优化的Pathways基线对比中，覆盖三种工作负载和8-64个TPU v7x芯片，其共址模式（LoCoJAX-CO）每步快1.3-3.6倍;在GSM8K上，8块芯片上的同步速度比Pathways在64上快一级。最后，光线管理的拆分模式（LoCoJAX-DA）在不同主机上重叠展开和训练，布置实验确定该重叠何时补偿设备分割和权重转移：仅当滚动主导该步并接收大部分主机时。在 NuminaMath-CoT 上，256 芯片，75% 的主机部署能比 LoCoJAX-CO 缩短约 15% 的步进时间。

StateAlign: Internalizing State-Aware Decision-Making for Reliable Multi-Turn Tool Agents via Reinforcement Learning

StateAlign：通过强化学习内化可靠的多回合工具代理的状态感知决策

• Authors: Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.16304
• Pdf link: https://arxiv.org/pdf/2611.16304
• Abstract Reinforcement learning has improved the tool-use capabilities of large language models. However, better task performance can coincide with increased tool hallucination. Reliable tool use requires decisions to follow the current task state: available information, tool capabilities, and task progress. Our analysis shows that agents sometimes execute when they should ask for information, retrieve it, or decline an infeasible request, even when reasoning identifies missing information. We introduce StateAlign, combining task-state construction with state-contrastive reinforcement learning. The data cover five task conditions and pair states with different action requirements. Outcome rewards guide task completion; token-level contrasts compare the same generated reasoning and actions under paired states to train outputs that fit the current state. Experiments across five base models from three families show improvements on BFCL v4 multi-turn, ToolSandbox, and When2Call. Ablations show that state construction improves performance and reliability under different reward designs. The complete method further reduces tool hallucination across three Qwen3 sizes. On Qwen3-8B, StateAlign improves BFCL v4 multi-turn accuracy from 45.5% to 55.2% and reduces no-tool hallucination on SimpleToolHalluBench from 62.2% under outcome-only training to 56.1%. Behavioral analyses show more consistent decisions, more complete multi-call execution, and stronger recovery after tool errors.
• 中文摘要 强化学习提升了大型语言模型的工具使用能力。然而，任务性能的提升可能伴随着工具幻觉的增加。可靠的工具使用需要决策遵循当前任务状态：可用信息、工具能力和任务进度。我们的分析显示，代理有时执行时应请求信息、检索信息或拒绝不可行请求，即使推理识别了缺失信息。我们引入了StateAlign，结合任务状态构建与状态对比强化学习。数据涵盖五个任务条件和不同动作需求的配对状态。结果奖励指导任务完成;代币级对比比较相同生成的推理和配对状态下的动作，以训练符合当前状态的输出。涵盖三个家族的五个基础模型实验显示，BFCL v4 multi-turn、ToolSandbox 和 When2Call 均有改进。消融显示状态构建在不同奖励设计下提升性能和可靠性。完整方法进一步减少了三种Qwen3尺寸的工具幻觉。在Qwen3-8B上，StateAlign将BFCL v4多轮准确率从45.5%提升至55.2%，SimpleToolHalluBench在仅结果训练下无工具幻觉率从62.2%降至56.1%。行为分析显示决策更一致，多次调用执行更完整，工具错误后恢复更强。

Memory-Efficient Parallel Policy Learning for Collapse-Reactive Retreat via Hybrid Terrain Modeling

通过混合地形建模实现的内存高效并行策略学习，用于崩溃反应式撤退

• Authors: Yuki Kadokawa, Ritsuki Matsunaga, Yuqi Yang, Edgar Anarossi, Takamitsu Matsubara
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.16337
• Pdf link: https://arxiv.org/pdf/2611.16337
• Abstract At disaster sites and on unstable slopes, robot contact can collapse uninspected terrain, requiring retreat upon support loss. Massively parallel reinforcement learning of collapse-reactive retreat requires independent state--action experiences. However, on shared rigid-body collapsible terrain, one robot's collapse alters others' contacts without direct interaction. We call this indirect interference. Duplicating terrain objects prevents this but is memory-intensive, limiting parallel sampling. Our key insight is that policy learning does not necessarily require rigid-body collapse modeling, which is feasible but computationally expensive; collapse-induced contact responses can suffice. We therefore propose a memory-efficient parallel policy-learning framework for collapse-reactive retreat via hybrid terrain modeling. Hybrid terrain modeling combines rigid-body simulation of non-collapsible terrain with virtual contact forces for collapsible regions. A State-Multiplexed Contact Model shares terrain geometry and a contact-response law while maintaining environment-specific collapse states, enabling independent experiences without terrain objects for collapsible regions. The framework trains a proprioceptive policy for both normal locomotion and collapse-reactive retreat. It supports over 100 times as many parallel environments as fine-resolution rigid-block terrain within the same GPU memory limit. This scalability and improved sample efficiency enable over 50 times faster policy convergence with comparable retreat performance. Sim-to-sim and real-robot experiments demonstrate retreat under multiple collapse conditions. A supplementary video is available at: this https URL
• 中文摘要 在灾害现场和不稳定坡面，机器人接触可能导致未检查地形坍缩，支撑丢失后需要撤退。坍缩反应性撤退的大规模并行强化学习需要独立的状态-动作体验。然而，在共享刚体可折叠地形中，一台机器人的坍缩会改变其他机器人的接触，而无需直接交互。我们称之为间接干扰。复制地形对象可以防止这种情况，但对内存需求大，限制并行采样。我们的关键见解是，策略学习不一定需要刚体坍缩建模，虽然可行但计算量大;坍缩诱导的接触响应即可。因此，我们提出了一个通过混合地形建模实现坍缩反应性撤退的内存高效并行策略学习框架。混合地形建模结合了不可折叠地形的刚体模拟与可折叠区域的虚拟接触力。状态多路复用接触模型共享地形几何和接触响应律，同时保持环境特定的坍缩状态，使可折叠区域能够独立体验无地形对象。该框架为正常移动和坍塌反应后退训练本体感受策略。它支持在同一GPU内存限制下，支持比细分辨率刚块地形多100倍以上的并行环境。这种可扩展性和提升的采样效率使策略收敛速度提升了50倍以上，且撤退性能相当。模拟对模拟和真实机器人实验展示了多种坍缩条件下的撤退。补充视频可在以下网站观看：此 https URL

Bait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward Function

诱饵：通过探究奖励函数缓解有害的代理强化学习

• Authors: Tiansheng Huang, Ananth Balashankar, Virat Shejwalkar, Santosh Pande, Ling Liu
• Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.16369
• Pdf link: https://arxiv.org/pdf/2611.16369
• Abstract We expose a safety vulnerability on nascent agentic reinforcement-learning-as-a-service. Specifically, with a cost as low as \$20, we show that safety alignment of a large language model (e.g., Inkling) can be completely subverted through rounds of harmful agentic reinforcement learning (agentic RL) with a toxic environment from its users. We also demonstrate that upon a successful harmful agentic RL attack via the service API, the attackers can potentially use the safety-degraded model to assist with real-world harmful activities, e.g., exploit software vulnerabilities and launch code re-use attacks. To mitigate harmful agentic RL, we propose Bait, a defense that mitigates the attack via proactively probing the reward function. Experimental results show that such a simple defense can sufficiently mitigate learning from a harmful reward function and can still learn from a benign reward function that promotes benign agentic tasks. Further analysis on both empirical and theoretical aspects shows that an adaptive attacker who assumes the knowledge of Bait defense cannot perfectly subvert it due to a systematic challenge. Source code is available at this https URL.
• 中文摘要 我们暴露了新兴代理强化学习即服务中的安全漏洞。具体来说，成本低至20美元，我们证明了大型语言模型（如Inkling）的安全对齐可以通过用户在有毒环境中进行有害的代理强化学习（代理强化学习）来完全破坏。我们还展示了，在通过服务API成功实施有害代理强化学习攻击后，攻击者可能利用安全降级模型协助现实世界的有害活动，例如利用软件漏洞和发起代码重用攻击。为减轻有害的代理强化学习，我们提出了诱饵防御，一种通过主动探测奖励函数来减轻攻击的防御。实验结果表明，这种简单的防御可以充分减轻有害奖励函数的学习，同时仍能从促进良性代理任务的良性奖励函数中学习。对经验和理论方面的进一步分析表明，假设具备诱饵防御知识的自适应攻击者，由于系统性挑战，无法完美颠覆其防御。源代码可在此 https 网址获取。

Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

递归自我提升的经验交接强化学习

• Authors: Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.16409
• Pdf link: https://arxiv.org/pdf/2611.16409
• Abstract It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.
• 中文摘要 通常通过标量奖励训练大型语言模型（LLMs）进行强化学习（RL）来执行代理任务。然而，这会丢弃代理在试验中体验到的环境细节信息（例如错误日志）。为了利用这些丰富的信息，近期研究将代理包裹在一个框架中，通过反思或压缩细化自身输入，并将过去经验交给未来的推广，但交付产物（例如剧本）的构建本身并未被学习。我们希望两者兼得优点：标量奖励带来梯度更新，同时利用环境信息传递经验。作为一个具体的测试平台，我们固定代理上下文工程（\ace{}）框架，训练一个模型以行动、反映和策划任务-局部剧本。我们的方法 \acerl{} 通过候选体验交接工件对未来行动者部署的下游影响评估，并在固定支架下协同优化角色特定行为（动作生成器、洞察反射器和操作手册策展者）。在各智能基准测试中，\acerl{} 实现了比基线方法更高的预期性能。结果表明，代理训练不仅应优化当前部署中的动作，还应优化未来推广继承的模型编写体验。

VOKE: Visual Object Kinematics Estimator for Dexterous In-Hand Manipulation

VOKE：用于灵巧手部操作的视觉对象运动学估计器

• Authors: Shengjia Zhu, Wenqiang Xu, Lin Shao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.16423
• Pdf link: https://arxiv.org/pdf/2611.16423
• Abstract Vision-based dexterous in-hand manipulation requires knowing not only where an object is, but also how it moves. However, most robotic manipulation systems rely on off-the-shelf visual perception models whose outputs are limited to static object masks, geometry, or 6D poses, leaving downstream controllers to infer object dynamics implicitly from noisy visual observations. We present VOKE, a kinematics-aware perception framework that augments existing object and hand-object pose estimators to recover a complete object kinematic state comprising pose, linear and angular velocity, and linear and angular acceleration. VOKE contains two core perception modules and a downstream application study. First, a Pluggable Kinematics Module (PKM) attaches to the output of a standard pose estimator and predicts the object's linear and angular velocity and acceleration from temporal visual features. Second, a Kinematics-guided Pose Refinement Module (KPRM) converts this motion vector into a second-order, motion-prior reference pose and refines the per-frame pose estimates. We evaluate VOKE on DexYCB and OakInk, where it consistently improves object pose estimation across three different backbones and substantially reduces velocity and acceleration estimation error compared with pose differencing. As a downstream application, we use Kinematics-Augmented Reinforcement Learning (KARL) to compare a kinematics-augmented policy with a pose-history baseline. After training the policies in simulation, we transfer the kinematics-augmented policy to the physical robot without policy fine-tuning, demonstrating the complete perception-to-control pathway in the target domain.
• 中文摘要 基于视觉的灵巧手控操作不仅需要知道物体的位置，还需要知道其运动方式。然而，大多数机器人操作系统依赖现成的视觉感知模型，其输出仅限于静态物体遮罩、几何体或6D姿态，因此下游控制器只能通过噪声视觉观察隐式推断物体动态。我们提出了VOKE，一种运动学感知框架，增强现有物体和手对象姿态估计器，恢复包含姿态、线性和角速度以及线性和角加速度的完整物体运动学状态。VOKE包含两个核心感知模块和一个下游应用研究。首先，一个可插拔运动学模块（PKM）连接到标准姿态估计器的输出，并从时间视觉特征预测物体的线性和角速度及加速度。其次，运动学引导姿态细化模块（KPRM）将该运动矢量转换为二阶、运动先验参考姿态，并优化每帧姿态估计。我们在DexYCB和OakInk上评估VOKE，其在三种不同骨干链上持续提升物体姿态估计，并显著降低了速度和加速度估计误差，相较姿态差分。作为下游应用，我们使用运动学增强强化学习（KARL）将运动学增强策略与姿态历史基线进行比较。在模拟中训练策略后，我们将运动学增强策略传输到物理机器人，无需策略微调，展示了目标域中完整的感知到控制路径。

HALight: Heterogeneous Multi-Agent Reinforcement Learning for Traffic Signal Control under Lane-Level Missing Observations

HALight：车道级缺失观测下交通信号控制的异构多智能体强化学习

• Authors: Chenkai Hu, Yingqian Zhang, Ming Xie, Yi Fang, Chengxiang Jin, Chao Li
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.16574
• Pdf link: https://arxiv.org/pdf/2611.16574
• Abstract Multi-agent reinforcement learning has shown great promise for traffic signal control. However, most existing methods rely on two key assumptions: (i) agents are assumed to be homogeneous with unified observation/action spaces; (ii) full and reliable lane-level traffic observations are available to all agents. In practice, urban road networks exhibit substantial structural heterogeneity (e.g., lane configurations and phase designs), while sensor failures and incomplete detector coverage often result in missing incoming lane measurements. Such issues can severely impair coordination and destabilize learning. To address these challenges, we propose HALight, a communication-enhanced heterogeneous multi-agent reinforcement learning framework for traffic signal control under lane-level missing observations. HALight supports intersection-specific observation and action modeling to accommodate heterogeneous intersections, and incorporates a lightweight one-hop neighborhood communication module to enrich spatial context when lane-level observations are partially unavailable. Moreover, HALight adopts a sequential policy update scheme to improve coordination among heterogeneous agents. Experimental results on two real-world road networks demonstrate that HALight outperforms representative homogeneous and heterogeneous baselines in terms of traffic efficiency and convergence behavior. The code and datasets used in this work are available at this https URL.
• 中文摘要 多智能体强化学习在交通信号控制方面展现出巨大潜力。然而，大多数现有方法依赖于两个关键假设：（i）假设智能体是同质的，具有统一的观察/动作空间;（ii）所有智能体均可获得完整且可靠的车道级交通观测。在实际应用中，城市道路网络表现出显著的结构异质性（如车道配置和相位设计），而传感器故障和检测器覆盖不完整常导致输入车道测量数据缺失。此类问题严重影响协调并破坏学习稳定性。为应对这些挑战，我们提出了HALight，一种基于通信增强的异构多智能体强化学习框架，用于车道级缺失观测下的交通信号控制。HALight支持路口特定观察和动作建模，以适应异构交叉口，并集成轻量级一跳邻域通信模块，在车道层级观测部分不可得时丰富空间上下文。此外，HALight采用顺序策略更新方案，以改善异构代理之间的协调。两个真实道路网络的实验结果表明，HALight在交通效率和收敛行为方面优于代表性的同质和异构基线。本工作中使用的代码和数据集可在该 https URL 访问。

Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models

预算中的安全：校准视觉语言模型受限GRPO的控制通道

• Authors: Bo Han
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.16628
• Pdf link: https://arxiv.org/pdf/2611.16628
• Abstract Reinforcement learning can teach vision-language models (VLMs) to decline harmful image-text requests, but how safe the result becomes is set indirectly, through reward weights found by trial runs. Stating safety as budgets on the rate of harmful assistance and the rate of over-refusal, enforced by Lagrange multipliers inside GRPO with an off-the-shelf scorer as the cost, is the natural fix, and it fails twice. The multiplier compares the scorer's mean with the budget, and scorers built for ranking get the mean wrong: a fixed-point analysis shows that the mean bias, not the budget, then decides whether the multiplier settles, saturates or switches off. And standard GRPO standardizes the penalized reward within each group, which cancels the multiplier before it reaches the gradient. Two-channel constrained GRPO repairs both: the base reward keeps the raw score for the within-group comparison, the penalty and the constraint estimate pass through a monotone map fitted on judge-labeled training responses, and the advantage keeps the multiplier. On a Qwen3-VL-2B policy with a training-free readout of an open 4B model as the scorer, held-out harm on 1,037 adversarial prompts falls from 8.9% under standard GRPO to 4.8%, lower in every seed, where the uncalibrated budget gives 7.6%; over-refusal stays at most 2% on 400 benign queries and MMBench is unchanged. The gain is largest where the intent sits in the image (10.1% -> 5.1%) and grows on the unseen HADES benchmark (28.9% -> 12.8%). Order-preserving distortions of the scores reproduce each predicted controller state in every seed, and the results hold under a map fitted on another judge's labels, an image-reading judge and a human audit.
• 中文摘要 强化学习可以教视觉语言模型（VLM）拒绝有害的图像文本请求，但结果的安全性通过试验获得的奖励权重间接决定。将安全定义为基于有害援助率和过度拒绝率的预算，由GRPO内的拉格朗日乘数强制执行，并以现成评分器为代价，是自然的解决办法，但失败了两次。乘数器将评分者的平均值与预算进行比较，而为排名设计的评分器会错误地计算平均值：不动点分析显示，平均偏误而非预算决定了乘数是稳定、饱和还是关闭。标准GRPO标准化了每个组内的惩罚奖励，在乘数达到梯度前取消。双通道受限GRPO修复了两者：基础奖励保留组内比较的原始得分，罚则和约束估计通过对评判标记训练响应拟合的单调映射，优势保持乘数。在Qwen3-VL-2B策略中，开4B模型的无训练读数作为评分者，1037个对抗提示的未预留伤害从标准GRPO的8.9%降至4.8%，每个种子中较低，未校准预算为7.6%;400个良性查询的过度拒绝最多2%，MMBench保持不变。在意图中意图（10.1% -> 5.1%）和未见的HADES基准（28.9% -> 12.8%）增长时，收益最大。保持顺序的分数失真在每个种子中重现每个预测的控制状态，结果在另一位评委、图像读取评委和人工审核标签上的映射下成立。

CADET: Construction of PCB Routing Instances with Executable Demonstrations

CADET：构建带有可执行演示的PCB布线实例

• Authors: Hyungseok Song, Won-Seok Choi, Seohui Bae, Han-Seul Jeong, Junseok Park, Youngjoon Park, Soonyoung Lee
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.16913
• Pdf link: https://arxiv.org/pdf/2611.16913
• Abstract Printed circuit board (PCB) routing is the stage of physical layout that finds paths connecting the placed pads of each net. Public datasets in this domain are few, and most contain only the final states of routed boards, not demonstrations of the electronic design automation (EDA) engine actions that produce them. To close this gap, we propose CADET (Constructing executAble Demonstrations for Engine-grounded Training), which jointly generates synthetic boards and demonstrations built from EDA engine actions. The resulting RouteDemos provides 100,000 synthetic boards routed with zero design rule violations (DRVs), together with their demonstrations. Their distributions over 17 statistics resemble those of 142 real reference boards. CADET trains a routing policy on these demonstrations with behavior cloning and fine-tunes it with reinforcement learning (RL). On real test boards, the policy raised the rate of complete DRV-free routing from 38.0% to 67.2% over an RL-only baseline. This rate was comparable to that of the open-source autorouter FreeRouting and exceeded those of the other baselines, including GPT-6 Astra. We release the dataset and its generation code as open source at this https URL.
• 中文摘要 印刷电路板（PCB）布线是物理布局中寻找连接每个网络垫板路径的阶段。该领域的公开数据集很少，大多数仅包含布线板的最终状态，而非产生这些电路板的电子设计自动化（EDA）引擎动作的演示。为弥补这一空白，我们提出了CADET（构建可执行的引擎接地训练演示），该系统联合生成合成板和由EDA引擎动作构建的演示。最终的RouteDemos提供了10万块合成电路板，布线时没有设计规则违规（DRV），并附带演示内容。它们在17个统计数据上的分布与142个真实参考板相似。CADET通过行为克隆训练这些演示的路由策略，并通过强化学习（RL）进行微调。在真实测试板上，该政策将完全无DRV路由率从38.0%提升至仅基于RL基线的67.2%。这一率与开源自动路由FreeRouting相当，且超过了包括GPT-6 Astra在内的其他基线。我们将数据集及其生成代码以开源方式发布，链接为 https URL。

Skill Adapter: Personalized Modification and Online Routing of General-Purpose Skills

技能适配器：通用技能的个性化修改与在线路由

• Authors: Qijia Zhuang, Yanyi Gou, Wenbin Ma, Yuxiang Ren
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.17007
• Pdf link: https://arxiv.org/pdf/2611.17007
• Abstract Large language model (LLM) agents use external skills to reuse task procedures and experience with tools. General-purpose skills describe shared execution procedures, while specific tool choices, operating practices, and output requirements vary across users. In long-horizon tasks, subgoals, available inputs, and environmental states change as execution progresses. Agents must select currently applicable skills from an expanding skill pool and prepare candidates for subsequent steps. We propose Skill Adapter, which enables personalized modification and online routing of general-purpose skills. The system distills local adjustments from user requirements and execution feedback and stores them as Adapters linked to general-purpose skills. The routing agent selects general-purpose skills and applicable Adapters based on the current subtask, execution progress, and environmental feedback, incrementally maintaining a task-level online skill library that the main LLM reads on demand. The system preloads candidates for subsequent nodes and rechecks their applicability once actual inputs are determined. Reinforcement learning optimizes the two types of decisions using modification gains and routing outcomes, respectively. We evaluate Skill Adapter on ALFWorld, WebShop, and SkillsBench. Compared with SkillRL, it improves task success rates by an average of 11.43 percentage points and reduces mean execution time on all three benchmarks. Skill Adapter retains its performance advantage when the skill pool grows to 1,000 skills.
• 中文摘要 大型语言模型（LLM）代理利用外部技能重复使用任务流程和工具经验。通用技能描述共享执行过程，而具体工具选择、操作实践和输出需求因用户而异。在长期任务中，子目标、可用输入和环境状态会随着执行进程而变化。代理必须从不断扩展的技能库中选择当前适用技能，并为后续步骤做好准备。我们提出了技能适配器，支持通用技能的个性化修改和在线路由。系统从用户需求和执行反馈中提取局部调整，并将其存储为与通用技能关联的适配器。路由代理根据当前子任务、执行进度和环境反馈选择通用技能和适用适配器，逐步维护任务级在线技能库，主LLM按需读取。系统预加载后续节点的候选节点，并在确定实际输入后重新检查其适用性。强化学习分别通过修改增益和路由结果优化这两种决策类型。我们在ALFWorld、WebShop和SkillsBench上评估了Skill Adapter。与SkillRL相比，它平均提升了11.43个百分点的任务成功率，并在三个基准测试中降低了平均执行时间。当技能池增长到1000个技能时，Skill Adapter依然保持性能优势。

From Delegation to Coordination: Learning Efficient Multi-Agent Deep Research

从委派到协调：学习高效的多智能体深度研究

• Authors: Yuyang Wu, Yang Liu, Zequn Xie, Junjie Wang, Dan Yang, Zehao Tan, Ding-Chu Zhang, Junming Yang, Yue Shen, Jian Wang, Jinjie Gu
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.17041
• Pdf link: https://arxiv.org/pdf/2611.17041
• Abstract Multi-agent deep-research systems use delegation to parallelize search and keep long tool traces out of the main agent's context. Yet delegation helps only when task allocation and subtask execution are coordinated: the main agent must issue useful subtasks, while subagents must return reliable evidence. Existing training either optimizes only the main agent or broadcasts the same team advantage to all agents, which can reward misleading subagent reports and penalize useful ones. We introduce LEAD-RL (Learning Effective Agent Delegation with Reinforcement Learning), a joint main-agent-subagent training framework with contribution-aware credit assignment. A training-time rubric evaluates the research utility of each subagent report and gates its inherited team advantage, while final-answer correctness remains the only team reward; no token, step, or speedup reward is used. Across five benchmarks and two model families, LEAD-RL improves answer quality. On BrowseComp, it improves accuracy by 2.49 percentage points and reduces critical-path cost by 24.5% over joint training with shared team credit. During validation, accuracy increases by 6.78 points at approximately constant critical-path cost. These results show that contribution-aware credit assignment improves the quality-efficiency trade-off of multi-agent deep research.
• 中文摘要 多智能体深度研究系统利用委托并行化搜索，避免长工具追踪进入主智能体上下文。然而，委托仅在任务分配和子任务执行协调时才有用：主智能体必须发布有用的子任务，而子智能体必须返回可靠证据。现有培训要么仅优化主智能体，要么向所有智能体广播相同的团队优势，这可能奖励误导性的子智能体报告并惩罚有用的报告。我们引入了LEAD-RL（带强化学习的有效智能体委派学习），这是一个带有贡献感知的信用分配的主代理-子代理联合培训框架。培训时间评分标准评估每个子代理报告的研究效用，并限制其继承的团队优势，而最终答案正确性仍是团队唯一的奖励;不使用代币、步骤或加速奖励。在五个基准测试和两个模型家族中，LEAD-RL提升了答案质量。在BrowseComp上，它比联合训练与共享团队积分相比，准确率提升了2.49个百分点，关键路径成本降低了24.5%。在验证过程中，在关键路径成本大致保持不变的情况下，准确率提升了6.78个百分点。这些结果表明，贡献感知型积分分配改善了多智能体深度研究的质量与效率权衡。

Support-Constrained Offline Policy Improvement for Pediatric ECMO from Limited Clinical Data

基于有限临床数据的儿童ECMO支持限制线下政策改进

• Authors: Fateme Golivand Darvishvand, Saurabh Mathur, Nikhilesh Prabhakar, Michael A. Skinner, Ameet Soni, Neel Shah, Ethan L Sanford, Lakshmi Raman, Sriraam Natarajan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.17134
• Pdf link: https://arxiv.org/pdf/2611.17134
• Abstract Managing children on ECMO (Extracorporeal Membrane Oxygenation) requires balancing short-term physiological stability with the long-term goal of preventing neurological injury. Standard reinforcement learning can propose risky, out-of-distribution actions, while imitation learning is limited to reproducing clinician behavior. We introduce Support-Constrained Fitted Q-Iteration (FQI-SC), an offline reinforcement learning framework that cautiously improves upon expert behavior by restricting optimization to empirically supported state-action pairs and defaulting to the clinician baseline in low-data regions. On 78 pediatric trajectories totaling over 6,000 hours of clinical data, we evaluate our framework using Fitted Q-Evaluation (FQE) and a behavioral disagreement. Additionally, we formalize the performance-safety trade-off by introducing a unified Balance Score ($S(\pi)$). Our empirical evaluation shows that support-constrained policy improves the Balance Score by 38\% relative to FQI, improves estimated value by 1.28 FQE units over imitation learning, and reduces disagreement with clinician actions by 71\% relative to FQI.
• 中文摘要 管理体外膜氧合（ECMO）儿童需要在短期生理稳定性与预防神经损伤的长期目标之间取得平衡。标准强化学习可能提出风险较高、分布外的行为，而模仿学习仅限于临床行为的再现。我们引入支持-约束拟合Q迭代（FQI-SC），这是一种离线强化学习框架，通过限制优化至实证支持的状态-行动对，并在数据不足区域默认临床医生基线，谨慎地改进专家行为。在78条儿科轨迹中，累计超过6000小时的临床数据，我们使用拟合Q评估（FQE）和行为分歧来评估我们的框架。此外，我们通过引入统一的平衡评分（$S（\pi）$）形式化了绩效与安全性的权衡。我们的实证评估显示，支持约束政策相较FQI提高了平衡分数38%的平衡分数，比模仿学习提升了1.28个FQE单位的估计值，并且相较于FQI，减少了71%的对临床医生行为的分歧。

Learning Dynamics of the Influencer's Game in Resource Landscapes

资源景观中影响者游戏的动态学习

• Authors: Mark Lovett Feng Fu
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Theoretical Economics (econ.TH); Physics and Society (physics.soc-ph)
• Arxiv link: https://arxiv.org/abs/2611.17148
• Pdf link: https://arxiv.org/pdf/2611.17148
• Abstract Across domains as varied as ecology, economics, and online platforms, individuals are incentivized to position themselves within resource-rich strategy spaces. In this work we introduce a new spatial game framework called the influencer's game. Within this new framework players' strategies are represented as positions within a trait space. Players then receive a share of a portion of a resource based on their relative influence at that resource's points traits. For the influencer's game, we show that there is a unique symmetric Nash equilibrium that exists under strict log-concavity of the influence kernel. Using the unique equilibrium we show that the stability of the system depends on the influence kernel's parameters, number of players, and the resource distribution shape. For any dimension we demonstrate a stability threshold that depends on both the scatter and curvature of the players. When this threshold is crossed, the symmetric equilibrium destabilizes and gives way to a variety of bifurcation types; including, specialist clusters, limit cycles, and sub-games that reproduce the larger game's structure. We ground these results using real-world data for resources as a demonstration of the game's application. Finally we extend the game dynamics to use multi-agent reinforcement learning and further show that the predicted bifurcation structure persists under partial observability and discretized strategy spaces. In all, the influencer's game provides a novel framework for predicting when competitive pressure drives convergence versus fragmentation with possible applications across domains.
• 中文摘要 在生态学、经济学和在线平台等多样领域，个体被激励将自己定位于资源丰富的战略空间中。本研究介绍了一个新的空间博弈框架，称为影响者博弈。在该新框架中，玩家的策略被表示为特质空间中的位置。玩家随后根据其在该资源点性特征上的相对影响力，获得资源的一部分分成。对于影响者博弈，我们证明存在一个独特的对称纳什均衡，该均衡存在于影响核严格对数凹性下。利用唯一均衡，我们表明系统的稳定性依赖于影响核的参数、玩家数量和资源分布形状。对于任意维度，我们展示了一个依赖于玩家散布和曲率的稳定性阈值。当跨越这一阈值时，对称均衡会不稳定，转而出现多种分岔类型;包括专门集群、极限环和子博弈，这些都重现了更大博弈的结构。我们用资源的真实世界数据来验证这些结果，作为博弈应用的演示。最后，我们将博弈动力学扩展到多智能体强化学习，进一步证明预测的分岔结构在部分可观测性和离散化策略空间下依然存在。总的来说，影响者的博弈为预测竞争压力何时驱动收敛与碎片化提供了新框架，并可能在跨领域应用。

Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

演员、法官还是教师？理解长视线代理中的决策模型

• Authors: Haochen Liu, Qianhao Zhu, Zixuan Wang, Dingming Li, Tongbo Chen, Peng Qiu, Yingqi Qu, Weiming Lu, Jun Xiao, Yongliang Shen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.17167
• Pdf link: https://arxiv.org/pdf/2611.17167
• Abstract Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at this https URL.
• 中文摘要 决策模型通过类型输出回答问题，比如每个选项的概率选择，因此一个模型可以选择代理的行动、评估已采取的行动，或对每个行动进行评分，以通过强化学习来教授策略。以往的研究主要将它们作为行为者来研究，通过它们的行为表现来判断其实用性。我们探问一个行为不佳的决策模型是否仍能有效教学。我们将三种决策模型和三种通用大型语言模型作为ALFWorld上的演员和评判进行比较，并在ALFWorld和WebShop上使用JEV作为教师。作为参与者，所有六个模型都被一个必须优先实现的子目标拉开。作为看到整个轨迹的评判者，同样的模型会认可通往该子目标的步骤，并且询问每个行动是否有帮助，而非任务是否成功，这区分了有用与无用的行动。作为教师，JEV将ALFWorld上的Qwen2.5-1.5B政策提升至82.9%，从56.8%提升至WebShop的71.4%。这些结果表明，演员表现不佳仍能教学，因为在剧集结束后判断动作能揭示表演必须预见的后果。代码可在此 https URL 获取。

Negative-Only Policy Optimization for One-Sided Verifiable Rewards

仅负面策略优化，以实现单方面可验证的奖励

• Authors: Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, Bo An
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.17287
• Pdf link: https://arxiv.org/pdf/2611.17287
• Abstract Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives. This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses. Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.
• 中文摘要 带可验证奖励的强化学习（RLVR）通常依赖一个能够可靠判断抽样解是否正确的验证器。然而，在许多现实环境中，完全验证通常不可得，而廉价检查只能伪造失败。我们将此机制形式化为单面可验证性（OSV）：已验证的负面是可靠的反例，但已验证的阴性仅为未被证伪的候选，且可能包含许多假阳性。这种不对称性使标准RLVR不可靠，因为强化已验证阳性可能直接奖励通过部分检查的捷径。为解决这个问题，我们提出了仅负面策略优化（NOPO），将已验证的正值视为未标记，仅通过抑制可信的已验证阴性来更新策略。具体来说，NOPO结合了参考相对负抑制损失与组级适配器，该适配器按未伪造响应的比例来扩展每次更新。在单元测试、约束检查和大型语言模型自我验证器中，当oracle奖励不可用时，NOPO在代码、逻辑和数学基准测试上的pass@k提升。这些提升在推理时间缩放和跨基准转移下依然有效。

Learning When Diffusion Language Models Should Regenerate

学习扩散语言模型何时应重新生成

• Authors: Minhae Oh, Jeonghye Kim, Jinwoo Koo, Yeongoon Kim, Jimin Seo, Minsik Seol, Jungwoo Lee
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.17320
• Pdf link: https://arxiv.org/pdf/2611.17320
• Abstract Remasking allows diffusion language models to revisit generated text, but effective correction requires knowing when and what to revise. Our diagnosis on Sudoku reveals two gaps. Hidden states contain error information that token confidence does not reliably expose, and remasking an isolated error usually reproduces it when the surrounding context remains fixed. These findings motivate DiVR , a verifier that predicts whether a partially generated sequence will reach a correct answer. Trained on the final correctness of the model's own trajectories, DiVR decides when to regenerate a block and which alternative to retain, while the generator itself stays unchanged. On five reasoning benchmarks with LLaDA-8B-Base, DiVR raises the average accuracy of the fine-tuned generator from 35.3\% to 43.4\% at $1.9$-$3.5\times$ the cost of greedy decoding, above search baselines that use $3.8$-$7.4\times$. It also complements post-training such as supervised fine-tuning and reinforcement learning, and generalizes to other generators, including Dream.
• 中文摘要 重掩蔽允许扩散语言模型重新审视生成文本，但有效纠正需要知道何时何物需要修正。我们在数独上的诊断揭示了两个空白。隐藏状态包含代币置信度无法可靠揭示的错误信息，而当周围上下文保持固定时，重新掩蔽孤立错误通常会重现该错误。这些发现激励了DiVR，一个验证器，用于预测部分生成序列是否会得到正确答案。基于模型自身轨迹的最终正确性训练，DiVR决定何时重生成块及保留哪个备选方案，而生成器本身保持不变。在五个推理基准测试中，DiVR将微调生成器的平均准确率从35.3%提升到43.4%，价格为1.9美元至3.5美元乘以贪婪解码成本，高于搜索基线的3.8美元至7.4美元倍数。它还补充了后期训练，如监督微调和强化学习，并可推广至包括Dream在内的其他生成器。

Internalizing Investment Mandates: A Neurosymbolic RL Approach to Portfolio Allocation

投资指令的内化：一种神经符号强化学习的投资组合配置方法

• Authors: Zeynep Yıldız, Satchit Chatterji, Ercan Atam, Erman Acar
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.17329
• Pdf link: https://arxiv.org/pdf/2611.17329
• Abstract Institutional portfolios operate under investment policy statements that specify asset allocation constraints, often conditioned on market regime information. Deep reinforcement learning has been shown to optimize financial performance but typically does not enforce rule compliance, making it unsuitable for real-world deployment. Existing neurosymbolic methods such as Probabilistic Logic Policy Gradient (PLPG) offer constraint-satisfying reinforcement learning (RL) via differentiable symbolic supervision but assume a finite, discrete action space. We present a PLPG-inspired continuous allocation method that estimates the probability that the policy's own executed action satisfies the investment mandate, using reparameterized Monte Carlo sampling, and adds it to the training objective. No post-hoc enforcement mechanism intercepts actions at inference time and compliance is learned by the agent. Audited directly on deterministic mean actions with no correction, the policy reduces violations of a six-rule regime-conditional mandate from 96-98% for unconstrained RL and 33.4% for a reward-penalty baseline to 4.8%, showing that a continuous-action neurosymbolic policy can substantially internalize a symbolic mandate.
• 中文摘要 机构投资组合遵循定义资产配置约束的投资政策声明，通常以市场机制信息为条件。深度强化学习已被证明能优化财务表现，但通常不强制执行规则合规，因此不适合实际部署。现有的神经符号方法如概率逻辑策略梯度（PLPG）通过可微分符号监督提供约束满足的强化学习（RL），但假设动作空间有限且离散。我们提出了一种受PLPG启发的连续配置方法，利用重参数化的蒙特卡洛采样估计策略自身执行动作满足投资任务的概率，并将其加入训练目标。没有事后强制机制在推断时拦截动作，代理学习合规性。该政策直接审计确定性平均行为且无修正，将六规则制度条件性强制令的违规率从无约束强化学习的96%-98%和奖励-惩罚基线的33.4%降至4.8%，表明连续行动神经符号政策可以显著内化象征性强制。

Keyword: diffusion policy

Inner Product Regularization for Imitation Learning

模仿学习的内积正则化

• Authors: Benjamin A. Christie, Tianyu Qiu, Amy H. Choi, David Fridovich-Keil, Dylan P. Losey
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.17159
• Pdf link: https://arxiv.org/pdf/2611.17159
• Abstract Robots that learn from demonstrations should extrapolate their policy despite state perturbations such as sensor noise, scene variation, and compounding errors. One approach to this robustness is regularization. Prior regularizers consider each state on its own and limit how sensitive the policy is to any perturbation at that state. By contrast, we explore how the policy's output shifts when given the same perturbation at different states. Our insight is that, locally, the robot should respond to state perturbations in a consistent manner: e.g., if changing an irrelevant object position does not impact the policy at one state, that same change should not shift the policy output at a nearby state. Applying this insight, we develop inner product regularization (InneR). We show that the inner product on action space induces an inner product on state space through the policy Jacobian (the Fisher information for stochastic policies), and that the null space of this inner product governs which state perturbations have a first-order effect on the policy. By clustering states whose inner products are aligned, averaging the inner products within each cluster, and minimizing the smallest eigenvalues of that average, InneR shapes the learned policy so that nearby states share a common null space. InneR enforces consistent responses to perturbations without limiting policy expressiveness, and can be appended to existing imitation learning pipelines for deterministic or stochastic policies. In simulated experiments, InneR makes learned policies more robust to irrelevant state dimensions and shifts between training and test distributions, whereas norm penalties often perform worse than behavior cloning. On a real robot arm performing three manipulation tasks, InneR raises the average success rate of a diffusion policy from 18% to 57% without additional data.
• 中文摘要 从演示中学习的机器人应当在状态扰动（如传感器噪声、场景变化和复合误差）的情况下，外推其策略。实现这种鲁棒性的一种方法是正则化。先验正则化者会单独考虑每个状态，并限制策略对该状态下任何扰动的敏感度。相比之下，我们探讨在不同状态下，策略输出在相同扰动时如何变化。我们的见解是，在局部上，机器人应以一致的方式响应状态扰动：例如，如果改变无关对象位置不影响某一状态的策略，那么同一变化不应改变邻近状态的策略输出。应用这一见解，我们发展了内积正则化（InneR）。我们证明，作用空间上的内积通过策略雅可比矩阵（随机策略的Fisher信息）诱导状态空间上的内积，而该内积的零空间决定哪些状态扰动对策略有一阶影响。通过聚类内积对齐状态，平均每个聚类内积，并最小化该平均值的最小特征值，InneR塑造了学习策略，使邻近状态共享一个共同的零空间。InneR在不限制策略表达性的情况下强制对扰动做出一致响应，并且可以附加到确定性或随机策略的模拟学习流水线中。在模拟实验中，InneR使学习策略对无关状态维度及训练与测试分布之间的变化更具鲁棒性，而范数惩罚通常表现不如行为克隆。在执行三项操作任务的真实机器人手臂上，InneR将扩散策略的平均成功率从18%提升到57%，且不需额外数据。