生成时间: 2026-11-19 01:01:58 (UTC+8); Arxiv 发布时间: 2026-11-18 20:00 EST (2026-11-19 09:00 UTC+8)
今天共有 42 篇相关文章
Keyword: reinforcement learning
Bookkeeping Is Not Judgement: EJBench, an Engineering Judgement Benchmark Built from a Conjugate Heat Transfer Design Campaign
簿记不是评判:EJBench,一项基于共轭热传导设计活动构建的工程判断基准
- Authors: James Sadler, Rotimi Alabi, Kotub Uddin
- Subjects: Subjects:
Computational Engineering, Finance, and Science (cs.CE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.21707
- Pdf link: https://arxiv.org/pdf/2611.21707
- Abstract
We introduce EJBench, a benchmark of engineering judgement: whether a model can predict how a design behaves at new conditions from a few earlier results, and decide on it. Unlike engineering benchmarks that test whether an agent can operate simulation or CAD software, EJBench is software-agnostic and asks what an engineer would conclude without running a simulation. Its tasks come from a conjugate heat transfer design campaign for battery cold plates, in three families: matching unlabelled test results to their operating points, choosing which stored runs to reveal before signing off a design, and ranking new duties by peak cell temperature. Every answer is taken from a converged simulation, not from an LLM judge. The best of ten frontier models, GPT-6 Astra, scores 0.71, and models state almost the same confidence on tasks they get wrong as on tasks they get right. We also post-train Qwen3.5-9B with LoRA and reinforcement learning, rewarded by the share of answer criteria the simulation results confirm; on the held-out test split its score rises from 0.071 to 0.106. We release the train, validation and test splits, the evaluation harness and the leaderboard.
- 中文摘要
我们介绍了EJBench,这是一项工程判断的基准:模型是否能根据早期几个结果预测设计在新条件下的表现,并据此作出决定。与测试代理是否能运行仿真或CAD软件的工程基准不同,EJBench是软件中立的,询问工程师在不运行模拟的情况下会得出什么结论。其任务来自电池冷板的共轭热传导设计,分为三类:将未标记测试结果与其工作点匹配、选择在签署设计前披露的存储运行,以及按峰值单元温度排序新任务。所有答案均来自汇聚模拟,而非大型语言模型评审。十个前沿模型中最好的GPT-6 Astra得分为0.71,模型对错误任务的信心几乎与正确任务相同。我们还在 Qwen3.5-9B 之后用 LoRA 和强化学习进行训练,并根据模拟结果所确认的答案标准比例来奖励;在保留的测试分段中,其分数从 0.071 上升到 0.106。我们发布了训练、验证和测试分段、评估工具和排行榜。
Learning In-Context Adaptivity for LLMs through Continual Gameplay
通过持续游戏学习LLM的上下文适应性
- Authors: Jiale Liu, Zun Li, John Schultz
- Subjects: Subjects:
Computer Science and Game Theory (cs.GT)
- Arxiv link: https://arxiv.org/abs/2611.21710
- Pdf link: https://arxiv.org/pdf/2611.21710
- Abstract
LLM agents are increasingly deployed in long-horizon tasks where they must explore, gather experience, and adapt their decisions accordingly. We name this capability in-context adaptivity: improving decisions across episodes from experience accumulated in context. We propose to evaluate it in repeated imperfect-information games against unknown opponents. In these games, fixed game-theoretically optimal policies cannot achieve maximum reward, whereas adapting to the opponent increases returns across games. We design a paired protocol that evaluates each agent with and without memory under identical conditions to isolate experience gain from single-game skill. Evaluation shows that memory brings little performance gain for open-weight models and inconsistent gains for frontier models. Our analysis of the agents' summaries and actions identifies a memory-doing gap in which models accurately record opponent tendencies yet rarely depart from default play. To close this gap, we construct a pool of exploitable opponents and post-train Qwen3.5-9B with supervised fine-tuning followed by trajectory-level reinforcement learning. We optimize full-match return across context segments so summary generation and action selection are trained together. On unseen opponents, our 9B RL policy doubles experience gain over supervised fine-tuning in Leduc Hold'em and increases it fivefold in No-Limit Hold'em. Across 35 opponents, our policy matches or surpasses frontier models, generalizes beyond its training horizon, and retains most of its gain on summaries from frontier models.
- 中文摘要
LLM代理越来越多地被部署在长期任务中,他们必须探索、积累经验并相应调整决策。我们将这种情境适应性称为“情境适应性”:通过在情境中积累的经验,改进各阶段的决策。我们建议在面对未知对手的重复不完全信息博弈中进行评估。在这些博弈中,固定的博弈理论最优策略无法获得最大回报,而适应对手则能提升博弈间的回报。我们设计了一套成对协议,在相同条件下评估每个有记忆和无记忆的智能体,以从单局技能中分离经验获得。评估显示,对开放权重模型带来的性能提升有限,而前沿模型则带来不一致的提升。我们对代理摘要和行为的分析发现了一个内存执行差距,模型准确记录对手倾向,但很少偏离默认玩法。为弥合这一差距,我们构建了一个可利用对手池,并对Qwen3.5-9B进行监督式微调,随后进行轨迹级强化学习。我们优化了跨上下文段的全匹配返回,因此总结生成和动作选择同步训练。对于未被看见的对手,我们的9B强化学习策略使经验值提升比Leduc德州扑克的监督微调翻倍,在无限注德州扑克中则提升五倍。在35名对手中,我们的策略匹配或超越前沿模型,推广超出训练范围,并将大部分优势保留在前沿模型的总结上。
Regularized Score Centering
正则化分数中心化
- Authors: Hao Bai, Chenlu Ye, Xuanchang Zhang, Aviral Kumar, Tong Zhang
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.21791
- Pdf link: https://arxiv.org/pdf/2611.21791
- Abstract
Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning. To stabilize off-policy learning, score centering (SC) removes the effect of reward baselines on the expected update. Our gradient diagnostics show less extreme gradients with SC than with importance sampling (IS), but SC can still fail when the learner moves far from the behavior policy that generated the data. To limit this policy mismatch, we introduce Regularized Score Centering (RSC), which adds a reverse Kullback-Leibler (KL) penalty to encourage the learner's next-token distribution to stay close to the behavior policy's next-token distribution. RSC works with or without truncated importance sampling (TIS), which caps token importance weights. Our theoretical analysis supports this approach, showing that KL regularization improves convergence guarantees. We evaluate dense and mixture-of-experts models on mathematical reasoning and agentic tasks, including online shopping, tool use, games, and fully asynchronous software engineering. RSC with TIS substantially improves sample efficiency across all agentic benchmarks, attaining higher training rewards with the same number of sampled responses. In final evaluations, it achieves top performance across all six agentic tasks, improving by 14.3% on AppWorld and reaching 9.9 times the strongest baseline's score on BabyAI.
- 中文摘要
异步强化学习提高了训练吞吐量,但来自陈旧策略的数据以及训练与推理之间的数值不匹配可能会破坏学习稳定性。为了稳定策略外学习,得分中心化(SC)消除了奖励基线对预期更新的影响。我们的梯度诊断显示,SC的梯度比使用重要性抽样(IS)时的极端梯度更小,但当学习者远离生成数据的行为策略时,SC仍然可能失败。为限制这种策略不匹配,我们引入了正则化分数中心化(RSC),它增加了反向库尔巴克-莱布勒(KL)惩罚,以鼓励学习者的下一代币分布接近行为策略的下一代币分布。RSC可配合或不使用截断重要性抽样(TIS),TIS对代币重要性权重有上限。我们的理论分析支持这一方法,表明KL正则化能改善收敛保证。我们评估了数学推理和代理任务(包括网购、工具使用、游戏以及完全异步软件工程)的密集模型和专家混合模型。带 TIS 的 RSC 在所有代理基准中显著提升了样本效率,在相同样本数量下获得更高的训练奖励。在最终评估中,它在所有六个代理任务中都表现出色,在 AppWorld 上提升了 14.3%,在 BabyAI 上达到了最强基线得分的 9.9 倍。
AR-WAM: efficient streaming world-action modeling for autonomous driving
AR-WAM:用于自动驾驶的高效流式世界行动建模
- Authors: Baiting Luo, Shatadal Mishra, Ahmadreza Moradipari, Nejib Ammar
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.21851
- Pdf link: https://arxiv.org/pdf/2611.21851
- Abstract
Autonomous driving requires anticipating how traffic will evolve while responding quickly to new observations. Video foundation models offer the first ability, but repeated history encoding and iterative video generation make them costly to use for driving. We present AR-WAM, an efficient streaming world-action model that combines autoregressive temporal modeling with single-pass future conditioning. Because its video backbone is causal, AR-WAM updates its visual context incrementally, reusing past computation and conditioning future prediction on observed motion. We further hypothesize that future features computed from recent observations can improve generalization in dynamic traffic without iterative video generation. AR-WAM therefore obtains these features in one forward pass and uses them to condition trajectory generation, and reinforcement learning then improves the policy through direct optimization of driving rewards. With a single front camera, AR-WAM achieves 91.5 PDMS after supervised training and 94.0 after reinforcement learning on NAVSIM-v1, outperforming the compared strong end-to-end world-action models. In a dynamic setting of continuous driving, supervised AR-WAM keeps its visual history across decisions and outperforms strong world-action baselines.
- 中文摘要
自动驾驶需要在快速响应新观测结果的同时,预判交通将如何演变。视频基础模型提供了首个能力,但由于重复历史编码和迭代视频生成,其在驾驶中使用成本较高。我们介绍了AR-WAM,一种高效的流式世界动作模型,结合了自回归时间建模与单次未来条件化。由于其视频骨干是因果的,AR-WAM会逐步更新视觉上下文,重用以往计算,并对观察到的运动进行预测。我们进一步假设,通过近期观测计算出的未来特征可以在无需迭代视频生成的情况下提升动态交通的泛化能力。因此,AR-WAM通过一次前向传递获得这些特征,并用它们来条件化轨迹生成,强化学习则通过直接优化驾驶奖励来改进策略。凭借单前置摄像头,AR-WAM在监督训练后实现91.5 PDMS,在NAVSIM-v1强化学习后达到94.0,优于与之相比的强力端到端世界行动模型。在连续驾驶的动态环境中,监督式 AR-WAM 在决策过程中保持其视觉历史,并优于强力的全球行动基线。
Adaptive Memory for Online Reinforcement Learning under Sparse Model Changes
在稀疏模型变更下的在线强化学习中的自适应记忆
- Authors: Sreejeet Maity, Sukanya Modak, Leonardo F. Toso
- Subjects: Subjects:
Machine Learning (cs.LG); Machine Learning (stat.ML)
- Arxiv link: https://arxiv.org/abs/2611.21879
- Pdf link: https://arxiv.org/pdf/2611.21879
- Abstract
We study the discounted reinforcement learning (RL) problem in a setting where the underlying environment changes between deployments. Each deployment provides $H$ observations from a fixed MDP. The reward function and transition kernel are fixed within each deployment. Across deployments, at most an $\epsilon >0$ fraction of the reward and transition kernel entries may change. In this setting, learning each model to deploy from scratch may discard useful observations from past deployments, whereas pooling all past observations can bias the estimates of entries that have already changed. We then propose an algorithm that for each state and action pair, selects how much history to reuse to estimate the reward and probabilities of the next state. We start with observations from the current deployment and add earlier deployment observations one at a time, the most recent first. After each addition, we check whether a single average reward agrees with the estimates from all histories considered until that point. We apply the same check to the distribution of next states. For each estimate, we leverage the last history that passes these aforementioned checks. The optimal control policy is then computed on top of the fitted MDP. We characterize the average gap between the optimal return and the return of the learner's policies over a finite number of deployments. We demonstrate that when $0< |S|\epsilon \leq 2$, with $|S|$ being the cardinality of the state space, our algorithm needs only the observations from the most recent $O(1/\epsilon)$ deployments, and achieves a long-run average gap of at most $\widetilde{O}(\sqrt{\epsilon/H})$.
- 中文摘要
我们在一个底层环境在不同部署之间发生变化的环境中研究了折现强化学习(RL)问题。每次部署都提供来自固定MDP的$H$观测值。奖励函数和过渡内核在每次部署中固定。在不同部署之间,奖励和过渡内核条目最多可能发生 $\epsilon >0% 的比例变化。在这种环境下,从零开始学习每个模型部署可能会丢弃过去部署中的有用观察值,而将所有过去观测数据汇总则可能偏向已变更条目的估计值。随后我们提出一个算法,针对每个状态和动作对,选择重用多少历史数据,以估算下一状态的奖励和概率。我们从当前部署的观察开始,逐一添加较早的部署观察值,先从最近的开始。每次增加后,我们检查单个平均奖励是否与迄今所有考虑的历史估计值一致。我们对下一个状态的分布应用相同的检查。对于每个估计值,我们利用通过上述检查的最后一次历史数据。然后在拟合后的MDP之上计算最优控制策略。我们描述了在有限次部署下,学习者策略的最佳回报与回报之间的平均差距。我们证明当 $0< |S|\epsilon \leq 2$,其中 $|S|$ 是状态空间的基数,我们的算法只需最近一次$O(1/\epsilon)$部署的观测值,长期平均差距最多为$\widetilde{O}(\sqrt{\epsilon/H})$。
World-Grounded Reinforcement Learning for Pollution-Aware Bird Movement Modeling Using a Multi-Instance DQN Framework
基于世界立足的强化学习,用于利用多实例DQN框架进行污染感知鸟类运动建模
- Authors: Shivin Mangal, Divyansh Chawla, Anshu Garg, Isshaan Singh
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.21895
- Pdf link: https://arxiv.org/pdf/2611.21895
- Abstract
Birds are modeled as environmentally sensitive agents whose movement reflects real-world air quality conditions, where AQI serves as a measurable world feedback signal representing pollution-driven consequences. Instead of relying on human preference or synthetic rewards, the system uses environmental outcomes as the primary training signal, enabling policy learning grounded in ecological impact. We propose a multi-instance reinforcement learning simulation in which agents navigate a 2D grid representation of India integrated with a precomputed AQI matrix and geographic state boundaries. The framework combines bird call metadata from the Xeno-canto dataset, spatial discretization, and real environmental data to learn adaptive navigation strategies under pollution constraints. To ensure scalability and efficiency, repeated real-time API queries are replaced by a structured AQI mapping that enables fast lookup during training and execution. Multiple baselines, including Monte Carlo, Q-Learning, and Value Iteration, are evaluated for comparison. Deep Q-Network demonstrates stronger stability, adaptability, and cumulative performance across varying environmental conditions, establishing a scalable foundation for world-feedback driven ecological decision modeling.
- 中文摘要
鸟类被建模为环境敏感的代理,其运动反映真实世界的空气质量状况,而AQI则作为可测量的世界反馈信号,代表污染驱动的后果。系统不依赖人类偏好或合成奖励,而是将环境结果作为主要训练信号,从而实现基于生态影响的政策学习。我们提出一种多实例强化学习模拟,在其中,代理在整合预先计算的AQI矩阵和地理州边界的印度二维网格中导航。该框架结合了来自Xeno-canto数据集的鸟鸣元数据、空间离散化和真实环境数据,在污染约束下学习自适应导航策略。为确保可扩展性和效率,重复的实时API查询被结构化的AQI映射所取代,从而在训练和执行过程中实现快速查找。对包括蒙特卡洛、Q-学习和值迭代在内的多个基线进行了评估以供比较。深度Q-Network在不同环境条件下展现出更强的稳定性、适应性和累积性能,为全球反馈驱动的生态决策建模奠定了可扩展的基础。
From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors
从人类视频到双手灵巧操作数据:基于信心引导的视频先验优化
- Authors: Yanghong Liu, Yang Chen, Kailang Xu, Bang Du, Feiyang Yang, Jiawei Xia, Zhengtao Yao, Yubiao Zhang, Masayoshi Tomizuka, Ruihai Wu, Wei Zhan
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.21908
- Pdf link: https://arxiv.org/pdf/2611.21908
- Abstract
Collecting demonstrations for bimanual dexterous manipulation is costly, motivating the use of human videos as a source of robot training data. However, uncertainty in video-derived motion estimates and differences between human and robot embodiments make direct motion transfer challenging. In this work, we introduce PriDex, a framework that transforms human-video Priors into executable robot Dexterous manipulation demonstrations through reinforcement learning in simulation. PriDex preserves task-relevant interaction structure while allowing embodiment-specific behavior to be regenerated for the robot. Human-object interactions provide affordance cues for selecting task-relevant, robot-specific grasps from geometry-based candidates. For motion refinement, object trajectories provide a task-level reference, while relative bimanual wrist trajectories offer complementary guidance for coordinated motion. Since object-motion estimates can become unreliable under occlusion or visual ambiguity, PriDex uses object-motion confidence to adaptively balance these two priors, emphasizing object-level guidance when confidence is high and increasing the contribution of wrist-level guidance as confidence decreases. Reinforcement learning refines these video-derived priors under the robot's kinematic and physical constraints, and the resulting policies are rolled out to generate robot demonstrations for training a vision-based diffusion policy. We evaluate PriDex on four bimanual dexterous manipulation tasks, including pouring, sweeping into a dustpan, bottle-cap unscrewing, and plate cleaning, assessing both simulation-based robot data generation and downstream visuomotor policy learning.
- 中文摘要
收集双手灵巧操作演示成本高昂,这促使人们将人类视频作为机器人训练数据的来源。然而,视频导出的运动估计不确定性以及人与机器人身体之间的差异使直接运动转移变得具有挑战性。在本研究中,我们介绍了PriDex,这是一个通过模拟强化学习将人-视频先验转换为可执行机器人灵巧操作演示的框架。PriDex 在允许为机器人重现具象特异行为的同时,保持与任务相关的交互结构。人与对象的互动为从几何候选对象中选择与任务相关、机器人特定的抓握提供了提示。对于运动细化,物体轨迹提供任务层级参考,而相对双手腕腕轨迹则为协调运动提供补充指导。由于物体运动估计在遮挡或视觉模糊时可能变得不可靠,PriDex 利用物体运动置信度在自适应性地平衡这两个先验,当置信度高时强调对象层面的引导,而当置信度下降时则增加手腕层面的引导。强化学习在机器人的运动学和物理约束下细化这些视频导出的先验,并推广出这些策略,生成用于训练基于视觉扩散策略的机器人演示。我们在四项双手灵活操作任务上评估了PriDex,包括倒酒、扫入簸箕、拧开瓶盖和清洗板子,同时评估基于模拟的机器人数据生成和下游视觉运动策略学习。
Safe Reinforcement Learning with Hard Constraints: Policy Gradient Methods with Intersample Guarantees
带有硬约束的安全强化学习:带有采样间保证的策略梯度方法
- Authors: Vipul K. Sharma, S. Sivaranjani
- Subjects: Subjects:
Systems and Control (eess.SY); Optimization and Control (math.OC)
- Arxiv link: https://arxiv.org/abs/2611.21987
- Pdf link: https://arxiv.org/pdf/2611.21987
- Abstract
We develop a policy-gradient (PG) reinforcement learning (RL) framework with hard inter-sample control barrier function (CBF)-based safety constraints for nonlinear sampled-data systems. We first design a class of stochastic policies with support truncated to sampling-period-dependent safe control sets, that guarantee safety-by-construction at sampling instants. We then propose a model-free PG algorithm that searches over the class of safe-by-design policies and establish sufficient conditions on sampling-period such that any action sampled from safe control sets remains safe throughout the sampling interval, providing inter-sample guarantees. We further establish finite-time convergence and sample complexity guarantees for the proposed PG algorithm and characterize its global optimality under gradient dominance. Finally, we illustrate inter-sample safety guarantees on a safe quadrotor navigation environment.
- 中文摘要
我们开发了一个基于硬样本间控制障碍函数(CBF)的安全约束的策略梯度(PG)强化学习(RL)框架,适用于非线性采样数据系统。我们首先设计了一类随机策略,其支持截断为采样周期依赖的安全控制集,保证采样瞬间的安全。随后,我们提出了一个无模型的 PG 算法,对安全设计策略类别进行搜索,并建立足够的采样周期条件,使得从安全控制集采样的任何动作在整个采样间隔期间保持安全,从而提供样本间保证。我们进一步为所提 PG 算法建立了有限时间收敛性和样本复杂度保证,并对其在梯度优势下的全局最优性进行了表征。最后,我们展示了在安全四旋翼导航环境中的样本间安全性保证。
An Explainable Agentic AI Framework for Distribution System Load Restoration
一个可解释的智能人工智能配电系统负载恢复框架
- Authors: S. M. Monzurul Hoque Chowdhury, Qinghua Li
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.22016
- Pdf link: https://arxiv.org/pdf/2611.22016
- Abstract
Distribution system load restoration is a sequential decision-making problem in which restoration actions must maximize served load, prioritize critical loads, and preserve power-flow feasibility. This paper proposes an explainable agentic artificial intelligence (AI) framework for distribution system load restoration. The proposed framework uses multiple large language model (LLM) agents for planner guidance, strategic plan generation, decision arbitration, memory-based feedback, and explanation generation. Unlike unconstrained LLM-based control, the proposed method restricts agent decisions to feasible restoration actions and evaluates all candidate plans using deterministic AC power-flow feasibility checks. The framework is evaluated on IEEE 33-bus and 69-bus distribution feeders, as well as a larger IEEE 118-bus test case, and compared against reinforcement learning and planning baselines. The results show that the proposed agentic framework achieves better restoration performance while producing structured explanations for operator-facing decision support. A direct comparison with a Single-LLM-Agent baseline further shows that the proposed multi-agent decomposition improves weighted restored load and critical-load recovery compared with a monolithic LLM decision structure.
- 中文摘要
配电系统负载恢复是一种顺序决策问题,其中恢复动作必须最大化服务负载、优先处理关键负载并保持电力流可行性。本文提出了一个可解释的智能人工智能(AI)框架用于配电系统负载恢复。该框架使用多个大型语言模型(LLM)代理进行规划指导、战略计划生成、决策仲裁、基于内存的反馈和解释生成。与无约束的基于LLM的控制不同,该方法将代理决策限制在可行的恢复行动上,并通过确定性交流电力流可行性检查评估所有候选方案。该框架在IEEE 33总线和69总线配电馈线以及更大规模的IEEE 118总线测试案例上进行评估,并与强化学习和规划基线进行比较。结果表明,所提代理框架在还原性能上表现更佳,同时为面向操作员的决策支持提供结构化解释。与单LLM代理基线直接比较进一步表明,所提多代理分解相比单体LLM决策结构,在加权恢复负载和临界负载恢复方面表现更好。
GP--RL--MPC: Uncertainty-Aware Reinforcement Learning-Augmented Model Predictive Control with Gaussian Process Postdecision State-Value Functions
GP--RL--MPC:不确定性感知强化学习增强模型预测控制,采用高斯过程后决策状态-值函数
- Authors: Jon Li, Jay H. Lee
- Subjects: Subjects:
Machine Learning (cs.LG); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.22038
- Pdf link: https://arxiv.org/pdf/2611.22038
- Abstract
We propose Gaussian Process Reinforcement Learning-Augmented Model Predictive Control (GP--RL--MPC), which uses a learned critic as the MPC terminal penalty to account for costs beyond the prediction horizon. Gaussian process regression (GPR) enables data-efficient critic learning from limited, noisy rollouts within optimistic approximate policy iteration (OAPI), with soft updates mitigating policy chattering. A tunable risk coefficient weights GP epistemic uncertainty in the terminal cost, regulating sensitivity to uncertain critic predictions. The postdecision state formulation shifts disturbance averaging offline to account for aleatoric uncertainty from the terminal transition onward. Under matching objectives and admissible actions, a one-step horizon recovers greedy value-space RL. After fitting GP hyperparameters by exact marginal likelihood, we construct a sparse deterministic training conditional (DTC) posterior using inducing points selected by pivoted Cholesky (PC), reducing online evaluation costs as the training dataset grows. In a continuous stirred-tank reactor (CSTR) subject to exogenous disturbances, GP--RL--MPC with an appropriately selected risk coefficient and policy screening reduces the sum of squared errors (SSE) by an average of 80.52% relative to an otherwise matched baseline MPC across five random seeds (sample standard deviation: 3.48%).
- 中文摘要
我们提出了高斯过程强化学习增强模型预测控制(GP--RL--MPC),它使用一个已学习的批评者作为MPC的终端惩罚,以考虑预测视界之外的成本。高斯过程回归(GPR)使得在乐观近似策略迭代(OAPI)内,通过有限且噪声大的展开实现数据高效的批评者学习,软性更新则减轻了策略抖动。可调风险系数将GP认知不确定性加权于终端成本,调节对不确定批评者预测的敏感度。决策后状态表述将扰动平均偏离,以考虑从终端过渡以后产生的偶然不确定性。在匹配目标和可接受行为下,一步视野恢复贪婪的价值空间强化学习。在按精确边际似然拟合 GP 超参数后,我们利用由枢轴 Cholesky(PC)选定的诱导点构建一个稀疏确定性训练条件(DTC)后验,随着训练数据集的增长降低在线评估成本。在连续搅拌罐(CSTR)中,受外源干扰影响的 GP-RL-MPC 配合适当选择的风险系数和策略筛选,相较于其他匹配的基线 MPC,在五个随机种子上平均将平方误差之和(SSE)降低 80.52%(样本标准差:3.48%)。
Addressing Exogenous Variability in Cooperative Multi-Agent Reinforcement Learning
解决合作多智能体强化学习中的外生变异性
- Authors: Seongmin Kim, Woohyeon Byeon, Jiwon Jeon, Seungyul Han, Youngchul Sung
- Subjects: Subjects:
Multiagent Systems (cs.MA); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.22067
- Pdf link: https://arxiv.org/pdf/2611.22067
- Abstract
Cooperative multi-agent reinforcement learning (MARL) often fails under exogenous variability such as shifts in opponents or environment regimes that cannot be controlled by the team but alter the transition dynamics. We formalize this challenge as Exogenous Dec-POMDP (ED-POMDP), in which an episode-level exogenous regime, selected independently of the team, indexes the dynamics, and the global state is partitioned into components that are directly responsive to the current team action and components that are not directly action-responsive at the current step. This formulation exposes mediated influence: while team actions do not directly update the latter at the current step, they can shape their later evolution through induced changes in the former. Based on this, we propose LEICA, a CTDE-compatible algorithm that learns history-conditioned context representations for both components and shapes policy updates using agent-wise, influence-weighted counterfactual intrinsic rewards. Across SMAX benchmarks with opponent strategy shifts, LEICA consistently improves both training performance and generalization to unseen opponents' strategies over existing baselines, supporting the usefulness of influence-weighted shaping under exogenous train-test regime shifts.
- 中文摘要
合作多智能体强化学习(MARL)常在外部变异性(如对手或环境环境变化等外生变异下失败,这些环境无法由团队控制,但会改变过渡动态)。我们将这一挑战形式化为外生 Dec-POMDP(ED-POMDP),其中一个独立于团队选择的集级外生状态对动态进行索引,并将全局状态划分为直接响应当前团队动作的组件和在当前步骤不直接响应动作的组件。这种表述揭示了中介影响:虽然团队行为在当前步骤不会直接更新后者,但可以通过诱发前者的变化来塑造其后续演变。基于此,我们提出了LEICA,一种兼容CTDE的算法,学习两组组件的历史条件上下文表示,并利用按智能体、影响力加权的反事实内在奖励塑造策略更新。在SMAX对手策略转变的基准测试中,LEICA持续提升训练性能和对未见对手策略的推广能力,支持在外生训练测试体制调整下影响力加权塑形的实用性。
When does deep reinforcement learning pay off in HVAC supervisory control? Four small Modelica case studies trained directly on FMUs
深度强化学习在暖通空调监督控制中何时能派上用场?四个直接在FMU上训练的小型Modelica案例研究
- Authors: Bingnan Zhao, Harshala Gammulle, Tharindu Fernando
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.22144
- Pdf link: https://arxiv.org/pdf/2611.22144
- Abstract
Deep reinforcement learning (DRL) is often proposed for the supervisory control of heating, ventilation and air-conditioning (HVAC) systems. The reported gains are, however, difficult to interpret, because the baselines are often weak and the attainable optimum is seldom known. This paper examines four small, reproducible case studies based on an office building in Brisbane, Australia: pre-cooling under a time-of-use tariff; chiller staging with condenser water reset; chilled water supply temperature reset with a wet cooling coil; and chilled water storage under time-of-use prices and a demand charge. The plants were modelled with the Modelica Buildings library, and in three of the cases a soft actor-critic agent was trained directly on the exported functional mock-up unit (FMU). Each agent was compared with tuned rule-based controllers and, where practicable, with a near-optimal reference from hourly exhaustive search or dynamic programming with perfect foresight. In the two cases without storage the agent reached the near-optimal reference, but the margin above a well-chosen rule was small: 1.3% for condenser water reset and 0.18% for chilled water reset. In the latter case two common reset rules consumed more energy than a fixed set point. With a chilled water storage tank the margin was considerably larger. The agent outperformed both rules (-7.8%, compared with -3.9% and -6.3%) and adopted midday charging during the solar off-peak period, yet it obtained only about 45% of the saving achieved by the perfect-foresight schedule and did little to limit the demand peak. In the pre-cooling case a hand-tuned rule remained superior (-12.0% compared with -5.6%). These results suggest that studies of DRL for HVAC control should state the margin between the best rule and a near-optimal reference. Some practical difficulties encountered in training on co-simulation FMUs are also reported.
- 中文摘要
深度强化学习(DRL)常被提出用于供暖、通风和空调(HVAC)系统的监督控制。然而,报告的收益难以解读,因为基线往往较弱,且几乎无法确定可达的最优。本文考察了基于澳大利亚布里斯班一座办公楼的四个小型、可重复的案例研究:按使用时间费率进行预冷;冷水机分段并重启冷凝器水;使用湿式冷却盘管重置冷水供水温度;以及按使用时间价格和需求收费进行冷水储存。这些工厂使用 Modelica Buildings 库建模,其中三个案例中,软演员-批评代理直接在导出的功能模型单元(FMU)上进行训练。每个代理都与调谐的基于规则的控制器进行比较,并在可行的情况下,采用每小时穷尽搜索或具备完美预见的动态规划所获得的近乎最优参考值。在这两种无储存的情况下,代理达到了近乎最优的参考值,但超出一个精心选择规则的余地很小:冷凝器水重置为1.3%,冷水重置为0.18%。在后一种情况下,两种常见的重置规则消耗的能量比固定设定点更高。在冷水储罐中,其余值明显更大。该代理在太阳能非高峰期采用了中午充电的表现优于这两种规则(-7.8%,相比之下为-3.9%和-6.3%),但其节能仅为完美预见计划所带来的约45%,对限制需求峰值作用有限。在预冷却情况下,手工调优规则依然优越(-12.0%,而-5.6%)。这些结果表明,在 HVAC 控制中的 DRL 研究应明确最佳规则与接近最优参考之间的差距。在使用协同模拟 FMU 的培训中也报告了一些实际困难。
What RL on Expert-Built Tasks Teaches Nemotron 3.5 Super
专家构建任务的RL教给Nemotron 3.5 Super的知识
- Authors: Sushant Mehta, Edwin Chen
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE)
- Arxiv link: https://arxiv.org/abs/2611.22161
- Pdf link: https://arxiv.org/pdf/2611.22161
- Abstract
We post-train NVIDIA's Nemotron 3.5 Super (120B total, 12B active parameters) with reinforcement learning (RL) on five expert-built task datasets, one at a time: real-world software engineering, complex constraints, long-horizon tool use, professional chart understanding and enterprise document reasoning. Each run trains a fresh rank-32 LoRA adapter with CISPO on Tinker, with no supervised stage, and each adapter is compared with the base model on the same tasks, harness and settings. On twelve headline benchmarks, eight built by other groups and four by Surge AI, the gains range from +4.5 points on IFBench to +13.8 on GDP_pdf, which rises from 10.4% to 24.2%; SWE-bench Pro V2 rises from 86.9% to 92.1%, Terminal-Bench 2.1 from 63.9% to 70.3%, MultiChallenge from 58.3% to 66.6%, AutomationBench from 14.8% to 27.5% and Chartography from 39.4% to 50.6%. The document adapter, trained without API tools, also gains +8.0 points on AutomationBench. In attempts read side by side with the base model's on tasks each adapter newly solves, the base model's failures recur in a few modes, and each adapter does more often what the base model missed: the tool-use adapter finds the data and the rules before acting, the coding adapter writes its own test or scratch files on SWE-bench Pro V2, the constraints adapter carries earlier turns into its answers, the chart adapter measures the marks in code, and the document adapter reads AutomationBench's spreadsheets.
- 中文摘要
我们对 NVIDIA 的 Nemotron 3.5 Super(总计 120B,12B 主动参数)进行后期训练,基于五个专家构建的任务数据集,逐一进行:真实软件工程、复杂约束、长期工具使用、专业图表理解和企业文档推理。每次运行都训练一个新的 32 级 LoRA 适配器,在 Tinker 上无监督阶段的 CISPO 系统,并将每个适配器与基础模型在相同任务、线束和设置上进行比较。在十二个主要基准测试中,其中八个由其他团队构建,四个由 Surge AI 构建,提升幅度从 IFBench 的 +4.5 点到 GDP_pdf 上的 +13.8 点,从 10.4% 上升到 24.2%;SWE-bench Pro V2 从 86.9% 提升到 92.1%,Terminal-Bench 2.1 从 63.9% 提升到 70.3%,MultiChallenge 从 58.3% 提升到 66.6%,AutomationBench 从 14.8% 提升到 27.5%,Chartography 从 39.4% 提升到 50.6%。文档适配器在未使用API工具的情况下训练,在AutomationBench上同样获得了+8.0点。在尝试与基础模型并排读取每个适配器新解决的任务时,基础模型的故障会在几种模式下反复出现,每个适配器更频繁地完成基础模型遗漏的任务:工具使用适配器在操作前查找数据和规则,编码适配器在 SWE-bench Pro V2 上编写自己的测试或临时文件,适配器之前携带的约束文件转化为答案,图表适配器测量代码中的标记,文档适配器读取 AutomationBench 的电子表格。
Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning
迈向基于模型的离线到在线强化学习中的乐观与悲观权衡
- Authors: Guochen Zhou, Yijun Yang, Qiqi Duan, Qing Su, Weiming Ou, Li Shen, Shihao Ji, Yuhui Shi, Zhisong Zhang
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.22172
- Pdf link: https://arxiv.org/pdf/2611.22172
- Abstract
Model-based offline-to-online reinforcement learning (RL) enables sample-efficient adaptation by leveraging offline pre-training for online fine-tuning. However, the distribution shifts between offline and online stages often hinder fine-tuning performance. Many existing methods encode the optimism-pessimism trade-off through a single-objective formulation. Selecting its trade-off coefficient through online fine-tuning can require costly bi-level optimization. We identify the balance established during offline training as a key challenge: in our motivating experiments, optimistic policies adapt better to novel tasks, whereas pessimistic policies perform better on the same online tasks. To address this challenge, we propose a bi-objective formulation that captures this trade-off, yielding a Pareto policy pool during offline training. This policy pool enables adaptive selection during online fine-tuning across diverse online tasks. To generate the pool, we introduce Multi-Objective Soft Actor-critIC (MOSAIC), which solves bi-objective problems and constructs diverse Pareto policies. After offline training, a contextual bandit algorithm hierarchically selects the most suitable policy for fine-tuning at each online interaction step. Empirically, our pipeline, Hierarchical Pareto Policy Pool (HiP3), achieves the strongest aggregate performance across both same and novel online tasks and maintains its advantage under stronger distribution shifts. Comprehensive ablations further clarify the roles of the policy pool, MOSAIC-based pool construction, and the online selection mechanism.
- 中文摘要
基于模型的离线到在线强化学习(RL)通过利用离线预训练进行在线微调,实现样本高效的适应。然而,离线与在线阶段之间的分布变化常常阻碍微调性能。许多现有方法通过单一目标表述编码乐观与悲观的权衡。通过在线微调选择其权衡系数可能需要代价高昂的双级优化。我们指出,离线训练中建立的平衡是一个关键挑战:在我们的激励实验中,乐观策略更能适应新任务,而悲观策略在相同在线任务上表现更好。为应对这一挑战,我们提出了一种双目标表述,捕捉这种权衡,在离线训练中生成帕累托策略池。该策略池在不同在线任务的在线微调过程中实现自适应选择。为生成该池,我们引入了多目标软演员批判IC(MOSAIC),该方法解决双目标问题并构建多样化的帕累托策略。经过离线训练后,上下文强盗算法在每个在线交互步骤中层级选择最合适的策略进行微调。从经验角度看,我们的流程——分层帕累托策略池(HiP3)在相同和新颖在线任务中实现了最强的综合性能,并在更强的分布变化下保持优势。全面的分解进一步明确了策略池、基于MOSAIC的池构建以及在线选择机制的作用。
Safely Mitigating Grid Imbalance: A Hybrid Distributional Wide-Margin Recourse Approach
安全缓解电网失衡:混合分配宽边际追索方法
- Authors: Shoaib Hussain, Mostafa Farrokhabadi, Hamidreza Zareipour
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.22220
- Pdf link: https://arxiv.org/pdf/2611.22220
- Abstract
This paper proposes the Multi-Agent Distributional Wide-Margin Policy Gradient (MADWMPG) framework: a dual-policy hybrid imitation and safe reinforcement learning (IL-SRL) method designed for safely mitigating grid imbalance in multiphase active distribution networks with distributed soft open points (SOPs). The framework comprises a wide-margin constraint barrier recourse (WCBR) module that seeks the maximal separation of constraint-compliant and unsafe actions and learns to proactively suppress exploratory violations. The framework addresses operational volatility using a novel adaptation of noncrossing quantile regression (NQR) distributional critics that track the complete distribution of policy returns. The set of behavior and recourse policies integrates an instantaneous physics-based and differentiable action projection mechanism that generates SOP constraint-compliant actions without relying on active learning. Policy safety is made responsive using an enhancement of conventional experience replay, and the safety performance of the framework is supported by probabilistic bounds derived in the paper. The proposed framework was trained and tested on a modified multiphase IEEE 123-bus test feeder. Results demonstrate its superiority to conventional reinforcement learning and SRL methods and show that it is highly competitive against the nonlinear AC optimal power flow.
- 中文摘要
本文提出了多智能体分布宽边际策略梯度(MADWMPG)框架:一种双策略混合模拟与安全强化学习(IL-SRL)方法,旨在安全缓解多阶段主动配电网络中分布式软开点(SOP)中的网格不平衡。该框架包含一个宽边距约束障碍追索(WCBR)模块,旨在最大限度地分离约束合规与不安全行为,并学习主动抑制探索性违规行为。该框架通过对非交叉分位数回归(NQR)分布批评器的创新适应,用于应对操作波动性,该批评点追踪策略结果的完整分布。该行为与资源策略集成了基于物理的瞬时可微动作预测机制,可在不依赖主动学习的情况下生成符合SOP约束的动作。通过增强传统经验回放,使政策安全性更具响应性,框架的安全性能则由论文中推导出的概率界限支持。所提出的框架在经过修改的 IEEE 123 总线测试馈源器上进行了训练和测试。结果表明其优于传统强化学习和SRL方法,并表明其在非线性交流最优功率流中具有高度竞争力。
From Temperature-Limited to Battery-Limited Legged Locomotion
从温度限制到电池限制的腿部移动
- Authors: Raphaël Ségas, Balázs Opra, Léo Ghafari, Thomas Stewart
- Subjects: Subjects:
Robotics (cs.RO); Machine Learning (cs.LG); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.22231
- Pdf link: https://arxiv.org/pdf/2611.22231
- Abstract
On legged robots, sustained locomotion often ends in thermal shutdown rather than battery depletion: the heat of walking concentrates in a few joint motors, which overheat long before the battery is empty. We train thermally aware locomotion policies by reinforcement learning without ever observing temperature, and demonstrate this on a Unitree Go2 quadruped. We first identify a coupled 13-node lumped RC thermal network, fitted on two runs of a single robot and calibrated against the measured electrical power, that predicts all twelve motor temperatures with a mean RMSE below 1 $^\circ$C across different robot instances, gaits, payloads, and speeds. Under a sustained gait, the temperatures the motors will settle at follow directly from the gait-averaged squared torques and joint speeds. Our core contribution is a reward that drives down the hottest predicted asymptotes: thermal time constants exceed the credit-assignment horizon of reinforcement learning by orders of magnitude, but predicted asymptotes respond to torque and joint speed within a single gait cycle, so the policy learns to lower and spread heat with no temperature input. On the robot, the policy turns sustained locomotion from temperature-limited to battery-limited: with a 5 kg payload at 1 m/s, a learned baseline reaches thermal shutdown after 19 minutes (1.1 km), whereas our policy runs the battery down after 174 minutes (10.0 km) with its hottest motor 14 $^\circ$C below the shutdown threshold in a 26 $^\circ$C room; on an unloaded factory circuit with stairs and ramps, the baseline overheats after 25 laps (6.2 km), while our policy completes 47 (11.2 km) on one battery with 22 $^\circ$C of margin.
- 中文摘要
在有腿机器人上,持续运动往往以热关机而非电池耗尽告终:行走时的热量集中在少数关节电机中,这些电机在电池耗尽前就已过热。我们通过强化学习训练热感知的运动策略,而无需观察温度,并在一台Unitree Go2四足机上进行了演示。我们首先识别出一个耦合的13节点集中RC热网络,安装在单一机器人的两次运行上,并根据测得的电力校准,能够预测所有12种电机温度,平均RMSE均低于1 $^\circ$C,适用于不同机器人实例、步态、有效载荷和速度。在持续步态下,电机稳定的温度直接基于步态平均的平方扭矩和关节速度。我们的核心贡献是一项奖励,能降低预测的最高温渐近点:热时间常数远远超过强化学习的学分分配时间范围,但预测的渐近线在单一步态周期内响应扭矩和关节速度,因此该策略学会在无温度输入的情况下降低并传播热量。在机器人上,策略将持续移动从温度限制转为电池限制:在5公斤有效载荷、速度1 m/s时,学习的基线在19分钟(1.1公里)后达到热关机;而我们的策略则在电池运行174分钟(10.0公里)后耗尽,且其最热电机低于关机阈值14 $^\circ$C,且在26 $^\circ$C的房间内;在无负载的工厂赛道上,有楼梯和坡道,基线在25圈(6.2公里)后过热,而我们的政策则是单电池完成47圈(11.2公里),剩余时间为22 $^\circ$C。
PCGRL for Complex Platformers: Synthesizing Playable and Diverse Levels via Inflated Envelopes and Forward Curricula
PCGRL适用于复杂平台游戏:通过膨胀的框架和前瞻性课程,综合可玩且多样化的关卡
- Authors: Di Wu
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22250
- Pdf link: https://arxiv.org/pdf/2611.22250
- Abstract
Procedural Content Generation via Reinforcement Learning (PCGRL) has emerged as a promising paradigm for automated game development. However, existing PCGRL frameworks still suffer from severe exploration bottlenecks and scaling limitations when faced with long-sequence generation tasks governed by strict rigid-body rules and non-linear kinematic boundaries (e.g., platformer constraints). Consequently, traditional approaches frequently fail to maintain reliable level solvability on large-scale canvases. To address these challenges, we present a physics-aware, curriculum-guided PCGRL framework. First, we construct an "Inflated Volume Envelope" mechanism, which provides a physics-constrained representation that incorporates continuous kinematic feasibility into the discrete state space of the MDP. Second, to alleviate the sparse reward challenge induced by extremely long decision sequences, we introduce intermediate state-transition feedback and a multi-stage forward extension curriculum learning framework, enabling informative value estimation and stable credit assignment over long-horizon trajectories. Finally, to resolve the issue where deterministic policies after convergence generate highly monotonous map paths with a lack of diversity, we activate high-entropy stochastic inference during the synthesis phase, regularized by a post-hoc parallel rejection filtering mechanism. Experimental results demonstrate that our framework significantly enhances structural diversity while maintaining high physical playability across multi-scale environments. Comparative evaluations across multi-scale canvases demonstrate that our framework maintains superior convergence stability and dimensional scalability, reliably synthesizing solvable levels where traditional PCGRL baselines degrade.
- 中文摘要
通过强化学习进行程序化内容生成(PCGRL)已成为自动化游戏开发的一种有前景的范式。然而,现有的PCGRL框架在面对受严格刚体规则和非线性运动学边界约束的长序列生成任务(如平台游戏约束)时,仍面临严重的探索瓶颈和扩展限制。因此,传统方法常常无法在大规模画布上保持可靠的层级可解性。为应对这些挑战,我们提出了一个具备物理意识、课程指导的PCGRL框架。首先,我们构建了一个“膨胀体积包络”机制,提供一种物理约束的表示方式,将连续的运动学可行性融入MDP的离散状态空间。其次,为缓解极长决策序列带来的稀疏奖励挑战,我们引入了中间状态转换反馈和多阶段前向延伸课程学习框架,实现了信息价值估计和长期轨迹上的稳定学分分配。最后,为解决收敛后确定性策略产生高度单调且缺乏多样性的映射路径的问题,我们在合成阶段激活高熵随机推断,并通过事后并行拒绝过滤机制进行正则化。实验结果表明,我们的框架在保持多尺度环境中高物理可玩性的同时,显著增强了结构多样性。跨多尺度画布的比较评估表明,我们的框架保持了卓越的收敛稳定性和维度扩展性,能够可靠合成传统PCGRL基线退化的可解层。
MCTS Is Counting the Wrong Thing: Learning Exploration Counts for LLM Reasoning
MCTS算错了东西:学习探索对LLM推理很重要
- Authors: Xuan Zhang, Zhijian Zhou, Zicheng Hu, Yan Feng, Hongyan Hao, Yuan Qi, Chao Qu
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22301
- Pdf link: https://arxiv.org/pdf/2611.22301
- Abstract
Monte Carlo tree search (MCTS) has powered landmark successes in reinforcement learning, yet its gains in LLM reasoning can remain modest despite substantial search effort. We identify an overlooked bottleneck: what MCTS counts. Standard UCT assigns independent visit counts to textual branches, allowing different formulations of the same reasoning strategy to repeatedly receive exploration bonuses. Our theoretical analysis shows that this count fragmentation can impair exploration efficiency, as quantified by increased cumulative regret. To address this problem, we introduce Learned Pseudo-Count MCTS (LPC-MCTS), which changes only the exploration statistic in UCT. A lightweight Coin-Flipping Network head on the policy's hidden representations estimates representation coverage, enabling related branches to share exploration evidence. Across 5 mathematical-reasoning and 7 search benchmarks, LPC-MCTS consistently outperforms strong MCTS baselines. Holding each LPC-MCTS-trained policy fixed, replacing standard MCTS counts at inference improves average mean@16 accuracy across 5 mathematical benchmarks by 3.79 percentage points on Qwen2.5-7B and 2.71 points on Qwen3-4B. Applying LPC-MCTS during both training and inference yields gains of 4.96 and 4.37 points over standard MCTS, respectively. Together, our theory and experiments show that changing what MCTS counts can substantially improve its exploration efficiency and reasoning performance.
- 中文摘要
蒙特卡洛树搜索(MCTS)在强化学习方面取得了里程碑式的成功,但尽管付出大量搜索努力,其在LLM推理方面的进步仍可能有限。我们指出了一个被忽视的瓶颈:MCTS计入的内容。标准UCT为文本分支分配独立访问次数,使同一推理策略的不同表述能够反复获得探索加成。我们的理论分析表明,这种计数碎片化会降低探索效率,这可通过累积遗憾的增加来量化。为解决这一问题,我们引入了学习性伪计数MCTS(LPC-MCTS),它仅改变UCT的探索统计量。在政策隐藏表征上的轻量级抛硬币网络负责人估算表征覆盖率,使相关分支能够共享探索证据。在5个数学推理基准和7个搜索基准中,LPC-MCTS始终优于强有力的MCTS基线。将每个LPC-MCTS训练策略固定,在推理中替换标准MCTS计数,Qwen2.5-7B中5个数学基准的平均mean@16准确率提升3.79个百分点,Qwen3-4B提升2.71个百分点。在训练和推理中同时应用LPC-MCTS,分别比标准MCTS提升4.96和4.37个百分点。我们的理论和实验共同表明,改变MCTS计数内容可以显著提升其探索效率和推理性能。
EviDARL: Evidence-Centric Dual-Agent Reasoning via Alternating Reinforcement Learning for Multi-Table Question Answering
EviDARL:通过交替强化学习实现多表问题答覆的以证据为中心的双代理推理
- Authors: Dunhan Li, Leilei Gan, Lirong Gao, Wenyan Fan, Yongwei Wang, Sheng Guo, Fei Wu
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22310
- Pdf link: https://arxiv.org/pdf/2611.22310
- Abstract
Despite advances in LLM-based table reasoning, most studies focus on single-table settings. Multi-table QA requires multi-hop reasoning over complex table relationships and evidence dispersed across long contexts, often without verifiable intermediate traces. To address these challenges, we introduce EviDA, an evidence-centric dual-agent framework comprising an Evidence Construction Agent and a Reasoning and Answering Agent. The Evidence Construction Agent generates and executes table-manipulation programs over complete tables to produce question-specific evidence and verifiable intermediate traces, simplifying subsequent reasoning. The Reasoning and Answering Agent then performs multi-hop reasoning over this evidence to derive final answers. We further introduce EviDARL, which combines agent-specific fine-grained rewards with parameter-efficient alternating reinforcement learning to improve collaboration between the agents. By optimizing each agent in turn while holding the other fixed, EviDARL enables bidirectional adaptation between evidence construction and answer reasoning. Experiments on MMQA and TQA-Bench demonstrate that EviDARL substantially outperforms competitive open-model and multi-table reasoning baselines under constrained model and training budgets.
- 中文摘要
尽管基于LLM的表格推理有所进展,大多数研究仍聚焦于单表场景。多表质检要求对复杂的表关系进行多跳推理,并将证据分散在长上下文中,往往没有可验证的中间痕迹。为应对这些挑战,我们引入了 EviDA,这是一个以证据为中心的双代理框架,由证据构建代理和推理与回答代理组成。证据构建代理在完整表格上生成并执行表格操作程序,生成针对问题的证据和可验证的中间痕迹,简化后续推理。推理与回答代理随后对这些证据进行多跳推理,以推导出最终答案。我们进一步引入了EviDARL,它结合了智能体专属的细粒度奖励与参数高效的交替强化学习,以提升代理之间的协作。通过依次优化每个代理,同时保持另一个固定,EviDARL 实现了证据构建与答案推理之间的双向适应。MMQA 和 TQA-Bench 的实验表明,在受限的模型和训练预算条件下,EviDARL 在竞争性的开放模型和多表推理基线上显著优于竞争性。
More Than a Score: Reinforcement Learning with Verbalized Feedback for Generative Recommendation
超越得分:通过口头反馈进行生成式推荐的强化学习
- Authors: Zipeng Chen, Xinyu Lin, Yuliang Sun, Sheng Zhang, Zeyu Wang, Xiaowen Shi, Junwei Pan, Chengguo Yin
- Subjects: Subjects:
Information Retrieval (cs.IR)
- Arxiv link: https://arxiv.org/abs/2611.22410
- Pdf link: https://arxiv.org/pdf/2611.22410
- Abstract
Large language models (LLMs) are increasingly used for generative recommendation, where recommended items are generated directly from users' interaction histories. To further improve recommendation quality, reinforcement learning (RL) has been applied to align these models with recommendation objectives. However, existing RL methods rely on scalar rewards to implicitly guide exploration. These rewards do not specify which aspects of a candidate conflict with the user's preferences or how the next candidate should differ. To address this gap, we propose VerRec, a verbalized feedback-guided RL framework for generative recommendation. Specifically, a user simulator generates personalized verbalized feedback based on the user's historical preferences and the candidate item profile. The policy uses this feedback to produce refined recommendations. Original and refined candidates form an expanded group and are jointly optimized with group relative policy optimization. The user simulator is only required during training and is removed at inference time. Experiments on three Amazon review datasets with two generative recommendation backbones demonstrate that VerRec improves recommendation accuracy and ranking quality. Further analyses show that verbalized guidance encourages exploration of more diverse candidates and helps the policy discover more rewarding recommendations during training.
- 中文摘要
大型语言模型(LLM)越来越多地被用于生成式推荐,即直接从用户的互动历史生成推荐项目。为进一步提升推荐质量,已应用强化学习(RL)将这些模型与推荐目标对齐。然而,现有的强化学习方法依赖标量奖励来隐式引导探索。这些奖励并未明确指出候选人的哪些方面与用户偏好相冲突,或下一个候选人应如何有所不同。为弥补这一空白,我们提出了VerRec,一种基于口头反馈引导的生成式推荐强化学习框架。具体来说,用户模拟器会根据用户的历史偏好和候选题目画像生成个性化的口头反馈。该政策利用这些反馈生成精细的推荐。原始候选人和精炼后候选人组成一个扩展的群体,并通过群体相对策略优化共同优化。用户模拟器仅在培训期间需要,推理时会被移除。在三个亚马逊评审数据集上,结合两个生成式推荐骨干的实验表明,VerRec 提高了推荐准确性和排名质量。进一步分析显示,口头指导鼓励探索更多样化的候选人,并帮助政策在培训期间发现更有价值的推荐。
Eyes in the Update? Learning Where Rewards Land in Multimodal Reasoning
更新中的眼睛?了解多模态推理中奖励的定位
- Authors: Lixuan He, Tianzhe Chu, Shulin Tian, Shikang Zheng, Feng Chen, Yi Ma
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22438
- Pdf link: https://arxiv.org/pdf/2611.22438
- Abstract
Multimodal post-training has moved from broad instruction tuning toward reinforcement learning for image and video reasoning, where outcome rewards can improve final-answer accuracy. Yet these rewards are assigned to complete generated trajectories, while visual evidence is used only at sparse and uneven positions inside the response. Recent work has begun to make multimodal RL more token-sensitive through grounding signals or external token-level weighting. We ask a more specific question: can the allocation of the same trajectory-level reward be inferred from the model's own multimodal computation, rather than from an auxiliary annotator, perception module, or process label? We propose Gaze-RL, a representation-driven policy optimization method that redistributes completion-level advantage through a latent reward landing field. The field is computed from internal cross-modal geometry and local predictive uncertainty, assigning larger update weight to positions that are both visually anchored and decision-bearing while remaining mean-one, so the total credit of each rollout is preserved and only its positionwise allocation changes. Gaze-RL requires no process supervision, auxiliary perception module, or architectural modification, and improves both Qwen2.5-VL and InternVL3.5 across a broad range of image and video benchmarks.
- 中文摘要
多模态后训练已从广泛指令调优转向用于图像和视频推理的强化学习,其中结果奖励可以提升最终答案的准确性。然而,这些奖励被分配给已生成的轨迹,而视觉证据仅在响应中稀疏且不均匀的位置使用。近期研究开始通过接地信号或外部代币级加权,使多模态强化学习更加敏感于代币。我们提出一个更具体的问题:同一轨迹级奖励的分配能否通过模型自身的多模态计算推断,而非通过辅助标注器、感知模块或过程标签?我们提出了 Gaze-RL,一种以表示为驱动的策略优化方法,通过潜在奖励着陆场重新分配完成层级的优势。该场基于内部跨模态几何和局部预测不确定性计算,赋予既具视觉锚定又具决策作用的位置更大的更新权重,同时保持均值为一,从而保持每次部署的总积分,仅其位置分配发生变化。Gaze-RL无需过程监督、辅助感知模块或架构修改,能够在广泛的图像和视频基准测试中提升Qwen2.5-VL和InternVL3.5的性能。
Autonomous Track Racing with a Commercially Available Humanoid: System Design and Real-World Deployment
使用商业化人形机器人进行自主赛道赛车:系统设计与实际部署
- Authors: Koki Inami, Takashi Sato, Hiroyuki Oka, Shota Takizawa, Yuya Takahashi, Amon Suzuki, Hayato Ono, Kohtaro Ihara, Yuki Noka, Shohei Matsugu
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.22445
- Pdf link: https://arxiv.org/pdf/2611.22445
- Abstract
Learning-based methods now let humanoids run robustly, and they have begun to compete in athletic events. Yet few reports describe a complete autonomous humanoid running system in an actual race, where the robot must follow a lane at a new venue, pace itself within hardware limits, and finish without intervention. We report such a system deployed on the Unitree G1, a commercially available general-purpose humanoid, in the 100 m, 400 m, and 1500 m events of the World Humanoid Robot Games 2026. The system integrates vision-based lane perception, path tracking, and a running policy trained by reinforcement learning with a human motion prior. To address the challenges of humanoid running, we stabilize head-camera images against body motion and use IMU-assisted leg odometry to bridge flight phases. To manage motor heating, we penalize hip-pitch torque during training, add heat-dissipation plates, and apply temperature-based speed limiting with event-specific pacing. The robot placed 13th of 74 in the 100 m, 10th of 44 in the 400 m first round before falling in the semifinal, and seventh of 35 in the 1500 m. In this deployment, thermal margin set the pace in every event, and the semifinal fall followed a lane-detection error.
- 中文摘要
基于学习的方法如今使类人机器人能够强健地奔跑,他们也开始参加体育赛事。然而,很少有报告描述在真实比赛中完全自主的人形跑步系统,机器人必须在新场地的一条车道上行驶,在硬件限制内控制节奏,并在不需干预的情况下完成比赛。我们报道了这样一套系统,已部署在Unitree G1——一款商业通用类人生物——在2026年世界类人机器人运动会的100米、400米和1500米项目中。该系统将基于视觉的车道感知、路径追踪以及通过强化学习训练的跑步策略与人类运动相结合。为应对类人跑步的挑战,我们通过头部摄像头图像与身体运动进行稳定,并利用IMU辅助腿部里程计来衔接飞行阶段。为管理运动发热,我们在训练时对髋部俯仰扭矩进行惩罚,添加散热板,并根据项目进行基于温度的速度限制。机器人在100米比赛中获得74人中的第13名,400米首轮44人中排名第10,半决赛中落败,1500米比赛中35人中排名第七。在这次部署中,热裕度决定了每个项目的节奏,半决赛的落败则因车道检测错误而降。
ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models
ReMaRL:学习在冻结扩散语言模型中解除和重新掩码
- Authors: Egor Shibaev, Danila Biktimirov, Ivan Klimov, Artem Makoian, Viacheslav Meshchaninov, Dmitry Vetrov
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.22576
- Pdf link: https://arxiv.org/pdf/2611.22576
- Abstract
Masked diffusion language models generate tokens in parallel, and the order in which tokens are unmasked affects final-answer accuracy. Several methods learn unmasking policies from task rewards, but these policies cannot remask earlier predictions. Remasking lets the model correct those predictions using newly generated context. Heuristic rules guide both unmasking and remasking, while supervised remaskers learn to detect token errors. Neither approach directly optimizes these decisions for final-answer correctness. To address this gap, we introduce ReMaRL (Remasking with Reinforcement Learning), a lightweight policy that learns which tokens to unmask, retain, or remask. We train this roughly 0.3M-parameter policy with Group Relative Policy Optimization (GRPO) from final-answer rewards while keeping the diffusion model frozen. With LLaDA-8B-Instruct, ReMaRL achieves the highest scores among the compared unmasking and remasking policies within the tested inference budgets of 4-32 model calls on math and 8-16 calls on code.
- 中文摘要
掩蔽扩散语言模型并行生成代币,而代币被解除掩码的顺序会影响最终答案的准确性。多种方法通过任务奖励学习解除掩码策略,但这些策略无法重新掩码早期预测。重新掩蔽让模型能够利用新生成的上下文修正这些预测。启发式规则指导卸罩和重新掩码,而监督式掩蔽器则学习检测标记错误。这两种方法都无法直接优化这些决策以保证最终答案的正确性。为弥补这一差距,我们引入了 ReMaRL(强化学习中的掩码),这是一种轻量级策略,学习哪些代币要解除掩码、保留或重新掩码。我们通过组相对策略优化(GRPO)从最终答案奖励训练这一约 30 万参数的策略,同时保持扩散模型冻结。借助LLaDA-8B-Instruct,ReMaRL在测试推理预算中,数学4-32次模型调用和代码8-16次调用的卸罩和重掩码策略中得分最高。
PACT: Plan-Action Consistency Training by Preserving Correct Plans and Repairing Actions in GUI Agents
PACT:通过保留正确的计划并修复 GUI 代理中的操作,实现计划与行动一致性训练
- Authors: Zhuohan Cai, Zhangxuan Gu, Runze Li, Beitong Zhou, Ming Wen, Hongbo Yu, Shuheng Shen, Changhua Meng
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.22591
- Pdf link: https://arxiv.org/pdf/2611.22591
- Abstract
GUI agents can formulate a correct immediate plan in their reasoning yet emit an action that fails to execute it, creating an execution gap. This recurring mismatch undermines reliable task completion, as a single execution error can derail a multi-step workflow. In outcome-supervised reinforcement learning, broadcasting a negative response-level advantage across all generated tokens risks penalizing valid planning together with erroneous execution. To address this credit-assignment conflict, we introduce PACT (Plan-Action Consistency Training), a training framework that combines reference-aligned plan protection and plan-aligned action repair without altering the original scalar action reward. Using annotated reference actions to identify execution gaps, PACT protects the verified plan by masking negative policy-gradient contributions across its token span. Keeping the generated reasoning unchanged as context, it then favors the reference action over the emitted erroneous action through a preference surrogate with stop-gradient pair weights. Compared with the outcome-only baseline, PACT raises AndroidWorld task success from 60.34% to 68.97% (+8.63 pp), while improving rule-based plan-action consistency on AndroidControl High from 83.71% to 89.74% (+6.03 pp). Adapting PACT to desktop agents also improves task success from 40.92% to 44.17% (+3.25 pp) on the full 369-task OSWorld benchmark. Together, these results establish PACT as an effective credit-assignment framework for bridging execution gaps and improving task completion across mobile and desktop GUI environments.
- 中文摘要
GUI 代理可能在推理中制定正确的即时计划,却发出未能执行的动作,从而造成执行缺口。这种反复出现的不匹配削弱了可靠的任务完成,因为一次执行错误就可能使多步骤工作流脱轨。在结果监督强化学习中,对所有生成的代币广播负的响应水平优势,有可能同时惩罚有效的规划和错误执行。为解决这种信用分配冲突,我们引入了 PACT(计划-行动一致性训练),这是一种结合参考对齐计划保护和计划对齐行动修复的培训框架,同时不改变原始标量动作奖励。通过注释引用动作识别执行缺口,PACT 通过掩盖负策略梯度贡献,保护已验证计划在其令牌跨度范围内。在保持生成推理作为上下文不变的情况下,它通过带有停止梯度对权重的偏好代理,优先选择参考动作而非发出的错误动作。与仅结果基线相比,PACT 将 AndroidWorld 任务成功率从 60.34% 提升到 68.97%(+8.63 pp),同时将 AndroidControl High 的基于规则的计划-行动一致性从 83.71% 提升到 89.74%(+6.03 pp)。将 PACT 适配到桌面代理后,在整个 369 个任务 OSWorld 基准测试中,任务成功率从 40.92% 提升到 44.17%(+3.25 pp)。综合来看,这些结果共同确立了 PACT 作为一个有效的积分分配框架,用于弥合执行差距、提升移动端和桌面 GUI 环境中的任务完成率。
Making the World React Realistically: A Closed-Loop Interactive World Model for Autonomous Driving
让世界真实地反应:一个闭环互动世界模型用于自动驾驶
- Authors: Jia Hu, Yang Chang, Panqu Wang, Zhijun Fu, Chang Liu, Changjian Yu, Jintao Lai
- Subjects: Subjects:
Robotics (cs.RO); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22691
- Pdf link: https://arxiv.org/pdf/2611.22691
- Abstract
World models (WMs) generate driving scenarios beyond recorded logs, enabling the training and evaluation of autonomous driving systems in unseen interactions. However, existing WMs struggle to ensure both behavioral realism and visual fidelity, since they overlook the relationship between vehicle interactions and the ego visual observations. We propose a closed-loop interactive WM that couples behavior modeling and visual rendering. First, imitation-learned background-vehicle (BV) policies are fine-tuned with multi-agent reinforcement learning in the closed loop against an end-to-end ego planner that reacts to the rendered images, so that their interaction behaviors match those in real logs. Second, the viewpoints exposed by these interactions guide a per-Gaussian diagnosis of each 3D Gaussian vehicle asset, which attenuates view-dependent colors on under-observed Gaussians and fills never-observed surfaces with generated Gaussians, while preserving the observed appearance. Third, signal lights are rendered according to the driving intentions of the vehicles, by an editor that learns the light states unobserved for a vehicle through counterfactual fine-tuning with a light-state classifier. On long-tail closed-loop scenarios, our method reduces the divergence from real-world interaction behaviors by 85% and improves novel-view vehicle PSNR by 2.0dB over the state-of-the-art WM. The significance of BV light signaling in WMs is shown, as the rendered signal lights conveying BV driving intentions indirectly impact ego driving behaviors in a quarter of the scenarios.
- 中文摘要
世界模型(WM)生成超越记录日志的驾驶场景,使自动驾驶系统能够在无形交互中进行训练和评估成为可能。然而,现有的 WM 难以同时保证行为真实性和视觉真实性,因为它们忽视了车辆交互与自我视觉观察之间的关系。我们提出了一种闭环交互式 WM,将行为建模与视觉渲染相结合。首先,在闭环中通过多智能体强化学习,对应端到端自我规划器对应渲染图像的自我规划器,对模仿学习后背景-车辆(BV)策略进行微调,使其交互行为与真实日志中保持一致。其次,这些交互所展现的视角指导每个三维高斯车辆资产的逐高斯诊断,该资产削弱了未被观察的高斯数据的视图依赖色彩,并在保持观察到的外观的同时,将从未观察到的表面填满生成的高斯。第三,信号灯由编辑器根据车辆的驾驶意图进行渲染,编辑器通过光状态分类器进行反事实微调,学习车辆未被观察到的光状态。在长尾闭环场景中,我们的方法将与现实世界交互行为的偏差减少了85%,并将新颖视野下的车辆PSNR比最先进的WM提升了2.0dB。在WM中,BV光信号的重要性得以体现,因为传达BV驾驶意图的信号灯在四分之一的场景中间接影响了自我驾驶行为。
Learning Accurate End-Effector Trajectory Tracking with Inertial Feedback for a Quadruped Manipulator
学习四足机械臂的准确末端执行器轨迹跟踪,并利用惯性反馈
- Authors: Dohyeon Kim, Seungyong Han, Yeongtae Jung
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.22695
- Pdf link: https://arxiv.org/pdf/2611.22695
- Abstract
Accurate end-effector trajectory tracking on quadruped manipulators is challenging because uncertainties in actuators, foot-ground contact, and geometric errors are propagated through the leg-arm kinematic chain. Detecting such errors requires sensing beyond joint encoders that can capture the effects of these uncertainties on the robot's motion, since joint encoders only provide measurements at the actuated joints and cannot fully observe mechanical errors such as backlash, dimensional inaccuracies, or structural deformation. This work addresses these challenges using a reinforcement learning framework with end-effector (EE) inertial feedback, targeting local high-accuracy manipulation while maintaining a nominal body pose and support configuration without intentional body motion or foot repositioning. The learned policy uses EE inertial measurements to compensate for tracking errors induced by uncertainties throughout the leg-arm system that cannot be fully inferred from proprioception alone. Simulation and real-world experiments on continuous EE trajectory tracking show overall improvements relative to nominal and proprioception-only baselines. In real-world experiments, the proposed policy reduces the MoCap-based RMSE by 17.3% and 28.0% relative to the nominal baseline under the two in-distribution (ID) command conditions, respectively, and by an additional 12.6% and 15.9% relative to the proprioception-only baseline. Furthermore, the trajectory-wise RMSE between FK-reconstructed and motion-capture-measured end-effector positions averaged 12.61+-2.15mm (mean+-standard deviation) across 24 controller-trajectory combinations, revealing a substantial discrepancy between nominal kinematic reconstruction and measured end-effector motion.
- 中文摘要
在四足操作手上精确追踪终端执行器轨迹具有挑战性,因为执行器、脚与地面接触以及几何误差的不确定性会通过腿臂运动链传递。检测此类误差需要超越关节编码器的感知能力,能够捕捉这些不确定性对机器人运动的影响,因为关节编码器仅在被驱动关节处提供测量,无法完全观察诸如反冲、尺寸不准确或结构变形等机械错误。本研究利用带有末端执行器(EE)惯性反馈的强化学习框架,针对局部高精度操作进行定位,同时保持名义上的身体姿势和支撑配置,无需有意运动或脚部重新定位。该策略利用EE惯性测量来补偿由整个腿-臂系统中因不确定性引起的追踪误差,而这些误差仅凭本体感觉无法完全推断。连续EE轨迹追踪的模拟和现实实验显示,相较于名义和仅本体感觉基线,整体上有改善。在真实实验中,拟议政策在两种分布内(ID)指令条件下,相较于名义基线分别降低了基于MoCap的RMSE17.3%和28.0%,相对于仅本体感觉基线则额外降低了12.6%和15.9%。此外,FK重建与动作捕捉测量的末端执行器位置之间的轨迹均方均值平均为12.61+-2.15毫米(平均+标准差),显示出名义运动学重建与测得末端执行器运动之间存在显著差异。
Better Rankers, Worse Retrievers: The Hidden Cost of RL Post-Training in Generative Recommendation
排名更佳,成绩较差的取回犬:生成式推荐训练后学习的隐性成本
- Authors: Yuheng Chen, Fan Zhang, Yixuan Xu, Chao Yang, Yang Xiang, Can Yang
- Subjects: Subjects:
Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22838
- Pdf link: https://arxiv.org/pdf/2611.22838
- Abstract
Generative recommenders decode the semantic ID of a user's next item with a language model, and post-training them with reinforcement learning (RL) after supervised fine-tuning (SFT) has become standard practice. The gains are mostly reported on the first ten results, yet in deployment such models hand hundreds of candidates to a downstream ranker or reward model. We show that RL post-training has a hidden cost: it improves accuracy at the top of the list (hit@K for small K) but, on all five datasets we study, lowers the recall of the candidate list (hit@K for K in the hundreds). The model becomes a better ranker and a worse retriever. Controlled experiments trace the loss to the negative advantages that RL assigns to the candidates it samples, the rollout. The root cause is that only one next item is logged per user, so RL treats every other candidate as wrong. Yet these candidates are the model's most plausible alternatives, and often the next items of other users with similar histories. Penalising them pushes down exactly what similar users choose next, and these items drop out of those users' candidate lists. This diagnosis points directly to a remedy: out-of-rollout anchoring, a single additional loss term that preserves the SFT model's ordering of the items outside the rollout. It keeps the gain at the top and restores recall to the SFT level on all five datasets, and it is effective under other RL configurations as well. To our knowledge, this is the first systematic study of this recall loss: we measure it, identify its cause, and show that, once the cause is known, a simple remedy suffices.
- 中文摘要
生成式推荐器用语言模型解码用户下一个项目的语义 ID,并在监督微调(SFT)后对其进行强化学习(RL)后进行后训练,已成为标准做法。这些提升主要集中在前十个结果上,但在部署时,这类模型会将数百名候选人交给下游排名或奖励模型。我们表明,RL后训练存在一个隐藏成本:它提高了列表顶部的准确性(小 K hit@K),但在我们研究的五个数据集中,候选列表的召回率降低(K 的回忆率hit@K以数百个)。模型成为更好的排名者,而更差的检索者。受控实验将损失追溯到RL赋予候选样本的负面优势——即推广。根本原因在于每个用户只记录一个下一个项目,因此RL将其他候选项目视为错误。然而,这些候选对象是模型中最合理的替代选项,且往往是其他有相似历史的用户的下一个项目。惩罚它们会让相似用户下一步选择的项目被压缩,这些项目从这些用户的候选列表中消失。这一诊断直接指向一种解决方案:推出后锚定,即一个额外的损失项,保持SFT模型对部署外项目的排序。它将增益保持在顶部,并在所有五个数据集上恢复到SFT水平,并且在其他强化学习配置下同样有效。据我们所知,这是首次对这种召回损失进行系统性研究:我们测量它,确定其原因,并证明一旦原因明确,只需简单的补救措施即可。
WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning Loops
瀑布式:采用进化奖励表述和自动化学习循环的自适应训练工作流程
- Authors: Eleftherios Triantafyllidis, Filippos Christianos, Zhibin Li, Bernd Bickel
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22854
- Pdf link: https://arxiv.org/pdf/2611.22854
- Abstract
Reward design remains a central bottleneck in reinforcement learning (RL), particularly in sparse-reward, long-horizon and partially observable settings requiring dependency chaining, memory, and deceptive affordance disambiguation. While recent foundation-model approaches reduce manual reward engineering, most assume goal-conditioning, task descriptions, expert demonstrations, or ground-truth metrics. These assumptions are problematic in partially observable environments, where task-relevant objectives, affordances, and dynamics must be discovered through interaction rather than disclosed a priori. We formalise this problem setting by introducing WATERFALL, an iterative population-based workflow for automated reward discovery under the strict observability constraints inherent in POMDP environments. WATERFALL achieves this by (i) synthesising diverse programmatic reward candidates via persona-conditioned generation, (ii) evaluating candidate behaviour from raw visual rollouts utilising a Swiss-system evolutionary tournament judged by vision-language models to identify elites, and (iii) leveraging longitudinal assessment histories to iteratively mutate, escalate, or prune candidates. We evaluate WATERFALL across fully observable and reveal-gated partially observable environments ranging from discrete to continuous, comparing against sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines. Our results indicate that WATERFALL discovers and refines reward programs without access to privileged information, outperforming baselines that retain their native information advantages, most clearly in the POMDP regime. Extensive ablations substantiate that visual evidence, tournament-based selection, and iterative refinement are jointly important. Enforcing this contract is what separates autonomous reward discovery from benchmark-privileged reward shaping.
- 中文摘要
奖励设计仍然是强化学习(RL)中的核心瓶颈,尤其是在稀疏奖励、长视野和部分可观测环境中,这些环境需要依赖链、记忆和欺骗性供给消除歧义。虽然最新的基础模型方法减少了人工奖励工程,但大多数假设目标条件、任务描述、专家演示或真实指标。这些假设在部分可观察环境中存在问题,因为任务相关的目标、可获得性和动态必须通过互动发现,而非事先披露。我们通过引入WATERFALL来形式化这一问题设定,这是一种基于群体的迭代工作流程,可在POMDP环境固有的严格可观察性约束下实现自动奖励发现。WATERFALL 通过(i)通过角色条件生成综合多样化的程序化奖励候选人,(ii)利用瑞士系统进化锦标赛,通过视觉语言模型评判的原始视觉推广评估候选人的行为,以及(iii)利用纵向评估历史,迭代变异、升级或修剪候选人。我们评估了WATERFALL在完全可观察和揭示门控的部分可观察环境中的表现,从离散到连续,并与稀疏奖励RL、内在动机和基础模型奖励合成基线进行比较。我们的结果表明,WATERFALL 能够在不获取特权信息的情况下发现并优化奖励项目,表现优于保留原生信息优势的基线,尤其是在 POMDP 体系中。广泛的消融证明了视觉证据、基于锦标赛的选择和迭代优化是共同重要的。执行这一契约正是将自主奖励发现与基于基准特权的奖励塑造区分开来的关键。
Outcome-Balanced PPO and Policy-Induced Flow Rectification for Flow-Based VLA Policies
基于流的 VLA 策略采用结果平衡的 PPO 和策略诱导的流整正
- Authors: Yuhan Wang, Ruiyang Wang, Jiaguan Zhu
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.22878
- Pdf link: https://arxiv.org/pdf/2611.22878
- Abstract
Reinforcement learning (RL) enables post-training of flow-based vision-language-action (VLA) policies by optimizing expected task return through environment interaction. However, this optimization raises two challenges: policy updates can alter the velocity-field geometry underlying the score approximation used for stochastic transitions, and sparse multi-task rollouts can yield uneven learning signals across tasks and advantage signs. We introduce an RL framework that combines Outcome-Balanced Proximal Policy Optimization (PPO) with policy-induced flow rectification. Outcome-Balanced PPO reweights positive advantages using task success-rate estimates and rescales positive and negative advantages while preserving their signs. Policy-induced rectification periodically fits a linear-bridge flow to ODE endpoint pairs generated by the current policy, refreshing the representation used in subsequent stochastic optimization. We evaluate the framework on simulated manipulation benchmarks and additionally study a component combination in a Real2Sim2Real task, examining task performance, optimization behavior, and flow consistency. The proposed framework improves success rates over the evaluated baselines across the simulated benchmarks.
- 中文摘要
强化学习(RL)通过通过环境交互优化预期任务回报,实现基于流程的视觉-语言-行动(VLA)策略的后期训练。然而,这种优化带来了两个挑战:策略更新可能改变用于随机转移的分数近似所依据的速度场几何结构,而稀疏的多任务展开则可能导致任务和优势符号间的学习信号不均。我们引入了一个强化学习框架,将结果平衡近端策略优化(PPO)与策略诱导的流修正相结合。结果平衡PPO利用任务成功率估计重新加权正优势,并在保持其符号的同时重新缩放正负优势。策略诱导的整正周期性拟合线性桥接流到由当前策略生成的ODE端点对,刷新后续随机优化所用的表示方式。我们在模拟操作基准测试上评估该框架,并研究Real2Sim2Real任务中的组件组合,考察任务性能、优化行为和流程一致性。所提出的框架在模拟基准测试中提高了成功率。
Learning Agent-Centric Vision-Language-Action World Models for Autonomous Driving
学习以代理为中心的视觉-语言-行动世界模型,用于自动驾驶
- Authors: Guoqing Wang, Pin Tang, Xiangxuan Ren, Chao Ma
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22880
- Pdf link: https://arxiv.org/pdf/2611.22880
- Abstract
Recent progress in integrating vision-language-action (VLA) models with world modeling has significantly advanced end-to-end autonomous driving by allowing systems to reason about imagined futures instead of merely reacting to current observations. However, most world VLA models generate future frames at the pixel level with plenty of dense visual tokens. Although such generated futures can be visually plausible, they preserve many driving-irrelevant appearance details, i.e., only a small portion of the tokens directly encode critical planning-relevant factors, such as agent geometry, motion dynamics, and interactions. This introduces a significant performance bottleneck. An important question thus arises: what world state should a world VLA model reason over to support planning beyond pixel-level prediction? We argue that planning-oriented world modeling should focus on a compact and structured agent-centric latent state that captures the interactions between objects in the driving scene over time. As such, we introduce AgentWorld, an agent-centric VLA world model that uses discrete agent tokens jointly for reasoning and trajectory planning. Each token is dynamically grounded to a surrounding agent associated with its object's state, such as semantic identity, 3D geometry, motion, and interaction context. AgentWorld learns these tokens through three key processes: explicit geometric grounding, reasoning-oriented supervised fine-tuning, and agent-oriented reinforcement learning. Extensive open-loop and closed-loop experiments under diverse driving scenarios show that AgentWorld enhances planning capabilities, fully exploits visual evidence and agent dynamics, and yields interpretable agent-centric world states.
- 中文摘要
近年来,视觉-语言-行动(VLA)模型与世界建模的集成显著推动了端到端自主驾驶的发展,使系统能够推理想象中的未来,而不仅仅是对当前观测做出反应。然而,大多数世界VLA模型在像素层面生成了带有大量密集视觉符号的未来帧。虽然这类生成的未来在视觉上可能合理,但它们保留了许多与驾驶无关的外观细节,也就是说,只有极小部分的代币直接编码了关键的规划相关因素,如代理几何形状、运动动力学和交互。这带来了一个显著的性能瓶颈。因此,一个重要问题出现了:一个世界VLA模型应考虑什么样的世界状态,以支持超越像素级预测的规划?我们认为,以规划为导向的世界建模应聚焦于一个紧凑且结构化的以代理为中心的潜在状态,以捕捉驾驶场景中物体间随时间的相互作用。因此,我们介绍了 AgentWorld,一种以代理为中心的 VLA 世界模型,结合使用离散代理令牌进行推理和轨迹规划。每个令牌动态地与其对象状态相关的周边代理建立基础,如语义身份、三维几何、运动和交互上下文。AgentWorld 通过三个关键过程学习这些令牌:显式几何基础化、基于推理的监督微调,以及面向智能体的强化学习。在多种驱动场景下进行的大量开环和闭环实验表明,AgentWorld 增强了规划能力,充分利用视觉证据和代理动态,并产生可解释的以代理为中心的世界状态。
Action-outcome coupling: a pre-investment feasibility audit for offline reinforcement learning in supply chain operations
行动-结果耦合:供应链运营中离线强化学习的投资前可行性审计
- Authors: Tung Dang, The Hung Phung, Son Lam Nguyen, Tu Nguyen
- Subjects: Subjects:
Machine Learning (cs.LG); Computational Engineering, Finance, and Science (cs.CE); Econometrics (econ.EM); Applications (stat.AP)
- Arxiv link: https://arxiv.org/abs/2611.22921
- Pdf link: https://arxiv.org/pdf/2611.22921
- Abstract
Prescriptive analytics initiatives in logistics, such as offline reinforcement learning and next-best-action systems, are often funded before anyone verifies that the available historical logs can support an improved decision policy. We formalise this pre-investment feasibility question via action-outcome coupling: a total-effect variance index measuring how much expected outcomes change with the logged action. Coupling provides an algorithm-independent upper bound on any policy's achievable value, expressed in monetary units per decision for both single-step and discounted sequential objectives. We also analyse the asymptotic false-licensing risk of an invalid workflow: fitting a value model and evaluating the model's greedy policy on the same data. The false-licence rate is driven mainly by the number of state cells over which selection is performed, rising from the nominal test level with one cell to near certainty with a few dozen, while overlap has a smaller effect. We then propose an audit that assigns logs to four tiers of decision support, separately for fixed and contextual policies, and returns an explicit stopping rationale. On a randomised e-commerce log, the audit endorses a fixed recommendation policy, certifying gains equal to 30 per cent of the theoretical ceiling; the remaining gap reflects bound looseness under an 80-action design. Two public logistics logs fail at different checkpoints: one because the target is an accounting identity of the action, and another because, with 27 clusters, estimated coupling stays below the design's detectability threshold even though a conservative offline learner certifies an improvement in 29 of 30 seeds.
- 中文摘要
物流领域的处方分析举措,如离线强化学习和次优行动系统,通常在有人验证可用历史日志能否支持改进决策政策之前就已获得资金支持。我们通过行动-结果耦合(Action -Outcome)形式化这一投资前可行性问题:一个总效应方差指数,衡量预期结果随记录动作变化的程度。耦合为任何策略的可实现价值提供一个与算法无关的上限,该值以每决策的货币单位表示,适用于单步和折现后的顺序目标。我们还分析了无效工作流的渐近虚假许可风险:拟合价值模型并评估该模型对同一数据的贪婪策略。虚假许可率主要由选择的状态单元格数量驱动,从标称测试水平的单个单元格增加到几乎确定的数十个单元格,而重叠效应较小。随后,我们提出一项审计方案,将日志分别分配给四个决策支持层级,分别针对固定策略和情境策略,并返回明确的停止理由。在随机化的电子商务日志上,审计认可了固定推荐政策,证明收益相当于理论上限的30%;剩余的差距反映了80个动作设计下的约束性松散。两个公开物流日志在不同的检查点失败:一个是因为目标是动作的会计身份,另一个是因为在27个集群中,即使保守的离线学习者在30个种子中有29个证明有所改善,估计耦合仍低于设计的可检测阈值。
When Do Typed Decision Models Work in Transportation? Evidence from Travel Choice and Traffic Signal Control
类型化决策模型在交通领域何时起作用?来自出行选择与交通信号控制的证据
- Authors: Hongyang Zhang, Boyuan Deng, Siyuan Feng, Kai Chen, Junbo Zhang
- Subjects: Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.22926
- Pdf link: https://arxiv.org/pdf/2611.22926
- Abstract
Typed decision models combine natural-language understanding with a structured interface that returns a selected action and an option-level probability distribution without generating lengthy textual responses. Their low inference cost and response latency make them attractive for transportation problems with predefined alternatives. This paper investigates their effectiveness, robustness, and efficiency in two representative transportation tasks: individual travel-mode choice prediction and sequential traffic signal control. We systematically evaluate TypeSafe Jev and GPT-6 Luna Decisions against conventional transportation methods, reinforcement learning algorithms, and four chat-based large language models. Experiments on the Swissmetro dataset show that typed decision models achieve lower prediction accuracy than traditional discrete choice models and the strongest chat-based baseline, while their performance varies substantially with input representation and question framing. In closed-loop traffic signal control experiments, typed decision models demonstrate competitive performance against established controllers in several scenarios, but exhibit performance degradation under more challenging network conditions. Furthermore, the typed models achieve lower inference latency and computational cost than the evaluated chat-based large language models, highlighting their potential for efficient, zero-shot transportation decision-making. Overall, these findings demonstrate both the potential and limitations of typed decision models, emphasizing the importance of input design, task characteristics, and robustness evaluation for their effective application in transportation systems.
- 中文摘要
类型决策模型结合了自然语言理解与结构化界面,返回所选动作和选项级概率分布,且不生成冗长文本回复。其低推断成本和响应延迟使其在具有预定义替代方案的交通问题中具有吸引力。本文探讨了它们在两种代表性交通任务中的有效性、稳健性和效率:个人出行方式选择预测和顺序交通信号控制。我们系统地将TypeSafe Jev和GPT-6 Luna决策与传统交通方法、强化学习算法以及四个基于聊天的大型语言模型进行比较。在Swissmetro数据集上的实验表明,类型决策模型的预测准确率低于传统离散选择模型和最强的基于聊天基线,而其性能随输入表示和问题框架变化显著。在闭环交通信号控制实验中,类型决策模型在多种场景下与成熟控制器竞争,但在更具挑战性的网络条件下表现下降。此外,类型模型相比基于聊天的大型语言模型实现了更低的推理延迟和计算成本,凸显了其在高效零样本交通决策方面的潜力。总体而言,这些发现展示了类型决策模型的潜力与局限性,强调输入设计、任务特性和稳健性评估对于其在交通系统中的有效应用至关重要。
Residual-Guided Lightweight Reinforcement Learning for Risk-Aware Trajectory Selection with a Frozen World Model
利用冻结世界模型实现风险感知轨迹选择的残差引导轻量级强化学习
- Authors: Jiaqi Wu, Yongqi Dong, Yijie Zhong, Maple Shao, Yide Tao, Linsen Cai, Weize Cai, Hai L. Vu, Alvaro Garcia Hernandez
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Emerging Technologies (cs.ET); Machine Learning (cs.LG); Robotics (cs.RO); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.22998
- Pdf link: https://arxiv.org/pdf/2611.22998
- Abstract
In automated driving, world models provide future scene predictions for evaluating candidate trajectories, but fixed selection rules may not capture scene-dependent trade-offs. We propose a residual-guided reinforcement learning (RL) framework for lightweight trajectory selection with a frozen world model and a fixed candidate bank. The framework uses proximal policy optimization (PPO) to fine-tune a joint policy over candidate selection and bounded trajectory residuals. A context-conditioned ranking network learns candidate preferences from feedback obtained by evaluating local perturbations of the selected trajectories. At inference, the selector combines learned preference and predicted risk ranks using fixed weights and returns the selected candidate trajectory without modification. Fine-tuning updates 16,106 parameters across the actor and critic, while the deployed selector retains only the 11,725-parameter ranking network, with the residual branch and critic used exclusively during training. In open-loop evaluation on the nuScenes validation set, the method reduces average Euclidean trajectory error (L2) by 3.99% relative to the RiskWorld model baseline without RL-based selection. Under matched budgets for feedback accesses and actor updates, residual exploration reduces L2 by 0.83% relative to the categorical PPO variant trained without residuals. The method also maintains lower L2 than RiskWorld when risk features are removed or shuffled.
- 中文摘要
在自动驾驶中,世界模型为评估候选轨迹提供未来场景预测,但固定选择规则可能无法捕捉场景依赖的权衡。我们提出了一个残差引导强化学习(RL)框架,用于轻量级轨迹选择,结合冻结世界模型和固定候选库。该框架利用近端策略优化(PPO)对候选选择和有界轨迹残差进行联合策略的微调。情境条件排序网络通过评估所选轨迹的局部扰动获得的反馈,学习候选偏好。在推理时,选择器使用固定权重将学习到的偏好和预测风险等级结合起来,返回所选候选轨迹而不作修改。微调会在演员和批评者之间更新 16,106 个参数,而部署的选择器仅保留 11,725 参数的排名网络,剩余分支和批评者仅在训练时使用。在 nuScenes 验证集的开环评估中,该方法相较于 RiskWorld 模型基线(不含基于 RL 选择)时,平均欧几里得轨迹误差(L2)降低了 3.99%。在反馈访问和演员更新的匹配预算下,残差探索相较于无残差训练的分类 PPO 变体,使 L2 降低 0.83%。当风险特征被移除或调整时,该方法还能保持低于 RiskWorld 的 L2。
M3OPD-Reasoner: An Open Recipe for Post-Training Multimodal Reasoning with Multi-Teacher On-Policy Distillation
M3OPD-推理器:多教师在策略上提炼的培训后多模态推理的开放配方
- Authors: Nguyen Quang Trung, Hoang Phan, Shuo Sun, Longyin Zhang, AiTi Aw
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.23034
- Pdf link: https://arxiv.org/pdf/2611.23034
- Abstract
Multi-Teacher On-Policy Distillation (MOPD) enables a single student to absorb the skills of several domain teachers by learning from its own generated responses. We present Multimodal Model Multi-Teacher On-Policy Distillation Reasoner (M3OPD-Reasoner), a fully open recipe for post-training vision-language models (VLMs) over three reasoning domains: image STEM, image perception, and video. We curate two datasets, M3OPD-SFT-1.35M for supervised fine-tuning (SFT) and M3OPD-RL-193K for reinforcement learning (RL) with verifiable rewards. On these data, we compare post-training recipes that start from Qwen3.5-9B: training one model on the mixed data of all domains (MixSFT and MixRL), and training one teacher per domain with Group Sequence Policy Optimization (GSPO), directly or after an SFT warm-up, and distilling the teachers into one student with MOPD. We evaluate on 21 image STEM, image perception, and video reasoning benchmarks after we decontaminate the training data against them. The comparison establishes MOPD of teachers trained with GSPO directly from Qwen3.5-9B as the best recipe: the student closely matches each teacher on its own domain and is on average above the mixed-training baselines on the same prompts. Since MOPD already merges the domain capabilities that MixRL would otherwise have to learn, its student is a stronger starting point for RL, and we ask whether the same data can carry the model further from there: we add MixRL on top of the MOPD student, on the same prompts and with the same settings, which, to our knowledge, no prior VLM recipe has studied. The same prompts still improve the MOPD student, with no collapse, and give our main model, M3OPD-Reasoner-9B. MOPD and MixRL can therefore work as consecutive stages that get more out of the same data. We release all data, the training code, and the checkpoints of every stage.
- 中文摘要
多教师策略提炼(MOPD)使单个学生通过自身生成的回答学习,吸收多位领域教师的技能。我们介绍多模态模型多教师策略提炼推理器(M3OPD-Reasoner),这是一套涵盖三个推理领域——图像STEM、图像感知和视频——的训练后视觉语言模型(VLMs)的完全开放配方。我们策划了两个数据集,M3OPD-SFT-1.35M用于监督式微调(SFT)和M3OPD-RL-193K用于带有可验证奖励的强化学习(RL)。基于这些数据,我们比较了从 Qwen3.5-9B 开始的培训后方案:在所有领域(MixSFT 和 MixRL)混合数据上训练一个模型,每个领域直接或在 SFT 热身后使用组序列策略优化(GSPO)培训一名教师,最后将教师提炼成一个学生,使用 MOPD。我们在对 21 个图像 STEM、图像感知和视频推理基准进行评估,并对其进行去污染后对应训练数据。比较结果确定,直接从 Qwen3.5-9B 使用 GSPO 培训的教师的 MOPD 是最佳配方:学生在各自领域与每位教师高度匹配,且在同一提示下平均高于混合培训基线。由于MOPD已经融合了MixRL本应学习的领域能力,其学生成为RL更强的起点,我们询问是否同样的数据还能进一步推动模型发展:我们在MOPD学生之上添加MixRL,使用相同的提示和设置,据我们所知,此前没有任何VLM配方研究过这些。相同的提示仍能提升MOPD学生,且不会出现崩溃,并给出我们的主模型M3OPD-Reasoner-9B。因此,MOPD和MixRL可以作为连续阶段工作,从同一数据中获得更多收益。我们发布所有数据、训练代码和每个阶段的检查点。
When Anchors Shift the Target: Advantage-Gated Adaptation for Diffusion Reinforcement Learning
当锚点转移目标时:优势门控适应用于扩散强化学习
- Authors: Pu Cao, Min Zhou, Ying Shu, Tiezheng Ge, Qing Song, Lu Yang
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.23038
- Pdf link: https://arxiv.org/pdf/2611.23038
- Abstract
Forward-process reinforcement learning adapts diffusion and flow-matching models by re-noising rewarded generations into regression targets. Because negative advantages induce concave updates with unbounded repulsive gradients, existing methods stabilize training using cached predictions from a moving policy copy, yet these anchors also alter what positive feedback teaches the model to fit. We formalize this conflict as anchor coupling: after separating frozen-reference regularizers, affine moving-anchor rules cannot eliminate negative curvature while preserving every positive regression target. We propose Advantage-Gated Adaptation (AGA) to decouple these roles. Positive samples retain standard target fitting, whose self-limiting update vanishes at the observed target independently of the moving anchor. Negative samples instead use the first-order tangent of the concave objective at the moving-policy prediction, preserving the local repulsion direction with a prediction-independent gradient. Within the quadratic-anchor family, these requirements uniquely determine the parameter-free gate $\omega(a)=\max(-a,0)$. Because the moving copy provides local linearization rather than retention, AGA assigns long-horizon behavior preservation to a frozen pretrained reference regularized at its operating classifier-free guidance scale, interpreted through an implicit-reward lens. Across compositional generation, text rendering, and preference alignment, our experiments show that displacing positive targets degrades visual quality and heightens sensitivity to policy-copy schedules, whereas AGA consistently improves the reward--quality trade-off.
- 中文摘要
前向过程强化学习通过将奖励生成重新噪点为回归目标,适应扩散和流匹配模型。由于负优势诱导凹更新且排斥梯度无界,现有方法利用来自移动策略副本的缓存预测来稳定训练,但这些锚点也改变了正反馈教导模型拟合的内容。我们将这种冲突形式化为锚耦合:在分离冻结参考正则化子后,仿射移动锚定规则无法在保持所有正回归目标的同时消除负曲率。我们提出优势门控适应(AGA)来解耦这些角色。正样本保留标准目标拟合,其自限更新在观察目标处独立于移动锚点消失。负样本则使用移动策略预测处凹目标的一阶正线,保持局部排斥方向且与预测无关梯度。在二次锚点族中,这些要求唯一决定了无参数门 $\omega(a)=\max(-a,0)$。由于移动副本提供局部线性化而非保留,AGA 将长期视界行为的保留赋予一个以运行无分类器指导尺度正则化的冻结预训练参考,并通过隐式奖励视角进行解释。在合成生成、文本渲染和偏好对齐等方面,我们的实验表明,位移正目标会降低视觉质量并增强对策略复制计划的敏感度,而 AGA 则持续改善了奖励-质量的权衡。
Deep Reinforcement Learning for Heat Pump and Underfloor Heating Control: An End-to-End Case Study
热泵与地暖控制的深度强化学习:端到端案例研究
- Authors: Antonio Manjavacas, David Suárez-Caro, Alejandro Campoy-Nieves, Guillermo Rey, Gonzalo Satrústegui, Miguel Molina-Solana
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.23063
- Pdf link: https://arxiv.org/pdf/2611.23063
- Abstract
Efficient control of underfloor heating (UFH) systems coupled with heat pumps (HPs) is challenging due to the high thermal inertia of buildings and the slow dynamic response of UFH systems. Deep Reinforcement Learning (DRL) offers a data-driven control approach capable of simultaneously optimizing energy consumption and thermal comfort. However, the application of DRL-based controllers in occupied buildings under real operating conditions remains limited. This work presents and end-to-end case study (development, evaluation, and real-world deployment) of a DRL-based controller for an integrated UFH-HP system. First, a simulation-based pilot study was conducted to assess different DRL algorithms, control strategies, and configurations. Subsequently, a digital twin of a real residential building was developed and used to train DRL agents under simulated operating conditions. The best-performing agent was then deployed in the physical building without additional online fine-tuning. The results show that the controller successfully maintained indoor temperatures within the prescribed comfort limits while dynamically adjusting the HP supply temperature and UFH valve positions. These findings demonstrate the feasibility of deploying DRL controllers in occupied buildings and highlight their robustness and the practical challenges associated with transferring control policies from digital environments to real heating systems.
- 中文摘要
由于建筑物热惯性较大且UFH系统动态响应缓慢,配合热泵(HP)进行高效控制具有挑战性。深度增强学习(DRL)提供一种数据驱动的控制方法,能够同时优化能耗和热舒适度。然而,基于DRL的控制器在实际操作条件下的建筑物应用仍然有限。本研究展示了基于DRL控制器的端到端案例研究(开发、评估及实际部署)用于集成UFH-HP系统的DRL控制器。首先,进行了基于仿真的试点研究,评估不同的DRL算法、控制策略和配置。随后,开发了真实住宅建筑的数字孪生,用于在模拟运行条件下训练DRL代理。性能最佳的代理随后部署到实体建筑中,无需额外在线微调。结果显示,控制器在动态调节 HP 供电温度和 UFH 阀门位置的同时,成功将室内温度维持在规定的舒适范围内。这些发现表明在有人居住的建筑中部署 DRL 控制器的可行性,并凸显了其坚固性以及将控制政策从数字环境转移到真实供暖系统所面临的实际挑战。
Learning to Undo: Transfer Reinforcement Learning under State Space Transformations
学习逆转:在状态空间转换下转移强化学习
- Authors: Mridul Mahajan, Aldo Pacchiano, Xuezhou Zhang
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.23081
- Pdf link: https://arxiv.org/pdf/2611.23081
- Abstract
Transfer learning in reinforcement learning (RL) has shown strong empirical success. In this work, we take a more principled perspective by studying when and how transferring knowledge between MDPs (a source and a target) can be provably beneficial. Specifically, we consider the case where there exists an undo map such that applying this map to the target's state space recovers the source exactly. We propose an algorithm that learns this map by matching state feature statistics gathered from both MDPs, and then uses it to transfer the source policy to the target. We theoretically justify the algorithm by analyzing the setting when the undo map is linear and the source is linearly-$Q^\star$ realizable, where our approach has strictly better sample complexity than tabula rasa RL in the target MDP. Empirically, we demonstrate that these benefits extend beyond this regime: on challenging continuous control tasks and Atari games, our method achieves significantly better sample efficiency. Overall, our results highlight how shared structure between tasks can be leveraged for efficient transfer of policies across environments.
- 中文摘要
强化学习(RL)中的迁移学习已展现出显著的实证成功。在本研究中,我们以更为原则性的视角研究在何时以及如何在MDP(源与目标)之间传递知识是可证明有益的。具体来说,我们考虑了这样一种情形:存在一个撤退映射,使得将该映射应用于目标的状态空间时,能够精确恢复源。我们提出了一种算法,通过匹配从两个MDP收集到的状态特征统计数据来学习该映射,然后利用它将源策略传递给目标。理论上,我们通过分析当撤销映射为线性且源$Q可实现时的设定来支持该算法,此时我们的方法在样本复杂度上严格优于目标MDP中的白纸化RL。通过实证,我们证明这些优势不仅限于这一模式:在具有挑战性的连续控制任务和Atari游戏中,我们的方法显著提升了样本效率。总体而言,我们的结果凸显了如何利用任务间的共享结构,实现跨环境高效策略转移。
CADENCE: Learning Acyclic Gaits for Model-Based Quadruped Locomotion
CADENCE:学习基于模型的四足行走的无环步态
- Authors: Owen Sullivan, Sourav Selvaraj, Mahdi Agheli
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.23102
- Pdf link: https://arxiv.org/pdf/2611.23102
- Abstract
Model-based quadruped controllers usually step on a cyclic gait, which lifts the legs in a fixed order on a fixed period, wherever the safe footholds are. We present CADENCE, a learned step planner with a small action space. Every 40 ms, it picks one of at most 65 discrete choices: no step, or one leg's step to a safe foothold. It also predicts a swing time. We train CADENCE by reinforcement learning on top of a convex model predictive controller (MPC). Nothing prescribes a gait: there is no gait schedule, and the reward has no gait term, only a cost per step. The one structural constraint is that at most two legs may swing at once. The policy we analyzed developed an acyclic gait, with no fixed period or stepping order, yet it still paired diagonal legs. An inverse footstep model (IFM), fitted on a controller's logged steps, turns each foothold into a command. With only the IFM refitted, on fewer than 20,000 steps, CADENCE ran frozen on OCS2's nonlinear MPC. In simulated trials on a range of terrains (26% to 49% voids) and speeds (0.1 m/s and 0.2 m/s), it succeeded in 85.1% (mean of five seeds), against 67.8% for OCS2-Trot, 65.5% for OCS2-Single, and 39.6% in a static walk ablation of CADENCE. At 49% voids, it succeeded in 72.9%, against 43.0% for OCS2-Trot.
- 中文摘要
基于模型的四足控制器通常会在循环步态上踩踏,这种步态在固定时间内以固定顺序抬起双腿,无论安全脚点所在。我们介绍CADENCE,一款拥有较小动作空间的学习式步幅规划器。每40毫秒,它从最多65个离散选项中选择一个:无步,或单条腿迈步到安全脚点。它还能预测摆动时间。我们通过在凸模型预测控制器(MPC)之上进行强化学习训练CADENCE。没有规定步态:没有步态计划,奖励没有步态项,只有每步的成本。唯一的结构性限制是最多两条腿可以同时摆动。我们分析的策略发展出一种无周期步态,没有固定周期或步伐顺序,但仍将对角腿配对。在控制器记录的步数上安装反向步步模型(IFM),将每个脚步点转化为命令。仅对IFM进行了重新调整,在少于2万步的情况下,CADENCE在OCS2的非线性MPC上运行时冻结。在模拟试验中,在不同地形(26%至49%空洞)和速度(0.1 m/s 和 0.2 m/s)上,成功率为85.1%(五种子的平均值),而OCS2-Trot为67.8%,OCS2-Single为65.5%,在静态CADENCE行走消融中为39.6%。在49%空隙时,成功率为72.9%,而OCS2-Trot为43.0%。
Get Hint-Augmented GRPO Done Right Through Activation Steering
通过激活引导,直接实现提示增强 GRPO 操作
- Authors: Khiem Le, Ting Hua, Nitesh V. Chawla
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.23124
- Pdf link: https://arxiv.org/pdf/2611.23124
- Abstract
GRPO has become the dominant reinforcement learning method for eliciting reasoning capability in LLMs using verifiable rewards. However, it suffers from a fundamental failure mode: when a training question exceeds the base model's current capability, all generated responses fail and receive zero reward, yielding no gradient signal and stalling further capability expansion. An emerging class of GRPO extensions, termed Hint-Augmented GRPO (HA-GRPO), adds a "hint" to the context to reduce the difficulty of the question, thereby producing a usable gradient signal. Nevertheless, existing HA-GRPO methods share a key issue: no hint is available at inference time to append to the question, creating a training-inference context mismatch that leads to inconsistent performance gains. In this work, we reinterpret the function of hinting -- not as reducing question difficulty, but as steering model behavior -- and propose ASH (Activation Steering for Hinting), a novel HA-GRPO method built on this view. During training, the model is presented with both hinted and plain questions, while a trainable steering vector is optimized to widen the margins between the model's generation probabilities for responses to the hinted version relative to the plain version, thereby capturing the question-agnostic component of the behavioral shift induced by the hint. During inference, the trained steering vector is directly injected into the model's activation, reproducing the hint-conditioned behavior without requiring any hint in the context. Experiments with four LLMs (Qwen3-1.7B/4B-Base, Llama-3.2-1B/3B-Instruct) show that ASH consistently improves pass@$k$ on competition-level mathematics benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves average pass@8 of Qwen3-1.7B-Base and Qwen3-4B-Base by \textbf{2.60}\% and \textbf{2.21}\%.
- 中文摘要
GRPO 已成为利用可验证奖励在 LLM 中诱导推理能力的主流强化学习方法。然而,它存在一个根本性的失败模式:当训练问题超过基模型当前能力时,所有生成的回答都会失败且无奖励,无法产生梯度信号,从而阻碍进一步的能力扩展。一种新兴的 GRPO 扩展称为提示增强 GRPO(HA-GRPO),会在上下文中添加“提示”以降低问题难度,从而产生可用的梯度信号。然而,现有的 HA-GRPO 方法存在一个关键问题:推理时无法提供提示以附加问题,导致训练与推理上下文不匹配,导致性能提升不一致。在本研究中,我们重新诠释提示的功能——不是降低问题难度,而是引导模型的行为——并提出了基于这一观点的新型HA-GRPO方法ASH(提示激活引导)。在训练过程中,模型同时呈现提示问题和普通问题,而可训练引导向量则优化以扩大模型对提示版本回答生成概率的差距,从而捕捉提示引发的行为转变中与问题无关的部分。在推理过程中,训练好的引导向量直接注入模型激活中,在上下文中无需提示即可再现提示条件行为。对四款大型语言模型(Qwen3-1.7B/4B-Base、Llama-3.2-1B/3B-Instruct)的实验表明,ASH在竞赛级数学基准(AMC、OlympiadBench、AIME24、AIME25)和非发行基准(Minerva、GPQA-Diamond)上持续提升pass@$k$。值得注意的是,它将Qwen3-1.7B-Base和Qwen3-4B-Base的平均pass@8提升幅度提升了\textbf{2.60}\%和\textbf{2.21}\%。
Learning Grounded Reasoning from Human Gaze
从人类凝视中学习扎根推理
- Authors: Gabriel Sarch, Declan Campbell, Nicholas Budny, Kia Ghods, Katerina Fragkiadaki, Jonathan D. Cohen, Brenden M. Lake, Taylor W. Webb
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.23209
- Pdf link: https://arxiv.org/pdf/2611.23209
- Abstract
Human visual reasoning is driven by systematic exploration and extraction of task-relevant visual evidence. Human gaze provides a window into this process, making gaze a useful source of guidance for visually-grounded reasoning in vision-language models (VLMs). However, human gaze also includes noise and redundancies that may transfer poorly to VLMs. In this work, we investigate how human gaze data can be effectively leveraged to improve visual reasoning in VLMs, while maintaining alignment with human behavior. We collect 7,109 human gaze trials from a diverse and challenging set of visual reasoning tasks, and train grounded reasoning models with different forms of gaze guidance. Across evaluations in chart understanding, visual search, and counting, we find that using human gaze as a prior for grounded reasoning consistently outperforms supervision using the full gaze sequence. Based on these findings, we introduce GGR (Gaze-Guided Reasoner), which achieves greater human alignment and more reliable visual grounding while matching or outperforming leading grounded-reasoning baselines in accuracy. Reinforcement learning further improves task accuracy, gaze alignment, and grounding quality despite using no gaze supervision. Behavioral analyses show that gaze supervised models exhibit more visual exploration before arriving at an answer. Our results suggest that the value of human gaze lies in guiding spatial exploration and shaping reasoning behavior without requiring complete imitation of human behavior.
- 中文摘要
人类视觉推理由系统性探索和提取与任务相关的视觉证据驱动。人类凝视为这一过程提供了一扇窗口,使凝视成为视觉语言模型(VLMs)中视觉基础推理的有用指导来源。然而,人类凝视也包含噪声和冗余,这些可能难以很好地转移到VLMs。本研究中,我们探讨如何在保持与人类行为对齐的前提下,有效利用人类凝视数据提升VLM中的视觉推理能力。我们收集了来自多样且具有挑战性的视觉推理任务的7,109项人类凝视试验,并训练了基于不同凝视引导形式的基础推理模型。在图表理解、视觉搜索和计数等评估中,我们发现,将人类凝视作为基础推理的先验,始终优于使用完整凝视序列的监督。基于这些发现,我们引入了GGR(凝视引导推理器),它实现了更高的人类对位和更可靠的视觉基础,同时在准确性上与领先的基础推理基线相匹配甚至优于。强化学习在不使用凝视监督的情况下,进一步提升任务准确性、凝视对位和扎根质量。行为分析显示,凝视监督模型在得出答案前展现出更多的视觉探索。我们的结果表明,人类凝视的价值在于引导空间探索和塑造推理行为,而无需完全模仿人类行为。
Keyword: diffusion policy
From Human Videos to Bimanual Dexterous Manipulation Data: Confidence-Guided RL Refinement of Video Priors
从人类视频到双手灵巧操作数据:基于信心引导的视频先验优化
- Authors: Yanghong Liu, Yang Chen, Kailang Xu, Bang Du, Feiyang Yang, Jiawei Xia, Zhengtao Yao, Yubiao Zhang, Masayoshi Tomizuka, Ruihai Wu, Wei Zhan
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.21908
- Pdf link: https://arxiv.org/pdf/2611.21908
- Abstract
Collecting demonstrations for bimanual dexterous manipulation is costly, motivating the use of human videos as a source of robot training data. However, uncertainty in video-derived motion estimates and differences between human and robot embodiments make direct motion transfer challenging. In this work, we introduce PriDex, a framework that transforms human-video Priors into executable robot Dexterous manipulation demonstrations through reinforcement learning in simulation. PriDex preserves task-relevant interaction structure while allowing embodiment-specific behavior to be regenerated for the robot. Human-object interactions provide affordance cues for selecting task-relevant, robot-specific grasps from geometry-based candidates. For motion refinement, object trajectories provide a task-level reference, while relative bimanual wrist trajectories offer complementary guidance for coordinated motion. Since object-motion estimates can become unreliable under occlusion or visual ambiguity, PriDex uses object-motion confidence to adaptively balance these two priors, emphasizing object-level guidance when confidence is high and increasing the contribution of wrist-level guidance as confidence decreases. Reinforcement learning refines these video-derived priors under the robot's kinematic and physical constraints, and the resulting policies are rolled out to generate robot demonstrations for training a vision-based diffusion policy. We evaluate PriDex on four bimanual dexterous manipulation tasks, including pouring, sweeping into a dustpan, bottle-cap unscrewing, and plate cleaning, assessing both simulation-based robot data generation and downstream visuomotor policy learning.
- 中文摘要
收集双手灵巧操作演示成本高昂,这促使人们将人类视频作为机器人训练数据的来源。然而,视频导出的运动估计不确定性以及人与机器人身体之间的差异使直接运动转移变得具有挑战性。在本研究中,我们介绍了PriDex,这是一个通过模拟强化学习将人-视频先验转换为可执行机器人灵巧操作演示的框架。PriDex 在允许为机器人重现具象特异行为的同时,保持与任务相关的交互结构。人与对象的互动为从几何候选对象中选择与任务相关、机器人特定的抓握提供了提示。对于运动细化,物体轨迹提供任务层级参考,而相对双手腕腕轨迹则为协调运动提供补充指导。由于物体运动估计在遮挡或视觉模糊时可能变得不可靠,PriDex 利用物体运动置信度在自适应性地平衡这两个先验,当置信度高时强调对象层面的引导,而当置信度下降时则增加手腕层面的引导。强化学习在机器人的运动学和物理约束下细化这些视频导出的先验,并推广出这些策略,生成用于训练基于视觉扩散策略的机器人演示。我们在四项双手灵活操作任务上评估了PriDex,包括倒酒、扫入簸箕、拧开瓶盖和清洗板子,同时评估基于模拟的机器人数据生成和下游视觉运动策略学习。
Body Schema Policy: Body as the Vision-Action Interface for Efficient Robot Loco-Manipulation
身体模式政策:身体作为高效机器人操作的视觉-动作接口
- Authors: Lechen Zhang, Chenyuan Chen, Muxin Zhou, Zewen He, Yoshihiko Nakamura
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.22713
- Pdf link: https://arxiv.org/pdf/2611.22713
- Abstract
Currently, robots can learn whole-body loco-manipulation through scale: the strongest policies have billions of parameters and still need task-specific fine-tuning. Humans have limited processing capacity, yet learn and act efficiently through a body schema that organizes what they see and how they move. We introduce Body Schema Policy (BSP), an imitation learning policy that builds such a schema into the vision-action interface. Its latent body schema places joint and link tokens on the robot's kinematic tree, where learned screw motions carry features between links. Through body-queried vision, each body part retrieves its own visual evidence, which conditions the denoising of its actions. With this inductive bias, BSP returns an action chunk faster than any policy we compare. Compact, body-specific visual readouts also emerge without spatial supervision: manipulation commands depend strongly on task objects, while locomotion commands respond more broadly to the scene. On nine simulated Unitree G1 whole-body tasks, BSP matches the $\Psi_0$ humanoid foundation model with $75\times$ fewer parameters and nearly doubles Diffusion Policy's success. The same holds on a real G1 and a Fetch mobile manipulator.
- 中文摘要
目前,机器人可以通过规模学习全身运动操作:最强的策略拥有数十亿参数,仍需针对任务进行微调。人类的处理能力有限,但通过一个组织他们所见和如何移动的身体图式,能够高效学习和行动。我们介绍了身体图式策略(BSP),这是一种模仿学习策略,将此类图式嵌入视觉-动作界面。其潜在的身体模式在机器人的运动学树上放置关节和链接标记,所学的螺丝运动在此树间传递特征。通过身体查询的视觉,每个身体部位获取自身的视觉证据,从而决定其动作的去噪。借助这种归纳偏向,BSP返回的动作块速度比我们比较的任何策略都快。紧凑、针对身体的视觉读数也会出现,无需空间监督:操作指令高度依赖任务对象,而运动指令则更广泛地响应场景。在九个模拟的 Unitree G1 全身任务中,BSP 以少 75 美元参数匹配了价值Psi_0$的人形基础模型,几乎将 Diffusion Policy 的成功率翻倍。真实的 G1 和 Fetch 移动机械臂同样适用。