生成时间: 2026-11-21 00:46:18 (UTC+8); Arxiv 发布时间: 2026-11-20 20:00 EST (2026-11-21 09:00 UTC+8)
今天共有 42 篇相关文章
Keyword: reinforcement learning
Forget Freely, Grep Later: Lossless, Index-Free Memory for Long-Horizon LLM Agents
Forget Freely, Grep Later:为长视野LLM代理提供无损、无索引内存
- Authors: Ziyan Yang, Taiwei Shi, Jieyu Zhao
- Subjects: Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25123
- Pdf link: https://arxiv.org/pdf/2611.25123
- Abstract
Long-horizon LLM agents often exceed their context windows, forcing them to discard information that may be needed later. Existing context management strategies rely on summarization, masking, or structured memory, requiring decisions about what to preserve or how to retrieve it before future needs are known. However, these methods can introduce lossy compression, potentially discarding details needed for future tasks. To address this limitation, we introduce GrepMem, a simple, lossless, and index-free memory mechanism built on two components: an append-only plain-text log of the agent's complete history and a regular-expression search tool for on-demand recall. Unlike structured memory systems, GrepMem requires no embeddings, retrieval index, memory extraction, or additional models. We train Qwen3.5-9B with reinforcement learning under five context management strategies and evaluate on multi-question research sessions, long-stream recall, and agent-trajectory question answering. GrepMem achieves 95-97% accuracy on ContextBench, versus at most 21% for lossy baselines, and outperforms SUPO's summarization by 8.8 percentage points on BCP-Multi. GrepMem scales effectively to increasingly long input streams while remaining on the accuracy-token-cost Pareto frontier across all five evaluation tasks. Our results show that a plain-text log and grep provide a simple but competitive approach to long-horizon agent memory, without requiring complex structured memory infrastructure.
- 中文摘要
长期视野的 LLM 代理往往超出其上下文窗口,迫使其丢弃可能日后需要的信息。现有的上下文管理策略依赖摘要、掩蔽或结构化内存,这需要在未来需求明朗之前做出决定,决定保留什么或如何检索。然而,这些方法可能会引入有损压缩,从而可能丢弃未来任务所需的细节。为解决这一限制,我们引入了 GrepMem,这是一种简单、无损且无索引的内存机制,基于两个组成部分:一个仅追加的纯文本日志,记录代理的完整历史,以及一个用于按需回忆的正则表达式搜索工具。与结构化记忆系统不同,GrepMem 不需要嵌入、检索索引、内存提取或额外模型。我们用五种上下文管理策略进行强化学习训练 Qwen3.5-9B,并评估多题研究、长流回忆和代理-轨迹问题解答。GrepMem 在 ContextBench 上准确率高达 95-97%,而在有损基线下最多只有 21%,并且在 BCP-Multi 上比 SUPO 的汇总高出 8.8 个百分点。GrepMem 能够有效扩展以适应越来越长的输入流,同时在所有五个评估任务中都保持在准确性、令牌成本、帕累托的前沿。我们的结果表明,明文日志和 grep 为长期代理内存提供了一种简单但具有竞争力的方法,无需复杂的结构化内存基础设施。
Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is
校准判决奖励:在一位了解其确定性有多大的法官那里训练语言模型
- Authors: Rudransh Agnihotri, Manasvi Kapoor
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.25135
- Pdf link: https://arxiv.org/pdf/2611.25135
- Abstract
Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.
- 中文摘要
强化学习(RL)只能教语言模型其奖励能识别的内容。答案检查器虽然精确,但需要参考答案;学习的奖励模型和 LLM 评判器在各处都能发挥作用,但它们的分数并非概率。我们研究了 RLCDR(校准决策奖励强化学习)。将一个 LLM 转化为决策模型,回答一个关于某个回答的打字问题(“最终答案是否正确?”“哪一步是第一个错误?”),并以概率分布方式进行一次正向读取,对期权顺序进行偏倚,并用大约一百个标签校准。在 QWEN3-1.7B 上使用 GRPO 进行 MATH 训练,这种无参考的奖励将MATH500准确率从 72.2% 提升到 76.8%(+4.65 分,p < 1e-4),匹配答案键的训练(76.9%),而随机奖励仅获得 +0.65。收益在困难层面上难以增长,从44.8%升至51.7%。在策略自身变化样本中,奖励保持校准状态:其预期校准误差是同一27B骨干网上软极大法官的一半(0.074对0.141),且未显示奖励被篡改的迹象。没有步进训练,同一模型发现第一个错误步骤时比专门的7B流程奖励模型更好(ProcessBench F1 0.772对0.722,20/20分割)。我们还绘制了方法不再足够用的问题:仅训练基础模型大多失败的问题,无参考的奖励仍提升策略(+1.9 MATH500点,p = 0.03),但由于概率对该分布过度自信,仍落后于答案键。我们提前提出六个假设,并用一个对照组来验证每个假设,用来反驳它们。
From Self-Play RL to Constrained Control: Distilling DiffMPC Students with DAgger
从自玩强化学习到受限控制:用DAgger提炼DiffMPC学生
- Authors: Luca M. Hartmann, Parinaz Naghizadeh
- Subjects: Subjects:
Robotics (cs.RO); Machine Learning (cs.LG); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.25197
- Pdf link: https://arxiv.org/pdf/2611.25197
- Abstract
Self-play reinforcement learning (RL) produces strong driving policies, but these policies are hard to verify or interpret. Differentiable model predictive control (DiffMPC) can enforce explicit constraints, but it is usually trained by imitating logged data or with costly RL. To add constraints to an already trained policy, we keep a PPO teacher's backbone and replace its action head with a DiffMPC that enforces road boundary half-planes and agent ellipses. We then distill the student with DAgger in the teacher's self-play simulator. Over 9 seeds, the constrained student of a smaller teacher matches its teacher on nuPlan and beats it on CARLA and on out-of-distribution Waymo scenes. For a stronger teacher, the student has fewer failures on nuPlan and Waymo, while staying level on CARLA. Its distillation takes no more than 30% of the teacher's training time.
- 中文摘要
自我游戏强化学习(RL)能生成强有力的驱动策略,但这些策略难以验证或解释。可微分模型预测控制(DiffMPC)可以强制执行显式约束,但通常通过模拟日志数据或使用昂贵的RL进行训练。为了给已训练好的策略增加约束,我们保留PPO教师的骨干,并用强制执行道路边界半平面和代理椭圆的DiffMPC替换其动作头。然后,我们在教师的自玩模拟器中用DAgger提炼学生。在9个种子中,受限的较小教师的学生在nuPlan上匹配其教师,并在CARLA和非分发的Waymo场景中击败对手。对于更强的教师,学生在nuPlan和Waymo上的失败次数更少,同时在CARLA上保持水平。其提炼时间不超过教师培训时间的30%。
ϕTD: Distributional Reinforcement Learning using Characteristic Functions
φTD:利用特征函数进行分布式强化学习
- Authors: Thomas Mousseau, Tyler Kastner, Davide Baldelli, Amir-massoud Farahmand
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.25208
- Pdf link: https://arxiv.org/pdf/2611.25208
- Abstract
Distributional Reinforcement Learning methods aim to learn the entire distribution of returns, yet current algorithmic approaches are limited to a narrow class of distributions. We propose $\varphi$TD, a method to overcome this limitation by computing the loss in the frequency domain. We demonstrate that this further unifies quantile and categorical approaches to distributional RL, and enables the learning of distributions that were previously impossible to capture, such as mixtures of continuous distributions. We empirically study $\varphi$TD, across synthetic benchmarks and large-scale deep RL environments and demonstrate that the flexibility obtained from wider parametric families can lead to improved performance.
- 中文摘要
分布强化学习方法旨在学习整个返回分布,但当前的算法方法仅限于一小类分布。我们提出了 $\varphi$TD,一种通过在频域计算损失来克服这一限制的方法。我们证明,这进一步统一了分位数和分类型分布RL的方法,并使得学习以前无法捕捉的分布成为可能,例如连续分布的混合。我们通过实证研究了 $\varphi$TD,跨综合基准测试和大规模深度强化学习环境,并证明更广泛的参数族所带来的灵活性能够提升性能。
Bridging vision-language reasoning and learned control for legged navigation in the wild
在野外以腿导航将视觉-语言推理与已学会控制相结合
- Authors: Takahiro Miki, Robin Schmid, Manthan Patel, Turcan Tuna, Julia Richter, Julian Nubert, Joonho Lee, Matthias Müller, Marco Hutter
- Subjects: Subjects:
Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.25215
- Pdf link: https://arxiv.org/pdf/2611.25215
- Abstract
Autonomous navigation in diverse real-world environments remains one of the greatest challenges in robotics. It demands both robust low-level motion control to handle diverse terrain and high-level reasoning to interpret environmental cues. We present ROAM (Robust Open-world Autonomous Mobility), a system that enables legged robots to perform kilometer-scale navigation in complex alpine and urban settings. ROAM combines the world knowledge of a Vision-Language Model (VLM), pre-trained on internet-scale data, with robust locomotion and local planning policies learned in simulation via reinforcement learning (RL). The VLM provides common-sense reasoning and semantic interpretation of the scene, assessing risks, interpreting signs, and producing semantic labels with associated navigation costs. An open-vocabulary segmentation module converts these labels into a dense, real-time risk map, which conditions two pre-trained RL policies responsible for agile locomotion and local navigation, effectively coupling high-level VLM reasoning with low-level physical control. Real-world experiments showed the system's ability to autonomously navigate a wide range of complex real-world environments. The robot completes kilometer-long hikes with minimal intervention, interprets signs to avoid danger, and evaluates its environment to identify safe areas on which to step. This work represents a step toward the reliable deployment of systems in unstructured environments. However, it also underscores the challenges that remain in VLM reliability and spatial reasoning, indicating that foundational models are a crucial component but not yet a complete solution for navigation in the wild.
- 中文摘要
在多样化的现实环境中实现自主导航仍然是机器人领域最大的挑战之一。它既需要强大的低级运动控制来应对多样地形,又需要高层次推理来解读环境线索。我们介绍了ROAM(稳健开放世界自主出行),这是一套系统,使有腿机器人能够在复杂的高山和城市环境中执行公里级导航。ROAM结合了基于互联网规模数据预训练的视觉-语言模型(VLM)的世界知识,以及通过强化学习(RL)在模拟中学习到的稳健的移动和局部规划策略。VLM提供常识推理和语义解读,包括评估风险、解读标志并生成语义标签及其相关的导航成本。一个开放词汇分割模块将这些标签转换为密集的实时风险地图,用于规范两种预训练的强化学习策略,负责敏捷移动和局部导航,有效将高级VLM推理与低层物理控制结合起来。真实实验显示,该系统能够自主导航各种复杂的现实环境。机器人能在极少干预的情况下完成数公里长的徒步,解读标志以避开危险,并评估环境以识别安全区域。这项工作代表了系统在非结构化环境中可靠部署的一步。然而,这也凸显了VLM可靠性和空间推理仍面临的挑战,表明基础模型是关键组成部分,但尚未成为野外导航的完整解决方案。
Ferrofluid Shape Control with Deep Reinforcement Learning
带深度增强学习的铁磁流体形状控制
- Authors: Anthony Czubarow, Antonio Terpin, Raffaello D'Andrea
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.25232
- Pdf link: https://arxiv.org/pdf/2611.25232
- Abstract
Ferrofluids are liquids containing magnetic nanoparticles, enabling shape control through external magnetic fields. This deformability supports their use as soft robotic bodies that can navigate confined spaces and manipulate objects. Their nonlinear continuum dynamics, however, make reliable control challenging. Existing work shows how spatial magnetic fields from electromagnet arrays can establish and maintain desired ferrofluid configurations. A complementary objective is to exploit the fluid's dynamics under limited actuation to reach prescribed shapes that may not be sustainable at equilibrium. Here, we use deep model-based reinforcement learning to control these transient shapes. Learning directly through physical interaction, our policies select current commands for a single fixed electromagnet based on visual observations, guiding the moving ferrofluid toward prescribed geometries. The learned policies repeatedly attain a variety of shapes spanning polygons, asterisks and pentominoes. These results demonstrate that learning directly from physical interaction enables reliable fluid shaping, opening new possibilities for controlling ferrofluid-based soft robots.
- 中文摘要
铁磁流体是含有磁性纳米粒子的液体,使得通过外部磁场进行形状控制成为可能。这种可变形性支持它们作为软机器人身体的使用,能够在狭小空间中导航并操控物体。然而,它们的非线性连续动力学使得可靠的控制变得具有挑战性。现有研究表明,电磁阵列产生的空间磁场如何建立并维持所需的铁磁流体构型。一个补充目标是在有限驱动下利用流体动力学,达到可能无法在平衡状态下可持续的规定形状。在这里,我们利用基于深度模型的强化学习来控制这些瞬态形状。通过物理交互直接学习,我们的策略基于视觉观察为单个固定电磁铁选择当前指令,引导运动铁磁流体朝着规定的几何形状移动。所学策略反复实现了涵盖多边形、星号和五米骨牌的多种形状。这些结果表明,直接从物理交互中学习能够实现可靠的流体塑形,为控制基于铁磁流体的软机器人打开了新可能。
WEVO: World Model-Enabled VLA Optimization for Whole-Body Loco-Manipulation
WEVO:用于全身运动操控的全球模型驱动VLA优化
- Authors: Qianzhong Chen, Muyuan Lin, Yuyin Sun, Mac Schwager, Cheng-Hao Kuo, Roberto Martín-Martín
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.25251
- Pdf link: https://arxiv.org/pdf/2611.25251
- Abstract
Reinforcement learning (RL) can improve vision-language-action (VLA) policies beyond imitation, but humanoid exploration is unsafe and simulator RL can exploit residual sim-to-real gaps. We present WEVO, a teleoperation-free framework for post-training a whole-body humanoid VLA in an action-conditioned video world model (WM). Simulated manipulation and locomotion play data improve contact-dynamics and viewpoint-change prediction. A VLA trained on synthetic demonstrations collects 20 autonomous real robot rollouts per task to align the WM with deployment observations. The VLA, whole-body controller (WBC), and video WM then form a closed-loop RL environment with dense progress rewards. Across three real robot tasks, play data improves every evaluated WM metric and downstream policy success; video WM RL raises mean success from 23.3% to 75.0%, compared with 48.3% for real robot RL. We therefore leverage the complementary strengths of both environments: simulation provides policy initialization and broad coverage, while the video WM offers a deployment-aligned environment for policy improvement. Project website: this https URL.
- 中文摘要
强化学习(RL)可以超越模仿地改进视觉-语言-行动(VLA)策略,但类人探索存在安全性,模拟器RL可能利用模拟与现实之间的残余差距。我们介绍了WEVO,这是一个无远程操作的框架,用于在动作条件视频世界模型(WM)中对全身类人生物VLA进行后期训练。模拟操作和运动游玩数据提升了接触动力学和视点变化预测。基于合成演示训练的VLA会收集每个任务20次自主的真实机器人部署,以使WM与部署观察结果对齐。VLA、全身控制器(WBC)和视频WM随后形成一个闭环RL环境,带来高强度的进步奖励。在三个真实机器人任务中,游戏数据提升了所有评估的WM指标及后续策略成功率;视频 WM RL 的平均成功率从 23.3% 提升到 75.0%,而真实机器人 RL 为 48.3%。因此,我们充分利用了两种环境的互补优势:仿真提供策略初始化和广泛覆盖,而视频 WM 则提供与部署对齐的环境以促进策略改进。项目网站:此 https URL。
Data-driven Linearization of Operational Dynamics of Soft Open Points in Power Distribution Networks
数据驱动的电力配电网络软开放点运营动态线性化
- Authors: Shoaib Hussain, Mostafa Farrokhabadi, Hamidreza Zareipour
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.25307
- Pdf link: https://arxiv.org/pdf/2611.25307
- Abstract
This paper proposes a novel data-driven linearization of operational power losses in a Soft Open Point (SOP). The proposed learning-based representation, combined with the standard segmentation of the quadratic device capacity inequalities, linearizes the operational dynamics of SOPs in unbalanced power distribution network applications without a significant drop in power loss estimation accuracy. The technique was tested in three applications using a three-phase extension of the IEEE 33-bus network. The case studies include a sizing application for a multi-terminal phase-changing soft open point (PCSOP), an operational optimization application for an active distribution network featuring a PCSOP, and a system simulation engine built for a Deep Reinforcement Learning (DRL) application that trains an agent to optimally dispatch a PCSOP in an unbalanced distribution network. The results of the three case studies were validated against the standard nonlinear AC load-flow representations and point to the efficacy of the proposed solution.
- 中文摘要
本文提出了一种基于数据驱动的软开路点(SOP)运行功率损耗线性化方法。该基于学习的表示方法结合对二次型设备容量不等式的标准分割,使不平衡配电网络应用中的SOP运行动态线性化,且功率损失估计精度不显著下降。该技术在三个应用中使用IEEE 33总线网络的三相扩展进行了测试。案例研究包括多端相变软开点(PCSOP)的尺寸测定应用、基于PCSOP的主动配电网络的运行优化应用,以及为深度强化学习(DRL)应用构建的系统仿真引擎,该应用训练代理在不平衡配电网络中最优调度PCSOP。三个案例研究的结果经过与标准非线性交流负载流表示的验证,表明所提方案的有效性。
Improving Imitation Policies for Multi-Phase Tasks through Residual Learning Allocation
通过剩余学习分配改进多阶段任务的模仿政策
- Authors: Clinton Enwerem, Saleh Kemal, John S. Baras, Calin Belta
- Subjects: Subjects:
Machine Learning (cs.LG); Robotics (cs.RO); Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.25320
- Pdf link: https://arxiv.org/pdf/2611.25320
- Abstract
Residual reinforcement learning corrects an imitation-trained base policy using a limited budget of environment interaction. In a task with several phases, this budget must be divided among the phases, and the phase in which the base policy fails is not necessarily the phase in which a correction most improves task completion. We formalize this distinction through the completion gain: the change in full-task completion probability when, during a single phase, a state-feedback controller replaces selected components of the base policy's action. We decompose the completion gain into the probability of reaching the phase and the conditional improvement in completion from the start of the phase, and we estimate it from paired episodes. The positive estimates determine which action components the residual policy corrects and how training episodes are distributed across phases. In stochastic point-mass docking experiments, most failures occurred in the final docking phase, whereas the largest completion gain occurred in the preceding channel phase; gain-based allocation improved completion over allocation proportional to failure counts and, in a variant with a longer approach, over uniform allocation at half of the budget. In simulated bimanual robotic assembly, the completion-gain estimates were positive only for insertion and release, although few failures occurred during release; in a second assembly configuration, a residual policy trained in the release phase improved completion from held-out states at the start of release by 16.9%.
- 中文摘要
残余强化学习通过有限的环境交互预算修正了模仿训练的基础策略。在一个包含多个阶段的任务中,这个预算必须在多个阶段之间分配,而基础策略失败的阶段不一定是修正最能提升任务完成率的阶段。我们通过完成收益来形式化这种区分:即在单一阶段,状态反馈控制器替换基础策略动作中的选定组成部分时,完成任务完成概率的变化。我们将完成增益分解为达到该阶段的概率以及从阶段开始时的条件性完成改善,并从成对的阶段进行估算。正估计决定了残余策略纠正的动作组成部分以及训练阶段在不同阶段的分布。在随机点质量对接实验中,大多数失败发生在最终对接阶段,而完成增益最大的发生在前一个通道阶段;基于收益的分配使完成度与故障次数成正比分配,在更长方法的变体中,比预算一半的均匀分配更高效。在模拟双手机器人装配中,完成-增益估计仅在插入和释放时为正,尽管释放过程中故障较少发生;在第二种装配配置中,释放阶段训练的残余策略使完成度从释放开始时的未完成状态提升了16.9%。
Beyond Weight Updates: Evolving Agent Harnesses for Domain Specialization
超越权重更新:域专化的代理工具演进
- Authors: BoHao Chen
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25380
- Pdf link: https://arxiv.org/pdf/2611.25380
- Abstract
Domain specialization of LLM agents is usually framed as a weight-update problem: collect trajectories, run reinforcement learning (RL) or supervised fine-tuning (SFT), and deploy a new checkpoint. We study a complementary axis: evolving the \textbf{agent harness}---the system prompt, tool schemas, skill library, and planning scaffolding---while keeping the policy frozen. We cast harness evolution as an RL-style outer loop and instantiate it with GEPA, a reflective evolutionary optimizer. The resulting harness adapters require no GPU training, are loadable per vertical domain, and compose with weight-based RL. Experiments with Codev, a Claude Code-style coding agent, on SWE-bench Verified, OfficeBench, and Terminal-Bench show a consistent pattern: weight updates give the strongest single-path gains, harness evolution recovers most of those gains at a small fraction of the compute, and combining both paths is best overall. We also analyze a failure mode of SFT-based specialization---shortened reasoning chains and increased truncation---and argue that harness adapters avoid it by construction. Our results position harness evolution as a practical, non-destructive complement to weight updates for vertical agents.
- 中文摘要
LLM代理的领域专门化通常被框架为权重更新问题:收集轨迹,运行强化学习(RL)或监督微调(SFT),并部署新的检查点。我们研究一个互补轴线:进化\textbf{agent harness}---系统提示符、工具架构、技能库和规划支架---同时保持策略冻结。我们将束演化定位为类似强化学习的外部循环,并用 GEPA 这一反思式进化优化器实现它。由此产生的束适配器无需 GPU 训练,可按垂直域加载,并与基于权重的 RL 组合。在 SWE-bench Verified、OfficeBench 和 Terminal-Bench 上使用 Codev 这种类似 Claude Code 的编码代理进行的实验显示出一致的模式:权重更新带来最强的单路径收益,束演化在计算量的极小部分内恢复大部分收益,而结合两条路径总体上效果最佳。我们还分析了基于 SFT 的专门化---推理链缩短和截断增加---并认为束适配器通过结构避免了这一现象。我们的研究结果将束演化定位为垂直代理权重更新的实用且非破坏性的补充。
Drifting, Discriminating, and Directioning for Action Coordination in Higher-dimensional Action Space
高维作用空间中的漂移、区分与定向作用协调
- Authors: Taisuke Kobayashi
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.25385
- Pdf link: https://arxiv.org/pdf/2611.25385
- Abstract
This paper proposes a lightweight policy model to mitigate the difficulty of handling higher-dimensional action spaces in reinforcement learning (RL), alongside an algorithm to effectively train it. To address this, we focus on drifting models, which can implicitly represent complex distributions and infer actions in a single forward pass. However, while drifting models can be straightforwardly trained via backpropagation of the Q-gradient, this alone cannot be expected to appropriately capture the desired action coordination. Therefore, this paper constructs a three-step learning algorithm to encourage the action coordination. First, the source distribution of the drifting model is co-optimized as a diagonal policy model. This conditions the drifting model to attach the coordination, which cannot be expressed by a diagonal distribution, as residuals. Second, a discriminator model before and after drifting is introduced with adversarial training for a regularization term. This actively promotes the extraction of the coordination that fundamentally separates the two distributions. Third, best-of-N sampling is applied for action selection during rollouts in order to give directionality toward/beyond the regions where the coordination is insufficiently extracted/sufficiently discovered. The proposed method consistently improves the sample efficiency and the final control performance over the conventional diagonal distribution modeling across three locomotion tasks with high-dimensional action spaces. Furthermore, the analysis indicates that this improvement is achieved by reducing variance in certain action dimensions, effectively coupling them with other actions as action coordination.
- 中文摘要
本文提出了一种轻量级策略模型,以减轻强化学习(RL)中处理高维动作空间的难度,同时结合一个有效训练该空间的算法。为解决此问题,我们重点研究漂移模型,这些模型可以隐式表示复杂分布,并在一次前向传递中推断作用。然而,虽然漂移模型可以通过Q梯度的反向传播直接训练,但仅凭此无法准确捕捉所需的动作协调。因此,本文构建了一个三步学习算法以促进动作协调。首先,漂移模型的源分布作为对角策略模型进行协同优化。这使漂移模型能够将无法用对角分布表达的协调作为残差附加。其次,通过对正则化项的对抗训练引入漂移前后判别模型。这积极促进了从根本上将两种分布分隔开的协调提取。第三,在展开过程中对行动选择应用最佳N取样,以赋予协调未充分提取或发现不足区域的方向性。所提出的方法在三个具有高维动作空间的移动任务中,持续提升样本效率和最终控制性能,相较于传统对角分布模型。此外,分析表明,这一改进通过降低某些动作维度的方差实现,有效地将它们与其他动作作为动作协调耦合。
Cone-Constrained Safe Policy Learning With Finite-Step Improvement Guarantees
锥约束安全策略学习,确保有限步骤改进
- Authors: Liangliang Hao, Wei Liu, Yunjian Xu
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.25426
- Pdf link: https://arxiv.org/pdf/2611.25426
- Abstract
One central challenge in safe reinforcement learning is improving task performance while reducing infeasibility risk, especially when long-term constraint consequences are difficult to represent and optimize. We propose the Constraint Aggregation Function (CAF), a learnable feasibility-oriented representation of long-horizon risk that satisfies a discounted Bellman recursion and admits a unique fixed point. The proposed CAF induces a risk advantage that measures the reduction in predicted unsafety relative to the current policy. Building on this representation, we develop Cone-Constrained Policy Optimization (CCPO), which uses the CAF-induced risk advantage to define a risk descent direction. The proposed CCPO resolves conflicts between task ascent and risk descent through gradient projection, selecting the final update from the relative interior of a constructed safety cone. Under standard smoothness assumptions and a step-size bound, this update guarantees simultaneous task improvement and risk reduction. Numerically, CCPO stays feasible with a 1.05% mean optimality gap in emergency braking and attains low constraint violations on safety-critical benchmarks.
- 中文摘要
安全强化学习的一个核心挑战是提升任务性能,同时降低不可行性风险,尤其是在长期约束后果难以表示和优化的情况下。我们提出约束聚合函数(CAF),这是一种可学习的、面向可行性的长期风险表示,满足折现贝尔曼递归且唯一不变点。所提CAF诱导风险优势,衡量相对于当前策略预测不安全的减少。基于该表示,我们开发了锥约束策略优化(CCPO),利用CAF诱导的风险优势定义风险下降方向。拟议的CCPO通过梯度投影解决任务上升与风险下降之间的冲突,从构建的安全锥相对内部选择最终更新。在标准平滑性假设和步长界限下,该更新保证任务改进和风险降低的同时实现。从数值上看,CCPO在紧急制动中保持可行性,平均最优性差距为1.05%,并在安全关键基准上实现低约束违规率。
PAIR-MED: Policy-reward Alternation for Iterative Refinement in Multilingual Medical Reasoning
PAIR-MED:多语种医学推理中用于迭代优化的政策-回报交替
- Authors: Fan Gao, Xiaohang Xu, Lang Huang, Nan Lu, Irene Li
- Subjects: Subjects:
Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.25446
- Pdf link: https://arxiv.org/pdf/2611.25446
- Abstract
Large language models demonstrate strong medical question-answering capabilities, yet reasoning performance remains uneven across languages. Process reward models (PRMs) provide step-level feedback for improving reasoning, but a fixed PRM may become increasingly misaligned with the trajectory distribution of an evolving policy during reinforcement learning, thereby reducing the reliability of its supervision. We propose a policy--reward alternating training framework that jointly adapts a medical PRM and optimizes a multilingual reasoning policy. Following supervised initialization, the current policy generates multilingual trajectories to update the PRM, which then provides process-level rewards for the next policy update. Medical step supervision maintains the PRM's medical foundation, while process, outcome, and language-related rewards guide policy learning. We develop multilingual medical PRMs (PAIR-PRM) and reasoning models (PAIR-MED) at 8B and 9B scales across 12 languages. On the medical subsets of Global-MMLU and MMLU-ProX, our models improve average accuracy by 3.01--7.47% over their corresponding Qwen backbones. On CUREMED-BENCH, PAIR-MED-9B improves the joint language consistency and logical accuracy metric by 12.88% and 7.45% over baselines for in-domain and out-of-domain languages. Further analyses examine PRM adaptation, multilingual reasoning quality, and the contributions of individual training components.
- 中文摘要
大型语言模型展现出强大的医疗问题答覆能力,但推理表现在不同语言间仍不均衡。过程奖励模型(PRM)为提升推理提供步骤级反馈,但固定的PRM在强化学习过程中可能越来越与策略演变的轨迹分布不匹配,从而降低其监督的可靠性。我们提出了一种策略-奖励交替训练框架,既对医疗PRM进行调整,又优化多语言推理策略。在监督初始化后,当前策略生成多语言轨迹以更新PRM,随后为下一次策略更新提供过程级奖励。医疗步骤监督维护了PRM的医学基础,同时过程、结果和语言相关奖励指导政策学习。我们开发了跨12种语言的8B和9B尺度的多语言医疗PRM(PAIR-PRM)和推理模型(PAIR-MED)。在 Global-MMLU 和 MMLU-ProX 的医学子集上,我们的模型平均准确率提升了 3.01 个——比对应的 Qwen 骨干提升 7.47%。在 CUREMED-BENCH 上,PAIR-MED-9B 在联合语言一致性和逻辑准确性指标上提升了 12.88%,相比域外语言基线提升了 7.45%。进一步分析还考察了PRM适应、多语言推理质量以及各个训练组件的贡献。
FamilySpace: From Emergent Chemical Semantics to Controllable Crystal Generation
FamilySpace:从新兴化学语义到可控晶体生成
- Authors: Haowei Hua, Irwin King, Wanyu Lin
- Subjects: Subjects:
Machine Learning (cs.LG); Materials Science (cond-mat.mtrl-sci)
- Arxiv link: https://arxiv.org/abs/2611.25452
- Pdf link: https://arxiv.org/pdf/2611.25452
- Abstract
Latent crystal generative models typically combine a variational autoencoder (VAE) for learning atom-wise representations with a flow model that transports a simple source distribution to the learned latent distribution. However, previous studies have mainly focused on the quality of generated crystals, leaving the structure and generative utility of the atom-wise latent space largely unexplored. In this work, we show that even a vanilla crystal VAE implicitly organizes atom-wise latent representations according to elemental families, with different families occupying largely separated regions of the latent space. We term this phenomenon emergent chemical semantics. This structure suggests that sampling from a family-specific latent region can bias the decoder toward atoms from the corresponding elemental family. Building on this insight, we propose FamilySpace, a framework for controlling the elemental-family composition of generated crystals. FamilySpace assigns each elemental family a distinct source component, forming a family-indexed multi-source distribution, and trains a flow model using source-target pairs matched by family. In inference, a user-specified family template selects the source component for each atom-wise representation, enabling direct control over the family composition of generated crystals. Experiments on MP-20 and MPTS-52 show that FamilySpace achieves competitive crystal-generation performance while matching prescribed elemental-family compositions. Moreover, selected family templates increase the fraction of generated crystals predicted to be stable or metastable, demonstrating that latent-space composition control can support targeted exploration of promising chemical regions without additional optimization techniques such as reinforcement learning. Code is available at this https URL
- 中文摘要
潜在晶体生成模型通常结合了用于学习原子分类表示的变分自编码器(VAE)与将简单源分布传输到学习得来的潜在分布的流模型。然而,此前的研究主要关注生成晶体的质量,对按原子分类的潜在空间的结构和生成效用大多未被深入探讨。在本研究中,我们表明即使是普通晶体VAE,也会隐式地根据元素家族组织按原子分类的潜在表示,不同家族占据潜空间中相对分离的区域。我们将这一现象称为涌现化学语义学。这一结构表明,从特定家族的潜区抽样可能会使解码器偏向对应元素家族的原子。基于这一见解,我们提出了FamilySpace框架,用于控制生成晶体的元素家族组成。FamilySpace 为每个元素家族分配一个独特的源成分,形成一个家族索引的多源分布,并利用按家族匹配的源-目标对训练流程模型。在推理中,用户指定的家族模板为每个按原子的表示选择源成分,从而实现对生成晶体家族组成的直接控制。在 MP-20 和 MPTS-52 上的实验表明,FamilySpace 在匹配规定元素家族组成的同时,能够实现具有竞争力的晶体生成性能。此外,选定的家族模板会增加预测为稳定或亚稳晶体的比例,表明潜空间组成控制可以在无需额外优化技术(如强化学习)的情况下,支持对有前景化学区域的有针对性探索。代码可在此 https URL 获取。
CANDOR: Contrastive Self-Denoising Rewards for Label-Free Text-to-Image Post-Training
坦诚:无标签文本转图像后训练的对比性自我去噪奖励
- Authors: Junhyeok Lee, Byung-Hoon Kim, Kyu Sung Choi
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.25457
- Pdf link: https://arxiv.org/pdf/2611.25457
- Abstract
Post-training text-to-image generators with reinforcement learning usually relies on external reward models that need annotated preferences or auxiliary evaluators and are prone to reward hacking. We show that the generator's own denoising accuracy, used by a recent intrinsic reward, is a poor proxy for prompt fidelity. Within a same-prompt group, this absolute self-confidence is dominated by image complexity and ranks correct samples above incorrect ones near chance. We introduce CANDOR (Contrastive Alignment from Null-referenced Denoising lOg-Ratio), the log-ratio of unconditional to conditional denoising errors under shared noise probes, which measures how much the prompt helps the model denoise its own sample. It cancels prompt-independent difficulty that scales both errors and ranks the better of two same-prompt samples higher in 60% and 63% of pairs on compositional generation and text rendering. As the only Flow-GRPO reward for a flow-matching model, CANDOR improves text rendering, aesthetic and human-preference scores over the base model and a prior absolute self-confidence reward, with text-rendering and preference gains that stronger classifier-free guidance does not reproduce. On public Flow-GRPO adapters trained on external rewards, CANDOR further raises PickScore and ImageReward, so it can complement those rewards. Code is available at this https URL.
- 中文摘要
带有强化学习的后训练文本转图像生成器通常依赖外部奖励模型,这些模型需要注释偏好或辅助评估器,且容易遭遇奖励被篡改。我们证明,生成器自身的去噪准确率(由近期内在奖励所利用)是提示忠实度的良好替代指标。在同一提示组内,这种绝对自信由图像复杂度主导,并且将正确样本排在错误样本之上,几乎概率更高。我们介绍了CANDOR(Null-referenced Denaising lOg-Ratio)对比比对,这是共享噪声探针下无条件与条件去噪错误的对数比,衡量提示词对模型自身样本去噪的帮助程度。它取消了与提示无关的难度,该难度会同时调整错误,并在合成生成和文本渲染配对中,将两个相同提示样本中较好的者排在60%和63%的对中更高。作为唯一用于流程匹配模型的 Flow-GRPO 奖励,CANDOR 在文本渲染、审美和人类偏好评分上优于基础模型和先前的绝对自信奖励,并带来文本渲染和偏好提升,而这些提升是无分类器指导无法复制的。在基于外部奖励训练的公共 Flow-GRPO 适配器上,CANDOR 进一步提升了 PickScore 和 ImageReward,从而补充这些奖励。代码可在此 https URL 获取。
Demonstration-Free Skill Composition for Navigation and Manipulation
无需演示的导航与操作技能组合
- Authors: Madeleine Nouri, Gelareh Mohammadi, Francisco Cruz
- Subjects: Subjects:
Robotics (cs.RO); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25462
- Pdf link: https://arxiv.org/pdf/2611.25462
- Abstract
Hierarchical reinforcement learning enables sequential tasks to be represented as compositions of separately trained skills. This paper evaluates demonstration-free hierarchical pipelines for MiniGrid navigation and robotic manipulation, comparing their training requirements and task performance with flat reinforcement learning baselines. The evaluation concerns complete pipelines with task-specific observation, reward and control structures. In navigation, two parameterised skills are reused across unseen FourRooms layouts under manually specified high-level plans. In manipulation, reaching, grasp alignment and lifting are trained separately, with downstream skills initialised from states collected during predecessor rollouts. Across unseen navigation layouts, the hierarchical pipeline substantially outperforms the evaluated flat DQN baselines while requiring fewer training interactions. In manipulation, the hierarchical chain requires fewer training steps than flat SAC and exhibits lower observed variation across seeds. The frozen Pick and Lift skills are further reused for stacking by training an additional Place skill.
- 中文摘要
分层强化学习使序列任务能够被表示为单独训练技能的组合。本文评估了无演示的 MiniGrid 导航和机器人操作层级流水线,将其训练需求和任务性能与平面强化学习基线进行比较。评估涉及具有任务特定观察、奖励和控制结构的完整流水线。在导航中,两个参数化技能在手动指定的高层计划下,在未看见的 FourRooms 布局中被重复使用。在操作中,伸展、抓握对齐和举举分别训练,下游技能则从前一轮部署中收集的状态初始化。在未见的导航布局中,分层流水线显著优于评估过的平面DQN基线,且所需训练交互更少。在操作中,分层链所需的训练步骤比平坦SAC少,且种子间观察到的变异更小。通过训练额外的“置点”技能,冷冻的“拣选与提举”技能还会被用于堆叠。
Learning Ankle Exoskeleton Control for Reactive Standing Balance Through Neuromusculoskeletal Simulation
通过神经肌肉骨骼模拟学习踝关节外骨骼控制,实现反应性站立平衡
- Authors: Maria T. Tagliaferri, Inseung Kang
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.25480
- Pdf link: https://arxiv.org/pdf/2611.25480
- Abstract
Deficits in reactive balance can increase fall risk among older adults and individuals with neuromuscular impairments, motivating the need for interventions that improve balance recovery. Wearable robotic exoskeletons offer a promising approach to addressing this need by augmenting balance control. However, developing controllers that deliver appropriately timed and scaled assistance remains challenging, as evaluating each candidate strategy requires repeated exposure to destabilizing perturbations. We propose a simulation-to-real framework that combines motion imitation and reinforcement learning to develop balance-assistive ankle exoskeleton control. A muscle-driven neuromusculoskeletal model was trained using a small set of motion-capture data from a single subject to reproduce kinematic and kinetic responses to backward-translation perturbations. This model served as a virtual human for training a reinforcement-learning exoskeleton controller. The resulting controller generated bilateral ankle assistance using only inertial sensor measurements, without explicit perturbation detection. We implemented the controller on-board a bilateral ankle exoskeleton and evaluated it in five healthy subjects whose data were not used for training. Assistance increased the perturbation magnitude that subjects could withstand without stepping by 11.8% and reduced peak center-of-mass displacement and velocity by 13.5% and 28.2%, respectively. Individual benefits varied with baseline recovery strategy and body mass. These results indicate that a single virtual human can yield transferable balance assistance, and that more diverse virtual humans could further improve its effectiveness.
- 中文摘要
反应性平衡缺陷会增加老年人和神经肌肉障碍者的跌倒风险,促使人们需要采取促进平衡恢复的干预措施。可穿戴机器人外骨骼通过增强平衡控制,为满足这一需求提供了有前景的方法。然而,开发能够在适当时间和尺度上提供辅助的控制器仍然具有挑战性,因为评估每种候选策略需要反复接触不稳定扰动。我们提出了一个模拟到真实的框架,结合运动模拟与强化学习,以开发平衡辅助踝关节外骨骼控制。利用单个受试者的少量动作捕捉数据训练了一个以肌肉驱动的神经肌肉骨骼模型,以再现对向平移扰动的运动学和动力学反应。该模型作为虚拟人用于训练强化学习外骨骼控制器。由此产生的控制器仅利用惯性传感器测量产生双侧踝关节辅助,且未明确检测扰动。我们将控制器安装在双侧踝关节外骨骼上,并在五名未被训练数据使用的健康受试者中进行评估。辅助使受试者在不跨步情况下能承受的扰动幅度增加了11.8%,峰值质心位移和速度分别降低了13.5%和28.2%。个体的益处因基线恢复策略和体重而异。这些结果表明,单个虚拟人能够提供可转移的平衡辅助,且更多样化的虚拟人能够进一步提升其效果。
Reinforcement-Aligned Probabilistic Weather Forecasting with Discrete Tokens
带有离散令牌的强化对准概率天气预报
- Authors: Jinhan Li, Zhisheng Chen, Yuxuan Li, Yuan Gao, Fan Xu, Hao Wu, Jinlong Du, Kun Wang
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.25570
- Pdf link: https://arxiv.org/pdf/2611.25570
- Abstract
Global medium-range weather forecasting must not only generate ensembles efficiently to characterize uncertainty in future states, but also serve downstream applications such as extreme-event warnings and cross-sphere coupled simulations. We introduce WeatherCraft, a framework that unifies probabilistic generation and reinforcement learning alignment through discrete weather tokens, bringing the GPT-style pretraining--post-training paradigm to global medium-range weather forecasting. A multiscale tokenizer represents atmospheric states as a coarse-to-fine token hierarchy, while a Transformer models causal dependencies across time and scale and predicts tokens within each scale in parallel. Direct sampling generates diverse ensembles, and exact token log-probabilities enable Group Relative Policy Optimization (GRPO) to maximize downstream task rewards without differentiating through evaluators or coupled models. On the ERA5 benchmark, WeatherCraft's eight-member ensemble mean reduces day-15 normalized RMSE by 17.6% relative to the strongest evaluated deterministic baseline and, with KV caching, achieves an approximately $18\times$ inference speedup over GenCast. Direct fair-CRPS adaptation improves ensemble probabilistic skill, while reward-based post-training improves forecasts of persistent land temperature extremes and coupled ocean heat extremes. This work establishes discrete autoregressive modeling as a general foundation for scalable, task-alignable Earth system foundation models.
- 中文摘要
全球中期天气预报不仅必须高效生成集合以表征未来状态的不确定性,还应服务于极端事件预警和跨球耦合模拟等下游应用。我们介绍了 WeatherCraft,这是一种通过离散天气代币统一概率生成和强化学习比对的框架,将 GPT 式的预训练-后训练范式引入全球中程天气预报。多尺度分词器将大气状态表示为粗到细的令牌层级,而 Transformer 则建模跨时间和尺度的因果依赖,并并行预测各尺度内的标记。直接抽样生成多样化的集合,精确的令牌对数概率使组相对策略优化(GRPO)能够最大化下游任务奖励,而无需通过评估器或耦合模型进行差异化。在ERA5基准测试中,WeatherCraft的八个成员集合平均值相较于最强的确定性基线将第15天归一化RMSE降低17.6%,并通过KV缓存,推理速度比GenCast提升约18美元。直接公平CRPS适应提升了集成概率技能,而基于奖励的后期训练则改进了持续陆地温度极端和耦合海洋极端的预测。这项工作确立了离散自回归建模作为可扩展、可任务对齐地球系统基础模型的通用基础。
Decision-Focused Reinforcement Learning for LLM Post-Training
面向LLM后期培训的决策导向强化学习
- Authors: Haotian Zhai, Leqi Liu
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25597
- Pdf link: https://arxiv.org/pdf/2611.25597
- Abstract
Large language models (LLMs) have shown promise as predictors across various tasks, benefiting from their ability to use information from unstructured context. However, when predictions serve as parameters of downstream optimization problems, higher prediction accuracy does not necessarily translate into lower decision costs. We propose decision-focused reinforcement learning (DFRL), a framework for post-training LLMs to predict optimization parameters using negative downstream decision regret as reward. DFRL trains LLM predictors to minimize the costs of the decisions induced by their predictions, without requiring downstream convexity or differentiability, and integrates readily into existing RL post-training frameworks. To address expensive reward evaluation, we develop two approaches. Heuristic warm-starting uses inexpensive approximate rewards during an initial phase before switching to refinement with exact rewards. Reward correction combines inexpensive estimates with randomized exact evaluations, preserving the expected policy gradient while reducing the expected number of exact solves per update. Experiments on three synthetic optimization problems demonstrate improvements over prediction-loss baselines, while a larger knapsack experiment shows faster early improvement under short training budgets. On inventory planning with H&M transactions and product descriptions, our best configuration, in which the LLM reasons before predicting, reduces regret by 31.4% relative to the SFT baseline and 12.4% relative to the strongest evaluated conventional forecasting baseline.
- 中文摘要
大型语言模型(LLM)在各种任务中展现出作为预测变量的潜力,受益于其利用非结构化上下文信息的能力。然而,当预测作为下游优化问题的参数时,更高的预测准确性未必意味着决策成本降低。我们提出了决策聚焦强化学习(DFRL),这是一种用于训练后 LLM 预测优化参数的框架,利用负的下游决策遗憾作为奖励。DFRL 训练 LLM 预测器,使其在不要求下游凸性或可微性的情况下,将预测引发的决策成本降至最低,并能轻松整合进现有的强化学习后期训练框架。为应对昂贵的奖励评估,我们开发了两种方法。启发式热启动在初始阶段使用廉价的近似奖励,然后转向精确奖励的精细化。奖励校正结合了廉价估计与随机精确评估,保持预期策略梯度,同时减少每次更新的精确求解次数。对三道合成优化问题的实验显示,其优于预测-损失基线,而更大规模的背包实验则显示在短期训练预算下,早期改进更快。在使用H&M交易和产品描述进行库存规划时,我们的最佳配置——即LLM先推理再预测——相较于SFT基线减少了31.4%,相对于最强评估的传统预测基线减少了12.4%。
Beyond Easy Clients: Difficulty- and Improvement-Aware Aggregation for Federated Reinforcement Learning in Robot Navigation
超越简单客户端:针对难度与改进的汇总,用于机器人导航中的联邦强化学习
- Authors: Ye Lin Tun, Chu Myaet Thwal, Eui-Nam Huh, Choong Seon Hong
- Subjects: Subjects:
Robotics (cs.RO); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25637
- Pdf link: https://arxiv.org/pdf/2611.25637
- Abstract
Federated reinforcement learning enables a fleet of robots to jointly learn a shared navigation policy without exchanging raw experience. In practice, however, the robots rarely operate in the same conditions: an open aisle is far easier to navigate than a cluttered room, and takes less learning to master. Standard federated aggregation rules ignore this. Uniform averaging weights every client equally, whether or not it is still learning, and weighting by episode return is worse, since easy environments permit high returns by construction. Both failures stem from what we call the \emph{plateau problem}: once a client masters its environment, its updates carry little new information even though its reported return remains high. We propose \emph{FedDIA}, which scores each client by how much its local training improves the global model it received, amplified by the difficulty of its environment. Difficulty is inferred from the model's own performance, without requiring any description of the environment. Both signals are measured against the global model, so they are comparable across clients. On a mapless navigation task with eight heterogeneous clients, FedDIA reaches a mean success rate of $75.5\%$ across training and held-out environments, against $68.7\%$ for the strongest federated baseline. Its advantage holds when the models are transferred to hand-authored warehouse and living-space layouts, or when one of the clients is defective.
- 中文摘要
联合强化学习使机器人队伍能够在不交换原始经验的情况下共同学习共享导航策略。然而,在实际操作中,这些机器人很少在相同条件下运行:开放的通道比杂乱的房间更容易导航,且掌握所需的学习也更少。标准的联合聚合规则忽略了这一点。均匀平均对每个客户端均等权重,无论其是否仍在学习,而按集数返回加权则更差,因为简单环境允许通过结构获得高回报。这两种失败都源于我们所称的\emph{平台问题}:客户端一旦掌握了环境,其更新几乎不带来新信息,尽管报告的回报依然很高。我们提出\emph{FedDIA},该方法根据客户端的本地训练对其接收到的全局模型的提升程度来评分,并被环境的难度放大。难度是根据模型自身表现推断的,无需描述环境。这两个信号都以全局模型进行衡量,因此在不同客户端之间具有可比性。在无地图导航任务中,针对八个异构客户端,FedDIA 在培训环境和保留环境中的平均成功率为 75.5 美元,而最强的联邦基线则为 68.7%$。当模型被转移到手工编写的仓库和居住空间布局,或其中一个客户端存在缺陷时,它的优势依然存在。
SiliconGym: A Versatile Reinforcement Learning Environment for Agentic EDA
SiliconGym:一个面向代理式EDA的多功能强化学习环境
- Authors: Hejia Zhang, Sheng Lu, Yitong Ma, Zhongming Yu, Zhijing Wu, Wentao Ni, Chia-Tung Ho, Jishen Zhao
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.25658
- Pdf link: https://arxiv.org/pdf/2611.25658
- Abstract
Reinforcement learning (RL) with electronic design automation (EDA) feedback is advancing large language models for hardware design and verification. Extending these gains to hardware-engineering agents requires environments that span tasks, preserve state across tool calls, and support large hierarchies alongside parallel rollouts. We present SiliconGym, the first versatile, cross-task and cross-toolchain RL environment for hardware design and verification. By decoupling versioned workspaces, budgeted execution, and acceptance-gated scoring, SiliconGym scales from module-level RTL, testbench, and PPA tasks to multi-round repository repair and full-hierarchy SoC flows. Streaming dispatch overlaps tool checks with generation, finishing a 256-answer RL batch 21.9% faster than native ChipSeek. Using interchangeable open-source and commercial backends, we show that open-source-trained functional correctness transfers cleanly to commercial simulation, while paired netlist evaluation reveals when open-source PPA gains survive commercial synthesis.
- 中文摘要
强化学习(RL)结合电子设计自动化(EDA)反馈,正在推动大型语言模型用于硬件设计和验证。将这些优势扩展到硬件工程代理,需要跨任务、在工具调用间保持状态,并支持大型层级结构并行部署的环境。我们介绍 SiliconGym,这是首个面向硬件设计与验证的多功能跨任务和跨工具链 RL 环境。通过解耦版本化工作区、预算执行和验收门槛评分,SiliconGym 从模块级 RTL、测试岗位和 PPA 任务扩展到多轮仓库修复和全层级 SoC 流程。流派度与生成工具检查重叠,完成256个回复的RL批次比原生ChipSeek快21.9%。利用可互换的开源与商业后端,我们展示了开源训练的函数正确性能干净利落地迁移到商业模拟,而配对网表评估则揭示开源PPA收益在商业合成后何时存活。
Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models
通过微调视觉语言模型实现目标引导强化学习
- Authors: Yajie Qi, Wei Wei, Lin Li, Lijun Zhang, Jianye Hao
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.25742
- Pdf link: https://arxiv.org/pdf/2611.25742
- Abstract
Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. Large language models (LLMs) can guide exploration by generating high-level goals, but methods using models without explicit task adaptation often rely on carefully designed prompts and may produce inconsistent or contextually inappropriate goals. We introduce CCGP-2M, a multimodal goal-preference dataset constructed from interaction trajectories in Craftax-Classic, and propose a vision-language model (VLM)-guided RL framework that learns environment-specific goal guidance from this dataset. The framework consists of two stages: offline VLM fine-tuning and online goal-guided RL. In the offline stage, we align the VLM with goal preferences using Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO), which combines token-level KL regularization with distributionally robust reweighting. Experiments on Craftax-Classic show that our method collects diamonds earlier than state-of-the-art baselines, achieves the highest score and reward, and attains the highest success rates on most achievements.
- 中文摘要
由于奖励稀疏且视野较长,探索仍然是 Craftax 类环境中强化学习(RL)代理面临的一大挑战。大型语言模型(LLM)可以通过生成高层次目标来引导探索,但使用无明确任务适应的模型方法往往依赖精心设计的提示,可能导致目标不一致或语境不合适。我们介绍了 CCGP-2M,这是一个由 Craftax-Classic 交互轨迹构建的多模态目标偏好数据集,并提出了一个由视觉语言模型(VLM)引导的 RL 框架,从该数据集中学习环境特定目标指导。该框架由两个阶段组成:离线 VLM 微调和在线目标引导 RL。在离线阶段,我们使用 Kullback-Leibler 令牌级直接偏好优化(KLTDPO)将 VLM 与目标偏好对齐,该方法结合了令牌级 KL 正则化与分布稳健的重权重处理。在 Craftax-Classic 上的实验显示,我们的方法比最先进的基线更早收集钻石,获得最高分数和奖励,并在大多数成就上取得最高的成功率。
ECHO: In-Context Behavior foundation model for Humanoid Whole-Body Control
ECHO:人形全身控制的情境行为基础模型
- Authors: Xingyi Wang, Bin Li, Jiyuan Shi, Chenyun Zhang, Weiji Xie, Xiangyu Miao, Junsong Wu, Wei Song, Shiqiang Zhu, Chenjia Bai
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.25778
- Pdf link: https://arxiv.org/pdf/2611.25778
- Abstract
Humanoid behavior foundation models (BFMs) acquire diverse skills through large-scale motion imitation and massively parallel reinforcement learning, yet broad behavioral capability does not ensure precise execution across physical conditions. Without distinguishing the underlying dynamics, policies may favor conservative compromises over environment-specific precision. We introduce ECHO, an in-context BFM that turns interaction experience into environment-specific control, separating general behavioral capability from dynamics-specific execution. Mixed and Annealed Sampling first establishes a behavior prior in a single pretraining stage by balancing targeted practice with broad motion coverage. Building on this prior, Context Representation and Residual Adaptation learns structured dynamics representations from interaction histories to guide residual corrections to the frozen prior, enabling adaptation without test-time parameter updates. Extensive experiments show that ECHO outperforms all baseline methods across diverse motions and physical conditions. Moreover, deployment on the Unitree G1 demonstrates whole-body teleoperation spanning extreme acrobatics and terrain interaction, alongside payload adaptation in manipulation and whole-body control. Page is available at this https URL
- 中文摘要
类人行为基础模型(BFM)通过大规模运动模拟和大规模并行强化学习获得多样化技能,但广泛的行为能力并不能保证在物理条件下的精确执行。在不区分底层动态的情况下,政策可能倾向于保守妥协而非环境特定精度。我们引入了 ECHO,一种情境内的 BFM,将交互体验转化为环境特定控制,将一般行为能力与动态特定执行分离。混合和退火采样首先在单一预训练阶段建立行为,通过平衡有针对性的练习与广泛的运动覆盖。在此基础上,情境表示与残余适应通过交互历史学习结构化动力学表示,指导对冻结先验的残余修正,实现测试时参数更新的适应。大量实验表明,ECHO 在各种运动和物理条件下表现优于所有基线方法。此外,在 Unitree G1 上的部署展示了跨越极限特技和地形交互的全身远程操作能力,同时还包括在操控和全身控制中对有效载荷的适应。页面可在此 https 网址获取
When Safety Rationales Hallucinate: Diagnosing and Grounding Multimodal Safety Guards
当安全理由出现幻觉时:诊断并接地多模态安全护卫
- Authors: Yingshuo Wang, Weixiang Zhao, Guancheng Li, Jiahe Guo, Yu Zhang, Yanyan Zhao
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25793
- Pdf link: https://arxiv.org/pdf/2611.25793
- Abstract
Multimodal safety guards increasingly generate rationales alongside safety predictions, yet whether these rationales are reliable remains largely unexplored. We systematically study rationale hallucination from two perspectives: visual rationale hallucination, where rationales contain unsupported visual facts, and safety attribution hallucination, where grounded content is assigned an incorrect safety interpretation. Across rationale-generation methods, multiple vision-language backbones, and safety datasets, we find that rationale hallucination is widespread and does not consistently diminish with higher classification accuracy or increased model scale. Further analysis of the Think-with-Image paradigm shows that hallucination is concentrated in fine-grained attributes and relations, while visual tools are often invoked infrequently or without sufficient spatial focus. Based on these findings, we propose Tool-Contrastive Grounding Reinforcement Learning (TCG-RL), which rewards tool use only when it improves rationale grounding over a tool-free reference. TCG-RL substantially reduces rationale hallucination while maintaining competitive safety prediction performance. By revealing the gap between prediction correctness and rationale reliability, our study motivates a shift in multimodal safety guarding from outcome-only evaluation toward safety decisions that are both accurate and verifiably grounded.
- 中文摘要
多模态安全防护越来越多地在安全预测的同时生成合理性,但这些合理性是否可靠仍大多未被深入探讨。我们系统地从两个角度研究理性幻觉:视觉理性幻觉,即理据中包含的无依无据的视觉事实;以及安全归因幻觉,即基于依据的内容被赋予错误的安全解释。在理据生成方法、多视觉语言骨干和安全数据集中,我们发现理性幻觉普遍存在,且随着分类准确度提高或模型规模增加,幻觉并不会持续减少。对“带图像思考”范式的进一步分析表明,幻觉主要集中在细粒度属性和关系中,而视觉工具往往被调用得很少或缺乏足够的空间聚焦。基于这些发现,我们提出了工具-对比基础强化学习(TCG-RL),该方法仅在工具使用改善理性基础时才奖励其使用,而非无工具参考。TCG-RL在保持竞争性安全性预测表现的同时,显著减少了理性幻觉。通过揭示预测正确性与合理可靠性之间的差距,本研究促使多模态安全防护从仅基于结果评估,转向既准确又可验证的安全决策。
EvoMem: Self-Evolving Skills for Time-Aware Long Video Memory Agents
EvoMem:为时间感知的长视频存储代理提供自我演进技能
- Authors: Yutong Guan, Meixing Shi, Haoran Sun, Bing Sun, Yuxiang Cai, Yankai Jiang
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.25889
- Pdf link: https://arxiv.org/pdf/2611.25889
- Abstract
Although memory-augmented architectures have expanded the ability of large vision-language models (LVLMs) to process long videos, existing retrieval processes still largely rely on global semantic matching and lack state-driven temporal control, often returning fragmented or redundant evidence and failing to maintain a coherent evidence chain. To address this challenge, we propose EvoMem, a time-aware long-video memory agent guided by self-evolving retrieval skills. EvoMem constructs a hierarchical time-indexed memory, where different types of evidence are aligned onto a unified video timeline; during retrieval, temporal cues from acquired evidence are further converted into explicit signals that constrain subsequent memory access. To address recurring procedural errors in long-video question answering, such as unclear retrieval targets, improper memory access paths, or premature stopping, EvoMem introduces retrieval skills to provide process-level guidance, enabling the agent to dynamically adjust retrieval content, memory operations, and target temporal ranges according to the current evidence state, while deciding when to stop. Furthermore, inspired by reinforcement learning, we formulate skill evolution as a reward-guided policy improvement process. Each candidate skill is treated as an external policy patch, and rewards are assigned according to answer-correctness state transitions on the validation set. In this way, effective skills are accepted and low-quality skills are rejected without updating model parameters. Experiments on four long-video question answering benchmarks show that EvoMem significantly outperforms existing baselines, achieving an average accuracy of 60.1% and an average improvement of 8.6 percentage points over the previous best methods, demonstrating its effectiveness for long-video reasoning.
- 中文摘要
尽管记忆增强架构扩展了大型视觉语言模型(LVLM)处理长视频的能力,但现有检索过程仍主要依赖全局语义匹配,缺乏状态驱动的时间控制,常常返回碎片化或冗余的证据,且无法维持连贯的证据链。为应对这一挑战,我们提出了 EvoMem,这是一种由自我演进检索技能引导的时间感知长视频记忆代理。EvoMem 构建了一个分层的时间索引记忆,将不同类型的证据对齐到统一的视频时间轴上;在检索过程中,来自获取证据的时间线索进一步转化为限制后续记忆访问的显性信号。为解决长视频题答中常见的程序错误,如检索目标不明确、记忆访问路径不当或过早停止,EvoMem 引入检索技能,提供进程级指导,使代理能够根据当前证据状态动态调整检索内容、记忆操作和目标时间范围,并决定何时停止。此外,受强化学习启发,我们将技能演化构建为一种以奖励为导向的策略改进过程。每个候选技能被视为外部策略补丁,奖励根据验证集上的答案正确性状态转变分配。通过这种方式,有效技能被接受,低质量技能被拒绝,且不更新模型参数。对四个长视频题型答题基准测试的实验显示,EvoMem 显著优于现有基线,平均准确率为 60.1%,较以往最佳方法平均提升 8.6 个百分点,证明其在长视频推理方面的有效性。
ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining
ROAM-GRPO:通过路由离模式方法挖掘强制GRPO中的推理多样性
- Authors: Jeff Duan, Matt Thomson
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.25926
- Pdf link: https://arxiv.org/pdf/2611.25926
- Abstract
Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.
- 中文摘要
监督式微调(SFT)与可验证奖励强化学习(RLVR)相结合,已成为复杂推理任务后培训LLM的主流方案,但已被证明会使策略陷入低多样性模式。重复的LLM输出几乎相同,pass@k准确率因此趋同或下降。我们提出了ROAM-GRPO(GRPO路由离模式方法挖掘),这是对GRPO算法的补充,鼓励探索不同推理策略,旨在提升评估过程中的输出创造力。ROAM-GRPO在训练过程中检测已解决的问题,并将其转入“非模式”轨道。模型被提示为此类问题提出不同的解决方案策略,这些策略由 LLM 评审验证其逻辑有效性和独特性,随后自我提炼进策略中。在 MATH lvl5 数据集的 1,000 个问题子集上训练 Qwen3.5-2b,ROAM-GRPO 实现在困难评估集上使峰值pass@16提升 4.7%,正确嵌入余弦距离约提升 1.5-2 倍。
Learning optimal gaits and styles for terrain-adaptive quadruped locomotion
学习地形适应型四足行走的最佳步态和风格
- Authors: Guillaume Bellegarda, Auke Ijspeert
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.25958
- Pdf link: https://arxiv.org/pdf/2611.25958
- Abstract
Animals use many different gaits to locomote at different speeds across varying terrains. A longstanding question in biology and robotics is: which gait is best? This question is most often addressed from an energy-efficiency perspective on flat terrain, leaving open how the answer depends on terrain and on the criterion optimized. Here we show that gait optimality is determined jointly by terrain, performance metric, and velocity. We use deep reinforcement learning and networks of abstract oscillators, modeling spinal central pattern generators, to learn 11 quadrupedal gaits across diverse challenging terrains, together with the locomotion style (base height, foot ground clearance, foot offsets) that each requires. A second policy then selects among them to optimize a user-specified criterion such as cost of transport, base stability, or joint smoothness. Across these performance metrics and within this repertoire, every gait is optimal under at least one terrain-metric-velocity combination. The framework adapts zero-shot to gait couplings and styles never seen during training, to terrains far beyond the training distribution, and to larger quadrupeds five times the training mass. With a single additional pitch-tracking reward term, the same gait couplings also support bipedal-mode locomotion. Our method provides a general and systematic way to produce expressive locomotion and to select the gait best suited to a given terrain and criterion. Project page: this https URL.
- 中文摘要
动物在不同地形上以不同速度行驶,使用多种不同的步态。生物学和机器人学中一个长期存在的问题是:哪种步态最好?这个问题通常从平坦地形的能源效率角度来探讨,但答案如何取决于地形以及优化后的标准。在这里,我们展示了步态最优性由地形、性能指标和速度共同决定。我们利用深度强化学习和抽象振荡器网络,模拟脊柱中央模式发生器,学习11种四足步态,适应多样且具挑战性的地形,以及每种方式所需的行走方式(基准高度、脚距、脚距偏差)。然后,第二种策略从中选择,以优化用户指定的标准,如运输成本、基底稳定性或关节平滑度。在这些性能指标中,在这些参数库中,每种步态在至少一种地形-指标-速度组合下都最优。该框架将零射程调整为训练中从未出现过的步态耦合和风格,适应远超训练分布的地形,以及训练质量五倍的大型四足动物。通过一个额外的音高追踪奖励项,相同的步态耦合还支持双足行走模式。我们的方法提供了一种通用且系统的方法,用于生成富有表现力的移动,并选择最适合特定地形和标准的步态。项目页面:https URL。
Policy Improvement-Guided Replay for Continual Reinforcement Learning
策略改进引导回放,实现持续强化学习
- Authors: Qin-Wen Luo, Haoxuan Gao, Sheng Ren, Xiang Chen, Sheng-Jun Huang
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.25989
- Pdf link: https://arxiv.org/pdf/2611.25989
- Abstract
Continual reinforcement learning (CRL) requires models to acquire new capabilities while retaining performance on previous tasks. A stability-oriented perspective common in conventional continual learning emphasizes knowledge preservation and interference mitigation. Complementing this perspective, we revisit CRL for vision-language models through policy improvement, a fundamental principle of RL, to preserve acquired performance gains. Using the performance difference lemma, we characterize historical performance changes through the current policy's expected advantages on historical task distributions, motivating replay of historical prompts with the current policy. To allocate limited replay memory, we propose Contrastive Likelihood Replay (CoLiR), which contrasts likelihood gains on correct responses from the final task policy with likelihood reductions on incorrect responses from the initial policy. Our analysis links correct-response likelihood gains to reference-policy advantages under an idealized KL-regularized optimal update and provides an endogenous-reward interpretation for general policy updates. CoLiR estimates scores from finite training rollouts and cached log-likelihoods, selects prompts at task boundaries, and replays them uniformly during subsequent training, with KL regularization restricted to replayed prompts. Memory construction only requires a small amount of additional cross-policy scoring but no additional response generation. Across three Qwen3-VL model sizes on MRCL, CoLiR achieves the highest mean final accuracy and mean task accuracy among the compared methods while maintaining competitive new-task performance.
- 中文摘要
持续强化学习(CRL)要求模型在保持先前任务性能的同时,获得新能力。传统持续学习中常见的稳定性导向视角强调知识保存和干扰缓解。在这一视角的补充下,我们通过策略改进——强化学习的基本原则——重新审视视觉语言模型的CRL,以保留既得的性能提升。利用性能差异引理,我们通过当前策略对历史任务分布的预期优势来描述历史性能变化,从而激励历史提示与当前策略的重放。为分配有限的重放记忆,我们提出了对比似然重放(CoLiR),对比最终任务策略正确响应时的似然增益与初始策略错误响应的似然减少。我们的分析将正确响应似然增益与理想KL正则化最优更新下的参考策略优势联系起来,并为一般策略更新提供了内生奖励解释。CoLiR通过有限训练部署和缓存日志似然估算分数,在任务边界选择提示,并在后续训练中统一重放提示,KL正则化仅限于重放提示。内存构建只需少量额外的跨策略评分,无需额外响应生成。在MRCL上的三种Qwen3-VL模型规模中,CoLiR在保持新任务性能的同时,实现了所有比较方法中最高的最终平均准确率和平均任务准确率。
Reference-Free Humanoid Skateboarding via Termination-Shaped Reinforcement Learning
通过终止形强化学习实现无参考人形滑板
- Authors: Ganeshvar Soenessardien, Michael Przystupa, Kevin Sebastian Luck
- Subjects: Subjects:
Robotics (cs.RO); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.26054
- Pdf link: https://arxiv.org/pdf/2611.26054
- Abstract
Humanoid locomotion research has advanced rapidly on walking and running, but \emph {object-mediated} locomotion, in which a robot must coordinate its body with a passive external object, remains difficult. One challenging task is skateboarding, which requires simultaneous interaction with the ground and a passive but movable platform to achieve forward motion. Current methods for tackling skateboarding with humanoid robots either require human demonstrations or utilise analytic steering models that require system identification. In this paper, we introduce a deep reinforcement learning framework that does not rely on prior data such as demonstrations and enables robots such as the K1 humanoid to learn to skateboard using model-free learning. The method combines task rewards with exploit-blocking terminations. We introduce a number of constraints restricting behaviours such as losing the planted board foot, standing with both feet on the board, excessive twisting, separation from the board, and unstable board motion. Their changing prevalence during training produces an implicit progression from stance control towards sustained robot--board locomotion.
- 中文摘要
类人生物运动研究在步行和跑步方面取得了快速进展,但物体介导的移动——即机器人必须将身体与被动外部物体协调——仍然困难重重。滑板是一项具有挑战性的任务,它需要同时与地面互动,并有一个被动但可移动的平台以实现前进运动。目前用于人形机器人滑板的技术要么需要人工演示,要么采用需要系统识别的分析引导模型。本文介绍了一个深度强化学习框架,不依赖以往数据(如演示),使像K1类人机器人这样的机器人能够通过无模型学习学习滑板。该方法结合了任务奖励与阻止漏洞的终止机制。我们引入了多项限制行为的限制,例如失去着地板脚、双脚站立时双脚踩在板上、过度扭转、与板子分离以及板子运动不稳定。训练过程中它们的出现率变化,导致从站姿控制向持续的机器人-板上移动的隐性进展。
Controlling the Humanoid Robot H1-2 With a Simulated Biologically Plausible Network
通过模拟生物合理网络控制类人机器人H1-2
- Authors: Fatima Atwi (LAAS-GEPETTO, Nantes Univ - ECN), Constant Roux (LAAS-GEPETTO), Christian Bergaud (LAAS-MEMS), Olivier Stasse (LAAS-GEPETTO)
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.26074
- Pdf link: https://arxiv.org/pdf/2611.26074
- Abstract
Artificial Spiking Neural Networks have gained remarkable attention in the last few years due to their proven ability in simulating biological neural cultures and producing similar activity and trends. Nevertheless, whether they can be trained for humanoid locomotion while retaining biological plausibility and robust performance at once remains an open question. We study this problem by proposing a biologically inspired spiking network, trained by different learning schemes and tested on the H1-2 Unitree Humanoid. These schemes differ in how recurrent weights are trained. Ordered from most to least biologically faithful, they are: a fixed reservoir mimicking an in-vitro neural culture, where recurrent weights are kept frozen and only a linear readout is trained, the same network but with its synapses trained by reward-modulated spike timing dependent plasticity and eligibility propagation, and finally backpropagation through time as a non biological baseline for comparison. All schemes relied on distillation of a reinforcement learning expert policy with dataset aggregation. We show that e-prop reached 100% survival rate matching backpropagation through time, whereas both the fixed reservoir and the reward-modulated STDP, though the closest to biological cultures, plateaued at a much lower performance (~43.7% and ~11% survival). We explore in further detail which trends a real culture could and could not produce until now. Finally we demonstrate brief bipedal stepping on a full size humanoid H1-2.
- 中文摘要
近年来,人工尖峰神经网络因其在模拟生物神经培养及产生类似活动和趋势方面的成熟能力而备受关注。然而,它们是否能在同时保持生物学合理性和稳健性能的前提下训练为类人移动,仍是一个悬而未决的问题。我们通过提出一个受生物启发的尖峰网络进行研究,该网络通过不同的学习方案进行训练,并在 H1-2 Unitree 类人生物上进行测试。这些方案在重复性权重的训练方式上有所不同。按生物忠实度从高到低排序,它们分别是:一个固定的储存库,模拟体外神经培养,将循环权重冻结,只训练线性读数;同一网络,但其突触通过奖励调制的尖峰时机、依赖可塑性和适格性传播训练;最后则是时间逆向传播,作为非生物基准进行比较。所有方案均依赖于强化学习专家策略的提炼,并结合数据集聚合。我们表明,e-prop 在时间推移中实现了与反向传播匹配的 100% 存活率,而固定储槽和奖励调制 STDP 虽然最接近生物培养,但表现都趋于平稳(~43.7% 和 ~11%)。我们进一步探讨了真实培养物迄今能产生和未能产生哪些趋势。最后,我们演示了在全尺寸人形 H1-2 上短暂的双足行走。
RL Can Only Reinforce What It Sees Succeed
现实只能强化它认为成功的事物
- Authors: Ruben Sanchez-Corcuera, Unai Zulaika, Aritz Bilbao-Jayo, Miguel Fernandez-de-Retana
- Subjects: Subjects:
Machine Learning (cs.LG); Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.26084
- Pdf link: https://arxiv.org/pdf/2611.26084
- Abstract
When a familiar rule stops working, an agent has to weigh what pre-training taught it against what it now observes. We ask how outcome-reward reinforcement learning (RL) changes this balance, in a puzzle world where both sides are known: small language models trained from scratch on games played by a Bayesian player, so that the prior and the right answer are exact. Base models hedge as their corpus did, and RL turns that hedging into decisions, in whichever direction the reward favours. But RL can only reinforce what it sees succeed. When the right action is rare, each run either learns it everywhere or not at all, and which runs take off is predicted before training by one count: the expected number of groups of attempts in which the action wins. About 35 such groups give even odds. The count holds for new and repeated problems and for groups of 4 to 256 attempts, and across learning rates, optimizers, GRPO variants and model sizes from 1.3M to 85M parameters, which move the threshold but not what is counted.
- 中文摘要
当熟悉的规则不再起作用时,代理必须权衡预训练教给它的内容与现在观察到的内容。我们探讨结果-奖励强化学习(RL)如何改变这种平衡,在一个双方都已知的谜题世界中:由贝叶斯玩家玩的游戏中从零训练出的小型语言模型,以确保先验和正确答案完全准确。基础模型对冲,就像其语料库所做的那样,而RL则将这种对冲转化为决策,朝着奖励有利的方向进行。但强化学习只能强化它看到成功的行为。当正确行动罕见时,每次运行要么在各处学会,要么根本不学,而在训练前,哪些运行启动的预测仅仅通过一个计数:该动作在预期中获胜的尝试组数。大约有35个这样的组给出了平衡的概率。计数适用于新问题和重复问题,以及4至256次尝试的组,以及涵盖学习率、优化器、GRPO变体和模型大小参数,范围从130万到8500万,这些参数只移动阈值但不影响计数范围。
SenseAlign: Learning Decision-Aligned Predictive Representations for Active Urban Sensing with UAVs
SenseAlign:学习无人机的主动城市感知决策对应预测表征
- Authors: Weijiang Xiong, Nikolas Geroliminis
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.26109
- Pdf link: https://arxiv.org/pdf/2611.26109
- Abstract
Active urban sensing orchestrates a mobile sensor fleet to proactively inspect a dynamic road network and gather critical information for system management. This problem simultaneously requires understanding the complex spatio-temporal evolution and making decisions with partial observations. In this work, we focus on active sensing in a large-scale urban road network with utilities grounded in dynamic traffic states, while prior work has rarely considered these factors jointly. We propose \emph{SenseAlign}, a Reinforcement Learning-based (RL) framework that learns decision-aligned predictive representations for coordinating an urban sensing drone fleet. SenseAlign first trains a privileged teacher agent that exploits future ground-truth traffic for optimal sensing choices. The student agent has an auxiliary branch that learns to predict future traffic with partial observations, in addition to the RL-trained actor and critic branches. During student learning, our framework matches the action distributions of the teacher under ground-truth and student-predicted future traffic, while keeping the teacher's observations unchanged. Such a design shapes the latent space for decision-making instead of prediction alone. At inference time, the student agent has no overhead as all this learning is done in training. We conduct comprehensive experiments with the real road network of central Barcelona containing 1570 road segments, and demonstrate that our method consistently outperforms the baselines on multiple utility formulations. The codebase will be made publicly available upon paper acceptance.
- 中文摘要
主动城市感知协调移动传感器车队,主动检查动态道路网络,收集系统管理的关键信息。这一问题同时需要理解复杂的时空演变,并基于部分观测做出决策。本研究重点关注大规模城市道路网络中的主动感测,这些网络的公用事业建立在动态交通状态中,而以往的研究很少将这些因素结合考虑。我们提出了\emph{SenseAlign},这是一个基于强化学习(RL)的框架,用于学习决策对齐的预测表征,以协调城市感测无人机车队。SenseAlign首先训练一名特权教师代理,利用未来地面真实流量实现最佳感知选择。学生代理拥有辅助分支,除实况训练的行为者分支和批评分支外,还能通过部分观测学习预测未来交通。在学生学习过程中,我们的框架匹配教师在地面真实和学生预测未来交通下的行动分布,同时保持教师的观察不变。这种设计塑造了决策的潜在空间,而不仅仅是预测。在推理阶段,学生代理没有额外开销,因为所有学习都在培训中完成。我们在巴塞罗那市中心的真实道路网络中进行了全面实验,该网络包含1570条路段,并证明我们的方法在多种效用表述中持续优于基线。代码库将在纸质接受后向公众公开。
The Stability Gap in Continual Reinforcement Learning
持续强化学习中的稳定性差距
- Authors: Sebastian Pusch, Matthia Sabatelli, Gido M. van de Ven
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.26252
- Pdf link: https://arxiv.org/pdf/2611.26252
- Abstract
Continual reinforcement learning (CRL) aims to enable agents to adapt to changing tasks while preserving previously acquired capabilities. Evaluation often relies on final aggregate metrics that can hide temporary performance degradation during adaptation. This work introduces a simple CRL benchmark that varies observation representations while preserving environment dynamics and rewards, supporting proof-of-concept demonstrations, inexpensive experimentation, and easily reproducible sanity checks. Using this benchmark, we demonstrate the stability gap in CRL: a substantial drop in prior-task performance after the introduction of new tasks, followed by partial or complete recovery. This pattern occurs despite the agent having complete access to past tasks.
- 中文摘要
持续强化学习(CRL)旨在使代理能够适应不断变化的任务,同时保留先前获得的能力。评估通常依赖最终的汇总指标,这些指标可以掩盖适应过程中的暂时性能下降。这项工作引入了一个简单的CRL基准,在保持环境动态和奖励的同时改变观察表征,支持概念验证演示、低成本的实验以及易于复现的合理性检查。利用该基准,我们展示了CRL中的稳定性缺口:新任务引入后,先前任务性能大幅下降,随后出现部分或完全恢复。即使代理对过去任务拥有完全访问权限,仍会出现这种模式。
LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Whole-Body Control
LocoMuJoCo:全身控制的综合模仿学习基准
- Authors: Firas Al-Hafez, Davide Tateo, Oleg Kaidanov, Aditya Bhatt, Artur Zelik, Zhengyi Luo, Jan Peters
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.26293
- Pdf link: https://arxiv.org/pdf/2611.26293
- Abstract
LocoMuJoCo is an open-source benchmark for imitation learning (IL) and reinforcement learning (RL) on whole-body control. The library couples MuJoCo (CPU) with MJX and the MjWarp backend (GPU) under a single API, exposes 9 humanoid, 3 (musculo-)skeletal, and 4 quadruped embodiments, and ships a curated native motion-capture dataset for every humanoid, to which it can also retarget AMASS and LAFAN1. A modular plug-in architecture makes observations, rewards, control modes, terrains, and domain randomization fully interchangeable, while compact JAX implementations of many RL and IL algorithms provide JIT-compiled end-to-end training, and the companion loco-mujoco-real repository provides sim-to-real deployment. We describe the design, the SMPL-to-robot and robot-to-robot retargeting pipeline, and a usage example. LocoMuJoCo is released under MIT at this https URL.
- 中文摘要
LocoMuJoCo 是一个开源的全体控制模拟学习(IL)和强化学习(RL)基准测试。该库将 MuJoCo(CPU)、MJX 和 MjWarp 后端(GPU)集成在一个 API 之下,提供 9 个类人形、3 个(肌肉)骨骼型和 4 个四足形态,并为每个类人生物提供精心策划的原生动作捕捉数据集,还可以将 AMASS 和 LAFAN1 重新定位到该数据集。模块化插件架构使得观察、奖励、控制模式、地形和领域随机化完全可互换,而许多 RL 和 IL 算法的紧凑 JAX 实现则提供 JIT 编译的端到端训练,配套的 loco-mujoco-real 仓库则提供模拟到真实的部署。我们介绍了其设计、SMPL到机器人以及机器人到机器人的重定向流程,以及一个使用示例。LocoMuJoCo 由 MIT 发布,网址为 https。
Actor-Predictor Architecture for Reinforcement Learning with Formal Specification Objectives
基于形式规范目标的强化学习演员预测器架构
- Authors: Josef Brozovich, Qining Zhang, Lei Ying
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.26313
- Pdf link: https://arxiv.org/pdf/2611.26313
- Abstract
One fundamental challenge in reinforcement learning (RL) is enforcing temporal safety constraints, such as satisfying formal specifications. Existing methods often transform logic formulas into automata for state augmentation and reward shaping. However, this approach assumes that automaton construction is both feasible and tractable, which limits its applicability to small specifications and linear temporal logic (LTL). This paper proposes Actor-Predictor, a framework that learns a surrogate representation of logic progress without explicitly constructing an automaton. The predictor continuously estimates whether a partial trajectory will satisfy the formal specification in a later step, and its learned features are provided as auxiliary inputs to the actor, which is then optimized using GRPO and RWR. We show theoretically that, under sufficient predictive accuracy, the learned representation is equivalent to the automaton state. Experiments on grid world and robotic tasks in Isaac Lab demonstrate that Actor-Predictor outperforms automaton-based baselines on LTL-constrained tasks. It also performs well on non-regular formal specifications, such as $(a^n b^n)$, for which finite-automaton-based approaches are not applicable.
- 中文摘要
强化学习(RL)中的一个根本挑战是强制执行时间安全约束,例如满足形式规范。现有方法通常将逻辑公式转化为用于状态增强和奖励塑造的自动机。然而,这种方法假设自动机构建既可行又可操作,这限制了其适用范围为小规格和线性时间逻辑(LTL)。本文提出了演员预测器框架,该框架在不显式构建自动机的情况下学习逻辑进展的替代表示。预测器会持续估计某条部分轨迹是否在后续步骤满足正式规范,其学到的特征作为辅助输入提供给演员,随后通过 GRPO 和 RWR 进行优化。我们理论上证明,在预测精度足够高的情况下,所学到的表示等价于自动机状态。在 Isaac Lab 中对网格世界和机器人任务的实验表明,Actor-Predictor 在 LTL 约束任务中优于基于自动机的基线。它在非正规形式规范(如 $(a^n b^n)$ 上表现良好,而有限自动机方法不适用此类参数。
Training LLMs to Follow Reasoning Instructions with Minimal Edits
培训大型语言模型以最小修改方式遵循推理指令
- Authors: Alexandra Yakovleva, Luat Gia Khoi Nguyen, Juho Kannala, Harri Valpola, Alexander Ilin
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.26342
- Pdf link: https://arxiv.org/pdf/2611.26342
- Abstract
Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.
- 中文摘要
大型语言模型越来越依赖扩展的推理痕迹,但用于塑造这种行为的训练信号仍然粗糙。对完整轨迹进行监督微调迫使模型模拟整个演示,过度限制其输出分布;而轨迹级强化学习仅提供弱学分分配,奖励或惩罚完成过程,却不明确其成功或失败。我们提出基于编辑的训练,这是一种通过局部文本修正从模型自身轨迹学习的通用框架。给定一个抽样完备,编辑者在指定标准下识别第一个失败点,并提出最小替换跨度。模型随后仅对这些编辑跨度进行监督。设计上,这种方式将监督集中在模型推理最先失效的地方,并旨在在其他地方保留其现有推理模式。该方法既适用于线下,也适用于线上,后者会不断调整监督机制以适应模型不断演变的失败模式。我们在 ReasonIF 基准测试上的 Qwen3-32B 之后,将这一方法应用于推理教学。与全轨迹模仿和轨迹级强化学习基线相比,基于编辑的训练显著提升了对推理指令的依从性,同时在很大程度上保留了模型在非针对性任务上的技能。这些结果表明,局部纠正是这些方法的有效补充,有助于塑造大型语言模型的推理方式。
Beam Model Technical Report
梁模型技术报告
- Authors: Reflection Team: Adam Abdulhamid, Osama Abuelsorour, Yasmin Alameddine, Carlo Alfano, Duarte Alves, Brandon Amos, Selvaraj Anandaraj, Arseni Anisimovich, George Antoniadis, Ioannis Antonoglou, Sheila Babayan, Neeraj Bahl, Märt Bakler, Vidhisha Balachandran, Cameron Banks, Thibaut Barroyer, Navroop Bath, Irwan Bello, Jeremy Berman, Fernanda Bernaldo, Mrityunjay Bhadauria, Sanyam Bhutani, Tolga Bolukbasi, Anudhyan Boral, Joseph Brennan, Ethan Brooks, Sam Buchanan, Anton Bulyenov, John Cai, Catherine Cang, Eason Chan, Elijah Chanakira, Nathan Chandrasekaran, Hao Chen, Hong-You Chen, Jeffrey Chen, Saurav Chhatrapati, Ting-Rui Chiang, Brian Tiger Chow, Aakanksha Chowdhery, Timothy Chung, Rob Claire, John D. Co-Reyes, Anthony Conte, Eduardo Coronado-Montoya, Julien Crozon, Drew Csillag, Joe Cummings, Jeff Da, Samira Daruki, Tim Dawson, Charlie Deck, Arash Delijani, Ousmane Dia, Randy Dodgen, Ryan Douglas, Omar El-Kishky, Ben Ellis, Gamaleldin Elsayed, Simon Fan, Patrick Fernandes, Gabriel Ferns, Nick Fram, Ankush Garg, Florent Gbelidji, Iaroslav Geraskin, Behrooz Ghorbani, Ivan Glushkov, Neil Gollapudi, Scott Goodfriend, Yoni Gozlan, Christopher Grimm, Suriya Gunasekar, Shruti Gupta, Joel Gustafson, Emre Guven, Jon Hackmann, Jeremy Hadidjojo, Jess Hamrick, Ethan Hellman, Joscha Hoche, Spurthi Amba Hombaiah, Uzair Honda, Zijian Hu, Dzmitry Huba, Nancy Hung, Vladimir Ivanov, Shomik Jain, Mojan Javaheripi, Kevin Martin Jose, Aymen Kallala, Amog Kamsetty, Min Hyung Kang, Steven Kapturowski, Ramtin Keramati, Samuel Kessler, Sabina Kim, Seonghyeon Kim, Ivan Komarov
- Subjects: Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.26358
- Pdf link: https://arxiv.org/pdf/2611.26358
- Abstract
Beam is a sparse Mixture-of-Experts (MoE) model with 501.3B total parameters and 23.4B active parameters, specifically built and optimized for coding, reasoning, and agentic workloads. Beam delivers frontier level reasoning, while requiring 3-4x less inference compute than many models of its class. Pretrained on 23.8T diverse, curated, high-quality tokens, Beam features a 1M-token context window achieved through a novel two-stage midtraining phase, built on verifiable, correctness-anchored data. Post-training leveraged a large-scale, fully asynchronous reinforcement learning (RL) campaign across 10.5K GB300 GPUs, generating over 100M rollouts over 4 weeks of training. We believe this is one of the largest scale RL runs conducted by any open lab to date. Across our evaluation suite, capabilities continued to improve as we increased RL compute, with no sign of a plateau. Supported by deep hardware-model co-design, FP8/NVFP4 quantization, and a 1.93B-parameter speculative decoding drafter (DFlash 2), Beam maximizes serving throughput with efficient, flexible deployment. Averaged across conversation turns and harm categories, Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, achieving the highest refusal/pivot rate on harmful requests among open-weight models on the Pareto frontier.
- 中文摘要
Beam 是一个稀疏的专家混合模型(MoE),拥有 501.3 亿总参数和 23.4 亿主动参数,专为编码、推理和代理工作负载构建和优化。Beam 提供前沿层面的推理能力,同时比同类许多模型需要的推理计算少 3-4 倍。Beam 基于 23.8T 的多样化、精心策划的高质量代币预训练,采用了基于可验证、以正确性为依据的数据为基础的创新两阶段中期训练阶段实现的 1M 代币上下文窗口。后期训练利用了在 10.5K GB300 GPU 上开展的大规模、完全异步强化学习(RL)活动,在 4 周的训练中实现了超过 1 亿次部署。我们认为,这是迄今为止任何开放实验室中规模最大的 RL 运行之一。在我们的评估套件中,随着增强学习计算的提升,能力持续提升,且未出现任何停滞迹象。在深度硬件-模型协同设计、FP8/NVFP4 量化以及 1.93B 参数的推测解码起草机(DFlash 2)的支持下,Beam 通过高效、灵活的部署最大化了服务吞吐量。按对话回合和伤害类别平均计算,Beam 在 88.9% 的有害请求上拒绝或转向,同时满足 81.5% 的无害请求,在帕累托前沿的开放权重模型中实现了最高的有害请求拒绝/转换率。
How to Effectively Manage the Energy in Residential Systems: Development and Comparison of Methods
如何有效管理住宅系统中的能源:方法的开发与比较
- Authors: Julian Ruddick
- Subjects: Subjects:
Systems and Control (eess.SY); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
- Arxiv link: https://arxiv.org/abs/2611.26412
- Pdf link: https://arxiv.org/pdf/2611.26412
- Abstract
The transition from fossil fuels presents significant challenges in energy management. To ensure a sustainable energy future, it is crucial to reduce energy consumption when possible and to synchronize electricity consumption and storage with the production from sources with low greenhouse gas emissions. This thesis explores a variety of energy management methods used to manage batteries, heat pumps, and other energy-related systems. The thesis presents three simulation cases alongside one real-world experiment to compare energy management done through simple rules, optimization, and machine learning methods. Furthermore, this thesis introduces TreeC, a novel machine learning-based method that learns understandable yet efficient energy management strategies, applicable to many cases. The three key contributions of this thesis can be summarized as follows: 1) The developed TreeC method demonstrates competitive performance compared to a variety of energy management approaches, such as model predictive control and reinforcement learning, across a diverse set of energy management problems. The understandability of TreeC's energy management strategies greatly improves the transparency and trustworthiness of machine learning-based energy management. 2) The real-world experiment highlights the need to validate energy management methods on actual hardware and not solely on simulations. Both TreeC and model predictive control performed worse than in simulations, due to, respectively, insufficiently representative training data and the real-time input data not being reliable enough. 3) Energy management algorithms reduce costs in electrified Belgian dwellings, particularly in poorly insulated dwellings, electric vehicle chargers, and photovoltaic installations. However, on average, these cost savings do not yet outweigh the current costs and effort required to implement such algorithms.
- 中文摘要
从化石燃料的转型在能源管理方面带来了重大挑战。为确保可持续的能源未来,关键在于尽可能减少能源消耗,并将电力消耗和储能与来自低温室气体排放来源的生产同步。本论文探讨了用于管理电池、热泵及其他能源相关系统的多种能源管理方法。论文提出了三个模拟案例及一个真实实验,比较通过简单规则、优化和机器学习方法实现的能源管理。此外,本文介绍了TreeC,一种基于机器学习的新型方法,该方法学习易于理解且高效的能源管理策略,适用于多种情境。本论文的三大关键贡献可总结为:1)开发的TreeC方法在多样化的能源管理问题中,与模型预测控制和强化学习等多种能源管理方法相比表现出竞争力。TreeC 能源管理策略的可理解性极大提升了基于机器学习的能源管理的透明度和可信度。2)现实实验凸显了在实际硬件上验证能源管理方法的必要性,而不仅仅是在模拟上。TreeC 和模型预测控制的表现均不如模拟,原因分别是训练数据的代表性不足以及实时输入数据不够可靠。3)能源管理算法降低了比利时电气化住宅的成本,尤其是在绝缘不良的住宅、电动汽车充电桩和光伏装置中。然而,平均来看,这些节省的成本尚未超过目前实施此类算法所需的成本和努力。
ZeroVLN: Learning to Navigate without Supervised Fine-Tuning of the VLM Brain
ZeroVLN:学习在不进行VLM大脑监督微调的情况下导航
- Authors: Zixuan Huang, Yang Zhou, Guli Zhang, Sunzhu Li, Hongyan Xie, Zehao Chen, Fuzhen Zhuang, Kaixuan Wang, Xiaozhi Chen, Yikun Ban, Deqing Wang
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.26414
- Pdf link: https://arxiv.org/pdf/2611.26414
- Abstract
Representative vision-language navigation (VLN) methods typically couple a vision-language model (VLM) brain with a learnable action head and fine-tune the VLM brain on navigation-domain data to predict geometric goals. However, such adaptation can substantially alter base VLM weights and degrade general visual capabilities. We introduce ZeroVLN, a slow-fast navigation framework that achieves effective navigation with zero navigation-domain SFT of the VLM brain. Rather than teaching the brain to predict geometric goals, we equip it with a semantic tool-calling interface for conveying movement decisions to an action expert in natural language. We first adapt the action expert to carry out these movement requests, then jointly improve the brain's movement decisions and the action expert's continuous execution through closed-loop reinforcement learning (RL). Trained on the same navigation episodes, ZeroVLN substantially outperforms representative SFT-based hierarchical VLN methods on standard VLN-CE (continuous environments) benchmarks, while inducing only minor changes to the brain's weights and largely preserving its general capabilities. To our knowledge, ZeroVLN is the first slow-fast VLN framework with zero navigation-domain SFT of the VLM brain, and the first to jointly optimize its slow and fast policies through closed-loop agentic RL in VLN-CE.
- 中文摘要
代表性的视觉-语言导航(VLN)方法通常将视觉-语言模型(VLM)大脑与可学习的动作头结合起来,并根据导航领域数据微调VLM大脑以预测几何目标。然而,这种适应可能会显著改变基础VLM权重,并降低整体视觉能力。我们介绍了ZeroVLN,这是一种慢快导航框架,通过VLM大脑的零导航域SFT实现有效导航。我们不教大脑预测几何目标,而是配备语义工具调用接口,用自然语言向行动专家传达运动决策。我们首先让动作专家适应执行这些运动请求,然后通过闭环强化学习(RL)共同改进大脑的运动决策和动作专家的持续执行。在相同的导航集上训练后,ZeroVLN 在标准 VLN-CE(连续环境)基准测试中显著优于代表性的基于 SFT 的分层 VLN 方法,同时仅对大脑权重做出轻微变化,基本保持了其总体能力。据我们所知,ZeroVLN 是首个具有 VLM 大脑零导航域 SFT 的慢快 VLN 框架,也是首个通过闭环代理 RL 在 VLN-CE 中联合优化其慢与快策略的框架。
Grid Cells as Spectral Successor Features: A Reusable Basis for Continual Reinforcement Learning
网格单元作为谱继承特征:持续强化学习的可重复使用基础
- Authors: Élodie Côté-Gauthier, Doina Precup, Isabeau Prémont-Schwarz
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.26504
- Pdf link: https://arxiv.org/pdf/2611.26504
- Abstract
Continual reinforcement learning requires agents to adapt when the task changes while reusing what they know about the environment. Successor features separate dynamics from task-specific reward, but their effectiveness depends on the choice of base features. Taking the identification of entorhinal grid cells with eigenvectors of the successor representation as a design principle, we build this basis from eigenvectors of the transition matrix of a fixed reference policy. With such a basis, the successor features at a state-action pair are a linear combination of the base features at that same pair, which is not true in general, and the coefficients are known in closed form: each successor feature is its base feature multiplied by $\theta_i = 1/(1-\gamma\lambda_i)$, where $\lambda_i$ is its eigenvalue. The long-term return is thus predicted from the same features as the immediate reward, no successor-feature network or replay buffer is needed, and adapting to a new task reduces to a linear regression on the reward. The agent acts greedily on the action values of the reference policy, which is one step of policy improvement over it and suffices to reach a single goal. In a two-goal gridworld, with the uniform random walk as reference policy, whose eigenvectors are the low-frequency Laplacian eigenvectors, agents built this way maintain near-maximal return across every change of goal, whereas SimpleSF and DQN lose most of their return at each change.
- 中文摘要
持续强化学习要求代理在任务变化时适应,同时重复利用他们对环境的已知知识。后继特征将动态与任务特定奖励区分开来,但其有效性取决于基础特征的选择。以将内流网格单元与继任表示的特征向量识别为设计原则,我们从固定参考策略的转移矩阵特征向量构建该基。基于此基,状态-动作对上的继任特征是该对基特征的线性组合,这在一般情况下并非如此,且系数以闭合形式存在:每个继任特征是其基础特征乘以 $\theta_i = 1/(1-\gamma\lambda_i)$,其中 $\lambda_i$ 为其特征值。因此,长期回报是基于与直接奖励相同的特征来预测的,无需后续特征网络或重放缓冲区,适应新任务可简化为对奖励的线性回归。代理贪婪地依赖参考策略的行动值,这只是对其策略改进的一步,足以达成单一目标。在一个双目标网格世界中,以均匀随机游走为参考策略,其特征向量为低频拉普拉斯特征向量,这样构建的代理在每次目标变化时都能保持近乎最大收益,而 SimpleSF 和 DQN 则在每次变化时损失大部分收益。
Thermal-Guided Ensemble Reinforcement Learning for Regime-Adaptive Short-Term GPU Power Forecasting in AI Data Centers
热引导集成强化学习用于 AI 数据中心的模式自适应短期 GPU 功率预测
- Authors: Yuanshuo Zhang, Ying Zhang, Ziying Wang, Fanping Sui
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.26527
- Pdf link: https://arxiv.org/pdf/2611.26527
- Abstract
AI data centers (AIDCs) introduce bursty and regime-dependent load patterns, posing significant challenges for reliable energy management and predictive grid control. To address these challenges, this letter proposes a thermal-guided ensemble reinforcement learning framework for probabilistic AIDC load forecasting. A soft actor-critic (SAC) agent with a simplex-preserving policy adaptively weights LSTM, TCN, and Transformer predictors at every time step based on their realized errors. A thermal-guided neural correction estimates the ensemble residual, while a sensitivity-projection regularizer enforces the non-negative sensitivity of power to GPU utilization and temperature. The corrected ensemble output is further processed by a Bayesian probabilistic layer to produce predictive distributions and quantify forecasting uncertainty. Case studies on 1-second MIT Supercloud telemetry show that the proposed method reduces the normalized mean absolute error to about 1% across all computing regimes, compared with 4-11% for deep learning-based and ensemble learning benchmarks, and converges faster with narrower prediction intervals than a DRL ensemble without thermal guidance.
- 中文摘要
AI数据中心(AIDC)引入了突发性和依赖于系统的负载模式,这对可靠的能源管理和预测电网控制带来了重大挑战。为应对这些挑战,本函提出了一种热引导集成强化学习框架,用于概率性AIDC负载预测。一个采用单纯形保持策略的软演员-批评(SAC)代理,根据实际误差在每个时间步对LSTM、TCN和Transformer预测变量进行自适应加权。热引导神经修正估计集合残差,而敏感度-投影正则器则强制执行对GPU利用率和温度的非负功率敏感度。修正后的集合输出会通过贝叶斯概率层进一步处理,以生成预测分布并量化预测不确定性。1秒MIT超级云遥测的案例研究显示,所提方法在所有计算模式下将归一化平均绝对误差降低至约1%,相比基于深度学习和集成学习基准的4-11%为低,且在更窄的预测区间下收敛速度比无热制导的日行学习集成更快。
RL Post-Training in Diffusion Transformers Is Activation Steering
在扩散变压器中进行的强化后培训是激活引导
- Authors: Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag
- Subjects: Subjects:
Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.26627
- Pdf link: https://arxiv.org/pdf/2611.26627
- Abstract
Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.
- 中文摘要
强化学习(RL)后期训练显著提升了扩散变换器的视觉质量、文本渲染和合成精度。由于渲染请求文本或组合多个对象等能力本质上依赖提示,自然假设改进需要学习复杂的、依赖输入的变换。为了分离出输入无关移位解释了多少这些增益,我们用刻意简化的参数化探测每个RL调优模型:在冻结基模型的块输出中加入一组固定的引导向量。在SD3.5-M上,这些向量所包含参数比RL调谐的LoRA少65倍(0.29M对188M)。我们对其拟合以匹配公开RL调优检查点的速度预测,但目标中没有任何奖励。尽管每个提示、标记和采样种子的向量都完全相同,但它们恢复了六个RL调优图像模型的90.0%-96.3%的奖励收益,在两个RL调优视频模型中恢复了89.8%和111.9%。在初始化为零并通过在线强化学习训练后,仅这些向量就超过了基于权重的RL在文本渲染(OCR)和人类偏好(PickScore)上SD3.5-M的奖励增益。因此,恒定的调整足以满足策略梯度RL训练后大部分收益:在不进行新的输入依赖变换的情况下,它会引导预训练生成器朝其已能计算的方向发展。
Keyword: diffusion policy
There is no result