生成时间: 2026-11-11 23:56:03 (UTC+8); Arxiv 发布时间: 2026-11-11 20:00 EST (2026-11-12 09:00 UTC+8)

今天共有 40 篇相关文章

Keyword: reinforcement learning

Topology-Aware Reinforcement Learning for CAD Decomposition toward All-Hexahedral Meshing

拓扑感知强化学习用于CAD分解以实现全六面体网格

• Authors: Jiaming Peng, Zhihao Zheng, Rui Wang, Qinglin Wang, Jing Xiao, Jie Liu, Na Lei, Xinhai Chen
• Subjects: Subjects: Graphics (cs.GR); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12930
• Pdf link: https://arxiv.org/pdf/2611.12930
• Abstract Automatic hexahedral meshing for complex CAD models has long been regarded as the "holy grail" of mesh generation. The fundamental challenge arises from feature misalignment, where orthogonal or interacting features coexist, making direct hexahedral meshing infeasible and requiring decomposition into simpler, meshable blocks. While existing decomposition methods, including frame-field pipelines and learning-based techniques, have reduced manual intervention, they can remain sensitive to geometric and topological complexity and may fail to produce valid decompositions for challenging feature configurations. Specifically, their action spaces are typically restricted to surface-based splitting, which often yields invalid or unmeshable decompositions. To address these limitations, we propose DecompRL, a deep reinforcement learning framework for CAD decomposition toward hexahedral meshing. We design a set of topology-aware internal-splitting actions that leverage both global topological structure and local geometric property to effectively separate misaligned features. Furthermore, we introduce geometric rewards together with meshability feedback to guide the learning process toward valid decompositions and high-quality meshes. Extensive experiments on ABC, Fusion360, NIST, and DeepCAD show that DecompRL achieves the highest success rate among the evaluated learning-based methods on three of the four benchmarks, while maintaining competitive mesh quality. The measured LMFF-to-DecompRL runtime ratio reaches up to two orders of magnitude; this compares the full LMFF procedure with DecompRL decomposition alone, excluding subsequent mesh generation. The largest SR improvement is observed on Fusion360, where DecompRL reaches a success rate of 59.6%.
• 中文摘要 复杂CAD模型的自动六面体网格化长期以来被视为网格生成的“圣杯”。根本性挑战源于特征错位，即正交或相互作用特征共存，使得直接六面体网格化不可行，需要分解为更简单、可网格的块。虽然现有分解方法，包括帧场流水线和基于学习的技术，手工干预减少，但它们可能对几何和拓扑复杂性敏感，且可能无法产生针对复杂特征配置的有效分解。具体来说，它们的作用空间通常限于基于曲面的分割，这常常导致无效或不可网格化的分解。为解决这些限制，我们提出了DecompRL，一种用于CAD分解以实现六面体网格化的深度强化学习框架。我们设计了一套拓扑感知的内部拆分动作，利用全局拓扑结构和局部几何性质，有效分离错位特征。此外，我们引入几何奖励和网格可行性反馈，引导学习过程朝着有效分解和高质量网格迈进。在ABC、Fusion360、NIST和DeepCAD上的大量实验显示，DecompRL在四个基准测试中有三个在评估的基于学习的方法中成功率最高，同时保持了具有竞争力的网格质量。测得的LMFF与DecompRL运行时间比达到两个数量级;这比较了完整的LMFF过程与单独的DecompRL分解过程（不含后续网格生成）。SR最大的提升出现在Fusion360上，DecompRL的成功率达到59.6%。

Reinforcement Learning based control of Cancer Chemotherapy treatments

基于强化学习的癌症化疗控制

• Authors: Jayakody Kankanamalage Chamani Shiranthika
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12940
• Pdf link: https://arxiv.org/pdf/2611.12940
• Abstract Reinforcement Learning has achieved tremendous success in recent years. Extending its scope from the traditional complex games such as Atari, Go, Pokeman, and Chess, reinforcement learning has caught researchers' attention on modelling real-life scenarios. Personalized medicine and the dynamic treatment regimen are two such preliminary research fields; researchers are focusing their mindsets currently. Estimating the optimal dynamic treatment regimens from the observational real clinical data is being popularized among the researchers, widening its scope towards more advanced versions of reinforcement learning algorithms. This research is about a research study that focused on reinforcement learning techniques that have been approached in the context of cancer chemotherapy treatments. Primarily this thesis addresses the result of the study by means of the state-of-the-art on reinforcement learning techniques used in generating the optimal policy for sequential treatments, including the existing online reinforcement learning approaches. Thereby a simulation for cancer chemotherapy via traditional Q-learning would be presented. Secondly, the study will extend towards using offline reinforcement learning in the optimal policy generation. Thus, the proposed solution follows a data-driven approach, which works on the real human colon cancer clinical data. The novelty of the proposed solution resides in the implementation of an offline reinforcement learning approach by means of a model-based architecture using the conservative Q-learning and the model-based offline policy optimization algorithm. As per the experimental results and the evaluations, it was revealed that the proposed Chemo RL model caters well the problem of cancer chemotherapy treatment scheduling and serves as a supportive model for the junior oncologists in deciding their treatment decisions.
• 中文摘要 强化学习近年来取得了巨大成功。强化学习从传统复杂游戏如雅达利、围棋、宝可人和国际象棋扩展，吸引了研究人员对现实场景建模的关注。个性化医疗和动态治疗方案是这两个初步研究领域;研究人员目前正聚焦于思维模式。从观察性真实临床数据估算最佳动态治疗方案正被研究人员推广，其范围也逐渐扩展到更高级的强化学习算法版本。本研究聚焦于癌症化疗背景下被采用的强化学习技术的研究。本论文主要通过先进的强化学习技术，探讨用于生成序列治疗最优策略的强化学习技术，包括现有的在线强化学习方法，研究成果。由此，将呈现传统Q学习中的癌症化疗模拟。其次，研究将扩展到在最优策略生成中使用离线强化学习。因此，所提出的解决方案采用数据驱动方法，基于真实的人类结肠癌临床数据。该方案的新颖之处在于通过基于模型的架构，采用保守的Q学习和基于模型的离线策略优化算法，实现离线强化学习方法。根据实验结果和评估，所提出的化疗强化学习模型很好地解决了癌症化疗治疗的排期问题，并为初级肿瘤科医生在决策治疗决策中提供了支持性模型。

Exploration via Exploitation: The Blessing of Reward Diversity in Personalized Federated RL

通过利用探索：个性化联邦强化学习中奖励多样性的祝福

• Authors: Jiin Woo, Gauri Joshi, Yuejie Chi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.12964
• Pdf link: https://arxiv.org/pdf/2611.12964
• Abstract While federated reinforcement learning (RL) has traditionally guaranteed high efficiency in homogeneous environments, collaboration is often hindered when agents possess distinct goals. In this paper, we flip this paradigm, framing reward heterogeneity not as a hurdle to consensus, but as a structural asset that catalyzes exploration. We first propose Personalized Federated Upper Confidence Bound Value Iteration (PF-UCBVI), which achieves optimal linear speedup by decoupling shared dynamics from personalized objectives. To eliminate the risks of explicit exploration, we then introduce Personalized Federated Exploration-Free Value Iteration (PF-EFVI), a purely greedy algorithm that leverages reward diversity to ensure state-action coverage. We prove that PF-EFVI attains logarithmic regret without explicit exploration bonuses under sufficient reward diversity. Our results reveal that agent disagreement shifts the exploration burden from temporal complexity to spatial diversity, enabling efficient collective learning without explicit exploration.
• 中文摘要 虽然联邦强化学习（RL）传统上保证了同质环境中的高效率，但当代理拥有明确目标时，协作往往受阻。本文将这一范式颠覆，将奖励异质性框架化为推动探索的结构性资产，而非共识障碍。我们首先提出了个性化联合上置信度值叠代（PF-UCBVI），通过将共享动态与个性化目标解耦实现最佳线性加速。为消除显式探索的风险，我们引入了个性化联邦探索无价值迭代（PF-EFVI），这是一种纯贪婪算法，利用奖励多样性确保状态动作覆盖。我们证明在足够多的奖励多样性下，PF-EFVI在无显式探索加成的情况下实现对数级遗憾。我们的结果显示，代理分歧将探索负担从时间复杂性转移到空间多样性，从而实现高效的集体学习而无需显式探索。

Branch-and-Commit for Frozen RL Agents: When and Where to Spend Test-Time Compute

冻结RL代理的分支与提交：何时何地将测试时计算投入

• Authors: Sahar Admoni, Assaf Hallak, Yftah Ziser, Ofra Amir
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13003
• Pdf link: https://arxiv.org/pdf/2611.13003
• Abstract Reinforcement learning agents are often deployed with policies that cannot be retrained or modified. In digital environments whose states can be copied and resumed, an operator has another option: sample several continuations from the current state, evaluate them, and continue from the endpoint of the selected branch. We call this procedure branch-and-commit. We study when it improves return, what produces the improvement, and where its computational budget should be spent. Across four environments, two frozen policies, five continuation verifiers, and a model-free selection rule, we find that the benefit of branching depends on dispersion: the difference between the best candidate return and the mean candidate return. Branching provides no benefit in low-dispersion settings, while increasing return by 12.8% in our main Craftax-Medium setting. Most of this gain comes not from selecting better actions, but from selecting among different stochastic realizations of the same action. Several competent verifiers achieve similar episode returns, suggesting that verifier quality is not the main constraint at our operating budget. Instead, computation is better allocated to states where useful dispersion is likely. On held-out seeds, gating branching using predicted dispersion retains 69% of the improvement from uniform branching while using 48% as many branching calls. These results characterize when branch-and-commit is useful and clarify what its gains represent.
• 中文摘要 强化学习代理通常部署的策略无法重新训练或修改。在状态可复制和恢复的数字环境中，操作员还有另一种选择：从当前状态抽样多个延续，评估它们，并从所选分支的端点继续。我们称此过程为分支并提交。我们研究何时提升收益、何物产生改进，以及计算预算应花在哪里。在四个环境、两个冻结策略、五个延续验证器和一个无模型选择规则中，我们发现分支的益处依赖于离散：即最佳候选回报与平均候选回报之间的差异。分支在低离散环境中无益处，而在我们主要的Craftax-Medium设置中，收益提升12.8%。大部分收益并非来自选择更好的动作，而是来自于在同一动作的不同随机实现中选择。几位合格的验证者实现了类似的集数回报，表明验证器质量并非我们运营预算的主要限制。相反，计算更适合分配到可能存在有用离散的状态。对于保留的种子，使用预测离散进行门控分支仍保留了69%的均匀分支改进，而分支调用次数减少了48%。这些结果描述了分支提交的有效性，并阐明了其收益。

FaST: Factored Synthesis of Transitions for Offline Reinforcement Learning

FaST：离线强化学习中的因式分解过渡综合

• Authors: Daniel Campa, Spandan Das, Srinjoy Das
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13013
• Pdf link: https://arxiv.org/pdf/2611.13013
• Abstract Synthetic data augmentation promises to scale offline reinforcement learning beyond fixed logged datasets, but leading diffusion-based generators model the transition tuple (state, action, next state, reward, terminal) as a monolith: no single factor can be natively redrawn in isolation. We present FaST (Factored Synthesis of Transitions), which factors transitions via the chain rule into state, action, dynamics, reward, and terminal stages, fitting each with an estimator matched to its statistical properties. Across twelve D4RL locomotion and maze benchmarks, TD3+BC and IQL agents trained on FaST transitions match the SynthER diffusion baseline while generating transitions over 90x faster. The explicit action-given-state conditional can also be re-queried for a fixed state, enabling targeted action synthesis: drawing N candidate actions per generated state and keeping the highest-advantage one under a value function fit to the same data, thereby synthesizing improved behavior during generation rather than filtering for it post-hoc. Best-of-10 synthesis beats training on real data on 11 of 12 medium and medium-replay settings (+8.9 D4RL points on average, up to +28.4), and is competitive with advantage-filtered SynthER at a fraction of its generation cost. On expert mixtures both mechanisms degrade identically, tracing the failure to the advantage estimate itself rather than to either targeting mechanism.
• 中文摘要 合成数据增强有望将离线强化学习扩展到固定记录数据集之外，但领先的基于扩散的生成器将过渡元组（状态、动作、下一个状态、奖励、终端）建模为一个整体：没有单一因素可以单独原生重绘。我们提出了FaST（过渡分解综合），通过链式规则将过渡分解为状态、动作、动态、奖励和终端阶段，并为每个阶段匹配其统计属性的估计量。在十二个D4RL移动和迷宫基准测试中，训练于FaST转变的TD3+BC和IQL代理匹配SynthER扩散基线，同时生成跃迁速度超过90倍。显式的动作-给定-状态条件句也可以重新查询固定状态，实现有针对性动作综合：每个生成状态抽取N个候选动作，并保持价值函数下最高优势的动作拟合于同一数据，从而在生成过程中综合改进行为，而非事后过滤。10局2胜综合优于在12种中等和中等重放设置中11种真实数据训练（平均+8.9 D4RL点，最高+28.4），且其生成成本仅为优势滤波SynthER的一小部分。在专家混合条件下，两种机制的劣化相同，失败原因主要归因于优势估计本身，而非任一定向机制。

GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning

GTAGym：真实的AAA游戏任务环境，用于评估与强化学习

• Authors: Jiajun Chen, Shicheng Fang, Jiahao Lu, Yuxin Wang, Xipeng Qiu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.13028
• Pdf link: https://arxiv.org/pdf/2611.13028
• Abstract AAA games offer a rich testbed for evaluating agent perception, reasoning, and control. However, evaluating whether agents can complete tasks in AAA games is challenging and often human-annotated because task outcomes are often difficult to verify automatically, let alone applying it for Reinforcement Learning (RL). We introduce GTAGym, an environment and framework for evaluation and RL on GTA V tasks. Its evaluation benchmark, GTAVBench, derived from GTA V main-story missions, contains 103 online tasks reconstructed across nine task families, alongside 47 image-based shooting and sniping tasks using GTA V imagery. A web server provides access to the online environment and evaluation interface, supporting parallel testing through screenshots and keyboard-mouse actions with automatic engine-state outcome checks. We evaluate six frontier LLMs on GTAVBench with textual task instructions and on its GTAVBench-AGI branch, where instructions appear only in the first screenshot and a harness automatically compresses interaction history. The best configuration achieves 70.9% overall accuracy in GTAVBench-AGI, yet aerial navigation, evasion and interception remain difficult. GTAGym also provides a scalable generator that samples task parameters within user-specified ranges and option sets under fixed templates, creating batches of 3D and image-based shooting environments for RL with automatic reward and termination signals. LoRA RL of Qwen3.8-27B on generated on-foot navigation tasks improves the in-domain performance and transfers to out-of-domain task categories, and native GTA V tasks from 15% to 23%.
• 中文摘要 AAA游戏为评估智能体的感知、推理和控制提供了丰富的测试平台。然而，评估智能体是否能完成AAA游戏中的任务具有挑战性，且常常需要人工注释，因为任务结果往往难以自动验证，更不用说应用于强化学习（RL）。我们介绍GTAGym，一个用于GTA V任务评估和强化学习的环境和框架。其评估基准GTAVBench源自GTA V主线任务，包含103个在线任务，跨越九个任务族重建，以及47个基于图像的射击和狙击任务，均基于GTA V图像。一个网页服务器提供在线环境和评估界面访问，支持通过截图和键盘鼠标操作进行并行测试，并自动进行引擎状态结果检查。我们在GTAVBench及其GTAVBench分支上评估了六个前沿大型语言模型，该分支仅在首张截图中出现指令，且线束自动压缩交互历史。最佳配置在GTAVBench-AGI中实现70.9%的总体准确率，但空中导航、规避和拦截依然困难。GTAGym还提供可扩展生成器，在用户指定范围和选项集内采样任务参数，基于固定模板，创建批量3D和基于图像的RL拍摄环境，并自动奖励和终止信号。Qwen3.8-27B的LoRA RL在生成的步行导航任务上提升了域内性能，并可转移至域外任务类别，并将原生GTA V任务从15%提升至23%。

Building World Models from Intrinsic Motivation

从内在动机构建世界模型

• Authors: Reece Keller, Declan Kutscher, Xaq Pitkow, Aran Nayebi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13113
• Pdf link: https://arxiv.org/pdf/2611.13113
• Abstract In the absence of extrinsic rewards or task supervision, human behavior is guided by intrinsic motivation---neural signals that drive curiosity, creativity, play, and other essential facets of intelligence. Intrinsic motivation plays an important developmental role in building robust world models that enable generalization and adaptation. Yet existing studies leave open how a broad range of intrinsic motivations interacts with world-model representation learning to support downstream control. This limits our understanding of which intrinsically collected experiences build reusable models. To address these gaps, we integrate 24 intrinsic objectives, including established methods and explicit variants spanning curiosity, entropy, and competency, into a common model-based reinforcement learning framework. We evaluate the world models they build through zero-shot, few-shot, and multi-task control in locomotion, manipulation, and open-ended gaming environments, using three representation-learning strategies: observation reconstruction, latent prediction, and redundancy reduction. Curiosity-driven exploration with observation reconstruction yields the strongest aggregate family profile, although the best individual pairing depends on the experiment. In compositional vision-based tasks, intrinsic motivation derived from both episodic and lifelong novelty---providing a complementary memory to the world model's generic recurrent state---achieves the best zero-shot transfer and few-shot adaptation. These findings motivate designing intrinsic motivation and representation learning jointly. Our results demonstrate world models as a powerful tool for measuring the quality of intrinsically motivated experience, and positions intrinsic motivation as a key ingredient for building world models autonomously.
• 中文摘要 在缺乏外在奖励或任务监督的情况下，人类行为由内在动机---神经信号引导，这些信号驱动好奇心、创造力、游戏性及其他智力的关键方面。内在动机在构建支持泛化和适应的稳健世界模型中发挥重要发展作用。然而，现有研究对广泛的内在动机如何与世界模型表征学习相互作用，以支持下游控制保持开放。这限制了我们对哪些内在收集经验构建可重复使用模型的理解。为弥补这些空白，我们将24个内在目标，包括既有方法及涵盖好奇心、熵和能力的显式变体，整合进一个通用的基于模型的强化学习框架。我们评估他们通过零次、少数次和多任务控制构建的世界模型，应用于运动、操控和开放式游戏环境中，采用三种表征学习策略：观察重建、潜在预测和冗余减少。基于好奇心的探索与观察重建产生最强的聚合家族谱，尽管最佳个体配对取决于实验。在基于构图视觉的任务中，源自情节性和终生新颖性的内在动机---提供对世界模型泛泛循环状态的补充记忆---实现了最佳的零时效转移和少样本适应。这些发现激励人们共同设计内在动机与表征学习。我们的结果表明世界模型是衡量内在动机体验质量的强大工具，并将内在动机定位为自主构建世界模型的关键要素。

Cognitive Multi-Agent Search and Rescue: Language-Grounded Hierarchical Reinforcement Learning for Collaborative Robotics

认知多智能体搜救：基于语言的协作机器人分层强化学习

• Authors: Sepideh Ravanbakhsh, Hamidreza Moradi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2611.13161
• Pdf link: https://arxiv.org/pdf/2611.13161
• Abstract Effective search and rescue (SAR) operations require autonomous systems capable of coordinating multiple heterogeneous agents, adapting to dynamic hazards, and aligning with human mission intent. Existing language-guided reinforcement learning (RL) methods primarily focus on single-agent instruction following and lack mechanisms for team-level coordination in complex, partially observable environments. This paper presents a Cognitive Multi-Agent Framework based on Language-Grounded Hierarchical Reinforcement Learning (LG-HRL) that integrates natural-language understanding, hierarchical policy decomposition, and a shared cognitive workspace to achieve cohesive multi-agent behavior in SAR scenarios. Human instructions are interpreted by a large language model (LLM) and transformed into structured semantic cues that guide high-level strategy selection and role assignment across UAV and UGV agents. A formal Language-Grounded Multi-Agent Hierarchical Decision Process (LG-MAHDP) is introduced to model the interplay between semantic grounding, hierarchical control, and distributed coordination. Comprehensive experiments in a simulated disaster environment show that LG-HRL improves learning speed by 2.3 times, increases coordination coherence by 37%, and enhances subgoal accuracy and hazard avoidance by 46% and 31%, respectively. The system achieves 20-30% higher victim rescue rates, demonstrates emergent role specialization, and adapts immediately to human verbal updates. Ablation studies confirm that both the shared cognitive workspace and language grounding are critical to overall system performance. Overall, this work establishes a new paradigm for language-guided multi-agent SAR robotics, demonstrating that semantic grounding combined with hierarchical reasoning enables safer, more efficient, and more interpretable autonomous rescue operations.
• 中文摘要 有效的搜救（SAR）行动需要能够协调多个异构智能体、适应动态危害并与人类任务意图保持一致的自主系统。现有的语言引导强化学习（RL）方法主要关注单智能体指令的跟随，缺乏复杂且部分可观察环境中团队级协调的机制。本文提出了基于语言基础层级强化学习（LG-HRL）的认知多智能体框架，整合自然语言理解、分层策略分解和共享认知工作空间，以实现SAR场景中的多智能体行为。人类指令由大型语言模型（LLM）解释，转化为结构化语义线索，指导无人机和UGV智能体的高级策略选择和角色分配。引入了正式的语言基础多智能体层级决策过程（LG-MAHDP），以模拟语义基础、层级控制与分布式协调之间的相互作用。在模拟灾难环境中的综合实验显示，LG-HRL将学习速度提升2.3倍，协调一致性提升37%，子目标准确性和危险规避分别提升46%和31%。该系统实现了20-30%的受害者救援率，展现出涌现的角色专精，并能立即适应人类语言更新。消融研究证实，共享认知工作空间和语言基础对整体系统性能至关重要。总体而言，这项工作为语言引导多智能体SAR机器人建立了新的范式，证明语义基础与层级推理结合，使自主救援行动更安全、更高效且更具解释性。

Making Strides in Symbolic Imitation Learning: Offline Demonstrations, Online Tuning

符号模仿学习取得进展：离线演示，在线调谐

• Authors: Bryce MacInnis, Malcolm I. Heywood
• Subjects: Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2611.13182
• Pdf link: https://arxiv.org/pdf/2611.13182
• Abstract Deep reinforcement learning (DRL) methods have been the dominant paradigm in continuous control tasks due to their efficacy in function approximation, while interpretable evolutionary methods have lagged largely behind for the opposite reason. In this work, we introduce Symbolic Imitation Learning, a framework for distilling an oracle policy through an offline dataset into a decision list with linear model motion primitives. We show that this, in conjunction, with online Augmented-Random Search (ARS) tuning of the constants and motion primitives approximates the performance of the oracle across 20 benchmark RL tasks and exceeds the oracle in 11 of those 20 tasks while simultaneously producing parsimonious, interpretable policies that domain experts can better understand.
• 中文摘要 深度强化学习（DRL）方法因其在函数近似方面的有效性，一直是连续控制任务中的主导范式，而可解释的进化方法则因相反原因大幅落后。本研究介绍了符号模仿学习，这是一种通过离线数据集将预言机策略提炼为带有线性模型运动原语的决策列表的框架。我们展示了结合在线增强随机搜索（ARS）对常数和运动原语的调优，近似了预言机在20个基准强化学习任务中的表现，并在其中11项任务中超过了预言机，同时产生了简洁且可解释的策略，让领域专家更容易理解。

Wrong the Same Way: Cross-Prompt Sharing Both Suppresses and Amplifies Reward Hacking in RLVR

同样错误：跨提示共享同时抑制并放大RLVR中的奖励黑客行为

• Authors: Michael Li, Rohan Pandey, Ray Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.13218
• Pdf link: https://arxiv.org/pdf/2611.13218
• Abstract Two accounts of how reinforcement learning with verifiable rewards (RLVR) interacts with an imperfect verifier coexist: in one, reward hacking is inherited from the base model; in the other, RL amplifies it. We study both in a controlled environment small enough to compute the entire response distribution exactly. We vary the universality of a leak, or the fraction of prompts that accept the same wrong response, and the verifier's static false-positive rate. At its KL-regularized optimum, RLVR acts as a filter that reweights the accepted set without altering its composition, so the hack rate is inherited exactly. Taking that as a null model, a known decomposition writes the drift from it as a sum over prompts of covariances between reward and the empirical tangent kernel. The terms from other prompts vanish without parameter sharing. With sharing, every prompt that accepts a wrong answer pushes it up and every prompt that rejects it pushes it down through the baseline, in proportion to the model's probability of producing it there. A shared-parameter policy thus drives a leak that few prompts accept below the base rate and amplifies one that most prompts accept. At every convergent learning rate under Adam, SGD, and sampled GRPO, hacking ends higher at high universality than at low, but whether either end crosses the base rate depends on the step size. Independent per-prompt copies remove the dependence. The direction of this push depends on neither the verifier's error rate nor universality alone, but on how much of the model's kernel-weighted probability of a wrong answer falls on prompts that accept it, which frozen base model controls support. Verifier fixes should target the mistakes the model repeats across prompts: two patches that remove equally many errors differ by up to 38 points of true accuracy. We propose this alignment as a hypothesis for why earlier studies disagree.
• 中文摘要 关于带有可验证奖励的强化学习（RLVR）如何与不完美验证者共存的两种说法：一种是从基础模型继承的奖励黑客行为;另一种是强化性编程（RL）放大了它。我们在一个足够小的受控环境中研究这两者，以精确计算整个响应分布。我们改变泄漏的普遍性，即接受相同错误响应的提示比例，以及验证者的静态误报率。在其KL正则化的最优状态下，RLVR充当过滤器，在不改变其组合的情况下重新加权被接受集合，因此黑客率是完全继承的。以此为零模型，已知分解将其漂移写为奖励与经验切核协方差提示的求和。其他提示词的项在不共享参数的情况下消失。共享时，每个接受错误答案的提示都会将其推高，拒绝的提示按模型生成该提示的概率按基线下推。共享参数策略导致泄漏，少有提示接受低于基础率，放大大多数提示接受的泄漏。在Adam、SGD和采样GRPO下的每个收敛学习率下，破解在高普适性时结束率高于低，但是否跨越基准率取决于步长。独立的每个提示符复制消除了依赖性。推送方向不仅取决于验证者的错误率或普遍性，还取决于模型中接受错误率的内核加权错误概率中，接受该提示的比例，而冻结基础模型控制支持。验证器修复应针对模型在提示中重复出现的错误：两个消除相同错误的补丁，真实准确度相差最多38个点。我们提出这种对齐假设，解释早期研究为何不一致。

Off-policy Learning with Excursion Policies

带游学政策的非政策学习

• Authors: Jiamin He, Mark Rowland, Zhaohan Daniel Guo, Hado van Hasselt, Diana Borsa
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.13275
• Pdf link: https://arxiv.org/pdf/2611.13275
• Abstract Off-policy reinforcement learning (RL) tries to learn the value of a target policy while data comes from a different behavior policy. This often requires trading off bias for variance or stability. One way to trade these off is to linearly interpolate between the behavior and target policies, but this represents only one point in a broader space of interpolation mechanisms. We introduce excursion policies--a structured alternative where the agent follows the target policy for a multi-step "excursion" before reverting to the behavior policy. We derive dynamic programming results for these policies, establishing the theoretical basis for their use in RL. We further develop provably sound off-policy temporal-difference (TD) algorithms for excursion value estimation. We find that excursion policies provide an efficiency-stability trade-off comparable to mixture policies in continuous control benchmarks, and our analysis illustrates conceptual differences in their behavior.
• 中文摘要 非策略强化学习（RL）试图在数据来自不同行为策略时，学习目标策略的价值。这通常需要在方差或稳定性之间进行权衡偏差。一种权衡方式是线性插值行为与目标策略之间，但这仅代表更广泛插值机制中的一个点。我们引入了行程策略——一种结构化替代方案，代理人在返回行为策略前，先遵循目标策略进行多步“行程”。我们推导出这些策略的动态规划结果，为其在强化学习中的应用奠定理论基础。我们还进一步开发了可证明合理的偏差时差（TD）算法用于行程值估计。我们发现行程策略提供了类似于连续控制基准中混合策略的效率与稳定性权衡，分析展示了它们行为上的概念差异。

An Analytical Model of Compute-limited Multistage Training Pipelines

计算限制多阶段训练流水线的解析模型

• Authors: Nishil Patel, Jin Hwa Lee, Basile Confavreux, Andrew Saxe
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13276
• Pdf link: https://arxiv.org/pdf/2611.13276
• Abstract Modern foundation models are trained through multiple stages, yet how these stages interact and how compute should be allocated across them remain poorly understood. We introduce an analytically tractable high-dimensional model combining unsupervised pretraining with a linear autoencoder, supervised fine-tuning (SFT) with noisy stepwise labels, and reinforcement learning (RL) with sparse episodic rewards. Using random matrix theory and an operator-projection closure, we derive approximate low-dimensional learning dynamics that closely match simulations. We show that finite pretraining data imposes a representation-dependent performance limit, with Bayes classification error scaling as the inverse square root of the pretraining sample ratio, and observe the same exponent in asymptotic SFT and RL errors. Better pretraining also reduces sensitivity to noisy supervision. SFT initialization shortens the initial RL plateau induced by sparse rewards, an effect we also observe in OLMo-2 1B on GSM8K. Longer SFT warm starts additionally improve asymptotic RL performance. Optimizing compute across all three stages reveals that the optimal allocation depends on supervision quality and total budget, with a non-monotonic dependence of the RL compute share on budget. Finally, a complementary compositional model shows how RL selectively strengthens task-relevant pretrained skills while largely preserving unused skills. Together, these results identify mechanisms linking representation quality, feedback structure, and compute allocation in multistage training.
• 中文摘要 现代基础模型通过多个阶段训练，但这些阶段的相互作用以及如何分配计算仍然不充分。我们引入了一个解析可解的高维模型，结合了无监督预训练与线性自编码器、带噪声的监督微调（SFT）以及带有稀疏情节奖励的强化学习（RL）。利用随机矩阵理论和算符投影闭包，我们推导出与模拟高度匹配的近似低维学习动态。我们表明有限预训练数据会施加一个依赖表示的性能限制，贝叶斯分类误差的尺度为预训练样本比的倒平方根，并在渐近SFT和RL误差中观察到相同的指数。更好的预训练还降低了对噪声监督的敏感性。SFT初始化缩短了由稀疏奖励引起的初始强化学习平台期，这一效应我们在GSM8K的OLMo-2 1B中也观察到。更长的SFT热启动还能提升渐近强化学习表现。在这三个阶段优化计算表明，最优分配取决于监督质量和总预算，且强化学习计算份额与预算的非单调依赖。最后，一个互补的组合模型展示了强化强化任务相关预训练技能，同时在很大程度上保留未使用的技能。这些结果共同识别了多阶段训练中连接表征质量、反馈结构和计算分配的机制。

Network-Co-Designed Dynamic Sparsification for Dependable Distributed Robot Learning over 3GPP Networks

网络联合设计的动态稀疏化，用于3GPP网络上的可靠分布式机器人学习

• Authors: Géza Szabó, Boldizsár Bunda
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.13282
• Pdf link: https://arxiv.org/pdf/2611.13282
• Abstract Robot fleets performing distributed learning often face volatile uplink conditions like bandwidth variation, packet loss, and latency spikes that delay or destabilize training. We propose a network-co-designed dynamic sparsification loop that couples model-update compression to network-facing byte budgets and time constraints. We describe two realizations: (i) a standards-aligned SA6/5GC mapping that leverages CAPIF with NEF/PCF/NWDAF, enriched with AI-specific semantics; and (ii) a VASEAL-style role-reversed loop where the learning system exposes learning-state feedback and receives bounded, time-limited control intents. We implement the mechanism in an asynchronous federated reinforcement learning framework with heterogeneous simulated quadruped robots and reproducible uplink impairments via emulated network links. Results indicate that substantial communication reduction is achievable under dynamic constraints without materially degrading learning progress in the studied setting.
• 中文摘要 执行分布式学习的机器人队伍常常面临带宽变化、丢包和延迟尖峰等波动性上行条件，这些都会延迟或破坏训练。我们提出了一个网络联合设计的动态稀疏化环路，将模型更新压缩与面向网络的字节预算和时间约束相结合。我们描述了两种实现方式：（i）符合标准的SA6/5GC映射，利用CAPIF与NEF/PCF/NWDAF结合，并丰富了AI特有语义;以及（ii）类似VASEAL的角色反转循环，学习系统暴露学习状态反馈并接收有界、时间限制的控制意图。我们将该机制实现为异步联合强化学习框架，采用异构模拟四足机器人和通过模拟网络链路可重复的上行障碍。结果表明，在动态约束下，在研究环境中可以实现显著的交流减少，而不会实质性地降低学习进展。

Enforcing Axioms for AI Alignment under Loss-Based Rules

在基于损失的规则下执行AI对齐公理

• Authors: Alexandros Hollender, Sonja Kraiczy
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13369
• Pdf link: https://arxiv.org/pdf/2611.13369
• Abstract Many alignment methods, most notably reinforcement learning from human feedback (RLHF), train reward models by minimizing a loss on pairwise comparisons of model responses. Inspired by principle-guided methods such as Constitutional AI, we treat each principle as a voter that compares responses according to that principle, for example, preferring the more helpful or less toxic response. We represent these voters as linear directions in a fixed embedding space, a simplifying assumption motivated by the Linear Representation Hypothesis. In this linear social choice model, Ge et al. (2024) showed that an optimal linear reward model can violate Pareto optimality (PO): From the principles-as-voters lens, this means a response A can be less helpful and more toxic than B, yet still receive a higher reward. We ask whether richer reward models or changes to comparison data can recover PO within loss-based training. We show that PO can still fail for polynomials of any fixed degree and one-hidden-layer ReLU multilayer perceptrons (MLPs) of any fixed width $ d\ge 2 $ with suitable finite parameter bounds. We then recover PO for loss-based training of linear rewards under suitable assumptions in an idealized setting with uniform comparison directions and equal comparison lengths. These results motivate dataset design as an approach to recovering PO in constitutional-style alignment while retaining loss-based training.
• 中文摘要 许多比对方法，最著名的是人类反馈强化学习（RLHF），通过最小化模型反应两两比较中的损失来训练奖励模型。受原则指导方法如宪法人工智能启发，我们将每个原则视为一个投票者，根据该原则比较不同的反应，例如偏好更有利或毒性较小的反应。我们将这些投票者表示为固定嵌入空间中的线性方向，这一简化假设源自线性表征假设。在该线性社会选择模型中，Ge等人（2024）证明最优线性奖励模型可能违反帕累托最优性（PO）：从原则即选民的视角来看，这意味着反应A可能比B更无助且更具毒性，但仍获得更高的奖励。我们探讨更丰富的奖励模型或对比较数据的修改是否能在基于损失的训练中恢复PO。我们证明，对于任意固定次数的多项式和任意固定宽度 $ d\ge 2 $ 的单隐藏层 ReLU 多层感知器（MLP），且参数有限，PO 仍然可能失败。随后，我们在理想化环境下，基于损失的线性奖励训练恢复了基于损失的 PO，且在理想化的环境中，采用统一比较方向和相等的比较长度。这些结果激励了数据集设计，作为一种在保留基于损失训练的前提下，以宪法风格对齐恢复 PO 的方法。

Discrete Model-Based Diffusion for Combinatorial Optimization in Multi-Robot Task Allocation

基于离散模型的扩散技术用于多机器人任务分配中的组合优化

• Authors: Haejoon Lee, Thomas Marshall Vielmetti, Xinyi Wang, Dimitra Panagou
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2611.13382
• Pdf link: https://arxiv.org/pdf/2611.13382
• Abstract Decision making in multi-agent and multi-robot systems often involves discrete decision variables, leading to NP-hard combinatorial optimization problems that must be solved within seconds for near real-time implementation. Denoising diffusion has emerged as a powerful tool for such problems, but existing approaches face a trade-off: Model-Based Diffusion (MBD) incorporates diffusion in sampling-based optimization (SBO) without training but only over continuous spaces, while existing diffusion-based combinatorial solvers operate over discrete spaces but require offline training. To this end, we propose Discrete Model-Based Diffusion (Discrete MBD), a training-free SBO method that directly optimizes discrete variables by approximating categorical reverse transition distributions through Monte Carlo sampling. We establish a finite-step lower bound on the probability of finding a global minimizer and show that this bound converges to one as the number of denoising steps increases. Extensive experiments on generic integer programs and heterogeneous multi-robot coalition formation and scheduling demonstrate that Discrete MBD outperforms SBO baselines in nearly all settings while achieving solution quality competitive with that of mixed-integer programming and reinforcement learning methods at a fraction of their computational cost.
• 中文摘要 多智能体和多机器人系统中的决策通常涉及离散决策变量，导致需要在几秒钟内解决的NP难组合优化问题，才能实现近实时。去噪扩散已成为此类问题的强大工具，但现有方法面临权衡：基于模型的扩散（MBD）在基于采样的优化（SBO）中包含扩散，无需训练，仅覆盖连续空间;而现有基于扩散的组合求解器则可在离散空间上工作，但需要离线训练。为此，我们提出了一种无训练的离散模型扩散（Discrete MBD）方法，通过蒙特卡洛采样近似类别反转移分布，直接优化离散变量。我们建立了找到全局最小化器的概率有限步下界，并证明随着去噪步数的增加，该下界收敛至1。对泛整数程序及异构多机器人联盟组建与调度的广泛实验表明，离散MBD在几乎所有场景下都优于SBO基线，同时以极低的计算成本实现与混合整数规划和强化学习方法媲美的解决方案质量。

Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling

利用心理物理注意力分布进行凝视增强奖励建模

• Authors: Chaeho Lee, Junyup Kim, Wansoo Kim, Young Min Jung, Sang Ho Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2611.13477
• Pdf link: https://arxiv.org/pdf/2611.13477
• Abstract Integrating human feedback into language models has gained increasing attention as a means to align model outputs with human preferences, with Reinforcement Learning from Human Feedback (RLHF) serving as a prominent framework for preference-based alignment. Recent studies have incorporated eye-tracking (ET) data as an additional supervisory signal for reward model training, grounding reward learning in human reading behavior. Building on this approach, we extend prior fixation-centric models by leveraging established findings from cognitive psychology on human letter recognition. Specifically, we model visual attention as a spatially graded distribution centered on the current gaze location and incorporate this representation into reward model training. Our results show that the reward model trained with the proposed gaze distribution achieves higher preference prediction accuracy than a baseline model. Moreover, some results from our experiments are qualitatively consistent with established findings on human attention during reading, suggesting that they potentially capture cognitively meaningful aspects of human attention allocation.
• 中文摘要 将人类反馈整合进语言模型，作为使模型输出与人类偏好对齐的手段，越来越受关注，其中人类反馈强化学习（RLHF）成为基于偏好对齐的重要框架。近期研究将眼动追踪（ET）数据纳入奖励模型训练的额外监督信号，将奖励学习扎根于人类阅读行为。基于此方法，我们利用认知心理学关于人字识别的既有发现，扩展了以往以注视为中心的模型。具体来说，我们将视觉注意力建模为以当前凝视位置为中心的空间分级分布，并将该表征纳入奖励模型训练中。我们的结果表明，采用所提凝视分布训练的奖励模型，在偏好预测准确度上优于基线模型。此外，我们的一些实验结果在定性上与人类阅读时注意力的既有发现一致，表明它们可能捕捉到人类注意力分配中认知意义的方面。

Sharp Sample Complexity of KL-Regularized Reinforcement Learning

KL正则化强化学习的锐利样本复杂度

• Authors: Andrew Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.13480
• Pdf link: https://arxiv.org/pdf/2611.13480
• Abstract KL-regularized reinforcement learning, the objective behind maximum-entropy RL and RL from human feedback, is variational inference in the graphical model of "control as inference": the regularized optimal policy is a posterior over trajectories, and the suboptimality of a policy is $\beta$ times a KL divergence between path measures. However, it is not known how many samples it takes to learn this posterior, even with a generative model. Earlier $1/\epsilon$ guarantees scale with $H^5$ in the effective horizon $H$, and no lower bound for MDPs depends on the temperature $\beta$. In this work, we show that for a discounted MDP with $K$ state-action pairs, the plug-in estimator finds a policy that is $\epsilon$-optimal from every initial state with $\widetilde{O}(KH^3/(\epsilon\max{\epsilon,\beta}) + KH)$ samples, and that no algorithm, even one that chooses where to sample, can do with fewer when $\epsilon \lesssim \min{H, H^2/\beta}$. For $\epsilon < \beta$ this is a $1/\epsilon$ rate with the same horizon dependence as unregularized RL. We also find the exact asymptotic constant of the plug-in estimator, a Fisher-weighted variance of the estimated soft Q-function. The estimator is locally asymptotically minimax among methods that sample every pair equally often, and we give the allocation of samples that is optimal among adaptive designs. Simulations on small MDPs match the rates and the constants.
• 中文摘要 最大熵强化学习和人类反馈强化学习的目标是“控制作为推断”图形模型中的变分推断：正则化最优策略是轨迹的后验，策略的次最优度是 $\beta$ 乘以路径测度间的 KL 散度。然而，即使使用生成模型，学习该后验需要多少样本尚不得而知。早期的 $1/\epsilon$ 保证有效视界 $H$ 的尺度为 $H^5$，且 MDP 的下界不依赖于温度 $\beta$。在本研究中，我们证明对于一个状态-动作对为$K$的折现MDP，插件估计器能从每个初始状态（$\widetilde{O}（KH^3/（\epsilon\max{\epsilon，\beta}）+ KH）$$ 样本中找到一个$\epsilon$最优的策略，且即使算法选择采样位置，也无法在$\epsilon \lesssim{H， H^2/\beta}$时使用更少的采样数。对于$\epsilon < \beta$，这是一个$1/\epsilon$的速率，视界依赖性与非正则化强化学习相同。我们还找到了插件估计器的精确渐近常数，即估计软Q函数的费舍尔加权方差。在采样每对均等待频率的方法中，估计器局部渐近极大，我们给出自适应设计中最优样本分配。小型MDP上的仿真匹配速率和常数。

ExRL: Sample-Efficient Online Adaptation of Long-Chunk Policies via Execution Length Control

ExRL：通过执行长度控制的样本高效在线适配长块策略

• Authors: Xu Luo, Jiaying Yang, Hao Wu, Junlin Xie, Zijin Hong, Ji Zhang, Lianli Gao, Jingkuan Song
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.13482
• Pdf link: https://arxiv.org/pdf/2611.13482
• Abstract Predicting long action chunks, often spanning dozens of timesteps, has become a key ingredient in increasingly capable robot policies. In this paper, we find that adapting these long-chunk policies with existing reinforcement learning (RL) methods can require many online samples due to the high dimensionality of the action chunks being optimized, making such adaptation impractical in real-world scenarios where interaction data are costly to collect. Our key insight is to shift adaptation from optimizing actions to optimizing how long to execute each predicted action chunk. This keeps the adaptation space low-dimensional while its expressivity grows with the chunk size. We thus introduce Execution at the Right Length (ExRL), a simple approach which augments the base policy with a lightweight adapter that predicts the execution length of each chunk and is trained using off-policy RL algorithms. Across simulation benchmarks and real-world robotic tasks, ExRL enables sample-efficient adaptation of a diverse range of long-chunk policies with minimal task-specific hyperparameter tuning.
• 中文摘要 预测跨越数十个时间步的长动作块已成为机器人策略日益强大的关键要素。本文发现，由于被优化动作块维度较高，调整这些长块策略时可能需要大量在线样本，因为被优化动作块维度较高，在实际场景中交互数据收集成本高昂时，这种适应不切实际。我们的关键见解是将适应从优化动作转向优化每个预测动作块的执行时间。这使适应空间保持低维，而其表达力随块大小增长。因此，我们引入了“正确长度执行”（ExRL），这是一种简单方法，通过轻量级适配器补充基础策略，预测每个块的执行长度，并使用非策略的强化学习算法进行训练。在模拟基准测试和现实机器人任务中，ExRL能够高效地适应多种长块策略，同时最小限度地调整任务特定的超参数。

Reasoning-Aware Relational Representation Learning for Open-Vocabulary Scene Graph Generation

推理感知关系表征学习用于开放词汇场景图生成

• Authors: Jiawei Xiao, Boya Wang, Longtian Qiu, Shan Ning, Xuming He
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.13496
• Pdf link: https://arxiv.org/pdf/2611.13496
• Abstract Open-Vocabulary Scene Graph Generation (OV-SGG) requires models to recognize visual relationships beyond the training vocabulary, yet existing methods often rely on dataset-specific object--predicate co-occurrence patterns. Under long-tailed distributions and noisy object localization, such reliance leads to biased, entangled relation representations, limiting both rare-relation recognition and generalization to unseen predicates. To address these challenges, we propose RaRe, a reasoning-aware relational representation learning framework that transforms MLLM-generated chain-of-thought (CoT) rationales into structured relation embeddings. Rather than using CoT only as an intermediate reasoning trace for final prediction, RaRe directly optimizes the rationale-derived representation space to improve inter-relation discriminability. Specifically, RaRe first aligns rationale embeddings with textualized relation descriptions through self-supervised contrastive learning, and then refines rationale generation with a specificity-oriented reinforcement learning objective that encourages semantically object-pair-specific relational evidence. Experiments on VG150 and PSG demonstrate improved category-balanced recognition over existing baselines, while the additional gains from learning vary across datasets and proposal settings. Code available at: this https URL.
• 中文摘要 开放词汇场景图生成（OV-SGG）要求模型识别训练词汇之外的视觉关系，但现有方法往往依赖数据集特定的对象-谓词共现模式。在长尾分布和噪声对象定位下，这种依赖导致偏置纠缠关系表示，限制了稀有关系识别和对未见谓词的推广。为应对这些挑战，我们提出了RaRe，一种推理感知关系表示学习框架，将MLLM生成的思维链（CoT）理据转化为结构化关系嵌入。RaRe不仅将CoT作为最终预测的中间推理痕迹，而是直接优化理性导出的表示空间，以提升关系间的辨别能力。具体来说，RaRe首先通过自监督对比学习将理据嵌入与文本化关系描述对齐，然后通过以特异性为导向的强化学习目标，优化理据生成，鼓励语义上对象对特异的关系证据。在VG150和PSG上的实验显示，在现有基线上实现了更好的类别平衡识别，而学习带来的额外收益则在不同数据集和提案设置中有所不同。代码可访问：此 https URL。

DeRep: Learning Depth Representations for End-to-End Navigation in Cluttered Environments

DeRep：在杂乱环境中学习端到端导航的深度表示

• Authors: Hanyu Jin, Kenji Shimada
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.13499
• Pdf link: https://arxiv.org/pdf/2611.13499
• Abstract End-to-end navigation in cluttered environments relies on compact visual representations to support collision avoidance and goal-directed motion. Existing methods typically learn these representations jointly with the policy or through observation reconstruction, which does not explicitly specify what information the representation should preserve. Moreover, they are usually evaluated with a single downstream policy learner, although the benefit of a representation may depend on how the policy is learned. It therefore remains unclear how different representation-learning objectives affect downstream navigation performance and robustness. This work proposes DeRep, a navigation-oriented representation-learning method that combines local bird's-eye-view reachability supervision with depth reconstruction. To compare representation objectives under controlled conditions, we pretrain encoders with a common architecture and freeze them during downstream learning. We evaluate the representations using a linear policy probe, flow-matching imitation learning, and reinforcement learning with PPO. DeRep achieves the lowest linear policy probe prediction error and outperforms reconstruction alone across all six tested PPO conditions. In imitation learning, both reachability-based objectives achieve higher mean success rates than reconstruction alone, with similar mean success rates between the reachability-only and joint objectives. These results demonstrate consistent benefits of incorporating reachability supervision into depth representation pretraining across different downstream learning methods and evaluation conditions. We further deploy the PPO policy with the frozen DeRep encoder in Gazebo and on a physical quadrotor without fine-tuning, demonstrating its capability for zero-shot sim-to-sim and sim-to-real transfer.
• 中文摘要 在杂乱环境中的端到端导航依赖紧凑的视觉表示来支持碰撞避免和目标导向运动。现有方法通常与策略共同学习这些表示，或通过观察重建来学习，观测重建并未明确指定应保留哪些信息。此外，这些表示通常由单个下游策略学习器评估，尽管表示的效益可能取决于策略的学习方式。因此，不同的表征学习目标如何影响下游导航的表现和鲁棒性仍不明确。本研究提出了DeRep，一种以导航为导向的表示学习方法，结合了局部鸟瞰视角的可达性监督与深度重建。为了在受控条件下比较表示目标，我们预训练了统一架构的编码器，并在下游学习时冻结它们。我们使用线性策略探针、流匹配模仿学习和PPO强化学习来评估这些表示。DeRep实现了最低线性策略探针预测误差，并在所有六种测试PPO条件下均优于单独重建。在模拟学习中，基于可达性目标的平均成功率均高于单纯重建，仅可达性目标与联合目标的平均成功率相近。这些结果证明了将可达性监督纳入深度表示预训练中，跨越不同下游学习方法和评估条件的持续优势。我们还进一步在Gazebo中部署了结合冻结DeRep编码器的PPO策略，并在物理四旋翼上未微调，展示了其零时次模拟对模拟和模拟到真实传输的能力。

Closing the BF16-FP16 Convergence Speed Gap in RL Post-Training with Stochastic Rounding

利用随机四舍五入缩小强化学习后训练中BF16-FP16收敛速度差距

• Authors: Mahdi Nazemi, Oleg Rybakov, Tijmen Blankevoort, Asit Mishra, Carlo del Mundo
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2611.13545
• Pdf link: https://arxiv.org/pdf/2611.13545
• Abstract BF16 is widely used to pretrain large language models because it has the same dynamic range as FP32 and converges similarly to FP32 training. However, in reinforcement learning with verifiable rewards, where learning rates are typically low, we find that standard mixed-precision BF16 training converges more slowly than FP16 training at the same learning rate. We attribute this gap to weight sticking: when FP32 shadow weights are quantized to BF16 using round-to-nearest, small updates often leave the BF16 weights used for rollout generation and training unchanged. Consequently, several optimizer steps may be needed before updates to the shadow weights are reflected in the BF16 weights. Raising the BF16 learning rate reduces weight sticking and matches FP16's convergence speed, but it also increases the KL divergence from the reference model. We instead apply stochastic rounding when quantizing the FP32 shadow weights to BF16. This approach matches FP16's convergence speed across different models and datasets while reducing the KL divergence relative to BF16 training with a larger learning rate. Unlike FP16, it keeps BF16's wider dynamic range and needs no loss scaling.
• 中文摘要 BF16 被广泛用于大型语言模型预训练，因为它与 FP32 具有相同的动态范围，且收敛方式类似 FP32 训练。然而，在具有可验证奖励的强化学习中，学习率通常较低，我们发现标准混合精度 BF16 训练收敛速度比 FP16 在相同学习率下收敛更慢。我们将此差距归因于权重粘附：当 FP32 阴影权重通过四舍五入到最近量化为 BF16 时，小幅更新通常使用于展开生成和训练的 BF16 权重保持不变。因此，在阴影权重更新反映在 BF16 权重中之前，可能需要多个优化步骤。提高 BF16 学习率减少权重粘附，并匹配 FP16 的收敛速度，但也增加了 KL 与参考模型的偏差。我们改为在量化FP32影子权重到BF16时应用随机四舍五入。这种方法在不同模型和数据集间匹配FP16的收敛速度，同时降低了相对于BF16训练的基层偏差，且学习率更高。与FP16不同，它保持了BF16更宽的动态范围，且无需损耗缩放。

Unified Optimization of Traffic Signal and Lane Channelization for an Isolated Intersection Using Reinforcement Learning

利用强化学习统一优化隔离路口的交通信号和车道通道化

• Authors: Jiwon Kang, Jiyoung Kim, Hyunseok Jun, Keemin Sohn
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13599
• Pdf link: https://arxiv.org/pdf/2611.13599
• Abstract Optimizing traffic signal control has been a long-lasting critical issue in traffic engineering, and a number of robust methodologies have emerged to solve the problem so far. On the other hand, the road design for lane channelization is a static problem that is difficult to solve jointly with the real-time traffic signal control. If traffic volume does not vary, it is possible that both problems can be solved in an integrated manner as a single static optimization problem. Recently, researchers have devised a mixed integer linear problem (MILP) to jointly determine the traffic signal and road width designs. However, this joint optimization problem must be solved repeatedly whenever the traffic flow pattern varies. Solving the MILP may also take impractical computing time as the problem size gets larger. We present a robust methodology for solving the joint optimization problem using reinforcement learning (RL) on a real-time basis. Abrupt changes in lane markings with an incompatible signal phase can jeopardize drivers. To sidestep such a dangerous action change, we train a model with human feedback to filter infeasible action transitions in a rule-based manner. We devised a deep Q-network that can accommodate joint actions for traffic signal and lane marking plans. To ensure the implementation of the proposed operation, a variable sign for lane markings is necessary at each stop line of an intersection. The experimental results confirm that the proposed methodology enhances the intersection performance by increasing throughput.
• 中文摘要 优化交通信号控制一直是交通工程中长期存在的关键问题，迄今已有多种稳健的方法论来解决该问题。另一方面，车道通道化的道路设计是一个静态问题，难以与实时交通信号控制共同解决。如果交通量不变，两个问题可以作为一个静态优化问题进行整合解决。最近，研究人员设计了一个混合整数线性问题（MILP），用于联合确定交通信号和道路宽度设计。然而，每当交通流模式变化时，该联合优化问题必须反复求解。随着问题规模扩大，求解MILP可能耗费不切实际的计算时间。我们提出了一种基于实时的强化学习（RL）联合优化问题的稳健方法论。车道标记的突然变化与信号相位不兼容会危及驾驶员安全。为避免如此危险的动作变更，我们训练了一个带有人工反馈的模型，以基于规则的方式过滤不可行的动作转换。我们设计了一个深度Q网络，支持交通信号和车道标记计划的联合操作。为确保拟议操作的实现，每个路口的停车线都需设置可变车道标记。实验结果证实，所提方法通过提高吞吐量提升了路口性能。

Curriculum proof repair with learned counterexamples

课程证明修复及学习反例

• Authors: Yuechun Sun, Jun Yang, Kexin Pei
• Subjects: Subjects: Programming Languages (cs.PL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13627
• Pdf link: https://arxiv.org/pdf/2611.13627
• Abstract Language models (LMs) have substantially lowered the barrier to applying formal verification for building reliable and secure software. Existing approaches commonly follow the verifier-in-the-loop paradigm, but the verifier often provides only coarse, binary pass/fail feedback, which offers little actionable signal on how the proof should be repaired. Recent work shows that counterexamples provide more actionable feedback by grounding proof failures in concrete program states, but generating useful ones at inference time requires costly iterative refinement. We present TopoVerus, an LM-based framework for Verus proof repair that internalizes counterexample-guided diagnosis as a model capability, producing useful counterexamples in a single pass. Our key insight is to train counterexample generation not as a standalone validation task, but as a data-driven diagnostic objective: a counterexample is useful when conditioning on it helps the model produce a safe, verifier-passing proof. Given a failed Verus proof and verifier error, TopoVerus generates a structured source-level counterexample, repair rationale, and repaired proof in one trajectory, optimized with reinforcement learning from Verus feedback and a proof-code safety gate. To guide the order of invariant repairs, TopoVerus introduces a curriculum with a topological graph reward based on invariant dependencies derived from inductiveness tests, which encourages counterexample generation to follow the same topological order. Across seven Verus benchmarks with 471 problems, TopoVerus achieves 61.6%/72.2% weighted-average Safe-Pass@1/Safe-Pass@3, outperforming the strongest baseline, Claude Sonnet 4.5, by 5.7/6.3 points. We also show that TopoVerus's learned counterexamples can directly augment the reasoning of existing LM-based proof repair approaches, improving their success rate by 64.3% compared to chain-of-thought diagnostics.
• 中文摘要 语言模型（LM）大大降低了在构建可靠和安全软件中应用形式验证的门槛。现有方法通常遵循验证者在环（verifier-in-the-loop）范式，但验证者通常只提供粗略的二元通过/失败反馈，这几乎没有提供证明应如何修复的可操作信号。最新研究表明，反例通过将证明失败扎根于具体程序状态，提供了更多可操作的反馈，但在推理时生成有用的反例则需要昂贵且不断迭代的改进。我们提出了TopoVerus，这是一个基于LM的Verus证明修复框架，将反例引导诊断内化为模型能力，在一次流程中生成有用的反例。我们的关键见解是训练反例生成，而非作为独立验证任务，而是作为数据驱动的诊断目标：当条件反射反例有助于模型产生安全且通过验证器的证明时，反例非常有用。在Verus证明和验证错误失败的情况下，TopoVerus生成结构化的源级反例、修复理由和修复证明，集中在一条轨迹中，并通过Verus反馈的强化学习和证明码安全门进行优化。为指导不变修复的顺序，TopoVerus引入了基于归纳性检验得出的不变依赖的拓扑图奖励课程，鼓励反例生成遵循相同的拓扑顺序。在七个Verus基准测试中，共471个问题，TopoVerus实现了61.6%/72.2%的加权平均安全Pass@1/安全Pass@3，比最强基线Claude Sonnet 4.5高出5.7/6.3分。我们还展示了TopoVerus学习到的反例可以直接增强现有基于LM的证明修复方法的推理，使其成功率相比思维链式诊断提升了64.3%。

Broad-Persistent Advising in Interactive Reinforcement Learning via Latent State Encoding

通过潜在状态编码实现交互式强化学习中的广泛持久指导

• Authors: Zhuoran Liang, Francisco Cruz
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.13712
• Pdf link: https://arxiv.org/pdf/2611.13712
• Abstract Deep Interactive Reinforcement Learning (Deep IntRL) accelerates reinforcement learning training by incorporating external advice. However, conventional Deep IntRL uses each piece of advice only in the current step and discards it after a single interaction. Persistent Advising and Broad-Persistent Advising (BPA) extend this process by storing advice in persistent memory and reusing it in similar states through state generalisation based on K-means clustering. This research proposes Encoder BPA, which employs a latent encoder to retrieve and reuse advice. In this framework, a trained encoder is used to map a state sequence into a latent representation at every time step. Advice can be retrieved through latent comparison, together with a top-(k) aggregation and a consistency condition to improve reuse stability. Experimental results show that Encoder BPA improves learning efficiency and advice utilisation at an early stage of training compared with different Deep IntRL approaches, especially in complex and partially observable environments.
• 中文摘要 深度交互强化学习（Deep IntRL）通过引入外部建议加速强化学习训练。然而，传统的深度IntRL仅在当前步骤中使用每条建议，并在单次交互后丢弃。持久指导和广泛持续指导（BPA）通过将建议存储在持久记忆中，并通过基于K-均值聚类的状态泛化在类似状态中重复使用，扩展了这一过程。本研究提出了编码器BPA，利用潜在编码器检索和重用建议。在该框架下，使用训练有素的编码器将状态序列映射为每个时间步的潜在表示。建议可以通过潜在比较、顶（k）聚合和一致性条件来检索，以提升重用稳定性。实验结果显示，与其他深度IntRL方法相比，Encoder BPA在训练早期阶段提升了学习效率和建议利用率，尤其是在复杂且部分可观测的环境中。

TRUER: Test-Time Reward Correction Using an Error-Aware Rectifier for Reinforcement Learning

TRUEER：使用错误感知整流器进行测试时奖励纠正，用于强化学习

• Authors: Mingxiang Shen, Jinhao Li, Jiaming Xu, Yibo Han, Guohao Dai, Yijia Zhang, Ningyi Xu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13714
• Pdf link: https://arxiv.org/pdf/2611.13714
• Abstract Test-time reinforcement learning enables language models to adapt using rewards derived from their own answers. However, majority voting can reinforce incorrect solutions even when correct alternatives are available. Our training audit shows that false positives (incorrect responses receiving positive rewards) dominate reward errors. We focus on a practical regime where ground-truth rewards are difficult to obtain at scale, while a limited labeled source set is available. We introduce TRUER, a framework that uses limited supervision to train a reward rectifier that distinguishes correctness from consensus. To achieve this goal, we train the rectifier with an emphasis on incorrect majorities and correct minorities, targeting cases where consensus and correctness disagree. After that, we freeze the rectifier and use it to correct rewards during subsequent test-time adaptation. By improving reward accuracy and thereby model performance, TRUER consistently outperforms naive TTRL in both metrics across all evaluated configurations.
• 中文摘要 测试时强化学习使语言模型能够利用自身答案得出的奖励进行适应。然而，即使有正确的替代方案，多数投票也可能强化错误的解法。我们的训练审计显示，假阳性（错误回答获得正向奖励）占主导。我们关注一种实用机制，在大规模获得实地真实奖励且有限标签源集可用的情况下。我们引入了TRUER框架，该框架利用有限监督训练奖励整流器，区分正确性与共识。为实现这一目标，我们训练整流器，重点关注错误多数和正确少数，针对共识与正确性不一致的情况。之后，我们冻结整流器，用于后续测试时间调整中的奖励纠正。通过提高奖励准确性和模型性能，TRUER在所有评估配置中均在这两个指标上持续优于朴素TTRL。

Omni-JEV: Text and Dual-Stream Speech Decisions

Omni-JEV：文本与双流语音决策

• Authors: Shengzhou Kong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13777
• Pdf link: https://arxiv.org/pdf/2611.13777
• Abstract A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history. Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy. We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions. An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs. Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD). We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality. Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text. This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.
• 中文摘要 用户话语并不唯一决定全双工对话中的下一个动作：其后果取决于助手的反应状态和交互历史。语音活动、话语完成和中断意图提供了相关证据，其对反应的影响需要明确的行动策略。我们在收集片段端点制定该任务，并引入了Omni-JEV，将文本或角色索引的用户和助手引用语音映射到五次交互动作中的概率。对齐语音对语音理解骨干分别编码声学角色，并与文本输入共享候选读出。双语弱监督支持交叉熵微调，随后进行概率敏感决策的强化学习（RLCD）。我们在匹配训练暴露下比较同一监督初始化的高斯-对口和类别选择RLCD，观察到分类与概率质量之间的权衡差异。高斯全能-JEV在音频测试中达到84.08%，文本96.55%。该通用动作界面支持全双工语音助手何时响应、保留当前语音、中断或恢复中断响应的决策。

Integrating Action Prefix Prediction and Alignment Across Modalities for Multimodal Reinforcement Learning

跨模态整合动作前缀预测与对齐以实现多模态强化学习

• Authors: Sijie Xu, Francisco Cruz
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.13782
• Pdf link: https://arxiv.org/pdf/2611.13782
• Abstract Learning representations from multimodal observations is difficult in reinforcement learning with limited interaction data. Multimodal Joint Predictive Representation combines multimodal latent prediction with contrastive learning across modalities, but its recursive rollout depends on earlier predictions and its pairwise alignment focuses on correspondence between modalities. We propose Improved Predictive Representation, which combines prediction from action prefixes with alignment at the observed anchor based on Contrastive Multimodal learning. At each horizon, it predicts future representations directly from the observed anchor and the corresponding sequence of actions. We evaluate two crop configurations: IPR-S with a minimum retained crop area of 10%, and IPR-H with 50%. Across five DeepMind Control Suite tasks, IPR-H achieves an average return across tasks that is 10.0% higher than the baseline with visual and depth images and 3.4% higher with visual images and position vectors. The relative performance of the two configurations varies across tasks. In Webots, both IPR configurations achieve higher mean final returns than the baseline without LiDAR, but not when LiDAR is available.
• 中文摘要 在有限的交互数据下，从多模态观察中学习表征较为困难。多模联合预测表征结合了多模态潜在预测与跨模态的对比学习，但其递归推广依赖于早期预测，且其两对比对重点关注模态间的对应关系。我们提出了改进的预测表征，结合了基于对比多模态学习的动作前缀预测与观察锚点的对齐。在每个视角，它直接从观察到的锚点及相应的动作序列预测未来表征。我们评估了两种裁剪配置：IPR-S，最小保留作物面积为10%，IPR-H，占50%。在五个DeepMind控制套件任务中，IPR-H在任务中平均回报比基线高出10.0%，视觉图像和位置向量高出3.4%。两种配置的相对性能因任务而异。在Webots中，两种IPR配置的平均最终回报都高于无激光雷达的基线，但当有激光雷达（LiDAR）可用时则不然。

EVAThink: Evidence Sufficiency-Guided Training for Adaptive Thinking in Video Understanding

EVAThink：视频理解中自适应思维的充分证据引导培训

• Authors: Changti Wu, Ruizhe Fan, Bin Wang, Lei Zhang, Yuxiang Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.13907
• Pdf link: https://arxiv.org/pdf/2611.13907
• Abstract Under a fixed frame budget, video understanding with vision-language models (VLMs) depends on whether the sampled frames provide sufficient evidence for the question. Sufficient frame evidence supports direct answering, whereas less informative observations may benefit from reasoning that connects the available cues. We introduce EVAThink, which enhances video understanding through evidence sufficiency-guided training for adaptive thinking. First, Recoverable Evidence Gap Mining constructs evidence-dense and evidence-sparse video views that share the same question and answer under an identical frame budget. It retains pairs where sparse views yield lower direct-answer accuracy than dense views but benefit from explicit reasoning, providing supervision for response-mode selection. Second, Evidence-Guided Two-Stage Reinforcement Learning uses stage-specific rewards to guide training with evidence-based mode preferences and then refine response efficiency, while first-token advantage regularization mitigates mode collapse. Trained on only 6.7K curated samples, EVAThink consistently improves upon the base model across video understanding benchmarks under both 32- and 64-frame budgets, while outperforming strong baselines on average. The code, model, and datasets are available at \href{this https URL}{EVAThink}.
• 中文摘要 在固定帧预算下，视觉语言模型（VLM）的视频理解取决于采样帧是否为问题提供了足够的证据。足够的帧证据支持直接回答，而信息量较少的观察则可能受益于连接可用线索的推理。我们引入了EVAThink，通过证据充分性引导训练增强视频理解，促进适应性思维。首先，可恢复证据差距挖掘构建了证据密集和证据稀疏的视频视图，在相同的帧预算下共享相同的问题和答案。它保留了稀疏视图导致直接答案准确率低于密集视图的配对，但受益于显式推理，从而为反应模式选择提供监督。其次，证据引导的两阶段强化学习使用阶段特定的奖励引导基于证据的模式偏好训练，然后优化响应效率，而第一代币优势正则化则减轻模式崩溃。仅在6700个精选样本上训练，EVAThink在32帧和64帧预算下的视频理解基准测试中持续改进基础模型，平均表现优于强基线。代码、模型和数据集可在 \href{this https URL}{EVAThink} 获取。

Temporal Behavior Trees for Reinforcement Learning: Specifications as Rewards, Curricula, and Diagnostics

强化学习的时序行为树：作为奖励、课程和诊断的规范

• Authors: Vladimir Krsmanovic, Bernd Finkbeiner
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.13993
• Pdf link: https://arxiv.org/pdf/2611.13993
• Abstract Reinforcement learning from formal specifications often separates task definition, reward shaping, curriculum design, and policy diagnostics into different mathematical objects. This fragmentation requires additional conversion steps and discards structure already present in the specification, such as subtask boundaries, ordering constraints, and quantitative measures. We introduce TBTCL (Temporal Behavior Tree Centric Learning), a framework that uses Temporal Behavior Trees (TBTs) as a unified interface for specification-guided reinforcement learning. A single TBT specifies the task, induces dense robustness-based rewards, defines a curriculum, and supports better understanding of learned policies. By adapting TBT syntax and semantics for specification-guided reinforcement learning, we provide a formalism that is more expressive than popular specification languages such as LTL and inherently hierarchical. We present algorithms to automatically extract a curriculum DAG and derive dense, robustness-based rewards directly from TBT semantics. Furthermore, we utilize TBT trace segmentation to build understandable competence profiles and automatically adapt the reward toward bottlenecks. We evaluate TBTCL on six specifications across discrete and continuous domains and show that Temporal Behavior Trees can serve as a specification, a reward, a curriculum, and a diagnostic.
• 中文摘要 强化学习从形式规范中通常将任务定义、奖励塑造、课程设计和策略诊断分离到不同的数学对象中。这种碎片化需要额外的转换步骤，并丢弃规范中已有的结构，如子任务边界、排序约束和定量度量。我们引入了TBTCL（以时序行为树为中心学习），这是一个使用时序行为树（TBT）作为规范引导强化学习的统一接口的框架。单个TBT指定任务，诱导密集且稳健的奖励，定义课程，并支持更好地理解已学策略。通过调整TBT的语法和语义以适应规范引导强化学习，我们提供了一种比LTL等流行规范语言更具表现力且本质上具有层级性的形式主义。我们提出了算法，能够自动提取课程DAG，并直接从TBT语义中推导出密集、基于鲁棒性的奖励。此外，我们利用TBT跟踪分割构建可理解的能力配置文件，并自动调整奖励以适应瓶颈。我们基于六个规格在离散和连续领域评估TBTCL，展示了时间行为树可以作为规范、奖励、课程和诊断工具。

Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards

保证金引导代币积分用于强化学习，并提供可验证的奖励

• Authors: Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan Manoharan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.14015
• Pdf link: https://arxiv.org/pdf/2611.14015
• Abstract In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin $M=p_1-p_2$ of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95\% CI $[4.1, 9.0]$), and a Qwen3-4B-Base replication improves for all three seeds ($+4.75$). The gain is largest on problems that GRPO solves only some of the time and grows with $k$ in pass@$k$, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
• 中文摘要 在可验证奖励的强化学习中，语言模型因正确最终答案获得奖励，GRPO将该奖励等量传递给解的每个代币。我们将模型犹豫处的功劳归入，采用称为MCPO（保证-信用政策优化）的方法。我们用策略下一代币分布的前2个margin $M=p_1-p_2$来衡量犹豫，映射到两个固定阈值的分数，并将每个解决方案相较于其代币的优势分散，使均值保持不变。margin是从训练器已执行的前向传递中读取的，因此该方法无需额外的模型评估和批次统计。我们用LoRA训练MiniCPM5-2B-Base，共200步，涉及2048个DAPO-Math问题，并在6个竞争数学基准进行评估。MCPO平均在五个数据种子中都比未修改的GRPO基线高出6.55个宏观avg@32点（95\% CI $[4.1， 9.0]$），而Qwen3-4B-Base的复制在三个种子中均有所提升（$+4.75$）。在GRPO只部分时间解决的问题上，增长幅度最大，且在pass@32时达到13.8分，达到$k pass@$k$的比例。MCPO模型还能写出两到三倍长的答案，写得更长的跑题得分更高;我们的实验未将这两种效应区分开来。

Resilient Integrated Access and Backhaul Network Planning via Graph Neural Network Curriculum Learning

通过图神经网络实现韧性综合接入和回程网络规划课程学习

• Authors: Jie Zhang, Haochen Sun, Mingzhang Zhou, Paul D. Mitchell, Hamed Ahmadi, Qiang Ni
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.14042
• Pdf link: https://arxiv.org/pdf/2611.14042
• Abstract Integrated Access and Backhaul (IAB) is a key enabler for cost-effective millimeter-wave (mmWave) densification, yet practical deployments are strongly constrained by the spatial availability of fiber-connected IAB donors. Variations in donor topologies reshape both the initial coverage footprint and the feasible multi-hop backhaul expansion patterns, causing planning policies trained on a single topology to generalize poorly. This paper studies cross-topology generalization for resilience-aware IAB deployment under coverage and transport feasibility requirements. We formulate IAB deployment as a constrained Markov decision process that couples access coverage satisfaction with multi-hop transport feasibility, including hop-by-hop rate-budget degradation and a minimum parent-connectivity requirement for fault tolerance. To solve the resulting large discrete planning problem, we propose a graph-based actor-critic approach that uses a Graph Attention Network v2 (GATv2) encoder with Proximal Policy Optimization (PPO), and we develop a multi-stage curriculum learning strategy that progressively increases constraint difficulty and scenario diversity for stable training and transferable policies. In a 60 GHz urban simulation over a 1000 m x 1000 m area with multiple donor layouts, the trained policy satisfies the minimum coverage requirement of 98% with fewer deployed IAB nodes than learning and heuristic baselines, and maintains higher coverage under progressive backhaul failures. The results demonstrate that curriculum-trained graph-based reinforcement learning can improve deployment efficiency and resilience while transferring across heterogeneous IAB donor topologies without retraining.
• 中文摘要 集成接入与回传（IAB）是实现具有成本效益的毫米波（mmWave）密度化的关键推动力，但实际部署受到光纤连接IAB供体空间可用性的强烈限制。捐赠者拓扑的变化重塑了初始覆盖覆盖范围和可行的多跳回传扩展模式，导致基于单一拓扑训练的规划政策难以推广。本文研究了在覆盖和传输可行性要求下，针对韧性感知IAB部署的跨拓扑泛化。我们将IAB部署表述为一个受限的马尔可夫决策过程，将接入覆盖满意度与多跳传输可行性结合，包括逐跳速率预算的下降以及容错的最低父连接要求。为解决由此产生的大型离散规划问题，我们提出了基于图的演员-批判者方法，使用图关注网络v2（GATv2）编码器配合近端策略优化（PPO），并开发了一套多阶段课程学习策略，逐步提升约束难度和场景多样性，以实现稳定训练和可转移策略。在60 GHz城市模拟中，覆盖1000米×1000米区域，采用多重供体布局，训练策略满足98%的最低覆盖率要求，部署的IAB节点数少于学习型和启发式基线，并在渐进回程失败时保持较高覆盖率。结果表明，基于课程训练的基于图的强化学习在跨异构IAB供体拓扑转移且无需再训练的情况下，能够提升部署效率和韧性。

Opportunity-Cost Option-Critic: Adaptive Switching Costs for Robust Hierarchical Reinforcement Learning

机会-成本选项-批评：稳健层级强化学习中的自适应切换成本

• Authors: Khaled Ammoura, Aya Kayal, Naseem Daher
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.14099
• Pdf link: https://arxiv.org/pdf/2611.14099
• Abstract Deliberation costs are commonly used in hierarchical reinforcement learning (HRL) to prevent options from collapsing to primitive actions. Existing approaches rely on fixed penalties or duration-based adaptations, both of which are indirect proxies for an option's utility. This paper argues that option persistence should instead depend on the opportunity cost of abandoning the current behavior. We propose the Opportunity-Cost Option-Critic ($OC^2$) framework, which defines a state-dependent deliberation cost using the positive value gap between the active option and its best alternative. This encourages persistence when the current option is clearly preferable and allows rapid switching when a better option emerges. We show that directly learning such costs leads to a degenerate zero-cost solution and introduces an auxiliary objective that aligns the learned cost with the value-gap target. The approach is evaluated in Four Rooms and Pinball domains under stationary, goal-shift, and geometry-shift conditions. In the Pinball goal-shift task, the $OC^2$ framework achieves a 52\% first-ten-episode success rate compared with 18\% for a fixed-cost baseline. The results suggest that opportunity-cost-based persistence improves adaptation in non-stationary environments.
• 中文摘要 审议成本常用于层级强化学习（HRL），以防止选项崩溃为原始动作。现有方法依赖固定罚则或基于持续时间的适应，这两者都是选项效用的间接代理。本文主张选择权持久性应依赖于放弃当前行为的机会成本。我们提出了机会-成本期权-批判者（$OC^2$）框架，利用主动期权与最佳替代方案之间的正价值差距定义状态依赖的审议成本。该方法鼓励当前期权明显更优时的持久性，并允许在出现更好选项时快速切换。我们证明，直接学习此类成本会导致退化的零成本解，并引入辅助目标，使学到的成本与价值缺口目标对齐。该方法在四个房间和弹球领域中，在静止、目标转移和几何转换条件下进行评估。在弹球目标转移任务中，$OC^2$框架在前十集的成功率为52/%，而固定成本基线为18/%。结果表明，基于机会-成本的持久性提升了在非固定环境中的适应能力。

Model-Free Runtime Measurement of Task-Relevant Information

无模型运行时间的任务相关信息测量

• Authors: Dimitris Panagopoulos, Adolfo Perrusquía, Yang Xing, Weisi Guo
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.14111
• Pdf link: https://arxiv.org/pdf/2611.14111
• Abstract Autonomous systems increasingly monitor physical kinematics, environmental states and task completion outcomes. While existing work provides methods to plan information seeking actions and interpret acquired observations, current approaches do not monitor whether the live process of information gathering is functioning properly. Consequently, such systems can remain operationally active without resolving outstanding mission requirements. To address this limitation, we introduce Acquisition-State Telemetry (AST), a model free architecture that evaluates ongoing information acquisition from observable evidence relative to declared task requirements. Specifically, a domain interface maps raw observations into task resolution coordinates and normalized comparative representations, which are processed by a fixed causal monitor. This monitor emits two continuous real time signals that quantify (i) prospective progress across unresolved requirements and (ii) informational friction generated when recurrent evidence yields negligible informational gain. We evaluated a single, unadjusted monitor configuration across eight diverse domains including search and rescue dialogue, spatial grid exploration, hierarchical capability acquisition, continuous robotic manipulation, and probabilistic intent estimation. We show that across these domains the friction channel distinguished unproductive stagnation from purposeful exploration prior to task completion, while remaining muted during post completion activity. Furthermore, in closed loop reinforcement learning experiments under early termination constraints, we demonstrate that penalizing informational friction allowed agents to sustain high mission success, whereas unpenalized policies and baselines with matched operational costs failed to complete required tasks.
• 中文摘要 自主系统越来越多地监测物理运动学、环境状态和任务完成结果。现有工作提供了规划信息寻求行动和解读已获得观测数据的方法，但当前方法未能监测实时信息收集过程的正常运行。因此，这类系统可以在不解决未解决任务需求的前提下保持运营活跃。为解决这一限制，我们引入了获取状态遥测（AST），这是一种无模型架构，评估可观察证据中持续的信息获取与声明任务需求之间的情况。具体来说，领域接口将原始观测值映射为任务解析坐标和归一化的比较表示，这些由固定的因果监视器处理。该监测器发出两个连续的实时信号，量化（i）未解决需求的前瞻性进展和（ii）当重复证据带来微乎其微的信息收益时产生的信息摩擦。我们评估了单一未调整的监控配置，涵盖八个不同领域，包括搜救对话、空间网格探索、层级能力获取、持续机器人操作和概率意图估计。我们表明，在这些领域中，摩擦通道区分了无效的停滞与任务完成前的有目的探索，而在完成后的活动中则保持静音。此外，在早期终止约束下的闭环强化学习实验中，我们证明惩罚信息摩擦使智能体能够保持高任务成功率，而未惩罚的政策和基线且运行成本匹配的则未能完成所需任务。

Koopman Model-Based Reinforcement Learning

基于模型的强化学习

• Authors: Wenjian Hao, Paulo C. Heredia, Shaoshuai Mou
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.14186
• Pdf link: https://arxiv.org/pdf/2611.14186
• Abstract We propose a Koopman model-based reinforcement learning (K-MBRL) algorithm for controlling nonlinear robotic systems with unknown dynamics. The K-MBRL learns a control policy by minimizing an approximate policy objective that combines the analytical immediate cost with an estimate of the future cost evaluated at the one-step predicted state. To this end, we develop a stochastic Koopman operator-based learning method to approximate the unknown nonlinear dynamics in a linear representation, while a critic function estimates the remaining future cost. This formulation offers two key advantages. First, the linear Koopman dynamics simplify the model-dependent policy gradient computation to a constant matrix product, enabling computationally efficient policy improvement. Second, one-step model propagation mitigates the accumulation of prediction errors associated with long-horizon rollouts while retaining long-term cost information through the critic. The learned policy enables real-time control through feedforward evaluation without online optimization. We establish policy learning convergence guarantees under standard stochastic approximation assumptions. Simulations on HalfCheetah and quadruped four-legged locomotion tasks demonstrate strong sample efficiency and improved performance compared with classical model-free and model-based reinforcement learning baselines. A pendulum swing-up task is used to assess policy gradient approximation accuracy and conduct ablation studies on the learning mechanism and key hyperparameters. The proposed K-MBRL is further validated on a physical quadruped robot performing a challenging sequence of maneuvers: rising to an upright posture supported by two rounded rear feet, maintaining balance during bipedal walking, and safely returning to the ground, as shown in Fig.1.
• 中文摘要 我们提出了一种基于模型的强化学习（K-MBRL）算法，用于控制具有未知动力学的非线性机器人系统。K-MBRL通过最小化一个近似策略目标来学习控制策略，该目标将分析即时成本与在一步预测状态下评估的未来成本估计相结合。为此，我们开发了一种基于随机的Koopman算子学习方法，用于线性表示中近似未知的非线性动力学，同时用批判函数估计剩余的未来成本。该表述有两个关键优势。首先，线性Koopman动力学将模型依赖的策略梯度计算简化为常数矩阵积，从而实现计算效率更高的策略改进。其次，单步模型传播减少了长期推广带来的预测误差积累，同时通过批判者保留长期成本信息。该学习策略通过前馈评估实现实时控制，无需在线优化。我们在标准随机近似假设下建立了策略学习收敛保证。半猎豹和四足四足行走任务的模拟显示出较经典无模型和基于模型的强化学习基线的显著样本效率和性能提升。摆动向上任务用于评估策略梯度近似准确性，并对学习机制及关键超参数进行消融研究。所提的K-MBRL在一台物理四足机器人上进行了进一步验证，机器人执行一系列具有挑战性的动作：由两只圆润后脚支撑的直立姿势，双足行走时保持平衡，安全返回地面，如图1所示。

Recurrent Modular Structure in Spiking Neural Network Reinforcement Learning

尖峰神经网络强化学习中的循环模块结构

• Authors: Vikram Ramavarapu, Himanshu Udupi, Ashish Gautam
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2611.14262
• Pdf link: https://arxiv.org/pdf/2611.14262
• Abstract Biological neural circuits are both modular and recurrent, with dense communities sparsely linked to one another. Spiking neural networks (SNNs) for reinforcement learning are usually feed-forward, and it is unclear how to obtain functional modules in a recurrent SNN or what they buy. In this work, we introduce a class of spiking networks, called Community Recurrent SNNs (CoRe SNNs): recurrent networks partitioned into communities with fixed roles. A controller community receives the stimulus, each action has its own community, and latent communities carry internal state. Spiking activity is then a walk between communities, and the action is the community the walk reaches. All learning is local and three-factor, and the recurrent communities are closer to cortical microcircuits than a feed-forward actor. We compare three regimes for obtaining the communities: dynamic modularity, in which a new learning rule we introduce, Map-STDP, encourages modular structure during learning; partial modularity, in which a stochastic block model (SBM) initialization enforces it partly; and full modularity, in which sealing enforces it fully. On CartPole, across three spiking neuron models, Map-STDP improves learning over the same network without it matches full modularity (sealed communities) on two of the three models, and makes the observed spike walk more compressible. Replacing the feed-forward actor of a TD-STDP actor-critic with recurrent action communities improves on the feed-forward network on CartPole and LunarLander, and matches it on Acrobot. How strongly to enforce modularity depends on the task: full modularity helps most on Acrobot, while on LunarLander the dynamic, partial and full regimes perform alike. Overall, these recurrent modular networks can match or even improve upon feed-forward approaches, with improvements in bio-plausibility and energy efficiency.
• 中文摘要 生物神经回路既是模块化的，也是循环的，密集的社区彼此稀疏连接。用于强化学习的尖峰神经网络（SNN）通常是前馈的，目前尚不清楚如何获得循环SNN中的功能模块或它们购买的是什么。在本研究中，我们介绍了一类尖峰网络，称为社区循环SNN（CoRe SNN）：将循环网络划分为具有固定角色的社区。控制社区接收刺激，每个行动都有自己的社区，潜在社区则携带内部状态。激增活动是社区之间的行走，行动则是行走所到达的社区。所有学习都是局部且三因素的，循环社区更接近皮层微回路而非前馈行为者。我们比较了三种获得群落的模式：动态模块化，其中我们引入的新学习规则Map-STDP鼓励学习过程中模块化结构;部分模块化，随机块模型（SBM）初始化部分强制执行;以及全模块化，封闭完全强制执行。在CartPole上，跨越三个尖峰神经元模型，Map-STDP改善了同一网络上的学习效果，但未匹配三个模型中的两个模型的全模块化（封闭群落），并使观察到的尖峰行走更具压缩性。用重复作用社区替代TD-STDP演员-批判者的前馈演员，改进了CartPole和LunarLander上的前馈网络，并在Acrobot上匹配。模块化的强制执行程度取决于任务：全模块化在 Acrobot 上帮助最大，而在 LunarLander 上动态、部分和全模式表现相似。总体而言，这些循环模块化网络能够匹配甚至改进前馈方法，提升生物可行性和能效性。

Uncertainty-Aware Hybrid Offline Reinforcement Learning with Online Action Selection Enabled by Sparse Gaussian-Process Critics

由稀疏高斯过程批评者支持的不确定性感知混合离线强化学习与在线动作选择

• Authors: Marco Chan, Christos Evripidou, Liqiu Dong, Benoit Chachuat, Mehmet Mercangöz (Sargent Centre for Process Systems Engineering, Department of Chemical Engineering, Imperial College London)
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.14374
• Pdf link: https://arxiv.org/pdf/2611.14374
• Abstract Start-ups and grade changes remain largely operator-driven, while developing and maintaining dynamic models for model-based control requires substantial engineering effort. Hybrid Offline Learning and Online Optimisation (HOFLON) uses one-step offline reinforcement learning to select actions from historical transition data. In its original implementation, the online objective combines the predicted return from a tree-ensemble critic with an adversarial-autoencoder (AAE) penalty for unfamiliar state-action pairs. We examine the effect of this penalty and replace the critic and AAE with a sparse variational Gaussian-process (sGP) critic. Actions maximise $\mu - \kappa\sigma$, an uncertainty-adjusted estimate of the return from one action followed by the historical operating practice. On the HOFLON polymerisation-reactor benchmark, removing the AAE from HOFLON led to 4 failed start-ups in 15 runs over five critics. At its published weight, the variation in the AAE penalty is four orders of magnitude smaller than the variation in the tree-ensemble critic. Nevertheless, it changes actions when the predicted returns of competing candidates are nearly equal, and paired runs with and without the penalty diverged from the first move. With a Matérn-5/2 sGP critic, $\kappa = 0$ led to failure in 13 of 15 runs, whereas with $\kappa = 3$ none of 15 runs (five critics, three noise realisations) failed. The median and lowest returns were 13,464 and 13,398 (HOFLON: 13,393 and 12,411). A squared-exponential sGP critic with learned inducing inputs needed $\kappa \geq 3$ to remain within the empirical data support in steady state, and then settled with a temperature offset of 2.0 K.
• 中文摘要 启动和等级变更仍主要由操作员驱动，而基于模型的动态控制模型开发和维护则需要大量工程投入。混合离线学习与在线优化（HOFLON）利用一步离线强化学习从历史转移数据中选择动作。在线目标在原始实现中结合了树系批评者的预测回报与对陌生状态-动作对的对抗自编码器（AAE）惩罚。我们考察了该惩罚的影响，并用稀疏变分高斯过程（sGP）批评者替代了批判者和AAE。动作最大化$\mu - \kappa\sigma$，这是对一个动作后历史操作操作回报的不确定性调整估计。在HOFLON聚合反应器基准测试中，移除AAE导致5名批评者15次运行中4次启动失败。在公布权重下，AAE惩罚的变异比树系批评者中的变化小四个数量级。然而，当竞争候选者的预测回报几乎相等，且有罚则和无惩罚的配对跑子与第一次走法有分歧时，它会改变行动。在Matérn-5/2 sGP批评者中，$\kappa = 0$导致15次运行中有13次失败，而$\kappa = 3$时，15次运行（五位批评者，三次噪声实现）中无一失败。中位数和最低回报分别为13,464和13,398（HOFLON：13,393和12,411）。一个平方指数级sGP批评者，带有学习诱导输入，需要$\kappa \geq 3$才能在稳态下保持在经验数据支持范围内，最终以2.0 K的温度偏移稳定。

An Energy-Based Perspective on Machine Learning

基于能源的机器学习视角

• Authors: Bo Dai, Yilun Du
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.14383
• Pdf link: https://arxiv.org/pdf/2611.14383
• Abstract Energy-based models (EBMs) define a distribution through a scalar energy, $p_\theta(x)\propto\exp(-E_\theta(x))$, and are usually presented as one model family among many. This tutorial has a broader goal: beyond introducing EBMs, it uses them as a single lens that connects the different aspects of machine learning. Any EBM is specified by three choices, the energy, which determines what the model represents; the inference procedure, which samples from or optimizes the energy; and the learning method, which fits the energy to data. We first develop these components, including sampling, variational inference, and learning by maximum likelihood, noise contrastive estimation, pseudolikelihood, score matching, and adversarial training, and show that the learning methods share one contrastive structure. We then show that discriminative models, generative models such as GANs, normalizing flows, diffusion and autoregressive models, representation learning methods such as contrastive learning, word2vec, BERT, denoising auto-encoders, and diffusion representations, reinforcement learning with EBM policies and world models, inverse reinforcement learning, and planning, as well as inference-time computation with verifiers, are all EBMs that differ in these three choices, mostly in how they handle the partition function and sampling. This view explains existing methods, organizes them into a common framework, and suggests new algorithms by replacing one component while keeping the others, with composition of energies as a principled way to combine learned models at inference time.
• 中文摘要 基于能量模型（EBM）定义一个标量能量分布，称为$p_\theta（x）\propto\exp（-E_\theta（x）））$，通常作为众多模型族中的一个呈现。本教程目标更广：除了介绍EBM，还将它们作为连接机器学习不同方面的单一视角。任何EBM由三个选项指定：能量，决定模型所代表的意义;推理过程，从能量中采样或优化;以及学习方法，将能量拟合到数据中。我们首先开发这些组成部分，包括抽样、变分推断、最大似然学习、噪声对比估计、伪似然、分数匹配和对抗训练，并展示这些学习方法共享一个对比结构。随后我们展示了判别模型、生成模型（如GAN）、归一化流、扩散和自回归模型、表示学习方法（如对比学习）、word2vec、BERT、去噪自编码器和扩散表示、基于EBM策略和世界模型的强化学习、逆强化学习与规划，以及带有验证器的推理时间计算，都是在这三种选择上有所不同的EBM，主要体现在分配函数和采样处理方式上。该观点解释了现有方法，将它们组织成统一框架，并建议通过替换一个组件而保留其他组件，以能量组合作为在推理时结合所学模型的原则性方法，提出新算法。

AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models

AutoScaffold：用于语言模型强化学习的AI引导支架

• Authors: Zijian Zhang, Rizhen Hu, Jiangweizhi Peng, Shiyang Li, Sujay Bhatt, Sihan Zeng, Mingyi Hong
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.14515
• Pdf link: https://arxiv.org/pdf/2611.14515
• Abstract Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. Training-time scaffolds, such as strategies or hints added to the prompt during training, offer a natural way to turn this information into guidance, but their design and adaptation are largely left to humans and tailored to specific domains. We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. A frozen teacher periodically examines this experience, diagnoses bottlenecks, and designs scaffolds in two forms: (i) strategies shared across a class of tasks and (ii) hints for individual hard tasks, adjusting the content and amount of support as the student learns. The scaffolds are removed at evaluation. We apply the same framework to ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation. When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match. On GPU kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to 5.0 points. On math, our student surpasses QuestA's reported accuracy using approximately one-third as many student training steps. Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves, and that its revised scaffolds help the student learn more than keeping the previous scaffold or using no scaffold.
• 中文摘要 带有可验证奖励的强化学习（RLVR）能带来丰富的训练体验，但结果奖励将每次尝试简化为单个数字，且大部分信息未被使用。培训时间支架，如在培训中加入的策略或提示，提供了将信息转化为指导的自然方式，但其设计和适应大多由人类完成，并针对特定领域量身定制。我们介绍了AutoScaffold，一种统一的AI对AI框架，将学生的培训体验转化为自适应的支架，几乎没有领域特定信息，且培训期间无需人工干预。一位冻结教师定期审查这种体验，诊断瓶颈，并设计两种形式的支架：（i）跨类任务共享的策略，（ii）针对单个困难任务的提示，根据学生学习调整内容和支持量。评估时支架被移除。我们将相同框架应用于ALFWorld、基于搜索的质量保证、数学推理和GPU内核生成。在无支架的情况下评估，使用AutoScaffold训练的学生在四个领域均优于使用相应强化学习基线训练的学生，ALFWorld成功率提升7.2分，平均质量保证匹配提升4.0分。在GPU内核生成方面，AutoScaffold使运行速度快于PyTorch参考的正确内核任务比例提升了最多5.0分。在数学方面，我们的学生用约三分之一的学生培训步骤，成功率超过了QuestA报告的准确率。分析显示，教师会识别每个任务类别中的不同失败点，并随着学生进步将支架从逐步工作流程转向有针对性修正，且修订后的支架帮助学生学习比保留之前的支架或不使用支架更多。

Keyword: diffusion policy

Camera equalisation of an 11.5x drop in environmental illuminance: four imitation-learning policies on an SO-101 arm

环境照度下降11.5倍的相机均衡：SO-101臂上的四种模仿学习策略

• Authors: Simon Cowling
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.13810
• Pdf link: https://arxiv.org/pdf/2611.13810
• Abstract Real-robot benchmarks of learned visuomotor policies often record room illuminance as the lighting condition. We show that, on a typical low-cost rig, room light and image light can be almost unrelated. Four imitation-learning policies (ACT, SmolVLA, Diffusion Policy and pi0.5) were trained on the same 50 demonstrations of a pick-and-place task on an SO-101 arm. Each was run for ten rollouts at about 520 lux and again at about 45 lux, an 11.5x reduction. Success was 20/40 in normal light and 19/40 in low light: a change of -2.5 percentage points (95% CI -23 to +19), with no evidence that the policies differed in sensitivity (interaction p = 0.86). The recorded video shows why: mean image brightness was only 3% lower in low light on the front camera and 19% lower on the wrist camera. Both cameras were set to automatic exposure and white balance, which most likely absorbed the difference; exposure was not locked in this experiment. An accidental repeat of one 10-rollout block reproduced its success count (7/10) but matched the first block at only 6 of 10 placements, consistent with chance. We conclude that lux alone does not describe what a policy sees. Camera settings, and image statistics compared with the training recordings, are the quantities to report and control. Data and code are public.
• 中文摘要 真实机器人的视觉运动策略基准通常将房间照度记录为照明条件。我们表明，在典型的低成本设备中，室内光和图像光几乎无关。四个模仿学习策略（ACT、SmolVLA、扩散策略和pi0.5）在SO-101臂上同样50次挑选放置任务演示上进行了训练。每个策略分别在约520 lux和约45 lux下运行了十次，降低了11.5倍。在正常光下成功率为20/40，在低光下为19/40：变化为-2.5个百分点（95% CI -23至+19），且无证据表明这些策略在敏感度上存在差异（交互作用p = 0.86）。录制的视频显示原因：前置摄像头低光环境下平均图像亮度仅低3%，腕部摄像头低19%。两台摄像头均设置为自动曝光和白平衡，这很可能吸收了差异;本实验中曝光未被锁定。一个10个滚动块的意外重复可复制其成功计数（7/10），但仅在10次放置中6个匹配，符合偶然性。我们得出结论，仅用lux无法描述策略所见。摄像头设置以及与训练记录的图像统计量是报告和控制的量。数据和代码是公开的。

From Prediction to Action: Learning Physical Context for Deformable Linear Object Manipulation

从预测到行动：学习物理上下文以实现可变形线性物体操作

• Authors: Chengxiao Dong, Alessio Caporali, Hongyu Lan, Gianluca Palli
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.14274
• Pdf link: https://arxiv.org/pdf/2611.14274
• Abstract A representation that accurately predicts a deformable object's response is not necessarily the one most useful for choosing manipulation actions. We study this distinction between predictive utility and decision utility in planar single-grasp shape control of Deformable Linear Objects (DLOs). As a common decision backbone, we introduce Geometry-Pointer-Flow (GPF), a geometry-grounded policy that separates grasp-location selection from grasp-conditioned motion generation. Its geometry-only variant reduces one-step executed shape error by 32.8% relative to a DLO-specific diffusion policy, while using 18.7 times fewer parameters and achieving approximately 193 times faster inference. On this backbone, we compare physical context represented by ground-truth parameters, estimated parameters, and learned interaction-history representations, for both forward prediction and executed control. For prediction, ground-truth physical parameters reduce next-shape error by 15.8% over geometry alone, and estimated parameters are slightly more predictive than interaction history. This ordering does not carry over to control: estimated parameters provide no detectable one-step control gain, whereas history-conditioned GPF reduces executed error by 9.9% relative to geometry-only GPF across 470 paired simulation trials. In closed loop, interaction history improves strict-threshold attainment in simulation and yields the strongest zero-shot performance on hardware. All policies are trained exclusively in simulation and transferred to the physical robot without real-world training or fine-tuning. These results show that physical context should be evaluated by the decisions it supports rather than by predictive accuracy alone. Code will be released upon acceptance.
• 中文摘要 准确预测可变形物体响应的表示不一定是选择操作动作最有用的。我们研究了变形线性物体（DLO）平面单抓形形状控制中预测效用与决策效用之间的区别。作为一种通用决策骨干，我们引入了几何-指针流（Geometry-Pointer-Flow，GPF），这是一种几何基础策略，将抓取位置选择与抓取条件运动生成分离。其仅几何变体相较于DLO特有扩散策略，将一步执行形状误差降低32.8%，同时使用少18.7倍的参数，实现约193倍的推断速度。基于该骨干，我们比较了前向预测和执行控制中由地面真实参数、估计参数和学习到的交互历史表示表示的物理上下文。在预测方面，地面真实物理参数仅几何体下可降低15.8%的次形状误差，估计参数的预测性略高于交互历史。这种排序不适用于控制：估计参数不提供可检测的一步控制增益，而历史条件GPF在470次配对模拟试验中相较几何GPF降低执行误差9.9%。在闭环中，交互历史提升了模拟中的严格阈值达成，并在硬件上实现最强的零时效性能。所有策略均仅在仿真中训练，并传输到物理机器人，无需真实训练或微调。这些结果表明，物理上下文应以其所支持的决策来评估，而非仅仅依赖预测准确性。代码将在接受后发布。