生成时间: 2026-07-15 18:05:01 (UTC+8); Arxiv 发布时间: 2026-07-15 20:00 EDT (2026-07-16 08:00 UTC+8)

今天共有 24 篇相关文章

Keyword: reinforcement learning

In-Context Reinforcement Learning under Non-Stationarity: A Survey

非平稳性下的情境强化学习：一项综述

• Authors: A Run, Ziluo Ding
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.11906
• Pdf link: https://arxiv.org/pdf/2607.11906
• Abstract The development of decision-pretrained transformers, algorithm distillation, long-context meta-RL, and retrieval-augmented agents has renewed interest in in-context reinforcement learning (ICRL): the ability of a pretrained or fine-tuned decision model to infer latent task rules and improve future behavior from interaction context, without test-time parameter updates. This line of work asks when trial-and-error evidence, rewards, transitions, demonstrations, feedback, or retrieved experience can make learning-like computation happen inside the context window. However, existing surveys of ICRL mainly organize the field around pretraining objectives, architectures, context formats, evaluation protocols, and theoretical mechanisms, while the non-stationary setting remains comparatively underexamined. In changing environments, accumulated context is not merely more evidence about a fixed task: the reward specification, transition kernel, observation channel, action interface, constraint model, or demonstration and memory distribution can fall out of alignment with the current regime. Previously useful context can therefore become stale, misleading, or useful again when an old regime returns. We survey non-stationary ICRL as the problem of adapting through context while deployed policy parameters remain fixed: the policy must infer both the current decision rule and which parts of its accumulated evidence still support that rule. We define non-stationary ICRL, relate it to meta-RL, decision sequence modeling, retrieval-augmented RL, value- and model-aware ICRL, and reward-feedback agents, and organize the literature along three questions: what changes, how the change unfolds, and how observable the change is to the agent.
• 中文摘要 决策预训练变换器、算法提炼、长上下文元强化学习（meta-RL）和检索增强智能体的发展，重新激发了对上下文强化学习（ICRL）的兴趣：即预训练或微调决策模型能够从交互上下文中推断潜在任务规则并改善未来行为，而无需测试时间参数更新。该研究方向探讨了试错证据、奖励、过渡、演示、反馈或检索经验何时能在上下文窗口内实现类似学习的计算。然而，现有的ICRL调查主要围绕预训练目标、架构、上下文格式、评估协议和理论机制来组织该领域，而非平稳环境则相对缺乏研究。在变化的环境中，积累的上下文不仅仅是关于固定任务的更多证据：奖励规范、过渡核、观察通道、动作界面、约束模型，或演示和内存分布可能与当前体系不符。因此，当旧体制回归时，原本有用的背景可能会变得陈旧、误导或再次有用。我们考察非平稳ICRL的问题，即在部署的政策参数保持固定的情况下，如何通过上下文适应：政策必须推断当前决策规则及其积累证据中哪些部分仍支持该规则。我们定义了非平稳ICRL，并将其与元强化学习、决策序列建模、检索增强强化学习、价值感知和模型感知ICRL以及奖励反馈代理联系起来，并围绕三个问题组织文献：变化是什么、变化如何展开，以及变化对智能体的可观察性。

Graph-Constrained Policy Learning for Extreme Clinical Code Prediction

极限临床代码预测的图约束策略学习

• Authors: Amritpal Singh, Sebastian Torres, Khawar Shakeel, Syed Ahmad Chan Bukhari
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.11954
• Pdf link: https://arxiv.org/pdf/2607.11954
• Abstract Clinical code prediction maps unstructured discharge summaries to ICD-10-CM leaf codes in a large, sparse, and deeply hierarchical label space. Most systems treat the task as flat multi-label classification, scoring codes independently and providing limited training signal for rare labels. We propose a graph-constrained traversal policy that formulates ICD prediction as a finite-horizon decision process over a pruned code hierarchy. A single language model descends the graph level by level, selecting valid child nodes until billable leaf codes are reached. This converts extreme multi-label prediction into sparse, hierarchy-aware subset decisions while guaranteeing structurally valid outputs. On MIMIC-IV discharge summaries, our best supervised policy, SFT-1+, achieves 0.709 micro-F1 on a curated 50-code subset and 0.527 micro-F1 on the full 15,761-code space, outperforming flat baselines including CAML, LAAT, and PLM-ICD. In the full setting, SFT-1+ improves over the strongest flat baseline by 0.044 micro-F1 and 0.157 macro-F1, suggesting that graph-constrained decomposition mitigates the rare-code bottleneck. A controlled factorial study evaluates architecture, training algorithm, and data budget. Across both scales, one shared policy matches a three-specialist cascade while avoiding its context-window overflow on 28-32% of full-space test notes. Increasing supervised trajectory data is the only intervention that consistently improves performance, while GRPO reinforcement learning provides no benefit over supervised continuation with matched data. These results show that simple graph-constrained policy learning can outperform more complex flat, cascaded, and reinforcement-learning alternatives for extreme clinical code prediction.
• 中文摘要 临床代码预测将非结构化出院摘要映射为ICD-10-CM叶片代码，存在一个庞大、稀疏且高度层级化的标签空间。大多数系统将任务视为平面多标签分类，独立评分代码，并对稀有标签提供有限的训练信号。我们提出了一种图约束的遍历策略，将ICD预测表述为一个有限视野的决策过程，覆盖修剪后的代码层级。单一语言模型逐层下降图，选择有效的子节点，直到达到可计费叶节点。这将极端多标签预测转化为稀疏、层级感知的子集决策，同时保证结构有效输出。在MIMIC-IV放电摘要中，我们最佳监督政策SFT-1+在精选的50码子集上实现了0.709微F1，在整15,761码空间中实现0.527微F1，优于包括CAML、LAAT和PLM-ICD在内的平坦基线。在完整环境下，SFT-1+在最强平坦基线上微F1提升0.044，宏F1提升0.157，表明图约束分解缓解了稀有代码瓶颈。受控因子研究评估架构、训练算法和数据预算。在这两个尺度中，一个共享策略匹配三专家级联，同时避免在28-32%的全空间测试笔记中出现上下文窗口溢出。增加监督轨迹数据是唯一能持续提升表现的干预措施，而GRPO强化学习相比有匹配数据的监督持续学习没有优势。这些结果表明，简单的图约束策略学习在极端临床代码预测方面，能够优于更复杂的平面、级联和强化学习替代方案。

Calibration-First Reward-Component Auditing for Reinforcement Learning Control in Smart Greenhouses

校准优先奖励组件审计用于智能温室强化学习控制

• Authors: Yuhui Bie, Guowei Xu, Yaojun Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.11959
• Pdf link: https://arxiv.org/pdf/2607.11959
• Abstract Greenhouse reinforcement learning can test climate-control ideas at a speed and scale that is difficult to achieve with crop experiments alone. For smart-greenhouse control, however, a single simulator return is not enough: a grower or control engineer also needs to know when the policy heats, enriches CO2, vents, manages humidity, deploys screens, or uses this http URL propose a reproducible calibration-first reward audit framework that keeps named greenhouse-control reward components comparable across simulator training, facility-adapted rollouts, logged Autonomous Greenhouse Challenge records, and actuator-rule distillation. In GreenLight-Gym, the framework decomposes the scalar reward into conditional temperature, CO2, humidity and vapor-pressure-deficit, screen, and actuation-proxy terms; adapts GreenLight to the second Autonomous Greenhouse Challenge logged climate traces; and scores the same components on logged greenhouse data.
• 中文摘要 温室强化学习能够以难以仅靠作物实验的速度和规模测试气候控制理念。然而，对于智能温室控制来说，单一模拟器回报还不够：种植者或控制工程师还需要知道政策何时加热、浓缩二氧化碳、排放、管理湿度、部署筛网或使用该 http URL 提出可重复的优先校准奖励审计框架，使命名的温室控制奖励组件在模拟器培训、设施适配推广、自主温室挑战记录中保持可比性， 以及执行器规则蒸馏。在GreenLight-Gym中，该框架将标量奖励分解为条件温度、二氧化碳、湿度和蒸气压缺失、屏幕和执行代理项;将GreenLight调整为第二次自主温室挑战记录的气候痕迹;并且在记录的温室数据中对相同的成分进行评分。

Robust In-Hand Manipulation via Priors in Reinforcement Learning and Mechanical Design

通过强化学习和机械设计中的先验经验，稳健的手部操作

• Authors: Yifei Chen, Shihan Lu, Ed Colgate, Kevin Lynch
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.12105
• Pdf link: https://arxiv.org/pdf/2607.12105
• Abstract In-hand manipulation without external sensing is challenging due to uncertainties from finger-object contacts and disturbances by gravity. While reinforcement learning has shown promise in learning complex finger gaiting, existing approaches do not prioritize maintaining well-conditioned grasps for sustained manipulation. We introduce two complementary physics priors for robust in-hand rolling: a global grasp-quality prior derived from classical grasp analysis and a local contact-geometry prior based on fingertip curvature. The grasp-quality prior is used as a dense reward-shaping term that encourages well-distributed contacts with improved worst-case wrench resistance. The contact-geometry prior is expressed in the fingertip geometry that mechanically shapes the contact interface toward task-aligned rolling while reducing off-axis drift. We evaluate the effect of these priors on learning in-hand rolling manipulation for a multifingered robotic hand manipulating three different objects at four palm orientations. Results show significant improvement in rotation efficiency, grasp stability, and disturbance rejection, suggesting that physics priors embedded in both learning and fingertip morphology improve task robustness and sim-to-real transfer. An overview video can be found at this https URL.
• 中文摘要 由于手指与物体接触的不确定性和重力干扰，手握操作在没有外部感知的情况下具有挑战性。虽然强化学习在学习复杂的手指步态方面展现出潜力，但现有方法并未优先考虑保持良好条件的抓握以实现持续操作。我们引入了两个互补的稳健手内滚动物理先验：一个基于经典抓取分析的全局抓取质量先验，以及基于指尖曲率的局部接触几何先验。抓握质量先验被用作一个密集的奖励整形项，鼓励均匀分布的接触，并改善最坏情况下的扳手阻力。接触几何先验表现为指尖几何形状，机械地将接触界面塑造成任务对齐的滚动，同时减少偏轴漂移。我们评估这些先验对多指机器人手在四个掌心方向下操作三种不同物体时的手中滚动操作学习的影响。结果显示，旋转效率、抓握稳定性和干扰抑制显著提升，表明嵌入学习和指尖形态中的物理先验提升了任务的鲁棒性和模拟到实际的迁移。该网址有一段概述视频。

Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

零环：将零强化学习扩展到一万亿参数以实现涌现推理

• Authors: Xinyu Tang, Gangqiang Cao, Yurou Liu, Yuliang Zhan, Xiaochong Lan, Yifan Li, Yuchen Yan, Han Peng, Zican Dong, Zhenduo Zhang, Tianshu Wang, Xinyu Kong, Zujie Wen, Wayne Xin Zhao, Zhiqiang Zhang, Jun Zhou
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.12395
• Pdf link: https://arxiv.org/pdf/2607.12395
• Abstract Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the "bitter lesson" of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.
• 中文摘要 带有可验证奖励且无需人工注释数据的强化学习，通常称为零强化学习，已成为引发思维链推理的强大范式。然而，由于计算限制，现有研究大多局限于小模型，导致训练动力学和大规模涌现能力尚未被充分探索。为了有意义地探索这一前沿，我们旨在从模型中引发高质量的推理行为。然而，我们发现朴素尺度常常面临可读性差、符号冗余和缺乏自适应推理深度的问题。为应对这些挑战，我们提出了一个稳定高效的训练流水线，结合了算法和系统优化，如截剪重要性抽样、训练-推断比校正和混合精度控制。我们的实验提供了三个关键发现，验证了缩放的“苦涩教训”：（1）缩放到1T参数显著提升样本效率和性能上限;（2）训练过程依次进行初步发现阶段，随后是磨砺阶段;以及（3）模型自发发展高级认知行为，包括拟人化、结构化格式、自我验证、平行推理和上下文焦虑，使手工设计的启发式变得多余。经过七项数学基准测试，Ring-2.5-1T-Zero 实现了竞争性能。此外，为了评估CoT质量超越最终答案正确性，我们提出了一个结构化评估框架，涵盖可理解性、可重复性和效率三个维度，我们的模型在产生结构化且简洁的推理迹方面展现了明显优势。通过分享我们观察到的涌现现象，我们希望为社区提供更深入的关于尺度行为的见解，尤其是在1万亿尺度上。

Steering Diffusion Models via Class-Contrastive Influence for Few-Shot Medical Classification

通过类别对比影响引导扩散模型，用于少数样本医学分类

• Authors: Jeeyung Kim, Erfan Esmaeili, Qiang Qiu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.12464
• Pdf link: https://arxiv.org/pdf/2607.12464
• Abstract When labeled data are scarce, off-the-shelf diffusion models can augment training sets for few-shot medical image classification, but not all generated samples are equally useful for the downstream task. Existing approaches largely improve synthetic data by increasing realism, diversity, or domain adaptation, while overlooking a more fundamental question: how should sample usefulness for classification be measured and optimized? We address this with Class-Contrastive Influence (C2I), a criterion that quantifies a sample's usefulness through its gradient-based influence on the classifier. We find that effective samples exhibit a strong C2I gap: their loss gradients align with validation gradients from the same class and oppose those from other classes. Our analysis further suggests that such high-C2I samples are hard, boundary-proximal examples that help refine the decision boundary and improve robustness. Building on this insight, we fine-tune diffusion models with reinforcement learning using a C2I-based reward to steer generation toward class-informative samples. Across several few-shot medical imaging benchmarks, C2I-guided generation improves downstream accuracy and robustness over diffusion-based augmentation baselines, showing that synthetic augmentation is most effective when guided by task usefulness rather than image quality alone.
• 中文摘要 当标记数据稀缺时，现成的扩散模型可以增强训练集，用于少量样本的医学图像分类，但并非所有生成的样本在后续任务中都同样有用。现有方法主要通过提高真实性、多样性或领域适应性来改善合成数据，同时忽视了一个更根本的问题：样本在分类中的实用性应如何测量和优化？我们用类别对比影响（C2I）来解决这个问题，这是一个通过基于梯度的样本对分类器影响来量化样本的有用性。我们发现有效样本表现出强烈的C2I差距：其损耗梯度与同一类别的验证梯度一致，且与其他类别的梯度相对。我们的分析进一步表明，这种高C2I样本是硬的、边界近端的例子，有助于细化决策边界并提高鲁棒性。基于这一见解，我们利用基于C2I的奖励，通过强化学习微调扩散模型，引导生成向具有类别信息量的样本。在多个少数样本医学影像基准中，C2I引导生成相较于基于扩散的增强基线提升了后续精度和稳健性，表明合成增强在以任务实用性为指导而非仅仅图像质量时效果最佳。

TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments

TRACE：可审计代理承诺的操作推理模式

• Authors: Edward Y. Chang, Emily J. Chang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.12480
• Pdf link: https://arxiv.org/pdf/2607.12480
• Abstract This paper defines TRACE (Typed Reasoning And Commitment Evidence): a typed, versioned schema for recording reasoning traces, a reference procedure for writing records against it, and one operating discipline, no durable state change without a record. The paper argues in three layers that reasoning is not in the language model: the autoregressive mechanism natively computes association; chain-of-thought and reinforcement learning inherit its limits; and the formal constructs of reasoning theory, from Socratic procedure to Pearl's ladder, are absent as machinery. The schema answers the absence with fields and tests: the TraceRecord and its causal specialization, an eight-stage reference writer, a gate-first measurement regime, the TRACE-Bench protocol, and the consumers, memory admission, plan gating, temporal regret, and verdict reuse, whose more auditable decisions are the measure of the record. A record-consumer contract states what a record guarantees and what a consumer must honor in return, making the schema an operational interface rather than a passive document. Two worked examples run in the main text: a music-lessons argument traced from sentence to typed verdict, separating association, intervention, and prescription; and a flood search-and-rescue vignette in which a predictive world model reports confident plan success that its own support and out-of-distribution scores contradict, so the record defers the commitment, requests a bounded observation, revises append-only, and clears a different branch. The vignette is illustrative, not empirical; closed-loop evaluation is left to future work, so the contribution is the schema and its contract, not a performance claim. Appendices carry the full schema, writer algorithms and cost model, clinical and policy illustrations, the benchmark protocol, convergence metrics, and usage scenarios.
• 中文摘要 本文定义了TRACE（类型推理与承诺证据）：用于记录推理痕迹的类型化版本化模式，一种用于对其写入记录的参考程序，以及一个操作性领域，没有记录则不存在持久状态变化。论文分为三层论证，推理不属于语言模型：自回归机制原生计算关联;思维链和强化学习继承了其局限性;而推理理论的形式结构，从苏格拉底程序到珍珠阶梯，作为机制都不存在。该模式通过字段和测试来回应缺失：TraceRecord及其因果专化、八阶段参考写入器、门优先测量体系、TRACE-Bench协议，以及消费者——记忆录取、计划门槛、时间遗憾和判决重用，这些决策更具可审计性，是记录的衡量标准。记录-消费者合同规定了记录所保证的内容以及消费者必须遵守的条件，使模式成为一个操作接口，而非被动文档。正文中有两个示例：一个从句子到打字判决的音乐课程论证，区分联想、干预和处方;以及一个洪水搜救小品，预测世界模型报告了自信的计划成功率，但其支持率和分配外评分相互矛盾，因此记录推迟承诺，请求有界观测，仅修订附加，并清除其他分支。该小品是说明性的，而非经验性的;闭环评估留待后续工作，因此贡献是模式及其契约，而非性能声明。附录包含完整的模式、编写算法和成本模型、临床和政策示例、基准协议、趋同指标及使用场景。

OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning

OOD-RL-Bench：强化学习中分布外检测的基准框架

• Authors: Emil Mittag, Richard Dazeley, Peter Vamplew
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.12523
• Pdf link: https://arxiv.org/pdf/2607.12523
• Abstract Reliable reinforcement learning (RL) agents must maintain operational integrity amidst sensor malfunctions, dynamic disturbances, and slow environmental shifts. The detection of out-of-distribution conditions is pivotal to determining when an agent's observations, transitions, or trajectory dynamics deviate from the assumptions underpinning its policy training. Current out-of-distribution (OOD) detection benchmarks typically evaluate image classifiers or static low-dimensional datasets, failing to account for the complex, action-dependent temporal structure inherent in RL trajectories. To address this gap, we present OOD-RL-Bench, a comprehensive and extensible framework designed to evaluate OOD detectors against categories of anomalies injected into RL trajectories. Detectors and anomaly injectors are integrated through shared interfaces and configuration, which allows new scoring methods and perturbation families to be evaluated without modification of the core benchmark loop. We evaluate the utility of the framework using a Deep Q-Network policy within the LunarLander-v3 environment. We assess the performance of each detector across a suite of anomaly types using matched-time AUROC, matched-time AUPRC, matched-time false-positive rate, detection delay, and segmented-onset metrics. Our analysis reveals significant performance variance across anomaly types: observation perturbations and regime switches are identified with high accuracy by several methods, while observation delay and action-conditioned dynamics remain difficult even when post-onset anomaly scores are compared against clean scores from the same timesteps. We make the framework, trained policy checkpoint, and complete results publicly available as a reproducible artefact.
• 中文摘要 可靠的强化学习（RL）代理必须在传感器故障、动态扰动和环境变化缓慢的情况下保持操作完整性。检测分布外条件对于判断代理的观察、转变或轨迹动态是否偏离其政策训练的假设至关重要。当前的非分布（OOD）检测基准通常评估图像分类器或静态低维数据集，未能考虑强化学习轨迹中复杂且依赖动作的时间结构。为弥补这一空白，我们提出了OOD-RL-Bench，这是一个全面且可扩展的框架，旨在评估针对注入强化轨道的异常类别的OOD探测器。探测器和异常注入器通过共享接口和配置集成，使得无需修改核心基准循环即可评估新的评分方法和微扰族。我们利用深度Q网络策略在LunarLander-v3环境中评估该框架的实用性。我们通过匹配时间AUROC、匹配时间AUPRC、匹配时间假阳性率、检测延迟和分段起始指标，评估各检测器在一系列异常类型中的表现。我们的分析显示，不同异常类型存在显著性能差异：多种方法能高精度识别观测扰动和状态切换，而即使将异常发生后评分与同一时间步的纯净分数进行比较，观察延迟和动作条件动力学仍然难以确定。我们将框架、经过训练的政策检查点和完整结果作为可复现的成果公开。

Environment Parameter Gradient Theorem for Policy-Environment Co-Design in Reinforcement Learning

强化学习中政策-环境共设计的环境参数梯度定理

• Authors: Amber Srivastava
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.12590
• Pdf link: https://arxiv.org/pdf/2607.12590
• Abstract Reinforcement learning (RL) is traditionally concerned with learning a control policy for a fixed environment. In many engineering systems, however, the environment itself is alterable: physical or operational parameters can be tuned to shape the transition dynamics and costs experienced by the agent. This motivates jointly optimizing both the policy and the environment design parameters. To this end, we establish an Environment Parameter Gradient Theorem -- a formal expression for the gradient of the value function with respect to environment parameters. The key theoretical device is a generalized action-value function $Q_{\pi,\xi}(s,a,\zeta)$, which comprises two copies of the environment parameters: $\zeta$ governs the cost and transition dynamics at the current state--action pair, while $\xi$ governs the future rollouts. This decoupling yields a tractable closed-form gradient expression and is essential to the theorem's derivation. Building on this result, we develop a model-free algorithm that simultaneously learns the optimal policy and the environment parameters. We demonstrate the efficacy of our framework on a UAV network design problem, where the optimal UAV placement (environment parameters) and communication routes (governed by the policy) are learned jointly to minimize the total communication cost in the network.
• 中文摘要 强化学习（RL）传统上关注的是学习固定环境的控制策略。然而，在许多工程系统中，环境本身是可变的：物理或操作参数可以调整，以塑造智能体所经历的过渡动态和成本。这促使人们共同优化政策和环境设计参数。为此，我们建立了环境参数梯度定理——这是价值函数相对于环境参数梯度的形式表达式。关键理论装置是一个广义动作-价值函数 $Q_{\pi，\习}（s，a，\zeta）$，包含两个环境参数副本：$\zeta$ 控制当前状态-动作对的成本和过渡动态，而 $\习$ 控制未来的推广。这种解耦得到一个可解的闭式梯度表达式，并且对定理的推导至关重要。基于此结果，我们开发了一种无模型算法，能够同时学习最优策略和环境参数。我们展示了该框架在无人机网络设计问题上的有效性，在该问题中，最优无人机布置（环境参数）和通信路径（受策略控制）是共同学习的，以最小化网络中的总通信成本。

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

小型语言和视觉语言模型中GRPO的学习率门控失败：一个受控空虚及其机制

• Authors: Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.12640
• Pdf link: https://arxiv.org/pdf/2607.12640
• Abstract Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent. We ask whether it adds skill to a small language and vision-language model web agent at the 4B to 8B scale, or whether it mostly reshapes behavior the supervised model already has. Across a control grid of 18 runs that varies learning rate, KL weight, seed, initialization, and clipping, no configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered. On the text track, moderate to high learning rates make it credibly worse. The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set-of-Marks screenshot observations, and scaling the backbone to 8B; the credible harm is a text-track finding and is only nominal under Set-of-Marks. To show that the null reflects the setting and not a broken pipeline, we run the identical harness, reward, and recipe on tasks whose reward is reachable by sampling, and there the success rate rises by 22 points with a paired interval that excludes zero. GRPO therefore helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one. We then explain the failure. A middle learning rate degrades the agent and a high one collapses it, and the two regimes form a double dissociation: grafting localizes the degrade regime to the attention and MLP blocks, while the collapse regime cannot be traced to any single group, and the embedding change that dominates the weight movement is causally inert. At 4B, effective rank in the late layers tracks capability in both directions; at 8B the two come apart. This coupling is specific to the smaller model, so we report it as scale-dependent.
• 中文摘要 带有可验证奖励的强化学习，尤其是群体相对策略优化（Group Relative Policy Optimization，GRPO），现在常在监督检查点上常规运行，希望能产生更强的代理。我们会问，它是否在4B到8B尺度上为小型语言和视觉语言模型的网络代理增加了技能，还是主要重塑了受监督模型已有的行为。在由18次运行组成的控制网格中，这些运行变化包括学习率、KL权重、种子、初始化和裁剪，没有任何配置能有效提升代理基本掌握任务的强监督基线的成功率。在文本方向上，中等到高的学习率反而让情况更差。空值在配对测试、25个评估种子、6个训练种子、配方变更、文本和标记集截图观察，以及骨干网缩放到8B下保持有效;可信的伤害是文本追踪的判断，在标记集下仅为名义上的。为了证明空值反映了设置而非流水线故障，我们对可通过采样获得奖励的任务运行相同的线束、奖励和配方，成功率提升22分，且配对区间排除零。因此，GRPO只有在有上升空间时才有帮助，也就是说抽样的政策已经比贪婪的政策更成功。然后我们解释失败的原因。中等学习率会使主体退化，高学习率则使其崩溃，这两种状态形成双重解离：嫁接将退化状态局限于注意力块和MLP块，而崩溃状态无法追溯到任何单一群体，主导权重移动的嵌入变化则是因果惰性的。在4B层，后期的有效等级与能力的双向同步;在8B时，两者分开。这种耦合是针对较小模型的特有，因此我们报告为尺度依赖。

From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation

从批评者到信心：基于语言的PPO定量预测与置信估计

• Authors: Mehak Dhaliwal, Rasta Tadayon, Andong Hua, Haewon Jeong, Yao Qin
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.12687
• Pdf link: https://arxiv.org/pdf/2607.12687
• Abstract LLMs can perform language-based quantitative prediction from unstructured inputs, but remain susceptible to hallucinations and overconfident errors, making it critical to know not only what a model predicts, but when its predictions can be trusted. We introduce CARE-PPO, a reinforcement learning framework that establishes a connection between loss prediction for uncertainty estimation and actor-critic PPO fine-tuning, enabling joint learning of accurate numerical estimates and reliable confidence signals in language-based quantitative prediction. CARE-PPO uses a Confidence-Aligned Reward for Estimation, defined as a function of prediction error, to provide dense error-aware feedback to the actor while inducing the critic to learn a value function aligned with prediction quality. During inference, we repurpose the critic as a confidence estimator. Across two real-world tasks in healthcare and finance and two Qwen-3 model scales (4B and 8B), CARE-PPO achieves strong quantitative prediction performance, while producing significantly better-aligned confidence estimates through the critic than logit-based and verbalized baselines. These gains persist under realistic out-of-distribution settings across domains, spanning linguistic and domain shifts. Finally, CARE-PPO reduces task-specific overfitting on general instruction-following prompts, consistent with the broader generalization advantages of RL fine-tuning over supervised approaches.
• 中文摘要 LLM可以从非结构化输入中进行基于语言的定量预测，但仍易受幻觉和过度自信误差的影响，因此不仅要了解模型预测的内容，还要了解其预测何时可信。我们介绍了CARE-PPO，一种强化学习框架，建立了不确定性估计损失预测与演员-批评者PPO微调之间的联系，使得基于语言的定量预测中能够共同学习准确的数值估计和可靠的置信信号。CARE-PPO使用信心对齐的估计奖励，定义为预测误差的函数，向参与者提供密集的误差感知反馈，同时引导批评者学习与预测质量相匹配的价值函数。在推断过程中，我们会将批评者重新定位为置信估计工具。在医疗和金融的两个真实任务以及两个Qwen-3模型量表（4B和8B）中，CARE-PPO实现了强劲的定量预测表现，同时通过批评者产生的置信度估计比基于logit和口头化基线更为契合。这些成果在跨领域现实的非分布环境下依然存在，涵盖语言和领域的转变。最后，CARE-PPO减少了一般指令跟随提示的任务特异性过拟合，这与强化学习微调相较于监督方法的广泛泛化优势相符。

Vision-Based Dribbling for Humanoid Soccer via Privileged Representation Learning

通过特权表征学习实现人形足球的基于视觉的盘带

• Authors: Flavio Maiorana, Valerio Spagnoli, Eugenio Bugli, Flavio Volpi, Daniele Affinita, Vincenzo Suriani, Daniele Nardi, Luca Iocchi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.12702
• Pdf link: https://arxiv.org/pdf/2607.12702
• Abstract Recent advances in humanoid robotics have highlighted the importance of deployable loco-manipulation skills. Dribbling a soccer ball while evading active opponents requires simultaneous balance, precise ball control, and awareness of a dynamic adversary under onboard sensing and real-time constraints. Existing approaches typically separate perception and motion, which can be effective in controlled settings but may fail under occlusions, fast ball movements, and complex opponent interactions, since perception is not directly optimized for control. We propose an integrated approach in which a temporal depth encoder is embedded into a reinforcement learning policy through a task-specific projection layer. We apply this framework to a simulated Booster T1 humanoid robot and show that it is possible to learn vision-based, opponent-aware dribbling directly from depth observations, without explicit state estimation or privileged scene information. The learned policy achieves 100% success in nominal target-driven dribbling and 96% success with a single static obstacle, while reaching 46% success against an actively moving ball-attacker opponent. These results demonstrate that the proposed framework supports robust vision-based dribbling in nominal and moderately dynamic settings, and provides a strong foundation for handling more challenging moving-adversary scenarios.
• 中文摘要 人形机器人技术的最新进展凸显了可部署机车操作技能的重要性。在躲避活跃对手的同时运球，需要同时具备平衡、精准控球，以及在机载感知和实时约束下对动态对手的感知。现有方法通常将感知与运动分开，这在受控环境中可能有效，但在遮挡、快速球的移动和复杂的对手互动中可能失效，因为感知并非直接优化以控制。我们提出一种综合方法，通过任务特定的投影层将时间深度编码器嵌入强化学习策略中。我们将该框架应用于模拟的Booster T1类人机器人，展示了可以直接从深度观察中学习基于视觉、对手感知的盘带，而无需显式的状态估计或特权场景信息。该策略在名义上的目标驱动盘带成功率为100%，对单个静态障碍成功率为96%，而对积极移动的进攻者则达到46%。这些结果表明，所提框架支持在正常且中度动态环境中的基于视觉的稳健运球，并为应对更具挑战性的移动对手场景奠定了坚实基础。

Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordination

约束感知聚合用于微电网能源协调中的联邦强化学习

• Authors: Usman Haider, Karl Mason
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.12763
• Pdf link: https://arxiv.org/pdf/2607.12763
• Abstract Federated Reinforcement Learning (FedRL) enables coordination of distributed energy resources without sharing raw local data, but standard aggregation methods such as FedAvg do not account for system-level constraints, often leading to unsafe global behavior. In this work, we study constraint-aware aggregation for federated reinforcement learning in distributed energy coordination. We propose aggregation rules that incorporate both local performance and estimated constraint violation into the server-side update. Among these, a simple penalty-based rule, $w_i \propto R_i - \alpha V_i$, consistently provides the most reliable trade-off between reward and safety, without requiring dual optimization or modifications to local training. \textcolor{black}{We evaluate our approach on DairyGridEnv, a benchmark modeling multiple farms coordinating battery storage under stochastic demand and a shared grid capacity constraint, and further assess robustness using real load-driven demand profiles from Finland and the German FIELD dataset. Across multiple seeds, penalty-based aggregation substantially reduces violations while improving reward relative to FedAvg in both synthetic and real load-driven settings.} A combined reward-violation scheme exposes a tunable trade-off via $\lambda$, but is less stable. These results demonstrate that lightweight aggregation strategies can substantially improve empirical safety in federated reinforcement learning while preserving standard communication protocols.
• 中文摘要 联邦强化学习（Federated Reinforcement Learning，FedRL）能够协调分布式能源资源，而无需共享原始本地数据，但像FedAvg这样的标准聚合方法并未考虑系统层级的约束，常常导致不安全的全局行为。本研究研究分布式能量协调中联邦强化学习中的约束感知聚合。我们提出将本地性能和估计的约束违规纳入服务器端更新的聚合规则。其中，一条基于惩罚的简单规则 $w_i \propto R_i - \alpha V_i$，始终提供了奖励与安全性之间最可靠的权衡，无需双重优化或局部训练修改。\textcolor{black}{我们评估了DairyGridEnv的基准测试，该基准测试模拟多个农场在随机需求和共享电网容量约束下协调电池储能，并利用芬兰和德国FIELD数据集的真实负载驱动需求剖面进一步评估鲁棒性。在多个种子中，基于惩罚的聚合在合成和真实负载驱动环境中，显著减少违规并提升了相对于FedAvg的奖励。}组合奖励-违规方案通过$\lambda$暴露出可调节的权衡，但稳定性较低。这些结果表明，轻量级聚合策略在保持标准通信协议的同时，可以显著提升联邦强化学习的实证安全性。

Directional Constraints for Efficient Exploration in Safe Reinforcement Learning

安全强化学习中高效探索的方向约束

• Authors: Paolo Magliano, Puze Liu, Jan Peters, Davide Tateo, Raffaello Camoriano
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.12784
• Pdf link: https://arxiv.org/pdf/2607.12784
• Abstract Reinforcement Learning has revolutionized the landscape of robotic research, allowing robust learning of complex robotic skills in simulation. However, real-world deployment in open-ended environments requires strong safety guarantees to prevent dangerous or harmful behaviors. Safe Reinforcement Learning methods address this requirement by enforcing safety constraints. Nevertheless, learning under constraints often reduces learning speed and could lead to suboptimal task performance, as the agent must solve a more complex constrained optimization problem compared to unconstrained settings. To tackle this issue, in this work, we propose an extension of the ATACOM framework, a state-of-the-art reliable safety layer that can be integrated with existing Reinforcement Learning algorithms to enforce constraints derived from prior knowledge of the system or learned directly from data. Our proposed method, named ATACOM Directional Constraints (ATACOM-DC), significantly improves the safety-performance trade-off by introducing directional constraints that distinguish between actions approaching and moving away from constraint boundaries, activating constraint enforcement only when necessary. We evaluate our method across a range of challenging robotic control tasks in simulation, analyzing both constraint-violation costs and achieved task performance. Code and additional material at this https URL.
• 中文摘要 强化学习彻底改变了机器人研究的格局，使得在模拟中稳健地学习复杂机器人技能。然而，在开放式环境中的实际部署需要强有力的安全保障，以防止危险或有害行为。安全强化学习方法通过强制执行安全约束来满足这一需求。然而，约束下学习常常降低学习速度，可能导致任务表现不优，因为代理必须解决比无约束环境更复杂的受限优化问题。为解决这一问题，本研究提出了ATACOM框架的扩展，该框架为一种最先进的可靠安全层，可与现有强化学习算法集成，以强制执行基于系统先验知识或直接从数据中学习的约束。我们提出的方法，名为ATACOM方向约束（ATACOM-DC），通过引入方向约束，区分接近和远离约束边界的动作，并在必要时激活约束强制，显著改善了安全与性能的权衡。我们在模拟中评估该方法涵盖一系列具有挑战性的机器人控制任务，分析约束违背成本和任务性能。代码及附加内容请见此HTTPS网址。

UniVR: Thinking in Visual Space for Unified Visual Reasoning

UniVR：视觉空间中的思维以实现统一视觉推理

• Authors: Zhongwei Ren, Yunchao Wei, Yao Zhao, Weibo Gong, Xiao Liu, Anran Wang, Xiangtai Li, Xiaojie Jin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.12800
• Pdf link: https://arxiv.org/pdf/2607.12800
• Abstract Learning broad world knowledge directly from raw visual data is a fundamental capability of intelligence. We introduce UniVR, the first investigation into simultaneously learning complex reasoning, fine-grained physical dynamics, and long-term planning from pure visual demonstrations. At its core, UniVR features VR-GRPO, a reinforcement learning paradigm with complementary global and step-level rewards. This approach enforces logical coherence and physical consistency throughout the reasoning process without requiring task-specific heuristics or image-text pairs. To train and evaluate UniVR, we construct VR-X, a large-scale benchmark curated from 16 diverse sources spanning long-horizon manipulation, spatial puzzles, and physical reasoning. It is the first comprehensive suite to assess these heterogeneous capabilities under a purely visual protocol. Remarkably, UniVR achieves up to a 25% improvement on VR-X, and its superior visual reasoning also boosts performance on various multimodal understanding benchmarks. These findings underscore the vast potential of reasoning within visual spaces, with all code, data, and models are open-sourced for further research.
• 中文摘要 直接从原始视觉数据学习广泛的世界知识是智能的基本能力。我们介绍了UniVR，这是首次通过纯视觉演示同时学习复杂推理、细粒度物理动力学和长期规划的研究。UniVR的核心是VR-GRPO，这是一种强化学习范式，提供全局和阶级奖励。这种方法在整个推理过程中强化逻辑连贯性和物理一致性，无需任务特定的启发式方法或图像-文本对。为了训练和评估UniVR，我们构建了VR-X，这是一个从16个涵盖长视野操控、空间谜题和物理推理的多样化来源中策划的大型基准测试。这是首个在纯视觉协议下评估这些异构能力的综合套件。值得注意的是，UniVR在VR-X基础上提升了多达25%，其更优越的视觉设计也提升了多模态理解基准测试的性能。这些发现凸显了视觉空间推理的巨大潜力，所有代码、数据和模型均为开源供进一步研究。

AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning

AVSCap：全模态视频字幕的视听协同协调

• Authors: Yanghai Wang, Jiahao Wang, Jiafu Tang, Yuanxing Zhang, Zhe Cao, Hanyan Bian, Zijie Zhang, Weiliang Luo, Zhiyu Pan, Zixuan Dong, Jiaheng Liu, Zhaoxiang Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.12820
• Pdf link: https://arxiv.org/pdf/2607.12820
• Abstract Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence before composing grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline capabilities, while sample-efficient reinforcement learning uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than increasing SFT data. Third, we introduce AVSCapBench, a benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.
• 中文摘要 全模态视频字幕不仅仅是将视觉字幕与音频转录结合：有用的字幕必须描述视觉动作、语音、音乐和音效如何共同演变。现有大型多模态模型常常在这一关系步骤上失败，将音频和视觉流视为松散耦合的观察，依赖自动语音识别，且对非语音声音及其与视觉事件的关联缺乏具体说明。我们介绍AVSCap，一个以显式跨模态事件绑定为核心的视听字幕框架。首先，我们构建了AVSCap-130K，这是一种三模态训练语料库，由解耦后融合的管道生成，锚定视觉和声学证据，然后生成基于基础的全模态字幕。其次，我们训练AVSCap-7B，一款7B字幕器，采用两阶段策略：监督微调建立基线能力，采样高效强化学习利用混合奖励优化声学完整性和视听协同。我们的缩放分析显示，强化学习带来的收益大于增加SFT数据。第三，我们引入AVSCapBench，这是一个基准测试，将字幕分解为视觉、音频和协同事件，并以细致事件回忆进行评估。AVSCapBench 和外部基准测试的实验显示，AVSCap-7B 提升了非语音音频覆盖和跨模态绑定，在评估的开源模型中整体表现最佳。

Verifier-Based Reinforcement Fine-Tuning of Reasoning Models for Thermal Energy Storage Control

基于验证器的强化对热能储存控制推理模型的微调

• Authors: Takumi Shioda, Kohei Terashima, Tatsuo Nagai
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.12856
• Pdf link: https://arxiv.org/pdf/2607.12856
• Abstract Buildings are expected to shift cooling loads in response to grid conditions. Thermal energy storage (TES) enables this shift, but scheduling it well requires planning hours ahead under storage constraints. Model predictive control (MPC) and reinforcement learning are difficult to scale across buildings. This study instead adapts an open-weight reasoning model through reinforcement learning with verifiable rewards (RLVR). We convert exact offline dynamic-programming (DP) action values into dense rewards for every candidate action. Using only 30 training prompts, reinforcement fine-tuning (RFT) trains the model as an upper-level scheduler that outputs hourly heat-pump setpoints from text-based states and forecasts. Evaluation uses a deliberately simple office-building TES benchmark where exact DP is tractable and the optimum is known. RFT reduces the open-weight model's emissions from 70.5 to 61.2 kg-CO2, close to the DP optimum of 60.8 kg-CO2. GPT-5 nearly matches DP and MPC without task-specific training, while GPT-4o, a non-reasoning LLM, produces higher emissions than the no-storage baseline, so inference-time reasoning appears important. Trace analysis shows that RFT mainly stabilizes observable planning patterns (candidate comparison, look-ahead, and feasibility checking) rather than creating a new strategy. Robustness and generalization tests clarify what transfers: the reinforced planning patterns persist under forecast errors and an unseen TES condition and carry over to a battery task, but its different structure limits the gains. DP-based verifiable rewards offer a practical way to adapt open-weight reasoning models to building storage scheduling. These results motivate higher-fidelity tests of whole-building control and scalable verifiers for city-scale energy management.
• 中文摘要 建筑物预计会根据电网条件调整冷却负荷。热能储存（TES）实现了这一转变，但要合理安排这一转变，需要在储能限制下提前规划数小时。模型预测控制（MPC）和强化学习难以跨建筑推广。本研究改为采用开放权重推理模型，采用可验证奖励的强化学习（RLVR）。我们将精确的离线动态规划（DP）动作值转换为每个候选动作的密集奖励。仅使用30个训练提示，强化微调（RFT）将模型训练为上层调度器，从文本状态和预报输出每小时热泵设定点。评估采用一个刻意简化的办公办公TES基准，精确DP可处理且最优方案已知。RFT将开放重模型的排放量从70.5降至61.2千克二氧化碳，接近DP最优值60.8千克二氧化碳。GPT-5几乎能与DP和MPC匹敌，无需特定任务训练，而GPT-4o作为非推理的LLM产生的排放高于无存储基线，因此推理时间推理显得很重要。追踪分析显示，RFT主要稳定可观察的规划模式（候选人比较、前瞻和可行性检查），而非创建新策略。鲁棒性和泛化测试澄清了哪些转移：强化的规划模式在预测误差和未见的TES条件下依然存在，并可延续到电池任务，但其结构不同限制了收益。基于DP的可验证奖励为开放权重推理模型应用于构建存储调度提供了实用方法。这些结果激励了对整栋建筑控制和城市级能源管理的可扩展验证器进行更高保真度测试。

Unveiling Complex Collective Behaviors from Simple Rewards

揭示复杂集体行为与简单奖励的关系

• Authors: Yize Mi, Jianan Li, Liang Li, Shiyu Zhao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.12861
• Pdf link: https://arxiv.org/pdf/2607.12861
• Abstract Multi-agent Reinforcement Learning (MARL) holds great potential for robot swarms, but the black-box nature of neural policies complicates strategic analysis, limiting multi-robot applications. Furthermore, complex swarm behaviors can surprisingly emerge from simple rewards without explicit aggregation incentives. Unveiling the mechanisms behind this emergence is critical, but the disconnection between simple rewards and collective behaviors exacerbates interpretability challenges. This paper aims to reveal the hidden mechanisms in this process. We propose a two-stage EEC (\LinkIII) explanatory framework. This includes a novel analytical tool called the Agent Response Map (ARM), which reveals agents' decision-making patterns across space and identifies regions of aggregation and avoidance. ARM reveals that the robots implicitly learn the geometric fields of the environment and utilize these structures as desired targets for coordinated movement. We validate this finding across two distinct tasks: a cooperative multi-robot shape assembly and a competitive predator-prey pursuit-evasion. 1) In the cooperative task, ARM identifies the unoccupied target interior as the desired destination for robot navigation. As the center becomes occupied, this target region automatically shifts toward the boundary, demonstrating the robots' capacity to autonomously explore unoccupied areas. 2) In the competitive task, ARM surprisingly identifies the boundary of the predators' Voronoi diagram as the convergence destination for prey agents. Together, these two tasks demonstrate the capability of ARM to discover the hidden geometric structures underlying MARL policies in robot swarms.
• 中文摘要 多智能体强化学习（MARL）在机器人群体中具有巨大潜力，但神经策略的黑箱特性使战略分析复杂，限制了多机器人的应用。此外，复杂的群体行为可以意外地从简单的奖励中产生，而无需明确的聚合激励。揭示这一现象背后的机制至关重要，但简单奖励与集体行为之间的脱节加剧了可解释性的挑战。本文旨在揭示这一过程中的隐藏机制。我们提出了一个两阶段的欧洲经济共同体（\LinkIII）解释框架。其中包括一种名为代理反应图（ARM）的新型分析工具，它揭示了代理在空间中的决策模式，并识别聚集和回避的区域。ARM显示，机器人隐式学习环境的几何场，并将这些结构作为协调运动的目标。我们在两个不同任务中验证了这一发现：协作多机器人形状组装和竞争性捕食者-猎物追逐-规避。1）在协同任务中，ARM将无人目标内部定位为机器人导航的目标目的地。当中心被占领时，该目标区域会自动向边界移动，展示了机器人自主探索无人区域的能力。2）在竞争性任务中，ARM令人惊讶地将捕食者的沃罗诺伊图边界定为猎物主体的汇聚目的地。这两项任务共同展示了ARM发现机器人群体中MARL政策背后隐藏几何结构的能力。

Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes

参数化动作马尔可夫决策过程的知识与梯度引导强化学习

• Authors: Jonas Ehrhardt, René Heesch, Oliver Niggemann
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.12924
• Pdf link: https://arxiv.org/pdf/2607.12924
• Abstract In this paper, we study Reinforcement Learning in Parametrized Action Markov Decision Processes (PAMDP), where each decision consists of a symbolic action and numerical parameters. In such settings Reinforcement Learning algorithms typically determine parameters with one-shot estimators, which makes their training sample inefficient. Though in most PAMDP environments explicit but incomplete knowledge (e.g., rules, safety constraints, or expert heuristics) is available, it is rarely directly used to increase the sample-efficiency of training Reinforcement Learning agents. We step into this gap and propose our novel Neuro-Symbolic Knowledge- and Gradient-Guided Reinforcement Learning (KGRL) algorithm. KGRL uses domain knowledge in a Datalog knowledge base to derive the set of applicable actions and feasible parameters for a given state. This allows it to prune non-applicable actions from the decision-space and constrain the parameter spaces of the remaining actions. We then use a gradient-based parameter refinement loop to estimate the optimal parameters during training and deployment of the agent. By recording activated rules along the trajectory, KGRL additionally provides local procedural explanations on the pruning of actions and constraining of parameters. Overall, KGRL guides the agent's exploration and deployment toward feasible and constraint-aware decisions, while increasing sample efficiency during training. KGRL outperforms state-of-the-art RL baselines for PAMDPs in both, sample efficiency and episodic return.
• 中文摘要 本文研究参数化动作马尔可夫决策过程（PAMDP）中的强化学习，其中每个决策由符号动作和数值参数组成。在这种情况下，强化学习算法通常用一次性估计器确定参数，这使得其训练样本效率较低。虽然在大多数 PAMDP 环境中，显式但不完整的知识（如规则、安全约束或专家启发式）可用，但很少直接用于提高训练强化学习代理的样本效率。我们填补这一空白，提出了我们创新的神经符号知识与梯度引导强化学习（KGRL）算法。KGRL利用Datalog知识库中的领域知识，推导给定状态适用的动作集合和可行参数。这使得它可以从决策空间中修剪不适用的动作，并限制剩余动作的参数空间。然后，我们使用基于梯度的参数优化循环，在智能体的训练和部署过程中估计最优参数。通过沿轨迹记录激活的规则，KGRL 还提供了关于动作修剪和参数约束的局部程序解释。总体而言，KGRL指导代理的探索和部署，朝着可行且约束意识的决策方向发展，同时提升训练中的样本效率。KGRL在样本效率和周期性回馈方面均优于最先进的强化学习基础PAMDPs。

ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning

ExToken：结构化探索，实现高效的视觉-语言-行动强化微调

• Authors: Yilun Kong, Yunpeng Qing, Guozheng Ma, Haoyu Wang, Li Shen, Zhi Hou, Dacheng Tao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.12931
• Pdf link: https://arxiv.org/pdf/2607.12931
• Abstract Reinforcement Learning (RL) has demonstrated significant potential for improving Vision-Language-Action (VLA) models on complex manipulation tasks. However, its practical scalability remains severely limited by the substantial cost of environmental interactions. In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks and reveal that trajectory diversity is fundamentally more important to sample efficiency than the sheer quantity of collected rollouts. Motivated by these insights, we introduce RL Exploration Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors derived from offline demonstrations for structured exploration. By conditioning the policy on different tokens during rollout collection, ExToken encourages the agent to explore diverse behavioral modes, substantially improving state-action coverage and exploration efficiency. To bridge exploration during training with deterministic inference at deployment, ExToken further incorporates a state-conditioned token selector that adaptively predicts effective behavioral modes for unseen scenarios. Extensive experiments across simulated and real-world robotic manipulation tasks demonstrate that ExToken consistently accelerates convergence, improves task performance, and exhibits strong robustness under highly constrained interaction budgets.
• 中文摘要 强化学习（RL）已展现出在复杂操作任务中改进视觉-语言-行动（VLA）模型的巨大潜力。然而，其实际可扩展性仍受环境相互作用带来的巨大代价严重限制。本研究首先探讨当前VLA-RL框架中的探索停滞瓶颈，并揭示轨迹多样性对样本效率的根本性比收集的推广数量更为重要。基于这些洞见，我们引入了强化学习探索代币（ExToken），这是一个简单而通用的框架，基于离线演示得出的离散行为先验来设定VLA策略，用于结构化探索。通过在部署收集时对不同代币进行策略条件，ExToken鼓励代理探索多样化的行为模式，显著提升状态动作覆盖率和探索效率。为了在训练中探索与部署时的确定性推断相结合，ExToken进一步集成了一个状态条件令牌选择器，能够自适应地预测未见场景下的有效行为模式。在模拟和现实机器人操作任务中的广泛实验表明，ExToken能够持续加速收敛，提升任务性能，并在高度受限的交互预算下展现出强大的鲁棒性。

RecRec: Latent Interests Recursive Reasoning for Sequential Recommendation

RecRec：潜在兴趣递归推理，用于顺序推荐

• Authors: Wenhao Deng, Junchen Fu, Hanwen Du, Alexandros Karatzoglou, Ioannis Arapakis, Hangjun Guo, Kaiwen Zheng, Yongxin Ni, Joemon M. Jose
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.12945
• Pdf link: https://arxiv.org/pdf/2607.12945
• Abstract Sequential recommender systems rely on a single forward pass to encode user interaction histories and predict the next item. Increasing inference-time computation through latent reasoning, with the model proceeding step by step before the final prediction, has been recently explored in sequential recommendation with promising results. However, how to structure the reasoning process for sequential recommendation remains an open question. Existing approaches couple reasoning and prediction in a single $d$-dimensional state, limiting reasoning depth and often relying on multi-stage pipelines with reinforcement learning (RL). We propose RecRec (Recursive Reasoning for Recommendation), an RL-free framework that decouples reasoning from prediction, overcoming the fixed $d$-dimensional state bottleneck of prior methods. RecRec consists of a Context Compressor and a Recursive Reasoner, trained in two simple supervised stages. The Context Compressor distills the backbone's hidden states into a small set of latent interests, with an Interest Diversity Regularizer encouraging each interest to capture a distinct aspect of user behavior. The Recursive Reasoner then refines these interests by reasoning in a separate intermediate latent space. Deep supervision lets the reasoning depth be freely adjusted at inference without retraining. On four real-world datasets, RecRec outperforms state-of-the-art reasoning-enhanced methods, and on three of four datasets, gains extend past the training-time depth. Our findings point to a decoupled, multi-vector recipe that unleashes latent reasoning from the single-state bottleneck of prior methods, suggesting reasoning-state structure as a design axis to explore further in sequential recommendation.
• 中文摘要 顺序推荐系统依赖单次前向传递来编码用户交互历史并预测下一条内容。通过潜在推理提升推理时间计算，模型在最终预测前逐步推进，最近已被序列推荐研究，取得了令人鼓舞的成果。然而，如何构建顺序推荐的推理过程仍是一个悬而未决的问题。现有方法将推理和预测结合在单一$d$维状态中，限制了推理深度，且常依赖多阶段流水线配合强化学习（RL）。我们提出了RecRec（递归推理推荐）框架，这是一种无强化学习的框架，能够将推理与预测解耦，克服以往方法中固定$d$维状态瓶颈。RecRec 由上下文压缩器和递归推理器组成，训练过程分为两个简单的监督阶段。上下文压缩器将骨干的隐藏状态提炼成一组潜在兴趣，兴趣多样性正则器鼓励每个兴趣捕捉用户行为的独特方面。递归推理者随后通过在一个独立的中间潜在空间中推理这些兴趣来细化这些兴趣。深度监督允许推理深度在推理时自由调整，无需重新训练。在四个真实数据集中，RecRec 的表现优于最先进的推理增强方法，且在四个数据集中有三个提升超出训练时间深度。我们的发现指向一种解耦、多向量的配方，能够从以往方法的单一状态瓶颈中释放潜在推理，建议以推理状态结构作为后续顺序推荐的设计轴。

TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale

TerraZero：大规模零演示自玩的程序化驾驶模拟

• Authors: Zhouchonghao Wu, Akshay Rangesh, Weixin Li, Wei-Jer Chang, Zachary Lee, Tim Wang, Wei Zhan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.13028
• Pdf link: https://arxiv.org/pdf/2607.13028
• Abstract Training robust autonomous driving agents requires a simulator that is fast enough for reinforcement learning at scale, realistic enough to ground behavior in real-world map structure, and diverse enough to cover the safety-critical long tail that logged data rarely contains. We present TerraZero, a procedural driving simulator and self-play training stack. A configurable C engine runs simulation on the CPU and policy inference on the GPU over a zero-copy path, sustaining 1.3M agent-steps per second on a single server-grade GPU, far faster than existing object-level simulators, while keeping fidelity lighter single-agent systems omit: heterogeneous agents, multiple dynamics models, and full traffic-rule enforcement. TerraZero treats logged data only as a source of real-world map geometry, populating each map with randomized rule-based road users and signal controllers and randomizing agent dynamics, rewards, and sizes per episode, so a map yields an unbounded set of scenarios. Every reported policy trains from scratch by reinforcement learning alone on a compute-efficient self-play recipe across GPUs, with zero human demonstrations and no fallback planner at inference. Policies generalize zero-shot across cities and datasets, including emergent left-hand-traffic driving without explicit supervision. As an ego policy, TerraZero is the first fully learned policy to top the InterPlan long-tail benchmark, ahead of larger learned planners; on routine-driving val14 it ranks among the best approaches and is the safest, posting the best collision and time-to-collision scores. On Waymo Open Sim Agents realism the same recipe outperforms other demonstration-free methods and is competitive with the strongest reference-anchored self-play method. One stack serves both roles: driving policies across dynamics for cars and trucks, and sim agents that jointly control vehicles, pedestrians, and cyclists.
• 中文摘要 训练稳健的自动驾驶智能体需要一个足够快速以大规模强化学习、足够真实以基于真实地图结构的行为，以及足够多样化以覆盖记录数据中很少包含的安全关键长尾的模拟器。我们介绍TerraZero，一款程序化驾驶模拟器和自玩训练堆栈。可配置的C引擎在CPU上运行仿真，在GPU上进行策略推断，采用零复制路径，在单一服务器级GPU上维持每秒130万代理步数，远快于现有对象级模拟器，同时保持了轻量级单代理系统不包括异构代理、多动态模型和完整流量规则强制执行。TerraZero仅将记录数据视为真实地图几何的来源，每张地图中填充随机规则的道路用户和信号控制器，并随机化每集代理的动态、奖励和规模，因此地图提供了无界的场景集合。每一个报告的策略都是从零开始训练，仅靠高效的计算效率自游戏配方，跨GPU进行，完全没有人工演示，推理时也没有备用规划器。政策将零机会在各城市和数据集中推广，包括无明确监管的紧急左侧车道驾驶。作为一种自我意识政策，TerraZero是首个完全成熟的政策，领先于更成熟的规划者，超过了InterPlan长尾基准;在常规驾驶的Val14上，它被评为最佳进场方式之一，也是最安全的，碰撞和碰撞时间评分都最好。在Waymo的Open Sim Agents上，这种真实性方法优于其他无演示方法，并且能与最强的参考锚定自玩方法竞争。一个堆栈兼具两种角色：跨动态控制汽车和卡车的驾驶政策，以及共同控制车辆、行人和骑行者的模拟代理。

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

DenseReward：通过失败合成实现机器人操作的密集奖励学习

• Authors: Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang, Mingxiao Huo, Yao Mu, Huaxiu Yao, Li Erran Li, Daniel Szafir, Mingyu Ding
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.13033
• Pdf link: https://arxiv.org/pdf/2607.13033
• Abstract Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Collecting failure trajectories typically requires laborious human effort, while pseudo-failures constructed by relabeling successful demonstrations fail to capture the diverse physical failure modes that arise during robot execution. Meanwhile, existing reward models often predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning. We release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning.
• 中文摘要 强化学习在超越模仿学习限制的过程中，有望改进机器人政策。然而，其实际应用仍受限于缺乏可靠的视觉语言奖励模型，这些模型能提供密集且信息丰富的反馈。两个关键挑战依然存在：大规模获取多样化的失败数据，以及获得超越稀疏轨迹级成功标签的细致奖励信号。收集失败轨迹通常需要大量人力劳动，而通过重新标记成功演示构建的伪失败，无法捕捉机器人执行过程中出现的多样物理失效模式。与此同时，现有的奖励模型常常预测稀疏的二元或轨迹级奖励，这对高效策略优化提供了有限的指导。我们介绍DenseReward，一种密集机器人奖励模型，解决了这两个挑战。为了训练DenseReward，我们开发了一个自动化故障数据生成流程，在无需人工标记的情况下，在仿真中综合物理真实的失效轨迹，涵盖碰撞、抓取失误、物体掉落和恢复行为等多种失效模式。DenseReward通过视觉观察和语言指令预测密集的帧级奖励分数，从而实现对整集任务进展的细致估计。实验表明，DenseReward在模拟和现实操作中，在密集奖励预测方面优于通用VLM和现有机器人奖励模型。我们还进一步证明，DenseReward为下游模型的预测控制和强化学习提供了有效的奖励指导。我们发布数据集、训练好的奖励模型和评估套件，以支持机器人学习中失效感知的密集奖励建模的开发。

Keyword: diffusion policy

MAMMOTH: A Multi-Modal End-to-End Policy for Off-Road Mobility Robust to Missing Modality

猛犸象：一项多模式端到端的越野出行政策，强有力于缺失模式

• Authors: Ahaan Kotian, Shivani Subramanyan, Suresh Sundaram
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.12965
• Pdf link: https://arxiv.org/pdf/2607.12965
• Abstract Reliable autonomous navigation in unstructured off-road environments remains a critical unsolved challenge due to extreme terrain diversity, drastic illumination variations and acute sensor degradation. Recent developments have approached the problem as a traversability costmap estimation or visual navigation task. However, many exhibit heavy reliance on RGB modality, leading to poor performance in varied illumination such as glares, shadows or low ambient light. Achieving robust generalization in such conditions requires integrating modalities that provide supplementary scene information. Such multi-modal methods suffer from a rigid dependency on the presence of near-perfect sensor inputs, leaving them unable to robustly handle sensor degradation or individual modality failure. To address these limitations, we introduce MAMMOTH (MAsking Multi-Modal inputs for Off-road Traversability Heuristic-informed navigation), a unified end-to-end navigation policy for robust off-road visual-goal-conditioned navigation and undirected exploration. Specifically, MAMMOTH efficiently fuses multi-modal observations (RGB, Thermal, 3D Pointcloud and Ego Velocity) and is trained with a modality dropout scheme, enabling it to generalize to missing modalities at inference time. Furthermore, we employ a diffusion policy to learn the joint conditional probability distribution of physically-grounded trajectories and a intrinsic traversability heuristic. MAMMOTH utilizes this heuristic to prefer safer, smoother trajectories. We validate MAMMOTH through extensive real-world robot experiments in distinct off-road environments, including night-time operation. Our results demonstrate superior performance, with significant improvements in collision avoidance, terrain-aware planning and generalization to missing modalities. The code and dataset used for this work will be made publicly available.
• 中文摘要 由于极端地形多样性、剧烈的照明变化和传感器严重劣化，在非结构化的越野环境中实现可靠的自主导航仍是一个关键且未解决的问题。近期发展将该问题视为可穿越性成本图估算或目视导航任务。然而，许多设备高度依赖RGB模态，导致在眩光、阴影或低环境光等多样照明下表现较差。在这种条件下实现稳健的泛化需要整合提供辅助场景信息的模态。此类多模态方法对近乎完美的传感器输入存在严格依赖，导致无法稳健地处理传感器劣化或单个模态失效。为解决这些限制，我们引入了MAMMOTH（MAsking多模态输入用于越野通行启发式导航），这是一套统一的端到端导航策略，支持稳健的越野视觉目标条件导航和无方向探索。具体来说，MAMMOTH高效融合多模态观测（RGB、热成像、三维点云和自我速度），并采用模态丢弃方案训练，使其能够在推断时推广到缺失模态。此外，我们采用扩散策略来学习物理基础轨迹的联合条件概率分布，并采用内在可遍历启发式。猛犸利用这一启发式方法，偏好更安全、更平稳的轨迹。我们通过在不同越野环境中的广泛现实机器人实验验证了MAMMOTH的安全性，包括夜间运行。我们的结果显示了更优越的性能，在碰撞避免、地形感知规划和对缺失模态的泛化方面取得了显著提升。用于本工作的代码和数据集将公开。