生成时间: 2026-07-23 18:25:11 (UTC+8); Arxiv 发布时间: 2026-07-23 20:00 EDT (2026-07-24 08:00 UTC+8)

今天共有 25 篇相关文章

Keyword: reinforcement learning

From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation

从轨迹到前缀：通过重放前缀和在线续写重用教师轨迹

• Authors: Yihan Wang, Zhong Guan, Haoran Sun, Jiale Huang, Likang Wu, Hongke Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.19395
• Pdf link: https://arxiv.org/pdf/2607.19395
• Abstract Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets. This is inefficient in long-horizon environments, where early decisions shape later states and rewards. We propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization form. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines, while ablations show that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at this https URL.
• 中文摘要 小型语言模型是交互代理的有吸引力的骨干，但从强教师轨迹中直接提炼，往往会将丰富的多回合行为变成一次性的模仿目标。这在长期视野环境中效率低下，因为早期决策会影响后续状态和奖励。我们提出了Prefix-GRPO，一种强化学习框架，将教师轨迹分解为与重放对齐的前缀查询和在线延续。每个前缀在环境中会被重放以恢复有效的中间状态，之后学生继续在线互动并获得任务奖励。与仅响应的 GRPO 不同，Prefix-GRPO 还会对重放前缀内的历史助手标记应用截断的策略更新，利用策略提炼的 SFT 检查点估算其旧的日志概率。这统一了前缀学习和持续学习在同一策略优化形式中。在TextCraft、BabyAI和ALFWorld上的实验显示，Prefix-GRPO能在小模型代理上优于蒸馏和标准强化学习基线，而消融则显示，单靠重放若无明确前缀-令牌优化是不够的。实现和重现脚本可在该 https URL 访问。

HyGRL: Adaptive Hybrid Graph Reasoning for Multi-Entity Questions

HyGRL：多实体问题的自适应混合图推理

• Authors: Junyi Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.19398
• Pdf link: https://arxiv.org/pdf/2607.19398
• Abstract Multi-entity compositional questions pose significant challenges to existing retrieval-augmented language models. Conventional methods fall into a dilemma: standard RAG lacks dynamic reasoning, traditional Graph-RAG is limited by structural sparsity, and LLM-constructed Graph-RAG incurs prohibitive costs. We propose \textbf{\fwa}, a unified framework that embeds unstructured text into structured knowledge graphs, creating a heterogeneous network for flexible evidence retrieval. Reasoning is formulated as adaptive structure induction, learned via a robust two-stage process: (1) imitation learning distills heuristic expert signals, and (2) reinforcement learning refines the policy using LLM-driven preference rewards. Experiments demonstrate that {\fwa} effectively merges textual richness with structural knowledge, outperforming SOTA baselines in answer accuracy and reasoning fidelity while maintaining extremely low token costs and near real-time inference((code available at this https URL) .
• 中文摘要 多实体组合问题对现有的检索增强语言模型构成了重大挑战。传统方法陷入两难：标准RAG缺乏动态推理，传统图-RAG受限于结构稀疏性，而LLM构建的图-RAG则承担着高昂的成本。我们提出了 \textbf{\fwa}，一个统一框架，将非结构化文本嵌入结构化知识图谱，创建一个异构网络，实现灵活的证据检索。推理被表述为自适应结构归纳，通过一个稳健的两阶段过程学习：（1）模仿学习提炼启发式专家信号，（2）强化学习利用LLM驱动的偏好奖励来优化策略。实验表明，{\fwa} 有效将文本丰富性与结构性知识结合，在答案准确性和推理准确度上优于SOTA基线，同时保持极低的令牌成本和近实时推断（代码可在此 https URL 获取）。

Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models

利用离线监督实现大规模视觉-语言-行动模型中高效且可推广的强化学习

• Authors: Dmitriy Poyarkov, Aleksei Staroverov, Aleksandr I. Panov
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.19399
• Pdf link: https://arxiv.org/pdf/2607.19399
• Abstract It is commonly observed that online reinforcement learning (RL) produces better-performing strategies than offline methods across a broad range of performance measures. In particular, RL-trained policies exhibit stronger out-of-distribution (OOD) behavior, where models trained only with imitation learning approaches often struggle. A recent study introduced an OOD-focused benchmark and reported that RL-trained vision-language-action (VLA) policies achieve noticeably better OOD performance and slightly better in-distribution (IND) performance than their counterparts trained with supervised fine-tuning (SFT). In this work, we investigate whether hybrid offline-online training can combine the advantages of both approaches. Specifically, we study RL methods regularized by offline supervision via either offline data or an offline-trained reference policy. We evaluate these approaches on the OOD benchmark and compare them with both offline-only training and standard RL. Our results show that although offline training achieves limited OOD performance by itself, incorporating offline supervision into RL preserves strong OOD capability while substantially improving training efficiency. In particular, the guided methods reach performance close to that of standard RL while requiring roughly half of the training budget. Rather than producing a trade-off between speed and OOD performance, the hybrid approach retains strong OOD capability while achieving this efficiency gain. Project page: this https URL
• 中文摘要 普遍观察到，在线强化学习（RL）在广泛的性能指标上比离线方法产生更优的策略。特别是，强化学习训练的策略表现出更强的非分布（OOD）行为，而仅用模仿学习方法训练的模型往往难以应对。一项最新研究引入了以OOD为重点的基准，报告称RL训练的视觉-语言-行动（VLA）策略在OOD表现和分布内（IND）表现上明显优于用监督微调（SFT）训练的策略。本研究探讨了线下混合培训是否能结合两种方法的优势。具体来说，我们研究通过离线数据或离线训练的参考策略进行正则化的强化学习方法。我们在OOD基准上评估这些方法，并与仅线下训练和标准强化学习进行比较。我们的结果表明，尽管离线训练本身在外工作表现有限，但将离线监督纳入强化学习能保持强大的外在角色能力，同时显著提升训练效率。特别是，指导方法的性能接近标准强化学习，但其培训预算约占一半。混合方法避免在速度与离场性能之间做出权衡，而是在实现效率提升的同时，保持了强大的离场能力。项目页面：此 https URL

REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning

REGEN：基于离线强化学习的专家到通才提炼的重放循环

• Authors: Yunjie Chen, Xiaoxin Chen, Fang Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.19450
• Pdf link: https://arxiv.org/pdf/2607.19450
• Abstract Large-scale online reinforcement learning (RL) is the predominant means of eliciting advanced abilities including long-term reasoning and agentic tool use in large language models (LLMs). However, continuing to scale it across vast task domains of interest remains challenging in both computational infrastructure and cost, especially when considering RL as merely a one-off learning stage. Recently, a widely used technique for distilling knowledge across various domains and training stages, multi-teacher on-policy distillation (MOPD), helps to decouple the RL stage, saving costs, while maintaining generality across vast domains. Nonetheless, similar to online RL, MOPD requires coupled inference and backward passes, which continues to limit its scalability and computational efficiency. To address these challenges, we propose REGEN: Replay-recycling for Expert-to-Generalist Distillation with Offline RL. Instead of distilling from multiple teacher models, REGEN trains a generalist by simply recycling the replay memory -- the free by-product of the teachers' specialized RL training -- and employing offline RL algorithms. REGEN completely decouples the rollout sampling from the backward training process and thus greatly reduces the training cost. Across mathematical reasoning, code generation, and instruction following, REGEN matches the accuracy of MOPD at substantially lower cost. It potentially turns online RL into a data synthesis process instead of a one-off learning stage, and can potentially be extended to large-scale post-training without requiring heavy computational load.
• 中文摘要 大规模在线强化学习（RL）是激发高级能力的主要手段，包括长期推理和智能工具的使用，适用于大型语言模型（LLMs）。然而，继续将其扩展到广泛感兴趣的任务领域，在计算基础设施和成本上仍具挑战性，尤其将强化学习视为一次性学习阶段。最近，一种广泛使用的技术——多教师策略提炼（MOPD）——用于在不同领域和培训阶段提炼知识，有助于脱钩强化学习阶段，节省成本，同时保持跨广泛领域的通用性。然而，与在线强化学习类似，MOPD需要耦合推断和后向传递，这仍然限制了其可扩展性和计算效率。为应对这些挑战，我们提出了REGEN：基于离线强化学习的专家到通用蒸馏的重放循环。REGEN不是从多个教师模型中提炼，而是通过简单地回收教师专业强化学习的免费副产品回放记忆，并采用离线强化学习算法来培训通才。REGEN完全将推广抽样与后向训练过程解耦，从而大幅降低了训练成本。在数学推理、代码生成和指令跟踪等方面，REGEN以显著更低的成本与MOPD相当。它有可能将在线强化学习转变为数据综合过程，而非一次性学习阶段，并且可以扩展到大规模的后期训练，而无需大量计算负载。

CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning

CHMAS：多智能体强化学习的耦合层级框架

• Authors: Dongming Wang, Jie Xu, Yanyu Zhang, Wei Ren
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.19555
• Pdf link: https://arxiv.org/pdf/2607.19555
• Abstract Multi-agent reinforcement learning (MARL) systems face fundamental challenges in balancing global coordination with local execution across different temporal scales. This paper introduces the Coupled Hierarchical Multi-Agent System (CHMAS), a novel framework that decomposes multi-agent decision-making into centralized strategic planning and distributed tactical execution with bidirectional information flow. The strategic layer integrates all agents' states with an exclusive global environmental state to generate guidance actions every $T$ timesteps, while tactical agents execute distributed policies augmented by strategic guidance and local neighborhood observations. Unlike existing hierarchical approaches with unidirectional control, CHMAS establishes a feedback mechanism where accumulated tactical rewards influence strategic objectives through a coupling coefficient $\lambda$, ensuring strategic plans remain grounded in tactical feasibility. To address the non-stationarity inherent in hierarchical learning, we propose an asynchronous update protocol where strategic parameters update every $N_f$ tactical episodes, allowing tactical policies to converge to quasi-stationary points between strategic changes. We present both a general bi-level formulation capturing full system dynamics and a tractable additive approximation enabling rigorous analysis. Theoretical analysis proves that this asynchronous scheme achieves $\mathcal{O}(\log K/\sqrt{K})$ convergence for the strategic layer after $K$ strategic updates under standard assumptions. Experimental validation in a multi-agent foraging domain demonstrates successful learning of spatially partitioned exploration strategies, with both layers converging stably despite hierarchical coupling.
• 中文摘要 多智能体强化学习（MARL）系统面临着在不同时间尺度上平衡全局协调与局部执行的根本挑战。本文介绍了耦合层级多智能体系统（CHMAS），这是一个新颖框架，将多智能体决策分解为集中战略规划和分布式战术执行，并实现双向信息流。战略层将所有代理的状态与一个排他性的全局环境状态集成，每 $T 美元时间步生成指导动作，而战术代理执行分布式策略，辅以战略指导和局部邻域观测。与现有单向控制的层级方法不同，CHMAS建立了一种反馈机制，通过耦合系数$\lambda$通过累积战术奖励影响战略目标，确保战略计划始终基于战术可行性。为了解决分层学习固有的非平稳性，我们提出了一种异步更新协议，其中战略参数每 $N_f$ 战术事件更新一次，使战术策略在战略变更之间收敛到准平稳点。我们既提出了一个涵盖全系统动力学的通用二能级表述，也提供了一个可处理的加法近似，便于严谨分析。理论分析证明，在标准假设下，该异步方案在战略层实现了$\mathcal{O}（\log K/\sqrt{K}）$收敛，经过$K$的战略更新。在多智能体采集领域的实验验证显示，空间划分的探索策略能够成功学习，尽管存在层级耦合，两层依然稳定收敛。

CRB-Driven Beamforming and Trajectory Optimization for UAV-assisted ISAC System

CRB驱动的波束成形与无人机辅助ISAC系统的轨迹优化

• Authors: Yi Yang, Qianqian Zhang, Huaxia Wang
• Subjects: Subjects: Information Theory (cs.IT); Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.19609
• Pdf link: https://arxiv.org/pdf/2607.19609
• Abstract In this paper, we study an unmanned aerial vehicle (UAV)-assisted integrated sensing and communication (ISAC) system, where a UAV enhances the sensing capability of a base station (BS) towards a target while ensuring reliable communication towards a downlink user. This architecture is practically attractive for future wireless networks due to the UAV's controllable mobility and adaptive sensing coverage in wireless environments. The sensing performance is characterized by the average Cramér-Rao bound (CRB), which quantifies the minimum variance of the unbiased angle-of-arrival estimation. To enhance the sensing performance, the UAV trajectory and beamforming parameters are jointly optimized under power and mobility constraints, while satisfying communication requirements to the downlink user. To address the resulting non-convex problem, we employ null-space projection for beamforming design and adopt deep reinforcement learning for the trajectory optimization over a discrete-time scale. In each time slot, beamforming is optimized based on the channel state information to improve CRB performance while mitigating interference between the BS and the communication user. Simulation results demonstrate that the proposed method significantly reduces the time-averaged CRB by over 10%, compared with the ISAC system without UAV assistance, and also achieves a higher sensing accuracy than both the fixed-UAV-trajectory and the maximum-ratio-transmission-based beamforming benchmarks.
• 中文摘要 本文研究了一种无人机（UAV）辅助的综合感测与通信（ISAC）系统，其中无人机增强基站（BS）对目标的感知能力，同时确保与下行用户的可靠通信。由于无人机在无线环境中具备可控的移动性和自适应感测覆盖，这种架构对未来的无线网络具有实际吸引力。传感性能由平均克拉梅尔-拉奥界限（CRB）表征，该界限量化了无偏到达角估计的最小方差。为提升传感性能，无人机轨迹和波束形成参数在功耗和机动性限制下联合优化，同时满足下行用户的通信需求。为解决由此产生的非凸问题，我们采用零空间投影进行波束成形设计，并采用深度强化学习以在离散时间尺度上优化轨迹。在每个时隙中，波束成形基于信道状态信息进行优化，以提升CRB性能，同时减少BS与通信用户之间的干扰。模拟结果表明，与无无人机辅助的ISAC系统相比，该方法显著降低了时间平均CRB超过10%，并且在传感精度上优于固定无人机轨迹和基于最大比辐射比的波束形成基准。

The Mechanism Matters: When Knowledge Graphs Help Reinforcement Learning

机制很重要：知识图谱何时有助于强化学习

• Authors: Mohammed Sameer Syed
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.19616
• Pdf link: https://arxiv.org/pdf/2607.19616
• Abstract Knowledge graphs (KGs) are widely used to inject prior knowledge into reinforcement learning (RL), yet the literature is dominated by single-domain, positive-result method papers, so we lack a systematic account of when KG structure helps an agent, when it is neutral, and when it hurts. We conduct a controlled study that independently varies the RL task, the injection mechanism (state features, action masking, or potential-based reward shaping), and KG quality. Using a synthetic, fully controllable KG over MiniGrid environments, we report three findings. First, on compositional sparse-reward tasks structured KG guidance improves sample efficiency and solve reliability (70% to 97% of seeds), and a shuffle control that permutes the KG's edges while preserving their count collapses the benefit toward baseline (masking p=0.0001; shaping p=0.006), so the gain is structural rather than generic regularization. Second, KG value scales with the amount of task-relevant knowledge the graph contains. Third, and most consequential, safety depends on the mechanism: soft, optimality-preserving injection benefits from correct knowledge and harmlessly ignores incorrect knowledge, whereas hard masking is brittle, forbidding essential actions when the KG is incomplete or corrupted and making a wrong KG worse than none. A UMLS-derived clinical case study on sepsis management under offline RL is a careful null, underscoring that benefits require task structure the chosen mechanism can exploit. Our results give practitioners concrete guidance on how, and how much, to trust a KG when using it to guide RL.
• 中文摘要 知识图谱（KGs）被广泛用于将先验知识注入强化学习（RL），但文献中主要以单域、积极结果的方法论文为主，因此我们缺乏系统性说明知识图谱何时帮助一个智能体，何时中立，何时有害。我们进行了一项对照研究，独立改变强化学习任务、注射机制（状态特征、动作掩蔽或基于电位的奖励塑造）以及KG质量。我们利用一个合成且完全可控的 MiniGrid 环境下的 KG，报告了三项发现。首先，在组合稀疏奖励任务中，结构化的KG指导提高了样本效率并提高解信度（70%到97%的种子），而在保持计数的同时置换KG边缘的洗牌控制则使收益向基线方向崩溃（掩蔽p=0.0001;塑形p=0.006），因此增益是结构性的而非通用的正则化。其次，KG值随图表中包含的任务相关知识量而增长。第三，也是最重要的，安全性取决于机制：软性、保持最优性的注射受益于正确的知识，且无害地忽略错误知识，而硬掩蔽则脆弱，禁止关键操作，当基因格不完整或损坏时，错误的基因格比没有更糟糕。基于UMLS的一项关于离线强化学习下败血症管理的临床案例研究是一个谨慎的无效研究，强调益处需要所选机制能够利用的任务结构。我们的研究结果为从业者提供了具体指导，说明在使用一枚 KG 来指导强化学习时，该如何以及在多大程度上信任它。

HypEMBER: Hypernetwork-based Ensemble for Robust Policy Learning of Parametrized Dynamical Systems

HypEMBER：基于超网络的集合，用于参数化动力系统的稳健策略学习

• Authors: Nicolò Botteghi, Gabriele Pascali, Urban Fasel, Andrea Manzoni
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.19628
• Pdf link: https://arxiv.org/pdf/2607.19628
• Abstract In this work we investigate reinforcement learning (RL) as a framework for the robust control of parametrized dynamical systems in presence of measurements and model uncertainties. High-dimensional state spaces, expensive numerical solvers, the partial knowledge of the governing equations, and the dependence on physical parameters that may be uncertain or difficult to estimate accurately, make the use of standard RL approaches computationally unfeasible. Indeed, lack of robustness and poor generalization across parameter variations are further amplified in presence of noisy or incomplete measurements, ultimately hampering control performance. To address these challenges, we introduce HypEMBER, a novel RL framework based on the combination of hypernetworks and ensemble learning. In the proposed approach, both the policy and value functions are represented through hypernetworks that generate the weights of the underlying models conditioned on the physical parameters of the system, thereby enabling parametric generalization across different dynamical regimes. In addition, an ensemble of policy and value approximators is employed to quantify epistemic uncertainty, leading to improved exploration strategies and enhanced robustness during and after training. The performance of the proposed framework is assessed on two representative parametrized control problems: (i) the one-dimensional Kuramoto-Sivashinsky equation and (ii) a particle-navigation task in a two-dimensional time-dependent gyre flow, focusing on robustness with respect to measurement noise and parameter misspecification. Numerical results demonstrate that HypEMBER consistently improves training stability and sample efficiency, while achieving superior robustness to uncertainties affecting both the system dynamics and the available observations, in comparison with state-of-the-art RL methods.
• 中文摘要 本研究探讨强化学习（RL）作为参数化动力系统在测量和模型不确定性存在下稳健控制的框架。高维状态空间、昂贵的数值求解器、对控制方程的部分了解，以及对物理参数的依赖，这些参数可能不确定或难以准确估计，使得标准强化学习方法在计算上不可行。事实上，在存在噪声或不完整测量时，缺乏鲁棒性和参数变异间的泛化能力会进一步放大，最终影响控制性能。为应对这些挑战，我们引入了HypEMBER，一种基于超网络与集合学习结合的新型强化学习框架。在所提方法中，策略函数和值函数都通过超网络表示，这些网络根据系统的物理参数生成底层模型的权重，从而实现跨不同动态体系的参数泛化。此外，采用一组策略和价值近似指标来量化认识论不确定性，从而改进探索策略并在培训期间及之后增强鲁棒性。所提出框架的性能评估基于两个具有代表性的参数化控制问题：（i）一维仓本-西瓦申斯基方程，以及（ii）在二维时变环流中的粒子导航任务，重点关注对测量噪声和参数错误的鲁棒性。数值结果表明，HypEMBER在训练稳定性和样本效率方面持续提升，同时在面对影响系统动态和可用观测的不确定性方面实现了更优的鲁棒性，相较于最先进的强化学习方法。

A Human-AI Teaming Framework for Deep Reinforcement Learning-Based Voltage Regulation in Distribution Networks

基于深度增强学习的配电网络电压调节人机团队框架

• Authors: Mahmuda Akter, Hamidreza Nazaripouya
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.19667
• Pdf link: https://arxiv.org/pdf/2607.19667
• Abstract The growing penetration of distributed energy resources (DERs) has increased the operational variability of distribution networks, making voltage regulation increasingly challenging. Conventional deep reinforcement learning (DRL) methods exhibit unsafe exploration behavior, slow convergence, and limited reliability, which restrict their applicability in safety-critical power system settings. This paper presents a human-interactive reinforcement learning framework that enhances the safety and robustness of autonomous voltage regulation. The proposed approach integrates a Soft Actor-Critic (SAC) agent with an adaptive Lagrange constraint mechanism to enforce voltage limits, while a human-guidance module provides sensitivity-based corrections through coordinated capacitor-bank and Battery Energy Storage System (BESS) dispatch. These corrections are incorporated into a human-regularized actor loss, enabling the policy to internalize safe and interpretable control behavior. The framework is implemented in the PowerGym-OpenDSS environment using the IEEE 13-node feeder. Simulation results show that the proposed Human-Interactive Lagrangian SAC (HI-LSAC) achieves significantly lower voltage-violation severity and reduced power losses compared with the other baseline methods.
• 中文摘要 分布式能源资源（DERs）日益普及，增加了配电网络的运营变异性，使得电压调节变得越来越具有挑战性。传统的深度强化学习（DRL）方法表现出不安全的探索行为、收敛缓慢和可靠性有限，限制了其在安全关键电力系统环境中的适用性。本文提出了一种人机交互强化学习框架，提升自主电压调节的安全性和稳健性。所提方法将软演员-批判者（SAC）代理与自适应拉格朗日约束机制相结合以强制电压限制，而人工指导模块则通过协调的电容组和电池储能系统（BESS）调度提供基于灵敏度的修正。这些修正被纳入人为正则化的行为者损失中，使策略能够内化安全且可解释的控制行为。该框架在PowerGym-OpenDSS环境中通过IEEE 13节点馈源实现。模拟结果显示，提出的人机交互拉格朗日SAC（HI-LSAC）相比其他基线方法显著降低了电压破坏严重度和更小的功率损耗。

SLPO: Scaling Latent Reasoning via a Surrogate Policy

SLPO：通过替代政策扩展潜在推理

• Authors: Runyang You, Zhiyuan Liu, Yongqi Li, Wenjie Li
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.19691
• Pdf link: https://arxiv.org/pdf/2607.19691
• Abstract Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@$k$ under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.
• 中文摘要 带有可验证奖励的强化学习已成为显式思维链推理器中测试时间缩放的主要配方。然而，这种扩展路径仍然计算成本较高，因为每个中间步骤都必须以语言令牌形式解码。潜在推理则以连续向量的形式承载中间计算，并且在更短的视野内已经与显式CoT相匹配甚至超越。尽管有这一承诺，潜在推理者仍主要依赖于模仿，而显性CoT已通过结果-奖励强化学习超越了模仿。潜在轨迹缺乏可处理的每步概然和固定思维预算下的自适应停止界面，因此结果奖励无法引发潜在的测试时间缩放。我们引入替代潜在策略优化（SLPO），将结果-奖励强化学习引入自回归潜在推理器：针对轨迹级信用分配的经验替代政策密度，以及一个由结果-奖励优化细化为可变视野政策的正确性监督停止头。在连续思考和软思考环境中，SLPO在并行采样下提高了Pass@$k美元，并将更长的潜在计算分配给更高确定性准确性的更难实例。

The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

《世界模特记，演员遗忘：持续模型强化的梦境彩排》

• Authors: Gurp Nijjer
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.19749
• Pdf link: https://arxiv.org/pdf/2607.19749
• Abstract Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.
• 中文摘要 DreamerV3家族的基于模型的强化学习代理在任务序列训练时，即使无界重放缓冲区保留了所有之前的经验，遗忘率也会灾难性地出现。我们提出一个持续强化学习文献假设答案但从未测量的问题：哪个组成部分忘记了？在从未清晰回放的情况下，预注册的组件级探针（n=3个种子）显示，世界模型基本上保留了旧任务的所有可测量内容——奖励辨别（保留率~1.0）、价值估计和终止结构——而行为者的行为却崩溃了。在这种状态下，遗忘是通道问题，而不是记忆问题。我们通过干预来证明这一点：当世界模型冻结且想象中的展开完全相同时，想象中的强化学习未能恢复失去的技能（0/3种子），而对世界模型自身分级梦境进行监督自我模仿则在3/3种子上恢复该技能，且环境无任何交互。训练期间交错进行，这种分级梦境彩排产生无任务标签、参数常数的持续学习器：保留3/3四任务链，保留3/3条八任务链，且匹配真实剧集克隆持续提升（配对差+0.13，自助95%置信区间[0.07， 0.24]，完全种子分离）。梦境评分步骤是负重的：我们描述两种评分失败模式，提供一个离线选择计量表，在它们污染结果之前捕捉到两者，并给出一个实现的第一个评分规则来封闭它们。所有实验都已预先注册了已承诺的协议;所有被驳斥的假说都被报告。

Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

Trace：一个多领域视觉推理的分类引导环境

• Authors: Md Tanvirul Alam
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.19790
• Pdf link: https://arxiv.org/pdf/2607.19790
• Abstract Reinforcement learning with verifiable rewards (RLVR) has substantially improved language-model reasoning, yet its extension to vision-language models remains constrained by the lack of training data that are simultaneously broad, exactly verifiable, and reproducible. We introduce Trace, a taxonomy-guided environment for multidomain visual reasoning. Trace factorizes task construction into a scene grammar and an executable task program, separating visual realization from answer computation. A shared semantic state determines the rendered image, prompt, typed answer, verifier state, and replayable instance trace. The resulting environment comprises 1,000 tasks over 277 scene grammars and 11 visual domains, with controlled semantic and visual variation. RLVR on 64,000 Trace instances improves the macro-average across 24 external benchmarks by 3.51 percentage points for Qwen2.5-VL-3B and 4.06 points for Qwen2.5-VL-7B, providing evidence that broad procedural training can transfer beyond the generated task distributions. Project page: this https URL.
• 中文摘要 带有可验证奖励的强化学习（RLVR）显著提升了语言模型推理能力，但其在视觉语言模型上的推广仍受限于缺乏既广泛、可精确验证又可重复的训练数据。我们介绍了Trace，一个以分类法为导向的多领域视觉推理环境。Trace将任务构建分解为场景语法和可执行任务程序，将视觉实现与答案计算分离。共享语义状态决定渲染后的图像、提示词、输入的答案、验证者状态和可重放的实例轨迹。最终的环境包含1000个任务，涵盖277个场景语法和11个视觉领域，并实现了受控的语义和视觉变化。在64,000个Trace实例上的RLVR使Qwen2.5-VL-3B在24个外部基准测试中提升了3.51个百分点，Qwen2.5-VL-7B提升了4.06个百分点，证明了广泛的程序化训练可以超越生成的任务分布。项目页面：这个 https URL。

Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

Dreamer-CPC：基于世界模型进行的信息学习，用于去中心化多智能体强化学习

• Authors: Taisuke Takayama, Naoto Yoshida, Tadahiro Taniguchi
• Subjects: Subjects: Multiagent Systems (cs.MA); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.19809
• Pdf link: https://arxiv.org/pdf/2607.19809
• Abstract In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. We propose Dreamer-CPC, a decentralized model-based MARL method that integrates message learning based on Collective Predictive Coding (CPC) into the world model of DreamerV3. Each agent independently maintains a world model and a message module, and infers and exchanges messages from the latent states of the world model that reflect the history of past observations and actions. We evaluated Dreamer-CPC in two environments: Observer, a non-cooperative information-sharing task, and CatchApple, a newly introduced task in which task-relevant observations are temporarily missing. In both environments, Dreamer-CPC outperformed IPPO-CPC, an existing CPC-based method that generates messages from current observations, as well as no-communication baselines. In particular, in CatchApple, Dreamer-CPC achieved 4 to 5 times the episode return of IPPO-CPC, demonstrating effective coordination where other methods fail due to missing observations. These results suggest that communication grounded in the latent dynamics of world models can support decentralized decision-making when current observations alone are insufficient.
• 中文摘要 在多智能体强化学习（MARL）中，智能体间通信在部分可观测性下有效提升性能。基于表征学习的方法使去中心化智能体能够基于自身观察学习信息，但它们仅依赖当前观察，无法传达长期积累的信息。我们提出了Dreamer-CPC，这是一种去中心化的基于模型的MARL方法，将基于集体预测编码（CPC）的信息学习整合进DreamerV3的世界模型中。每个智能体独立维护一个世界模型和一个消息模块，并从世界模型的潜在状态推断和交换反映过去观察和行为历史的信息。我们在两种环境中评估了Dreamer-CPC：观察者（Observer），一个非合作的信息共享任务，以及CatchApple，一个新引入的任务，任务中暂时缺失与任务相关的观测值。在这两种环境下，Dreamer-CPC 都优于基于 CPC 的 IPPO-CPC，后者基于当前观测数据和无通信基线生成消息。特别是在CatchApple中，Dreamer-CPC的回归集数是IPPO-CPC的4到5倍，展示了在其他方法因缺失观测而失败时的有效协调。这些结果表明，基于世界模型潜在动态的沟通可以在现有观测不足时支持去中心化决策。

Rewarding Better Thinking for LLM Preference Alignment

奖励更好的思维以实现LLM偏好对齐

• Authors: Xubo Liu, Wenya Guo, Ruxue Yan, Xinying Qian, Ying Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.19824
• Pdf link: https://arxiv.org/pdf/2607.19824
• Abstract LLM preference alignment aims to optimize models toward human preferences across diverse user instructions. Reinforcement learning has become a major post-training approach for this goal, but existing proxy rewards are often outcome-level, mainly evaluating the final response while providing limited guidance for the reasoning trajectory. This can make credit assignment coarse when multiple responses receive similar final scores, leaving trajectory-level preferences under-specified. To address this limitation, we propose Thinking Checklist Reward (TCR), a process-oriented reward for RL-based preference alignment. TCR converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses the preference-implied considerations. To reduce overlap with outcome-level supervision, TCR further introduces an exponential moving average (EMA) residual formulation to isolate a complementary thinking surplus beyond what is predictable from the outcome reward. Experiments on five models from three model families show that TCR consistently improves alignment performance across diverse benchmarks, with ablations further validating the importance of EMA-based residual formulation and sample-specific checklist supervision.
• 中文摘要 LLM偏好对齐旨在优化模型，以适应多样化用户指令中的人类偏好。强化学习已成为实现这一目标的重要训练后方法，但现有的代理奖励通常是基于结果的，主要评估最终反应，同时对推理轨迹提供有限指导。当多个回答最终得分相似时，学分分配可能变得粗糙，导致路径层级偏好被忽视。为解决这一限制，我们提出了思维清单奖励（TCR），这是一种基于强化学习的偏好对齐过程导向的奖励。TCR将偏好对转换为样本特定的思维检查表，并用它们评估生成的推理痕迹是否涵盖了偏好隐含的考虑。为减少与结果层级监督的重叠，TCR进一步引入指数移动平均（EMA）残差表述，以隔离超出可预测结果奖励范围之外的互补思维剩余。对三组模型的五个模型实验显示，TCR在不同基准测试中持续提升比对性能，消融进一步验证了基于EMA的残余制式和样本特定检查表监督的重要性。

EA-Nav: Learning Safe Visual Navigation Policies with Embodiment Awareness

EA-Nav：学习具身意识的安全视觉导航政策

• Authors: Jialu Zhang, Yong Du, Xianda Guo, Shunwang Sun, Xinqi Liu, Yue Sun, Guodong Lu, Wei Sui, Jituo Li
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.19880
• Pdf link: https://arxiv.org/pdf/2607.19880
• Abstract Cross-embodiment navigation is a key challenge in embodied intelligence. Due to differences in embodiment, the same visual observation may imply different actions for different agents, making prediction ambiguous when relying solely on vision. Existing studies mainly rely on reinforcement learning, which requires large-scale interaction and careful reward design, making it difficult to support scalable pretraining and real-world adaptation. In contrast, imitation-learning-based approaches remain limited. To address these challenges, we propose an imitation-learning-based embodiment-aware navigation framework with a modular multi-stage design. In pretraining, we construct a cross-embodiment navigation dataset from Internet videos and introduce embodiment geometry as conditional tokens to reduce action ambiguity under the same observation. In fine-tuning, we design a multimodal information injection mechanism based on a decoupled architecture. Specifically, we design a trajectory augmentation strategy to generate high-risk samples, which are used to train spatial perception and risk-aware correction separately, thereby explicitly incorporating embodiment geometry for safe navigation. Experimental results show that the proposed method effectively improves navigation performance across different embodiment settings, demonstrating the effectiveness of incorporating embodiment geometry into embodied navigation.
• 中文摘要 跨身体导航是具身智能中的一个关键挑战。由于身体的差异，同一视觉观察可能对不同代理意味着不同的动作，仅依赖视觉时预测变得模糊。现有研究主要依赖强化学习，这需要大规模互动和精心设计的奖励，这使得支持可扩展的预训练和现实世界适应变得困难。相比之下，基于模仿学习的方法仍然有限。为应对这些挑战，我们提出了一个基于模仿学习、具身感知导航框架的模块化多阶段设计。在预训练中，我们从互联网视频构建交叉身体导航数据集，并引入具象几何作为条件令牌，以减少同一观察下的动作歧义。在微调中，我们设计了基于解耦架构的多模态信息注入机制。具体来说，我们设计了一种轨迹增强策略，生成高风险样本，分别用于训练空间感知和风险感知修正，从而明确纳入具体几何结构以实现安全导航。实验结果表明，所提方法在不同具象环境中有效提升导航性能，展示了将具形几何融入具象导航的有效性。

MOF-Sleuth: Tool-Grounded Reward Alignment for Explainable Fine-Grained MOF CIF Auditing

MOF-Sleuth：工具基础奖励对齐，实现可解释的细粒度MOF审计

• Authors: Yu Liu, Zhiwei Yang, Diandian Guo, Kun Peng, Fangfang Yuan, Cong Cao, Chaozhuo Li, Zhiyuan Ma, Yanbing Liu, Guobin Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.19935
• Pdf link: https://arxiv.org/pdf/2607.19935
• Abstract Large metal-organic framework (MOF) databases support simulation, screening, and machine learning through crystallographic information files (CIFs). Subtle chemical and structural errors in these inputs can compromise downstream results and hinder manual inspection. LLM advances in computational chemistry offer paths beyond predictive screening toward fine-grained diagnosis with evidence-grounded explanations. However, two challenges remain: (i) limited fine-grained attribution: MOF-specific validators and machine-learning models scale detection but provide fixed checks, readiness scores, or coarse labels rather than evidence-grounded explanations; and (ii) unreliable CIF reasoning: direct LLM auditing is costly and unreliable because chemical evidence is implicit across atom-site records and requires geometric, connectivity, occupancy, and charge calculations. Both stem from weak coupling between chemical evidence and language-model explanation. We introduce MOF-Sleuth, a reinforcement-guided CIF auditing agent with two modules: a deterministic Forensic Lab and a Sleuth reasoning engine. The Lab derives composition, geometry, connectivity, occupancy, coordination, and charge evidence, and Sleuth uses this evidence to produce an evidence-grounded explanation, error types, and a binary decision. Reward-guided reinforcement learning (RL) turns tool measurements into chemical explanation-level supervision, rewarding not only the final answer but also cited chemical evidence and evidence-supported diagnoses. We introduce Chemically Grounded Diagnosis (Chem-GD), a metric that assesses whether a correct diagnosis is explained by factual, relevant CIF-derived evidence. Across four benchmarks, MOF-Sleuth establishes state-of-the-art performance among LLM-based approaches and MOF-specific machine-learning methods, demonstrating gains in detection, attribution, and grounded explanation quality.
• 中文摘要 大型金属有机框架（MOF）数据库支持通过晶体信息文件（CIF）进行仿真、筛选和机器学习。这些输入中的细微化学和结构错误会影响后续结果，阻碍人工检查。计算化学领域的LLM进展为超越预测性筛查、以证据为基础的细致诊断提供了道路。然而，仍有两个挑战：（i）细致归因有限：MOF专属验证器和机器学习模型虽能扩大检测规模，但提供固定检查、准备度评分或粗略标签，而非基于证据的解释;以及（ii）不可靠的CIF推理：直接进行LLM审计既昂贵又不可靠，因为化学证据隐含于原子位点记录中，且需要几何、连通性、占用性和电荷计算。这两者都源于化学证据与语言模型解释之间的弱耦合。我们介绍MOF-Sleuth，一款基于强化的CIF审计代理，包含两个模块：确定性取证实验室和侦探推理引擎。实验室推导成分、几何、连通性、占用率、协调性和电荷证据，Sleuth利用这些证据提出基于证据的解释、错误类型和二元决策。奖励引导强化学习（RL）将工具测量转化为化学解释层面的监督，不仅奖励最终答案，还奖励引用的化学证据和证据支持的诊断。我们引入了化学基础诊断（Chem-GD），这是一个评估正确诊断是否由事实且相关的CIF来源证据解释的指标。在四个基准测试中，MOF-Sleuth 在基于LLM的方法和MOF特定机器学习方法中建立了最先进的性能，展示了在检测、归因和基础解释质量方面的提升。

Towards Reliable C-to-Rust Translation with Rule-Guided Reasoning and Reinforcement Learning

迈向可靠的C到Rust翻译，结合规则引导推理和强化学习

• Authors: Feng Luo, Jiachen Liu, Cuiyun Gao, Jia Feng, Kui Liu
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.19966
• Pdf link: https://arxiv.org/pdf/2607.19966
• Abstract The migration of legacy C programs to Rust has become an important direction for improving software memory safety while alleviating the high cost of manual rewriting. Leveraging large language models (LLMs) for automated C-to-Rust translation has emerged as a promising direction. However, existing LLM-based approaches remain limited. On the one hand, LLMs exhibit limited capability in identifying Rust-specific rules, and inadequate handling of Rust syntax often results in incorrect translations. On the other hand, existing LLMs often struggle to accurately capture the semantics of complex code, resulting in incorrect translations. To address these challenges, we propose a Translation fRAmework Via rule-guided reasoning and rEinforcement Learning, namely TRAVEL, consisting of two modules. The first module employs Monte Carlo Tree Search (MCTS)-based reasoning path construction guided by Rust-specific rules, steering the search toward translation steps that respect the syntactic rules that LLMs frequently violate. The second module introduces reinforcement learning that couples execution feedback with reasoning-quality signals, encouraging the model to construct reasoning paths that accurately capture program semantics, thereby ensuring that the generated Rust code preserves the intended behavior of the original C program. We evaluate TRAVEL on three datasets: xCodeEval (a public benchmark), OS-Bench (functions collected from the Linux kernel), and HW-Bench (an industrial dataset from Huawei). On xCodeEval, TRAVEL outperforms all baselines across three backbone LLMs. In particular, compared to the strongest prompting baseline IRENE, TRAVEL improves computational accuracy (CA) by 26.22% and compilation success rate (CSR) by 18.77%. On HW-Bench and OS-Bench, TRAVEL further improves CSR by 18.28% and 16.51%, respectively, while reducing unsafe rate (UR) by 13.06% and 13.08%, respectively.
• 中文摘要 将遗留的C程序迁移到Rust已成为提升软件内存安全、同时减轻手动重写高昂成本的重要方向。利用大型语言模型（LLMs）进行自动化C到Rust翻译已成为一个有前景的方向。然而，现有的基于LLM的方法仍然有限。一方面，LLM在识别Rust特定规则方面能力有限，且对Rust语法处理不当常导致翻译错误。另一方面，现有的大型语言模型常常难以准确捕捉复杂代码的语义，导致翻译错误。为应对这些挑战，我们提出了一种通过规则引导推理和强化学习进行翻译 fRamework 的方案，即 TRAVEL，由两个模块组成。第一个模块采用基于蒙特卡洛树搜索（MCTS）的推理路径构建，并以Rust专用规则为指导，引导搜索朝向尊重LLM常违反的语法规则的翻译步骤。第二个模块引入了强化学习，将执行反馈与推理质量信号结合，鼓励模型构建准确捕捉程序语义的推理路径，从而确保生成的Rust代码保持原始C程序的预期行为。我们基于三个数据集评估TRAVEL：xCodeEval（公开基准测试）、OS-Bench（从Linux内核收集的函数）和HW-Bench（华为的工业数据集）。在xCodeEval中，TRAVEL在三个骨干大型语言模型中表现优于所有基线。特别是，与最强的提示基线IRENE相比，TRAVEL提高了计算准确率（CA）26.22%，编译成功率（CSR）提升了18.77%。在HW-Bench和OS-Bench上，TRAVEL分别提升了18.28%和16.51%的企业社会责任，同时分别降低了13.06%和13.08%的不安全率（UR）。

Coordinating from Memory: Graph-Structured Experience Reuse for Multi-Agent Adaptation in Dynamic Manufacturing

记忆协调：动态制造中多智能体适应中的图结构体验重用

• Authors: Chengxiao Dai, Zhanhui Lin, Zhaokun Yan, Youyang Ni, Chenjun Lei, Luyan Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.19985
• Pdf link: https://arxiv.org/pdf/2607.19985
• Abstract Dynamic manufacturing environments require multi-agent systems to coordinate effectively under frequent operational disturbances such as machine failures, urgent job arrivals, and processing time variations. Existing multi-agent reinforcement learning approaches treat each disturbance episode independently, discarding valuable coordination experience that could accelerate future adaptation. In this paper, we propose a Graph-Structured Experiential Memory (GSEM) framework for multi-agent coordination in dynamic manufacturing. The framework encodes historical coordination episodes as heterogeneous relational graphs that capture task dependencies, machine states, and inter-agent collaboration patterns. When a new disturbance occurs, a graph neural network-based retrieval mechanism identifies structurally similar past episodes, enabling experience-guided policy adaptation rather than learning from scratch. Experiments on dynamic flexible job-shop scheduling benchmarks with three disturbance types show that GSEM reduces makespan by 4.1%-10.0% and adaptation time by 33%-38% compared to the strongest memory-augmented baseline, with the advantage increasing under higher disturbance frequency. Ablation studies and cross-disturbance transfer experiments further validate the necessity of graph-structured encoding and similarity-based retrieval and demonstrate the cross-disturbance generalizability of learned coordination patterns.
• 中文摘要 动态制造环境需要多智能体系统在频繁的操作干扰下有效协调，如机器故障、紧急任务到达和处理时间变化。现有的多智能体强化学习方法独立处理每个干扰事件，丢弃了可加速未来适应的宝贵协调经验。本文提出了一种图结构化体验记忆（GSEM）框架，用于动态制造中的多智能体协调。该框架将历史协调事件编码为异构关系图，捕捉任务依赖关系、机器状态和代理间协作模式。当新的干扰发生时，基于图的神经网络检索机制识别结构相似的过去发作，从而实现基于经验引导的策略调整，而非从零学习。在动态灵活的作业车间调度基准测试中，使用三种干扰类型进行实验显示，GSEM相比最强的内存增强基线，完成时间减少了4.1%-10.0%，适应时间减少了33%-38%，且在更高干扰频率下优势进一步增强。消融研究和交叉干扰转移实验进一步验证了图结构编码和基于相似度的检索的必要性，并展示了学习协调模式的交叉干扰泛化性。

Generalized Kalman filter based temporal difference reinforcement learning

基于广义卡尔曼滤波器的时间差分强化学习

• Authors: Vasos Arnaoutis, Eric Lutters, Bojana Rosić
• Subjects: Subjects: Machine Learning (cs.LG); Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2607.20010
• Pdf link: https://arxiv.org/pdf/2607.20010
• Abstract In this paper, we present a generalized temporal-difference (TD) reinforcement learning framework based on the theory of conditional expectations. The value and action-value (Q-value) functions are treated as uncertain quantities, and their estimation is formulated as a stochastic inference problem. Unlike classical Kalman-based temporal-difference learning, which relies on linear-Gaussian assumptions, the proposed formulation is derived directly from the conditional expectation framework and naturally extends to nonlinear models and non-Gaussian probability distributions. The proposed method recursively estimates not only the conditional expectation of the value function but also its second probabilistic moment, thereby quantifying the uncertainty associated with the learned value function throughout the learning process. To obtain a computationally tractable algorithm, the stochastic problem is discretized using either polynomial chaos expansions or ensemble-based approximations, providing efficient representations of the underlying random variables. The proposed framework is demonstrated on two optimal control problems: a linear mass--spring--damper system and a nonlinear heat conduction problem in a closed cavity. The numerical examples illustrate the capability of the proposed method to accurately estimate both the value function and its associated uncertainty, while extending classical Kalman-based temporal-difference learning to a broader class of stochastic systems.
• 中文摘要 本文提出了基于条件期望理论的广义时间差分（TD）强化学习框架。值函数和作用值（Q值）函数被视为不确定量，其估计被表述为随机推断问题。与依赖线性高斯假设的经典卡尔曼时间差分学习不同，所提出的表述直接源自条件期望框架，并自然扩展到非线性模型和非高斯概率分布。所提方法递归地不仅估计价值函数的条件期望，还估计其第二概率矩，从而量化学习过程中与学习价值函数相关的不确定性。为了获得一个计算可解的算法，随机问题通过多项式混沌展开或基于集合的近似离散化，从而高效地表示底层随机变量。所提出的框架在两个最优控制问题上进行了演示：线性质量-弹簧-阻尼系统和闭合腔内的非线性热传导问题。数值示例展示了所提方法准确估计价值函数及其相关不确定性的能力，同时将经典基于卡尔曼的时间差分学习扩展到更广泛的随机系统类别。

Active Inference as a Convex Markov Decision Process

主动推断作为凸马尔可夫决策过程

• Authors: Nikola Milosevic, Nicolás Hinrichs, Nico Scherf
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2607.20152
• Pdf link: https://arxiv.org/pdf/2607.20152
• Abstract Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.
• 中文摘要 主动推断（AIF）将适应行为框定为期望自由能（EFE）的最小化，将认识论和语用目标结合在单一变分原理中。我们将 AIF 框架为策略优化，并证明对于闭环控制策略，EFE 最小化可以表述为凸马尔可夫决策过程（MDP）。在该表述中，语用项在预测状态边际中线性，因此等价于潜在多层次学习中的奖励最大化，而认识值引入了非线性成分，将EFE最小化与标准强化学习区分开来。这一观点进一步揭示了主动推理作为一种依赖策略（表演性）奖励的认识驱动力。我们分析了EFE的有限视界、贴现和平均奖励表述，并推导出一种镜像下降（MD）算法，该算法在当前状态边际附近局部线性化目标，产生与actor-critic方法和动态规划兼容的策略依赖奖励。最后，我们认为将世界模型学习与策略优化结合，使主动推理具备了执行性强化学习的结构，为主动推理奠定现代强化学习与优化理论基础的路径，包括收敛分析和原则性策略改进保证。

OLEDLM: A Unified Language Model for OLED Molecular Design

OLEDLM：OLED分子设计的统一语言模型

• Authors: Fukang Wen, Yuchong Tang, Jingyuan Li, Beichen Wang, Yixuan Jiang, Xiaoyi Jiang, Yaxuan Liu, Shunyu Wang, Zuoqiang Shi, Yi Zhu, Yanan Zhu, Pipi Hu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.20194
• Pdf link: https://arxiv.org/pdf/2607.20194
• Abstract The development of organic light-emitting diode (OLED) materials faces the compounded challenges of an astronomically large chemical space, stringent quantum-chemical constraints, and a scarcity of labeled data. Although the question of OLED generation is important, few models have been trained effectively for this specific domain. We propose an inverse molecular design framework based on causal language models: given target optoelectronic properties (e.g., excitation energy, oscillator strength), our model directly generates OLED SMILES sequences satisfying the specified constraints. We employ a multi-stage strategy: first, we establish a foundational chemical language model using a LLaMA-style transformer architecture. To the best of our knowledge, this represents the first successful adaptation of LLMs specifically for the OLED domain, bridging the gap between generic molecular generation and the stringent structural requirements of optoelectronic materials. Second, we fine-tune property predictors based on a BERT model pre-trained on our large-scale OLED dataset. Then, we perform Reinforcement Learning on our fine-tuned model, leveraging our property predictor, for better SMILES generation. Finally, through DFT verification, we demonstrate that our framework can efficiently navigate the OLED chemical space, generating novel candidates with high structural validity and optimized optoelectronic properties.
• 中文摘要 有机发光二极管（OLED）材料的开发面临着天文数字般庞大的化学空间、严格的量子化学限制以及标记数据稀缺的复杂挑战。尽管OLED生成的问题很重要，但很少有模型能有效训练该领域。我们提出了基于因果语言模型的逆分子设计框架：给定目标光电子特性（如激发能量、振荡器强度），我们的模型直接生成满足指定约束的 OLED SMILES 序列。我们采用多阶段策略：首先，我们采用LLaMA风格的变换器架构建立基础化学语言模型。据我们所知，这是首次成功将LLMs专门应用于OLED领域，弥合了通用分子生成与光电子材料严格结构要求之间的桥梁。其次，我们基于预训练的BERT模型对性质预测器进行微调。然后，我们利用属性预测器对微调模型进行强化学习，以更好地生成SMILES。最后，通过DFT验证，我们证明了我们的框架能够高效地适应OLED化学领域，生成具有高结构效度和优化光电子性能的新型候选产品。

Notes to Self: Can LLMs Benefit from Experiential Abstractions?

给自己的笔记：大型语言模型能从体验式抽象中受益吗？

• Authors: Chang Liu, Xinyu Li, Artur Dubrawski
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.20372
• Pdf link: https://arxiv.org/pdf/2607.20372
• Abstract Humans distill experience into reusable abstractions, e.g., strategies and cautionary reminders, and apply them to gradually solve problems more effectively. We study whether Large Language Models (LLMs) can similarly benefit from such experiential abstractions. From LLMs' solution traces on the MATH training set, a stronger teacher or the LLMs themselves extract natural-language abstractions into a retrievable library. We explore two usage modes: (1) inference-time retrieval and (2) reinforcement learning (RL) with abstraction-augmented training prompts. Experiential abstractions improve LLM performance on mathematical and logical reasoning benchmarks. Self-extracted abstractions match teacher-extracted ones, and our abstraction usage framework can transfer to other datasets and models. These findings suggest LLMs can extract and apply experiential abstractions much as humans leverage distilled experience.
• 中文摘要 人类将经验提炼成可重复使用的抽象，例如策略和警示提醒，并应用它们以更有效地逐步解决问题。我们研究大型语言模型（LLMs）是否也能从这种体验式抽象中受益。从大型语言模型在MATH训练集中的解法追踪中，更强的教师或大型语言模型本身会将自然语言抽象提取成可检索的库。我们探讨两种使用模式：（1）推理时间检索和（2）基于抽象增强训练提示的强化学习（RL）。体验式抽象提升了大型语言模型在数学和逻辑推理基准测试中的表现。自提取的抽象与教师提取的匹配，我们的抽象使用框架可以迁移到其他数据集和模型。这些发现表明，大型语言模型可以像人类利用提炼经验一样提取并应用体验抽象。

PercepCap: Video Captioner with Structured Spatio-Temporal Perception

PercepCap：具备结构化时空感知的视频字幕

• Authors: Yifan Xu, Zihao Wang, Zhixiao Wang, Jiaming Zhang, Yichun Yang, Desen Meng, Yuanxing Zhang, Pengfei Wan, Limin Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.20389
• Pdf link: https://arxiv.org/pdf/2607.20389
• Abstract Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.
• 中文摘要 视频字幕需要对视频进行细致的时空理解，包括对物体位置的空间感知以及事件发生的时间感知。现有的MLLM通常直接从视频输入生成字幕，而不暴露描述背后的感知证据。因此，时空感知的错误仅在最终字幕中被观察到，这使得直接识别潜在的感知错误变得困难。为解决这些问题，我们介绍了PercepCap，一种感知感知视频字幕框架，在生成最终字幕前将感知证据明确化。具体来说，PercepCap遵循感知-描述生成链，模型首先生成包含物体轨迹和时间事件的时空感知痕迹，然后根据感知证据生成最终的字幕。为此，我们设计了两阶段的培训策略。感知后描述监督微调将模型从仅生成字幕调整为所提的感知-描述链，而感知-基于强化学习则通过对感知链和最终字幕的联合奖励优化感知追踪和字幕质量。为了支持我们的两阶段培训，我们引入了字幕锚定感知数据构建。该流程通过先生成仅带字幕的描述，提取提及的对象和事件，并用框和时间戳将它们扎根回视频中，构建SFT和RL训练数据。这会产生与字幕对齐的感知数据，提供扎实的训练基础，确保显式感知追踪和最终字幕指向相同的对象和事件。在直接字幕和字幕到质量保证评估中，PercepCap 持续优于 Qwen3-VL 基线，展现出领先的字幕质量。

Towards Miniature Humanoid Tele-Loco-Manipulation Using Virtual Reality and Reinforcement Learning

迈向利用虚拟现实和强化学习的微型人形远程机车操控

• Authors: Nicolas Kosanovic, Jordan Dowdy, Jean Chagas Vaz
• Subjects: Subjects: Robotics (cs.RO); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.20399
• Pdf link: https://arxiv.org/pdf/2607.20399
• Abstract Full-sized humanoid robot capabilities have grown exponentially in recent years, aiming towards general-purpose deployment in human environments. A popular control method used by manufacturers utilizes Virtual Reality for upper-body teleoperation and Reinforcement Learning for lower-body balance and locomotion control. As a result, a single remote operator can see, manipulate, and navigate about a real, distant physical environment. This powerful control stack is often relegated to expensive full-sized robots, many of which are inaccessible to the research community. Miniature humanoids are more prevalent, but employ less biomimicry in their design (e.g. fewer sensors, Degrees of Freedom, etc) and lack similar developments. This paper describes a compliant full-body telepresence control stack developed from the ground up for miniature humanoids. Framework experimentation on ROBOTIS OP3 hardware showcases walking at speeds up to 0.45 m/s independent of arm motions. Tele-loco-manipulation is demonstrated via a cube relocation experiment with an expert human operator. On average, the teleoperated system moved 2 different 40 g cubes within 10 mins, walking a total distance of 5 m. Overall, the developed system shows potential for miniature humanoid tele-loco-manipulation.
• 中文摘要 近年来，全尺寸类人机器人能力呈指数级增长，目标是实现在人类环境中的通用部署。制造商常用的一种控制方法是利用虚拟现实实现上肢远程操作，强化学习用于下半身平衡和运动控制。因此，单个远程操作员就能看到、操作并导航真实的遥远物理环境。这种强大的控制堆栈通常被限制在昂贵的全尺寸机器人上，而许多机器人对研究界来说是无法接触的。微型类人生物更为普遍，但其设计中仿生元素较少（如传感器、自由度等较少），且缺乏类似的发展。本文介绍了一套从零开始为微型人形生物开发的合规全身远程临场控制栈。在ROBOTIS OP3硬件上的框架实验展示了行走速度可达0.45米/秒，独立于手臂运动。远程机车操作通过由专家操作员进行立方体移动实验演示。遥控系统平均在10分钟内移动两个不同的40克立方体，总步距为5米。总体而言，该系统展现出微型类人远程机车操控的潜力。

Keyword: diffusion policy

Diffusion ReRoll: Revisable Denoising for Robotic Sequential Prediction

扩散重叠：可修正去噪用于机器人顺序预测

• Authors: Seonsoo Kim, Seongil Hong, Jun-Gill Kang
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.19919
• Pdf link: https://arxiv.org/pdf/2607.19919
• Abstract We propose Diffusion ReRoll, a diffusion-based framework for robotic sequential prediction that enables revisable denoising over horizons. Existing diffusion-based sequence predictors typically perform a single monotonic denoising process. In contrast, Diffusion ReRoll selectively re-noises regions that have become locally stable while the remaining regions continue denoising, so the re-noised regions can be refined again using context from the rest of the horizon. This structured re-noising enables iterative cross-horizon revision, allowing earlier and later segments to revise one another, while maintaining local consistency. We evaluate Diffusion ReRoll against full-sequence diffusion and causal denoising based on Diffusion Forcing across long-horizon planning, policy learning, and unified video-action modeling. On OGBench PointMaze and AntMaze, Diffusion ReRoll achieves relative gains in average success rate of 21% over Diffusion Forcing in matched guidance-based planning and 23% over Diffuser in matched goal-inpainting. In diffusion-policy-style action prediction, Diffusion ReRoll improves average success by 56.5% relative to Diffusion Policy across different prediction horizons and history lengths on the LIBERO-10 multi-task benchmark. In unified video-action prediction, Diffusion ReRoll improves policy and inverse dynamics performance, especially under out-of-distribution evaluation, and achieves the best action-video consistency. These results support structured re-noising as an effective mechanism for revisable robotic sequence generation.
• 中文摘要 我们提出了Diffusion ReRoll，一种基于扩散的机器人顺序预测框架，能够在视界内实现可修正的去噪。现有基于扩散的序列预测器通常执行单调去噪过程。相比之下，扩散重叠会选择性地重新复噪那些局部稳定的区域，而其余区域则继续去噪，因此可利用视界其他部分的上下文重新细化被重新传噪的区域。这种结构化的重新噪声支持迭代跨视界修订，允许较早和较晚的段相互修正，同时保持局部一致性。我们基于扩散强迫，在长期规划、政策学习和统一视频-动作建模中评估扩散重叠与全序列扩散和因果去噪。在OGBench PointMaze和AntMaze中，扩散重掷在匹配导引规划中平均成功率相较21%，在匹配目标绘制中比扩散器提升23%。在扩散策略风格的动作预测中，扩散重掷在LIBERO-10多任务基准测试中，在不同预测视野和历史长度下，平均成功率相较扩散策略提升了56.5%。在统一的视频-动作预测中，扩散重叠提升策略和反动态性能，尤其是在非分发评估下，并实现最佳动作-视频一致性。这些结果支持结构化重噪声作为可修订机器人序列生成的有效机制。