生成时间: 2026-08-06 18:41:31 (UTC+8); Arxiv 发布时间: 2026-08-06 20:00 EDT (2026-08-07 08:00 UTC+8)

今天共有 24 篇相关文章

Keyword: reinforcement learning

Perception Before Reasoning: Dynamic Latent Reasoning for Video Understanding and Question Answering

推理前的感知：视频理解与问答中的动态潜在推理

• Authors: Haotian Xia, Zilin Xiao, Junbo Zou, Vicente Ordonez, Hanjie Chen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04124
• Pdf link: https://arxiv.org/pdf/2608.04124
• Abstract Video question answering requires models to ground language queries in visual evidence and, when necessary, reason over that evidence across time. Existing methods typically rely on long textual chain-of-thought rationales, even though many questions can be answered as soon as the relevant object, action, or frame is localized. We propose Dynamic Latent Reasoning (DyLaR), which first grounds a question in a short block of perception latents (continuous hidden states that encode query-relevant visual evidence), and then adaptively decides whether to append reasoning latents (continuous thoughts that reason over this evidence in latent space) before answering. DyLaR learns this behavior by grounding perception latents in verified visual evidence and distilling verified rationales into reasoning latents, followed by reinforcement learning that further refines when to reason. Across nine video benchmarks and four multimodal language model backbones, DyLaR improves average accuracy over same-backbone baselines while generating fewer than 20 tokens per query. On Qwen3-VL-4B, for example, DyLaR improves average accuracy over Qwen3-VL-4B-Thinking from 54.0 to 58.2 while reducing response length from 1,220.7 to 18.5 tokens per query. Ablations further show that grounded perception latents, rationale-supervised reasoning latents, and adaptive routing each improve accuracy.
• 中文摘要 视频问答需要模型将语言查询建立在视觉证据基础上，并在必要时跨时间推理这些证据。现有方法通常依赖长篇文本的思维链推理，尽管许多问题一旦相关对象、动作或框架定位即可得到回答。我们提出了动态潜在推理（DyLaR），它首先将问题建立在一小块感知潜在状态（编码查询相关视觉证据的连续隐藏状态），然后自适应地决定是否附加推理潜在因素（在潜在空间中推理这些证据的连续思维），然后再回答。DyLaR通过将感知潜在因素扎根于已验证的视觉证据，并将验证的理由提炼为推理潜在因素，随后进行强化学习，进一步精炼何时推理来学习这种行为。在九个视频基准测试和四个多模态语言模型骨干中，DyLaR在同一骨干基线下提升平均准确率，同时每个查询生成的token少于20个。例如，在Qwen3-VL-4B上，DyLaR将Qwen3-VL-4B-Thinking的平均准确率从54.0提升到58.2，同时将每查询的响应长度从1,220.7个减少到18.5个。消融进一步表明，基于地面感知的潜在因素、理性监督的推理潜在因素和自适应路由都能提升准确性。

Trident : How to Break Deep Reinforcement Learning Cyber Defenses (Agentic)

三叉戟：如何突破深度强化学习网络防御（Agentic）

• Authors: Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi, Sanggeon Yun, Hyunwoo Oh, SungHeon Jeong, Nathaniel D. Bastian, Mahdi Imani, Mohsen Imani
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.04317
• Pdf link: https://arxiv.org/pdf/2608.04317
• Abstract Autonomous cyber defense systems based on Deep Reinforcement Learning (DRL) have attracted significant research attention, yet remain evaluated almost exclusively against static, heuristic red agents, leaving their robustness against adaptive threats critically understudied. Meanwhile, recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) have improved LLM reasoning, but their integration into cybersecurity remains elusive due to the absence of suitable benchmark environments and interaction datasets. To bridge this gap, we introduce Trident, an agentic LLM red teaming framework comprising three components: a dynamic benchmark with isolated sandbox servers spanning CybORG CAGE 4 and CyberWheel, a dataset comprises over 13,000 high-fidelity red-blue interaction trajectories for RLVR, and a ``Code-as-Policy'' RLVR agentic architecture Trident Agentic). The latter reformulates red agent training as a contextual bandit via a tripartite Log Summarizer--Planner--Coder design, where a trainable Planner generates complete attack strategies from compressed execution logs, which a frozen Coder translates into executable Python policies deployed against live DRL defenders. Empirical evaluations reveal a fundamental brittleness in existing defenses: with a single trainable 7B planner, Trident reduces blue agent defensive performance by an average of 522% compared to static red agent baselines while autonomously discovering emergent behaviors such as decoy avoidance and adaptive state prioritization that static heuristics entirely fail to uncover.
• 中文摘要 基于深度强化学习（DRL）的自主网络防御系统吸引了大量研究关注，但几乎只针对静态、启发式红色代理进行评估，其对适应性威胁的鲁棒性研究严重不足。与此同时，近期在可验证奖励强化学习（RLVR）方面取得了改进，但由于缺乏合适的基准环境和交互数据集，其与网络安全的整合仍然难以实现。为弥合这一空白，我们引入了Trident，一个代理型LLM红队框架，包含三个部分：一个包含CybORG CAGE 4和CyberWheel的隔离沙箱服务器的动态基准测试，一个包含13,000多个高保真红蓝交互轨迹的RLVR数据集，以及一个“代码即策略”RLVR代理架构Trident Agentic。后者将红代理训练重新表述为上下文盗贼，采用三方日志摘要器-规划师-编码器设计，可训练的策划器从压缩执行日志生成完整的攻击策略，冻结的编码器将其转化为针对实时DRL防御者的可执行Python策略。实证评估揭示了现有防御的根本脆弱性：使用单一可训练的7B规划器，Trident可将蓝色代理的防御性能平均降低522%，同时自主发现静态启发式完全无法发现的诱饵回避和自适应状态优先级等突发行为。

ATLAS: Adaptive Topological Learning with Abstract Successors for Continual Learning

ATLAS：带有抽象继承者的自适应拓扑学习以实现持续学习

• Authors: R. Blake Lawlor, Daniel S. Brown
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04334
• Pdf link: https://arxiv.org/pdf/2608.04334
• Abstract Contemporary model-free reinforcement learning algorithms can achieve very high performance, but have low sample efficiency and are not robust to changes in the environment. Model-based algorithms have much higher sample efficiency, but still fail when the environment shifts. This paper introduces Adaptive Topological Learning with Abstract Successors (ATLAS) to combat these challenges. ATLAS uses a Grow When Required network with Successor Features in order to achieve high sample efficiency while also robustly tackling catastrophic forgetting. We evaluate ATLAS in spatial navigation tasks, benchmarking its performance against common on-policy and off-policy algorithms. Our empirical results demonstrate that by structurally decoupling transition dynamics from the reward signal, ATLAS achieves near-instantaneous adaptation to new goals and can exhibit positive backward transfer, significantly outperforming baseline methods in non-stationary environments.
• 中文摘要 现代无模型强化学习算法能实现非常高的性能，但样本效率较低，且对环境变化缺乏鲁棒性。基于模型的算法样本效率更高，但环境变化时仍会失败。本文介绍了带有抽象继承器的自适应拓扑学习（ATLAS）以应对这些挑战。ATLAS采用带有后继特征的“必要时增长”网络，以实现高采样效率，同时坚实应对灾难性遗忘问题。我们在空间导航任务中评估ATLAS，并对其性能与常见的开策略和非策略算法进行基准对比。我们的实证结果表明，通过结构性地将过渡动态与奖励信号解耦，ATLAS几乎瞬时适应新目标，并能表现出积极的向后转移，在非平稳环境中显著优于基线方法。

Generative Optimization for Incentivized Advertising with Global Level Constraints

针对全球层级约束的激励广告生成优化

• Authors: Gege Chen, Ning Luo, Hao Jiang, Da Li, Wenzheng Shu, Teng Sha, Yanxiang Zeng, Wenxin Tai, Fan Zhou, Xialong Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04421
• Pdf link: https://arxiv.org/pdf/2608.04421
• Abstract Incentivized advertising allocates monetary or virtual rewards to drive user engagement, where a key challenge is optimizing continuous incentive magnitudes under strict global constraints. This problem is complicated by high-frequency interactions, delayed feedback, and non-Markovian user dynamics such as fatigue, which limit the effectiveness of existing uplift modeling and constrained reinforcement learning approaches. To address these challenges, we propose GOAL, a constraint-aware generative framework that formulates incentive allocation as a conditional sequence generation problem. GOAL directly generates incentive magnitudes conditioned on user histories and system-level global pressure, and integrates a hierarchical causal state encoder to capture both local behavioral dynamics and long-range dependencies. To enable flexible constraint control, we introduce \textbf{S}afe \textbf{C}onstrained \textbf{P}olicy \textbf{O}ptimization (SCPO), which learns a single generative policy that generalizes across a spectrum of ROI constraints without retraining. Experiments on large-scale real-world data and a synthetic fatigue-aware environment show that GOAL improves long-term revenue and user retention while substantially reducing ROI violation rates compared to strong baselines.
• 中文摘要 激励广告通过金钱或虚拟奖励来推动用户参与度，其中一个关键挑战是在严格的全局约束下优化连续激励幅度。该问题因高频交互、延迟反馈以及非马尔可夫用户动态（如疲劳）而复杂化，这些限制了现有提升建模和受限强化学习方法的有效性。为应对这些挑战，我们提出了GOAL，一种约束感知型生成框架，将激励分配表述为条件序列生成问题。GOAL直接根据用户历史和系统级全局压力生成激励幅度，并集成分层因果状态编码器，以捕捉局部行为动态和长程依赖关系。为实现灵活约束控制，我们引入了 \textbf{S}afe \textbf{C}onstrained \textbf{P}olicy \textbf{O}ptimization （SCPO），该策略学习单一生成策略，可跨越多种 ROI 约束谱系而无重训练。在大规模真实世界数据和合成疲劳感知环境中的实验表明，GOAL 在显著降低 ROI 违规率的同时，提升了长期收入和用户留存率，相较于强基线。

ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation

ToolArtist：使用统一多模态模型进行代理图像生成的工具

• Authors: Jiahao Zhao, Xiaomin Yu, Zhongxiang Sun, Fengwei Teng, Chengwei Qin, Xiaobin Hu, Jun Xu, Shuicheng Yan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.04436
• Pdf link: https://arxiv.org/pdf/2608.04436
• Abstract Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.
• 中文摘要 文本到图像（T2I）模型可以生成视觉上引人注目的图像，但在需要复杂语义理解、多步推理和外部世界知识整合的开放世界任务上仍然受限。现有努力将代理能力引入图像生成，但它们要么规定固定的工作流程，要么只将开放世界图像生成过程的部分置于代理控制之下。因此，推理、工具调用和图像生成并非由单一策略协调。我们提出了ToolArtist，一种通过后期训练统一多模态模型（UMM）获得的完全智能体图像生成模型。ToolArtist动态协调推理、外部工具使用和原生图像生成，并统一策略。在监督微调（SFT）期间，我们为教师代理配备了搜索工具和图像生成工具。然后我们将收集到的轨迹转换为兼容UMM的格式，图像生成工具被隐藏，而生成的图像则被保留。在强化学习（RL）过程中，我们为UMM开发了一个代理式强化学习基础设施，并引入了理性-行为-绘制GRPO（RAD-GRPO），该技术利用互补的意图和质量奖励共同优化模型。实验显示，将整个开放世界图像生成过程置于智能体策略之下，能够持续优于使用固定流水线或仅部分智能体控制组件的方法。我们发布了训练数据和完整的训练后基础设施。

MCHA: A Memory-Centric Hierarchical Architecture for Parallel-Sequential Computing

MCHA：一种以内存为中心的并行顺序计算分层架构

• Authors: Daijing Shi, Hongxiao Zhao, Yihan Fu, Zhan Chen, Jiayi Li, Yihang Zhu, Anjunyi Fan, Yaoyu Tao, Yuchao Yang, Bonan Yan
• Subjects: Subjects: Hardware Architecture (cs.AR); Distributed, Parallel, and Cluster Computing (cs.DC); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.04443
• Pdf link: https://arxiv.org/pdf/2608.04443
• Abstract Emerging workloads, such as Multi-Agent Reinforcement Learning (MARL), large-scale neuromorphic computing, and probabilistic graphical models, intrinsically exhibit parallel-sequential computing patterns. While these tasks demand massive parallelism to achieve high throughput, they are severely bottlenecked by irregular data access patterns centralized to main memory. Consequently, conventional architectures face fundamental limitations when executing these workloads, primarily manifesting as global buffer saturation and memory-bound bottlenecks. To address these challenges, we propose the Memory-Centric Hierarchical Architecture (MCHA), a reconfigurable hardware solution tailored for parallel-sequential execution. MCHA leverages a hierarchical communication strategy that facilitates distributed, inter-core data routing, thereby significantly reducing the bandwidth burden on the global memory. Complementing the hardware, MCHA introduces a novel parallel-sequential programming model that utilizes event-driven conditional triggers to effectively hide data transmission latency within the execution pipeline. We benchmark MCHA against a diverse suite of parallel-sequential tasks, including MARL, motor variable control, and Markov random fields. Validated through our open-source, cycle-accurate simulator, MCHA demonstrates performance speedups ranging from 153.06$\times$ to 2456.96$\times$ over NVIDIA A100 GPUs on MARL workloads, while maintaining robust programming flexibility across other application domains. Furthermore, the architecture successfully reduces main memory access from 96% to 5.44%. When synthesized in a 28 nm process, the MCHA implementation occupies an area footprint of 2.92mm$^2$ and consumes 115.36 mW of power at 200 MHz. MCHA is open-sourced at this https URL.
• 中文摘要 新兴工作负载，如多智能体强化学习（MARL）、大规模神经形态计算和概率图形模型，本质上表现出并行顺序计算模式。虽然这些任务需要大规模并行以实现高吞吐量，但它们被集中在主存的数据访问模式严重限制。因此，传统架构在执行这些工作负载时面临根本性限制，主要表现为全局缓冲区饱和和内存瓶颈。为应对这些挑战，我们提出了以内存为中心的层级架构（MCHA），这是一种可重构的硬件解决方案，专为并行顺序执行量身定制。MCHA采用分层通信策略，促进分布式核心间数据路由，显著降低全局内存带宽负担。作为硬件的补充，MCHA引入了一种新颖的并行顺序编程模型，利用事件驱动的条件触发器，有效隐藏执行流水线中的数据传输延迟。我们将MCHA与多种并行顺序任务（包括MARL、电机变量控制和马尔可夫随机场）进行基准测试。通过我们的开源、周期精确模拟器验证，MCHA在MARL工作负载上相比NVIDIA A100 GPU实现了从153.06美元到2456.96美元/时间美元的性能提升，同时保持了在其他应用领域的强健编程灵活性。此外，该架构成功将主存访问率从96%降至5.44%。在28纳米工艺合成时，MCHA实现占用面积为2.92mm$^2$，在200 MHz频率下耗电115.36毫瓦。MCHA开源于此https URL。

Energy Efficient AI-Enabled Wireless Sensor Networks for Mission Critical Environments: A Systematic Review across Smart Grid, AI, and Urban Infrastructure Applications

适用于关键任务环境的节能AI无线传感器网络：跨智能电网、人工智能及城市基础设施应用的系统综述

• Authors: Alexandros Gazis, Valeri Mladenov, Kleanthi SantamourI, Stylianos Pappas
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2608.04499
• Pdf link: https://arxiv.org/pdf/2608.04499
• Abstract Advanced wireless sensor networks powered by artificial intelligence are increasingly required for applications demanding continuous monitoring, autonomous operation, reliable communication, and fast decision support. This systematic review examines recent work from 2023 to 2026 on energy-efficient, AI-enabled wireless sensor networks (WSNs) in mission-critical environments, with particular focus on power electronics, and urban infrastructure systems. The authors synthesise a corpus of 50 DOI indexed studies satisfying inclusion criteria that received qualitative thematic coding and comparative analysis. Other references were only cited to provide historical, methodological, or technical context and were not included in the systematic review corpus. As such, our results show that AI can improve WSN energy behaviour through routing and clustering, edge AI, reinforcement learning, fuzzy logic, metaheuristic optimisation, and AI-based security. At the same time, energy efficiency cannot be treated as an isolated performance target. In mission-critical systems, security, latency, and reliability are closely interlinked requirements. The review concludes that future work should move away from optimising protocols in isolation, and instead focus on building lightweight, explainable, secure, and field-tested AI-driven WSN architectures suited to real operational environments.
• 中文摘要 由人工智能驱动的先进无线传感器网络日益被要求持续监控、自主运行、可靠通信和快速决策支持的应用所必需。本系统综述回顾了2023年至2026年间关于关键任务环境中节能、人工智能驱动的无线传感器网络（WSN）的最新研究，特别关注电力电子和城市基础设施系统。作者综合了50项符合纳入标准的DOI索引研究语料库，这些研究进行了定性主题编码和比较分析。其他参考文献仅用于提供历史、方法论或技术背景，未纳入系统综述语料库。因此，我们的结果表明，人工智能可以通过路由与聚类、边缘人工智能、强化学习、模糊逻辑、元启发式优化以及基于人工智能的安全来改善WSN能源行为。同时，能源效率不能被视为孤立的性能目标。在关键任务系统中，安全性、延迟和可靠性是紧密相关的需求。综述总结指出，未来的工作应避免孤立地优化协议，而应专注于构建轻量级、可解释、安全且经过实地测试的AI驱动WSN架构，以适应真实运营环境。

EASy: Towards Efficient LLM-Based Agentic System

EASy：迈向高效的基于大型语言模型的智能系统

• Authors: Junnan Liu, Linhao Luo, Thuy-Trang Vu, Gholamreza Haffari
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04588
• Pdf link: https://arxiv.org/pdf/2608.04588
• Abstract Agentic systems have emerged as a promising paradigm for solving complex tasks by coordinating specialized LLM-based agents. However, most existing systems primarily optimize task success while giving limited consideration to execution efficiency under practical constraints such as executor capability and computational cost. Existing router-based methods have limited ability to reason over rich, evolving task contexts, multi-step dependencies, and intermediate execution feedback, and often generalize poorly to unseen executors. We propose EASy, a trainable agentic framework that jointly optimizes task performance and computational efficiency through reinforcement learning. EASy equips an LLM-based orchestrator with explicit knowledge of the capability and cost profiles of heterogeneous executors, enabling context-sensitive coordination beyond performance-only routing. It further introduces a milestone-plan-act workflow that decomposes complex tasks into manageable milestones, constructs dependency-aware execution graphs, assigns suitable executors, and parallelizes independent steps while adapting subsequent decisions to intermediate outcomes. To train the orchestrator, we develop a tree-structured rollout procedure that explores alternative milestone decompositions and execution plans, together with multi-component rewards that capture task correctness, execution efficiency, and trajectory completeness. Extensive experiments on mathematical reasoning, embodied decision-making, and deep research benchmarks show that EASy consistently achieves stronger performance-efficiency trade-offs than strong agentic baselines.
• 中文摘要 代理系统已成为通过协调基于LLM的专业代理解决复杂任务的有前景范式。然而，大多数现有系统主要优化任务成功，同时在执行者能力和计算成本等实际约束下有限度考虑执行效率。现有基于路由器的方法在丰富且不断演变的任务上下文、多步依赖和中间执行反馈上推理能力有限，且常常难以泛化到看不见的执行者。我们提出了EASy，一种可训练的代理框架，通过强化学习共同优化任务性能和计算效率。EASy为基于LLM的编排器配备了对异构执行者能力和成本特征的明确了解，实现超越仅性能路由的上下文敏感协调。它还引入了里程碑-计划-行动工作流程，将复杂任务分解为可管理的里程碑，构建依赖感知执行图，分配合适的执行者，并行化独立步骤，同时调整后续决策以适应中间结果。为培训编排器，我们开发了树状结构的推广流程，探索替代里程碑分解和执行计划，并结合多元奖励，捕捉任务正确性、执行效率和轨迹完整性。数学推理、具身决策和深入研究基准的广泛实验表明，EASy在性能与效率权衡上持续优于强代理基线。

Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning

在推理模型中评估心智理论：稳健性胜于推理

• Authors: Ian B. de Haan, Peter van der Putten, Max van Duijn
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.04646
• Pdf link: https://arxiv.org/pdf/2608.04646
• Abstract Large language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests, prompting debate about the nature and validity of the underlying capabilities. At the same time, reasoning-oriented LLMs trained via reinforcement learning with verifiable rewards have demonstrated notable improvements across a range of benchmarks. In this work, we examine the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments together with results from established benchmarks. We observe that reasoning models consistently exhibit increased robustness to prompt variations and task perturbations. Our analysis suggests these gains come at least partly from models being more robust at reaching the correct answer under prompt and task variation. We read this as evidence for a robustness-based account rather than for a new ToM-specific ability.
• 中文摘要 大型语言模型（LLMs）最近在心智理论（ToM）测试中表现出强劲表现，引发了关于其底层能力性质和有效性的讨论。与此同时，通过强化学习训练并带有可验证奖励的推理导向LLM在多个基准测试中展现了显著改进。本研究通过对机器心理学实验的新颖改编结合既有基准测试结果，分析了这些推理模型在ToM任务中的行为。我们观察到推理模型在提示变化和任务扰动方面持续表现出更高的鲁棒性。我们的分析表明，这些提升至少部分来自于模型在提示和任务变异下更稳健地地得出正确答案。我们将此视为基于稳健性账户的证据，而非针对新的ToM专属能力。

Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning

人工内疚校准：神经基础奖励塑造，用于亲社会多代理强化学习

• Authors: Aaditya Mehta, Arya Shah
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04663
• Pdf link: https://arxiv.org/pdf/2608.04663
• Abstract Cooperative multi-agent reinforcement learning often adds social terms to individual rewards, yet the scale of those terms is usually chosen by hand. We ask whether a guilt signal can instead be calibrated from human neural and behavioural data and transferred to artificial agents. Using the public SoDec responsibility fMRI dataset (40 participants), we fit a subject-fixed-effects regression of momentary-happiness changes on outcome-type counts and recover a guilt weight as the Partner-negative minus Social-negative contrast ($\hat{w}=1.118$, Cohen's $d=0.214$). We embed this weight in a two-agent Social Lottery environment and train independent Proximal Policy Optimization actor-critics under four shaping regimes: neurally calibrated, uniform constant, zero (selfish), and a unit-coefficient oracle. Across 1{,}000 evaluation episodes per condition, the calibrated agents track the human Social safe-choice rate most closely ($0.459$ vs.\ human $0.484$; $\mathrm{KL}=0.0012$), while the other three conditions deviate by one to three orders of magnitude in KL. Human neurobehavioural priors can therefore act as quantitative constraints on prosocial reward shaping.
• 中文摘要 合作多智能体强化学习通常会在个体奖励中添加社会术语，但这些词的尺度通常是人工选择的。我们询问是否可以从人类神经和行为数据中校准内疚信号，并转移到人工智能体上。利用公开的SoDec责任fMRI数据集（40名参与者），我们拟合了一个主体固定效应回归，拟合了结果类型计数中瞬间幸福变化的主体固定效应回归，并恢复了负罪感权重为伴侣负面减去社会负对比（$\hat{w}=1.118$，Cohen's Initial$d=0.214$）。我们将该权重嵌入双代理社会彩票环境中，并在四个塑造模式下训练独立的近端策略优化行为者-批评者：神经校准、均匀常数、零（自私）和单位系数预言机。在每个条件的1{，}000次评估中，校准后的代理人最接近人类社会安全选择率（$0.459$ 对人类$0.484$;$\mathrm{KL}=0.0012$），而其他三种条件在KL中偏差1到3个数量级。人类神经行为先验因此可以作为亲社会奖励塑造的定量约束。

Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

教导MLLM说“不”：通过拒绝校准GRPO进行广义指称表达理解

• Authors: Xuzheng Yang, Jun Ling, Tao Huang, Caiyan Qin, Peng Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04698
• Pdf link: https://arxiv.org/pdf/2608.04698
• Abstract We tackle the challenging yet underexplored task of Generalized Referring Expression Comprehension (GREC), which requires a model to localize the object described by a textual expression when it exists (positive sample) and to refuse output when it does not (negative sample). Although Multimodal Large Language Models (MLLMs) excel at localizing existing objects, they often fail to reject nonexistent ones due to the absence of negative samples during training, producing hallucinated bounding boxes. Existing post-training approaches such as supervised fine-tuning (SFT) and reinforcement learning (RL) enhance refusal behavior but usually degrade localization accuracy on positive samples, undermining the model's core competence. To address this, we propose Refusal-Calibrated Group Relative Policy Optimization (RC-GRPO), a calibrated RL strategy that strengthens the refusal ability of MLLMs while preserving localization performance. It enforces "None" outputs in rollouts for valid advantage estimation on negative samples and applies a penalty to prevent over-refusal on positives, achieving a balanced trade-off between accuracy and reliability. A second-stage reasoning reinforcement further consolidates causal understanding and interpretability. Experiments on three GREC benchmarks demonstrate that RC-GRPO attains superior localization accuracy while maintaining strong refusal capability.
• 中文摘要 我们解决了具有挑战性但尚未充分探索的广义指称表达理解（GREC）任务，该方法要求模型在文本表达式描述对象存在时定位该对象（正样本），在不存在时拒绝输出（负样本）。尽管多模态大型语言模型（MLLMs）在定位现有对象方面表现出色，但由于训练过程中缺乏负样本，常常无法拒绝不存在的对象，导致产生幻觉的边界框。现有的训练后方法如监督微调（SFT）和强化学习（RL）增强了拒绝行为，但通常降低正样本的定位准确性，削弱模型的核心能力。为此，我们提出了拒绝校准群体相对策略优化（RC-GRPO），这是一种校准的强化学习策略，增强MLLM的拒绝能力，同时保持本地化性能。它在推广中强制执行“无”输出以实现负样本的有效优势估计，并对正样本施加惩罚以防止过度拒绝，实现准确性与可靠性之间的平衡权衡。第二阶段推理强化进一步巩固了因果理解和可解释性。在三个GREC基准测试上的实验表明，RC-GRPO在保持强拒绝能力的同时，实现了更优越的定位精度。

Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation

带有观察校准自蒸馏的能动强化学习

• Authors: Yi Yang, Cong Qin, Xiaodan Liu, Chishui Chen, Qing Dong, Yan Zhang, Cao Liu, Zhao Yang, Lu Pan, Jiaye Lin, Yi Feng
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.04788
• Pdf link: https://arxiv.org/pdf/2608.04788
• Abstract Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at this https URL.
• 中文摘要 大型语言模型代理通常通过强化学习训练，且奖励轨迹级奖励稀疏，这对单个代币更新的强度提供了有限的指导。策略上自我蒸馏（OPSD）通过在特权重放视图下重新评分生成的代币，以获得密集的代币级监督来解决这个问题。然而，我们发现了一个混淆问题：所得支持可能同时反映了重放视图中包含的特权信息和重放支架引发的分数变化，这使得难以将支持具体归因于该信息。当未来环境观测作为特权信息时，这个问题尤为明显，因为重放这些观测需要重建一个扩展支架，而支架本身也会扰动代币评分。为解决这一混淆，我们提出了观察校准自蒸馏（OCSD），对比两种结构匹配的回放视图——完整和观察-消融视图，仅在实际未来观测值是否存在上有所不同，以推导出一个观测残差，以抵消回放支架共享的分数变化。OCSD随后将该残差应用于调制高不确定性步长的令牌级GRPO更新，同时保持轨迹级更新方向。在ALFWorld、WebShop和Search-QA上，跨越三种Qwen3模型尺度的实验显示，OCSD始终优于强基线。诊断分析进一步证实，校准后的残差更符合局部环境反馈。我们的代码公开访问此https URL。

Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

特权但有偏见：受PI条件反射的教师如何打破自我提炼

• Authors: Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.04794
• Pdf link: https://arxiv.org/pdf/2608.04794
• Abstract Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.
• 中文摘要 自我提炼（SD）已成为一种计算高效的强化学习替代方案，并具有可验证的奖励：自学者基于关于答案的特权信息（PI），如参考解，为从未见过该信息的学生提供密集的每个代币监督。然而，报告的收益几乎完全来自狭窄、低难度的设置，留下一个基本问题：作为唯一目标，没有奖励项，SD是否能教到什么？我们在简单设置下重现SDPO报告的收益，然后将相同的设置应用于困难任务，发现它没有。无论是问答、数学、编码和多回合智能工具使用，还是推理模式、模型大小和PI形式，以及SDPO和OPSD配方下，每个代币的损失稳步下降，而验证准确性没有提升，通常还在下降。我们通过从损失到模型的单一因果链来解释这种失败。链条始于PI偏误：教师看到某个特定参考解后，其每个标记的目标被拉向该轨迹，而非整体正确性，这一效应我们用PI偏差评分量化。训练使学生在各处匹配该目标，其目标几乎对推销是否正确视为无效，其损失主要落在信息量较少的标记上，如停止词、标点、不确定性标记，而非决定答案的标记;在正确展开中，探索性标记产生最高的分歧，因此惩罚推理所需的犹豫。结果是学生更平淡、不够果断，推理能力也不优越：作为唯一目标，SD优化了一个与任务成功脱钩的信号。

PRIMAL3: Pathfinding via Reinforcement and Imitation Multi-Agent Learning - Leveraging LaCAM3

PRIMAL3：通过强化与模仿多智能体学习实现路径寻路——利用LaCAM3

• Authors: Chengyang He, Tanishq Duhan, Gadiel Sznaier Camps, Fangyuan Wang, Yuhong Cao, Jiankai Sun, Ge Sun, Mac Schwager, Guillaume Sartoretti
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.04905
• Pdf link: https://arxiv.org/pdf/2608.04905
• Abstract We present PRIMAL3, an ultra-large-scale learning-based framework for multi-agent pathfinding (MAPF) that integrates reinforcement learning, topology-aware communication, LaCAM3-guided training, and PIBT-based action refinement. PRIMAL3 targets failures at topologically critical states, where agents must coordinate decisively around bottlenecks, dead ends, and persistent conflicts. Each agent is represented using features derived from cut vertices, dead-end regions, shortest-path distances, and blocking estimates. Two complementary graphs capture agent interactions: a same-direction following graph propagates multihop context along compatible paths, while a different-direction conflict graph differentiates agents competing for shared space through masked attention and relative features. During training, we propose to let policy entropy identify uncertain agents, for which LaCAM3 provides confidence-triggered action interventions and label-smoothed imitation targets. During execution, a priority-aware PIBT module refines the proposed joint actions using persistent, learned, and distance-aware priorities together with policy-aware fallback preferences while maintaining collision-free execution. The resulting framework combines learned exploration with structured expert guidance without requiring LaCAM3 at inference. Experiments demonstrate that PRIMAL3 substantially outperforms state-of-the-art learning-based baselines and scales to ultra-large instances with up to city-level 100,000 agents. Real-world experiments further demonstrate the feasibility of deploying PRIMAL3 on physical robotic systems and ablation studies validate the individual contributions the components we proposed. Project page: this https URL
• 中文摘要 我们提出了PRIMAL3，一个基于多智能体路径寻觅（MAPF）的超大规模学习框架，集成了强化学习、拓扑感知通信、LaCAM3引导训练和基于PIBT的动作细化。PRIMAL3针对拓扑关键状态的失败，即代理必须果断协调，绕过瓶颈、死胡同和持续冲突。每个代理通过截顶点、死胡同区域、最短路径距离和阻塞估计得出的特征表示。两个互补的图捕捉代理交互：同向跟随图沿兼容路径传播多跳上下文，而不同方向冲突图通过掩蔽注意力和相对特征区分争夺共享空间的代理。在培训过程中，我们提议让策略熵识别不确定的代理，LaCAM3 对此提供基于置信度的动作干预和标签平滑的仿制目标。在执行过程中，一个优先级感知的 PIBT 模块通过持久、学习和距离感知的优先级以及策略感知的备选偏好，优化拟议的联合动作，同时保持无碰撞执行。最终框架结合了学习探索与结构化专家指导，无需在推断时使用 LaCAM3。实验显示PRIMAL3远超最先进的基于学习的基线，并可扩展至拥有高达 10 万代理的超大型实例。实际实验进一步展示了在物理机器人系统上部署 PRIMAL3 的可行性，消融研究验证了我们提出组件的各个贡献。项目页面：此 https URL

State2State: Environment-Derived Mid-Training for LLM Agents

State2State：面向LLM代理的环境衍生中期培训

• Authors: Xuanyu Lei, Yiqi Zhu, Chenliang Li, Kaiming Liu, Peng Li, Ming Yan, Jieping Ye, Ya-Qin Zhang, Yang Liu
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.04934
• Pdf link: https://arxiv.org/pdf/2608.04934
• Abstract Training LLM agents commonly relies on supervised fine-tuning from expert trajectories or online reinforcement learning over human-specified tasks with handcrafted verifiers. Though effective, both remain bottlenecked by externally specified tasks and supervision signals, limiting the scalability and diversity of agent training. We study an environment learning paradigm in which agents acquire interaction and manipulation capabilities solely through environment interaction, without externally specified tasks. We propose State2State, an environment-derived mid-training method that converts explored environment states into training objectives, challenging agents to reach a specified target state. By deriving tasks from environment exploration and verifying success through rule-based state matching, State2State provides scalable and verifiable training objectives without expert supervision or manual task design. Experiments on ALFWorld and ScienceWorld show that State2State improves agent performance as a standalone environment-learning stage in most settings. As initialization for downstream RL, it further improves final performance and learning efficiency, with promising evidence of cross-environment generalization.
• 中文摘要 培训LLM代理通常依赖于专家轨迹的监督微调或在线强化学习，替代人工定制的验证器任务。尽管有效，但两者仍受外部指定任务和监督信号的瓶颈，限制了代理训练的可扩展性和多样性。我们研究一种环境学习范式，代理仅通过环境互动获得交互和操作能力，无需外部指定任务。我们提出了State2State，一种基于环境的中期训练方法，将探索的环境状态转换为训练目标，挑战代理达到指定目标状态。通过从环境探索中推导任务并通过基于规则的状态匹配验证成功，State2State提供可扩展且可验证的训练目标，无需专家监督或人工任务设计。ALFWorld和ScienceWorld上的实验表明，State2State作为大多数环境中独立的环境学习阶段能提升代理性能。作为下游强化学习的初始化，它进一步提升了最终性能和学习效率，并提供了有希望的跨环境泛化证据。

SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts

SpecRoll：为推测强化学习推广的快速-慢速验证器-反馈适配

• Authors: Nhat Minh Pham, Duy Tung Doan, Thi Duyen Ngo, Vinh Van Nguyen, Khac-Hoai Nam Bui
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.04962
• Pdf link: https://arxiv.org/pdf/2608.04962
• Abstract Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but autoregressive rollout generation remains a major efficiency bottleneck. Speculative decoding can accelerate generation, yet applying it during RL is difficult because the target policy continually evolves: static proposers become stale, while frequent drafter updates add substantial overhead. We introduce SpecRoll, a speculative rollout engine that preserves the target model's sampling distribution while adapting at two timescales. Lightweight future-token heads generate parallel proposals, while our proposed Reflex module uses delayed verifier feedback to perform bounded, trajectory-local hidden-state corrections without backpropagation. A complementary slow path updates the head parameters only when sustained degradation is detected. SpecRoll combines these mechanisms with concurrency-aware sparse-tree verification and exact target verification, leaving the target rollout distribution and GRPO objective unchanged. Across five models ranging from 1.5B to 14B and three mathematical reasoning datasets, SpecRoll achieves 1.26-2.15x generation speedup and 1.21-2.04x end-to-end speedup over vanilla GRPO. It also outperforms FastGRPO in both generation and end-to-end time across all 15 matched settings, with an average pairwise end-to-end gain of 1.18x. Controlled ablations show that the fast and slow adaptation paths provide complementary benefits. Our source code is available at this https URL.
• 中文摘要 强化学习（RL）后训练提升了大型语言模型的推理能力，但自回归的推展生成仍是一个主要的效率瓶颈。推测解码可以加速生成，但在强化学习中应用它较为困难，因为目标策略不断变化：静态提议者变得陈旧，频繁的起草者更新则增加了大量开销。我们介绍了SpecRoll，一种在两个时间尺度上保持目标模型采样分布同时适应的推测推展引擎。轻量级未来代币头生成并行提案，而我们提出的Reflex模块利用延迟验证器反馈，执行有界、轨迹局部隐藏状态修正且无反向传播。一个补充的慢路径仅在检测到持续退化时更新头参数。SpecRoll将这些机制与并发感知的稀疏树验证和精确目标验证结合，保持目标推广分布和GRPO目标不变。在五个模型（1.5B到14B）和三个数学推理数据集中，SpecRoll相比原版GRPO实现了1.26-2.15倍的生成加速和1.21-2.04倍的端到端加速。在所有15种匹配环境中，它在生成和端到端时间上都优于FastGRPO，平均两端端到端增益为1.18倍。受控消融显示，快速和慢速适应路径提供了互补的好处。我们的源代码可在此 https 网址获取。

WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

WorldCycle：面向长期视野视频世界模型的自我验证强化学习

• Authors: Bohai Gu, Yueyang Yuan, Taiyi Wu, Dazhao Du, Jian Liu, Xiaoyi Pang, Jie Zhang, Xiaocheng Lu, Haobin Zhong, Xiaotong Zhao, Alan Zhao, Song Guo
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.04964
• Pdf link: https://arxiv.org/pdf/2608.04964
• Abstract Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4x over the base model, providing a vital foundation for physically grounded world models.
• 中文摘要 交互式视频世界模型对于长期视野规划和探索至关重要，但它们存在叠加错误。训练后方法如强化学习（RL）可以改进这些模型，但它们会遇到验证瓶颈：对于任意动作序列，没有用于测量长期漂移的基础真实未来状态。我们的关键见解是，可逆动作循环使这种验证成为可能：与其逆序列组成的序列必须解析返回初始状态，从而实现对长视界正确性的无注释监督。在此基础上，我们介绍了WorldCycle，这是一个自我验证的强化学习框架，它从普通动作序列构建闭合动作周期及其重复执行，并优化了两种互补奖励：空间闭合奖励强化镜像前向和反向段之间的对称性，以及时间一致性奖励，使状态在重复周期执行间对齐。这些奖励迫使模型将动作学习为一致的状态算符，而非记忆的时间模式，并自然扩展到基础模型处理不佳的分布外复合动作周期。我们还发布了CycleBench，这是一个复杂动作结构下状态返回能力的诊断基准。WorldCycle将状态返回漂移减少了多达44%，复合动作准确率几乎提升了基础模型的4倍，为物理基础世界模型提供了重要基础。

ORACLE: A Multi-Objective Reinforcement Learning-Based Analog Circuit Design Optimizer with Large Language Models-Guided Exploration

ORACLE：一款基于大型语言模型引导探索的多目标强化学习模拟电路设计优化器

• Authors: Osei Brempong, Mohammed Ayman Habib, Vivan Poddar, Morteza Fayazi
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04999
• Pdf link: https://arxiv.org/pdf/2608.04999
• Abstract Analog circuit design automation using reinforcement learning (RL) has emerged as a promising approach for reducing manual effort. However, many existing RL-based methods focus on single-objective optimization. Even methods designed for multi-objective (MO) problems often reduce multiple design specifications to a single scalar reward. This simplification limits the ability to capture the true Pareto trade-off among competing objectives and often leads to suboptimal designs. Moreover, requiring the model to be retrained from scratch whenever the desired MO specifications change remains a key limitation. To address these challenges, we present ORACLE, an open-source RL-based framework for MO analog circuit design optimization that replaces scalar reward optimization with vector-valued learning and preference-aware conditioning. ORACLE represents a true MO analog circuit design optimizer that uses a preference vector to specify the relative weights of multiple objectives, enabling a single trained model to generate designs across diverse trade-off settings without retraining. We further propose two preference-guidance strategies, namely normalized-weight guidance and cosine-aligned guidance, to improve convergence. In addition, we incorporate a large language model (LLM)-guided action selection mechanism to filter actions that are likely to lead to suboptimal designs or increased runtime. Our results show that, on multiple circuit topologies with 2,000 test cases, ORACLE reduces runtime by 20.4x - 104.4x compared to state-of-the-art approaches. It also meets 99.9% of the 2,000 target specifications, and achieves 5.1x - 318.6x better figure of merit in the resulting output specs.
• 中文摘要 利用强化学习（RL）进行模拟电路设计自动化已成为减少手工劳动的有前景方法。然而，许多现有基于强化学习的方法仍专注于单目标优化。即使是为多目标（MO）问题设计的方法，也常常将多个设计规格简化为单一标量奖励。这种简化限制了捕捉竞争目标之间真实帕累托权衡的能力，常常导致设计不理想。此外，每当期望的移动量参数发生变化时，必须从零重新训练模型仍是一个关键限制。为应对这些挑战，我们提出了ORACLE，一个开源的基于RL的MO模拟电路设计优化框架，它用向量值学习和偏好感知条件替代标量奖励优化。ORACLE代表了一个真正的MO模拟电路设计优化器，使用偏好向量指定多个目标的相对权重，使单一训练模型能够在不同权衡设置下生成设计，无需重新训练。我们还提出了两种偏好-指导策略，即归一化权重引导和余弦对齐指导，以提升收敛性。此外，我们还采用了大型语言模型（LLM）引导的动作选择机制，以过滤可能导致次优设计或延长运行时间的动作。我们的结果表明，在拥有2000个测试用例的多电路拓扑下，ORACLE相比最先进方法，可将运行时间缩短20.4倍至104.4倍。它还满足了2000个目标规格中的99.9%，最终输出规格的优点提升了5.1倍至318.6倍。

Exact Model-Free Policy Iteration for Co-safe LTL Planning

共同安全LTL规划的精确无模型政策迭代

• Authors: Zetong Xuan, Yu Wang
• Subjects: Subjects: Systems and Control (eess.SY); Formal Languages and Automata Theory (cs.FL); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.05047
• Pdf link: https://arxiv.org/pdf/2608.05047
• Abstract This work studies model-free reinforcement learning for co-safe linear temporal logic (sc-LTL) objectives in finite Markov decision processes, which can be reduced to maximal reachability objectives via the standard product construction. For this problem, direct sample-based bootstrap methods (e.g., TD or Q-learning) may fail to converge to optimal policies due to the noncontractive nature and nonuniqueness of solutions to the Bellman equation. We develop a new two-step model-free reinforcement learning method that first uses a discounted surrogate to identify a clamp set that resolves this nonuniqueness, and then applies undiscounted policy evaluation and greedy policy improvement with guarantees of finding an optimal solution. We prove almost-sure convergence of the policy evaluation step and finite termination of the policy iteration algorithm at an optimal policy. These theoretical results are validated through numerical experiments on a stochastic grid world.
• 中文摘要 本研究研究有限马尔可夫决策过程中共安全线性时序逻辑（sc-LTL）目标的无模型强化学习，通过标准乘积构造可简化为最大可达性目标。针对该问题，基于样本的自助方法（如TD或Q学习）可能因Bellman方程解的非收缩性质和非唯一性而收敛至最优策略。我们开发了一种新的两步无模型强化学习方法，首先使用折现代理识别解决该非唯一性的夹边集，然后应用无缺席策略评估和贪婪策略改进，保证找到最优解。我们证明了策略评估步骤几乎必然收敛，且策略迭代算法在最优策略处有限终止。这些理论结果通过随机网格世界的数值实验得到验证。

Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learning

优化政策可学习内容：可恢复性意识的推广干预学习

• Authors: Zheyuan Zhang, Manqing Mao, Hong Wang, Zhuoer Wang, Samson Koelle, Jie Yuan, Yanjun Lin, James Feng, Nikki Lijing Kuang, Yanfang Ye, Wei Niu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.05080
• Pdf link: https://arxiv.org/pdf/2608.05080
• Abstract Critic-free group-based reinforcement learning has become a scalable approach for post-training large language models. However, most existing methods allocate the same number of rollouts to every task and trajectory state, even though some rollouts provide much more useful learning signals than others. Recent work has started to treat rollout generation as an adaptive decision, but two important limitations remain. First, intervention strategies are often based on fixed heuristics and therefore cannot adjust as the policy changes during training. Second, these methods usually decide only how many rollouts to generate, without explicitly controlling where and how to intervene. To address these limitations, we propose Recoverability-Aware Intervention Learning (RAIL), a training-time framework that learns how to generate rollouts based on the improvement produced by each intervention. RAIL models intervention selection as an online contextual-bandit problem and trains a recoverability controller using intervention traces collected through a shadow-to-live procedure. This allows the controller to keep learning while the underlying policy evolves. We evaluate RAIL in terms of effectiveness, adaptivity, expressiveness, and efficiency. Across multiple settings, RAIL consistently improves performance under limited rollout budgets. These results show that recoverability-aware intervention provides a principled way to generate more informative and less redundant rollouts, leading to stronger learning signals during post-training.
• 中文摘要 无批评的基于群体的强化学习已成为训练后大型语言模型的可扩展方法。然而，大多数现有方法为每个任务和轨迹状态分配相同数量的推广，尽管某些推广比其他的提供更有价值的学习信号。近期研究开始将推广生成视为适应性决策，但仍存在两个重要局限性。首先，干预策略通常基于固定启发式，因此无法随着策略变化在训练中调整。其次，这些方法通常仅决定生成多少次推广，而未明确控制干预地点和方式。为解决这些局限性，我们提出了可恢复性感知干预学习（RAIL）框架，这是一种训练时间框架，学习如何根据每次干预带来的改进生成推广。RAIL将干预选择建模为在线上下文盗贼问题，并通过影子到实际过程收集的干预痕迹训练可恢复性控制器。这使得控制者在底层策略演进的同时，能够持续学习。我们从有效性、适应性、表达力和效率方面评估RAIL。在多种环境中，RAIL在有限的部署预算下持续提升性能。这些结果表明，具恢复性感知的干预提供了一种有原则的方式，能够生成更具信息量且冗余更少的推广，从而在培训后获得更强的学习信号。

ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment

ABSeeker：通过回溯信用分配培训长远搜索代理

• Authors: Yijun Lu, Rui Ye, Jiajun Wang, Yuwen Du, Tian Jin, Songhua Liu, Siheng Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.05102
• Pdf link: https://arxiv.org/pdf/2608.05102
• Abstract Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer. However, existing methods for training these agents typically treat all steps within a trajectory uniformly during both supervised fine-tuning (SFT) and reinforcement learning (RL), failing to distinguish useful actions from erroneous or redundant ones. In this paper, we propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents by converting sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions. Specifically, given a potentially obscure query and its corresponding ground-truth answer, ABC first performs Answer-Backtracked Clue Recovery, which traces back from the answer to recover intermediate clues required to solve the question. It then applies Clue-Anchored Step Scoring to evaluate each search step against these clues, converting sparse binary outcome supervision into dense step-level rewards. Based on these rewards, we develop ABC-SFT, which reweights the loss of each turn, and ABC-GRPO, which uses the step-level scores as rewards in GRPO. Building on this framework, we train ABSeeker based on Qwen3.5-4B with only 8.5k examples. ABSeeker achieves 37.3% on BrowseComp and 39.1% on BrowseComp-ZH. With context management, the scores further improve to 55.3% and 52.9%, respectively, significantly outperforming same-scale (4B) agents and even matching the performance of larger ones (approximately 30B). These results demonstrate the effectiveness of answer-backtracked step-level credit assignment for training long-horizon search agents.
• 中文摘要 长视野搜索代理必须进行多次顺序操作（步骤），以搜索、检索、验证和整合证据以达成最终答案。然而，现有的训练方法通常在监督微调（SFT）和强化学习（RL）中统一处理轨迹内的所有步骤，未能区分有用的动作与错误或冗余的动作。本文提出了“答案回溯信用分配”（ABC），这是一种细粒度的信用分配框架，通过将稀疏的轨迹级结果转换为密集的步骤级监督，奖励有用动作（即使是失败的轨迹），同时抑制错误或冗余的行为，从而训练长视野搜索代理。具体来说，给定一个可能晦涩的查询及其对应的真实答案，ABC首先进行答案回溯线索恢复，从答案回溯以恢复解决问题所需的中间线索。然后应用线索锚定步骤评分，将每个搜索步骤与这些线索进行评估，将稀疏的二元结果监督转化为密集的步骤级奖励。基于这些奖励，我们开发了ABC-SFT，重新加权每回合的损失，以及ABC-GRPO，将步骤级得分作为GRPO中的奖励。基于该框架，我们基于Qwen3.5-4B训练ABSeeker，仅有8.5k样本。ABSeeker在BrowseComp上达到37.3%，在BrowseComp-ZH上达到39.1%。通过上下文管理，分数进一步提升至55.3%和52.9%，显著优于同尺度（4B）代理，甚至与大型代理（约30B）相当。这些结果展示了回答回溯阶级学分分配在长视野搜索代理培训中的有效性。

Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming

特工对特工：自动即时注入红队的智能系统

• Authors: Yanting Wang, Chenlong Yin, Runpeng Geng, Jinyuan Jia
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2608.05108
• Pdf link: https://arxiv.org/pdf/2608.05108
• Abstract Prompt injection poses significant security risks to LLM agents. Efficient and effective red-teaming is therefore critical, both for evaluating these risks and for collecting training data to improve defenses. Existing state-of-the-art prompt injection red-teaming methods primarily rely on reinforcement learning (RL), producing attacker models that often generalize poorly to new target LLMs. In this work, we develop PIMiner, an agentic system for prompt injection red-teaming. During training, PIMiner is trained on a sequence of (dataset, target model) pairs and builds a strategy library from scratch. At test time, the learned strategy library can be directly transferred to a previously unseen target LLM without additional training. PIMiner requires only a small number of queries to a target agent (e.g., 10) per test sample. Experimental results demonstrate that PIMiner achieves strong performance. On IPIArena, it attains a 76.2% ASR against Gemini-2.5-Pro, 61.9% ASR against GPT-5.1, and 42.9% ASR against Claude-Sonnet-4.5. On AgentDojo, it achieves an 86.7% ASR against Gemini-2.5-Pro, 53.3% ASR against GPT-5.1, and 40.0% ASR against Claude-Sonnet-4.5.
• 中文摘要 提示注入对LLM代理构成重大安全风险。因此，高效且有效的红队化至关重要，既能评估这些风险，也用于收集训练数据以提升防御能力。现有最先进的提示注入红队方法主要依赖强化学习（RL），导致攻击者模型常常难以推广到新的目标LLMs。本研究中，我们开发了PIMiner，一种用于即时注入红队的代理系统。在训练过程中，PIMiner 会基于一系列（数据集、目标模型）对进行训练，并从零构建策略库。测试时，所学的策略库可以直接转移到此前未见过的目标LLM，无需额外训练。PIMiner 每个测试样本只需对目标代理进行少量查询（例如10次）。实验结果表明，PIMiner 实现了强劲的性能。在IPIArena上，它对Gemini-2.5-Pro的ASR为76.2%，对GPT-5.1为61.9%，对Claude-Sonnet-4.5为42.9%。在AgentDojo上，对Gemini-2.5-Pro的ASR为86.7%，对GPT-5.1为53.3%，对Claude-Sonnet-4.5为40.0%。

Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning

奖励结构塑造了情节探索与强化学习中神经记忆之间的互动

• Authors: Jai Malegaonkar, Rohan Patil, Henrik I. Christensen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.05111
• Pdf link: https://arxiv.org/pdf/2608.05111
• Abstract In partially observable reinforcement learning, agents face a dual bottleneck: they must explore to encounter rewarding states and retain that experience in memory to optimize their policies. Exploration bonuses and memory architectures are traditionally evaluated in isolation, leaving their interaction unmeasured, and standard notions of sparse reward conflate temporal signal density with what the reward actually supervises. We present a controlled study crossing episodic exploration bonuses with diverse neural memory architectures across three environments that vary how the content of memory is acquired. An identical bonus signal yields three distinct interaction patterns: it amplifies architectural capacity differences where memory content must be actively discovered and retained unsupervised; equalizes architectures to a shared ceiling where the content, once sought out, is a single reward-supervised cue; and is null where the observation stream is purely scheduled. Controlled reward manipulations verify that these patterns track reward structure rather than density: a dense reward neutralizes a bonus only if it directly supervises the required latent memory, and a small avoidable penalty on exploratory actions (leaving the optimum unchanged) induces policy convergence to suboptimal stationary states, which either bonus resolves. We then formalize reward sparsity with observation-anchored reward machines, separating structural sparsity (an automaton reproduces the return without the task-required history) from potential sparsity (the one-step reward misprices local exploratory actions); the resulting vocabulary organizes the three regimes by the retention burden each task exposes. Together, these results show exploration and memory are complements, not substitutes: a bonus induces exposure, and only memory converts exposure into return.
• 中文摘要 在部分可观察的强化学习中，代理面临双重瓶颈：必须探索以遇到奖励状态，并将该体验保留在记忆中以优化策略。探索奖励和记忆架构传统上是孤立评估的，未测量它们的交互，而标准的稀疏奖励概念将时间信号密度与奖励实际监督的内容混淆。我们展示了一项控制研究，将情节探索奖励与三种环境中不同的神经记忆架构交叉，这些结构改变了记忆内容的获取方式。相同的奖励信号产生三种不同的交互模式：放大了必须主动发现并保留记忆内容时的架构容量差异;将结构平衡到共享天花板，其中内容一旦被寻找，即为单一的奖励监督提示;在观察流纯调度时为零。受控的奖励操作验证了这些模式跟踪的是奖励结构而非密度：高密度奖励只有在直接监督所需的潜在记忆时才会中和奖励，而对探索性行动（保持最优状态不变）施加小幅可避免的惩罚则诱导策略趋同到次优的平稳状态，从而通过奖励解决。随后，我们用观察锚定的奖励机形式化奖励稀疏，将结构性稀疏（自动机在没有任务所需历史的情况下再现回报）与潜在稀疏（一步奖励错误地评估局部探索行为）分离;由此产生的词汇根据每个任务暴露的保留负担组织了这三种模式。这些结果共同表明，探索和记忆是互补而非替代：奖励诱导暴露，只有记忆将暴露转化为回报。

Argus: A General-Purpose Agentic Runtime for Long-Horizon Reasoning

Argus：用于长期视野推理的通用智能运行时间

• Authors: Boxiu Li, Zimo Wen, Yijia Fan, Junxiang Lei, Sufeng Guo, Jiaao Wu, Ruize Tang, Mukai Li, Yifei Shen, Xiaoyu Chen, Wanbo Zhang, Runjing Gu, Yifei Gao, Yuheng Wu, Xuyao Huang, Zelong Zhao, Jiachen Zhang, Shibo Hu, Hangxi Guo, Yilin Chen, Yuzhe Zhang, Fan Yang, Chuan Wen, Xian Zhang, Xuanhe Zhou, Zhijie Deng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.05144
• Pdf link: https://arxiv.org/pdf/2608.05144
• Abstract Long-horizon reasoning requires an agentic runtime that can persist when evidence supports its current approach and pivot when measurements reveal failure, hidden constraints, or a misspecified objective. We present Argus, a persistent, self-evolving runtime in which Manager, Planner, Engineer, and Reviewer execute bounded missions over durable project state. Argus separates stable user intent from operational objectives, constraints, and verification criteria, and admits memories, skills, procedures, verifiers, routing decisions, and rejected routes only after role-owned review and, when available, task-native verification. Model weights remain fixed; self-evolution occurs through persistent runtime state and control policy, with autonomous execution between operator-owned escalation points. Across seven GPT-5.5 benchmark arenas, Argus achieves about 78% on SWE-Bench Pro versus 59% for Direct Copilot while using 1.41 times the aggregate tokens. After verification-gated self-evolution, mature SWE-Bench waves use 21% fewer solve-input tokens and 15% less active workflow time per task than startup waves, while recording 34 verifier recoveries and 22 strict review-loop rescues. Argus also reaches 76.8% on AARRI-Bench and a 28.0-point gap on mathematical data synthesis, with competitive GPU-kernel and language-model-training results. Beyond benchmarks, an optimized RWKV6 kernel was merged upstream; a multi-day mathematics campaign retained falsified routes and proof-backed frontier updates; and six paper pipelines completed 254 missions with 16 stage rollbacks. These results show that a fixed-weight, self-evolving harness can revise, recover, and accumulate verified approaches while producing structured trajectories for future supervised and reinforcement learning.
• 中文摘要 长视野推理需要一个能持续存在的智能运行时，当证据支持其当前方法时，该运行时间能够持续存在，并在测量发现失败、隐藏约束或目标错误时转向枢轴。我们介绍Argus，一个持久且自我演进的运行时，管理者、规划者、工程师和审核员在持久项目状态上执行有界任务。Argus将稳定的用户意图与运营目标、约束和验证标准分离，只有在角色拥有的审查和任务原生验证（如有）后，才允许记忆、技能、程序、验证者、路由决策和拒绝路线。模型权重保持固定;自我演进通过持续运行时状态和控制策略实现，并在操作员拥有的升级点之间实现自主执行。在七个GPT-5.5基准测试领域中，Argus在SWE-Bench Pro上实现约78%，而Direct Copilot为59%，且使用1.41倍的汇总代币。经过验证门控自我演化后，成熟的SWE-Bench波次使用了21%的求解输入令牌，每任务的活跃工作流时间比启动波减少21%，同时记录了34次验证者恢复和22次严格审查循环救援。Argus在AARRI-Bench上达到76.8%，数学数据综合方面落后28.0分，GPU内核和语言模型训练成绩具有竞争力。除了基准测试外，上游合并了一个优化后的RWKV6内核;多日数学活动保留了伪造路径和有证明支持的前沿更新;六条纸质流水线完成了254个任务，完成了16个阶段回滚。这些结果表明，固定权重、自我演化的线束能够修正、恢复并积累经过验证的方法，同时为未来的监督和强化学习生成结构化轨迹。

Keyword: diffusion policy

There is no result